In the Beginning, Gray created the Wanderer...

The World Wide Web was started originally proposed in 1989 and the first implementation appeared in 1990. The Web however, did not gain any widespread popular use until NCSA Mosaic became available in early 1993.

The year was 1993. While Bill Clinton was sending his first email as United States President, Matthew Gray of the Massachusetts Institute of Technology wanted to measure the growth of the World Wide Web. The result: the Web's first robot, the Wanderer . "I wrote the Wanderer to systematically traverse the Web and collect sites," Gray said. "I was initially motivated primarily to discover new sites, as the Web was still a relatively small place. As the Web grew rapidly, the focus quickly changed to charting the growth of the Web." Upgrades allowed URLs to be captured in a database called Wandex.

The Wanderer was also a troublemaker, though, causing noticeable lag time across the 'Net by accessing pages "hundreds of times a day," according to Galaxy. Even after the problem was solved, people were uncertain about robots.

Shortly after, Amsterdam native Martijn Koster created ALIWeb, which proudly proclaims to be the world's oldest search engine. Users searched by submitting a form that queried the ALIWeb database. Users could also submit their sites to the database. Fields included "description" and "keywords."

In February 1993, six Stanford undergrad students (five of whom were hackers) created Architext, which looked closer at word relationships and later gave birth to Excite. Architext allowed for webmasters to allow " advanced concept-based searching" of documents to their sites. The unfortunate thing about Architext, though, was that at the time spiders could not determine relationships between links, making it hard for searchers to find what they were looking for.

At the end of that year, three bot-fed search engines arose nearly at the same time. Jumpstation and the World Wide Web Worm (WWWW) retrieved title and header information. The WWWW had "a database of over 100,000 multimedia objects." (Compared to Google's database, that's not much, but back then, it was something to brag about.) Meanwhile, the Repository-Based Software Engineering (RBSE) program, which was administered by NASA, had a spider and a type of ranking system.

In early 1994, Stanford's David Filo and Jerry Yang, Ph.D, released a little directory with a funny name - Yahoo! Filo and Young claim they consider themselves "yahoos" although rumor has it the name is short for Yet Another Hierarchical Officious Oracle. Although Yahoo! was originally created for personal reasons, it soon became what WebmasterWorld has called a "sentimental favorite" to the public. As the directory grew, it became searchable and included descriptions with URLs.

Around the same time, WebCrawler, which crawled and indexed entire pages, was released by Brian Pinkerton of the University of Washington. The WebCrawler index was meant to be as broad as possible, and within a few months WebCrawler had served 1 million queries. "It's a great tool for locating several different starting points for exploring by hand," Pinkerton said in its release announcement. Although it started out as a desktop application rather than a Web service, WebCrawler soon became immensely popular and in 1995 was purchased by AOL. The WebCrawler purchase would provide Web access to AOL, which at the time had less than 1 million users. In return, AOL's resources made WebCrawler's future look promising.

At Carnegie Mellon University's School of Computer Science, Dr. Michael L. Mauldin introduced a beta of Lycos to the public. With Lycos, he intended to provide a search capability to the World Wide Web. Even back in those humble beginnings Mauldin knew Lycos was more than a "hopeless dream." It soon grew into an incorporated company, acquiring companies such as Tripod, Wisewire Corp., WhoWhere Inc., and Wired Digital (owner of HotBot) along the way. Lycos introduced language search and a redesigned search page in 1998. However, 1999 proved to be an even bigger year. During that time, Lycos joined the Open Directory Project (ODP), created a search tool for USAToday.com, launched multimedia search and more.

Part 2-->

Originally posted: webpronews.com - article