Webmasters and information vendors began optimizing Sites for search engines inside the mid-nineteen nineties, as the first engines like google were being cataloging the early Internet. Indexing: The data gathered because of the crawlers is arranged and stored in a big database known as an index.