Many people whilst using the internet turn to search engines such as Google, Bing and Yahoo! to find what they are looking for quickly and easily. The information that search engines have may consist of web pages, images, information and other types of files. In the summer of 1993, no search engine existed yet for the web1 simply because there were not that many websites! During the early development of the web, there was a list of webservers edited by Tim Berners-Lee however today the web hs a huge amount of information – As of May 2009 over 109.5 million websites existed2.

Finding the information that you need today is like finding a needle in a haystack which is where search engines help us. We all turn to search engines to quickly find information in fact more and more of us are even doing this on the move via our mobile phones.

How do Search Engines find the information?

Search engines use automated programs, often referred to as robots or web-crawlers to crawl the web for information.These are computer programs that browses the World Wide Web in a automated manner or in an orderly fashion. Web crawlers are also often called bots, Web spiders or Web robots. Perhaps the easiest term to use is a web spider due to the way they work a web spider will visit a site then crawl all the other links on a page, if this page includes links to an external site the spider will often go off and crawl the next site in the same manner.

Links in the spidering process

Links from external websites are key to the way search engines work. In a nutshell the search engines are like giant voting systems if a site has hundreds of links from another site then this is often seen as a positive vote for that site from another. Some sites carry more ‘weight’ than others in this vote well established sites and sites with a great deal of authority such as university sites, the BBC, CNN for example will be seen as ‘authoritative’ by the search engines and a link from these to a site carry’s much more ‘votes’ than a link from a lower ranked site. Wikipedia has an excellent article on how Google’s Page Rank works.

Relevancy in the spidering process

The search engine spiders also look for relevancy from links. For example this link how Google’s Page Rank works goes to a Wikipedia page on how Google’s Page Rank works so it is seen as relevant by the search engines if it linked to a site about football for example the search engine would give the link a high vote. This is why links that say ‘click here’ are not only non descriptive for the user they have no relevancy to the search engines.

Search Engine References

1,2 Wikipedia