It seems that Google uses more than just an algorithm to produce its search results, and employs people to review specific queries.

Google’s Director of Research Peter Norvig has been talking about the methods the search engine uses to ensure the quality and accuracy of its results.

According to Norvig:
“Another way we do it [improve the quality of search results] is to randomly select specific queries and hire people to say how good our results are. These are just contractors that we hire who give their judgment.”

“We train them on how to identify spam and other bad sites, and then we record their judgments and track against that. It’s more of a gold standard because it’s someone giving a real opinion, but of course, since there’s a human in the loop, we can’t afford to do as much of it.”

This is not previously unknown, but Google has been coy in talking about the issue, though the search engine’s Marissa Mayer revealed in July that it has 10,000 human evaluators that manually review the results.

Norvig also revealed some other ways that Google ensures the quality of its results:

  • Google tracks user clicks – if they click on the first result in a search query and don’t come back for another search, this is considered a sign that the search engine has done its job properly, and the user has found what they were looking for.
  • As well as hiring human evaluators, Google does user experience tests in laboratories and in peoples’ homes, learning from the way people are using the site.
  • Perhaps Google is responding to human-edited search engines like the recently launched Mahalo, which claims that it can produce better results with a team of editors than Google can manage with its algorithm.

The problem with this theory is that, it cannot hope to index anywhere near the amount of webpages that Google can