What is a Search Engine Evaluator? Exam, Pay & More
Job Description
A search engine evaluator is an independent contractor that provides feedback and critical insights for search engine companies like Google and Bing. It is an entry-level, work-from-home type of job that anyone can apply for. No prior experience or specific technical background is required from candidates.
The core of the job is to rate the quality of webpages and relevance of search results and their feedback is used to test improvements on the search engine's algorithms.
It's important to stress that responses from raters are NOT used as an input to Google's algorithm. In other words, although raters contribute to improving Google Search, they don’t directly impact how search results are ranked.
Search evaluation is one of the most sought-after and legitimate ways to make money at home while surfing the web.
Requirements
To be eligible for the search engine evaluator role, candidates must have:
- High-speed internet access and the availability to work from home
- Relatively new computer and smartphone
- Excellent analytical abilities and web research skills
- In-depth and contemporary knowledge of the local culture
- Excellent comprehension and written communication skills in English and local language (in case you are from a non-English speaking country)
- University degree is desired, but not mandatory
- You must be at least 18 years old
Since you are looking for information about search engine evaluation, you may also like to register on these other websites that offer work from home opportunities.
Search Engine Evaluation Projects & Job Titles
A search engine evaluator job or project is called different names depending on the company that is offering the job (e.g. Appen, TELUS International), the client (e.g. Google, Bing, Apple) and the search engine service that the work is about (e.g. Google Web Search, Apple Maps, Google Ads).
At Appen, as one of its top entry-level Appen jobs, a search engine evaluator is referred to by different names depending on the project that the work is about. TELUS International AI (formerly Lionbridge AI), on the other hand, uses different job titles for its search engine evaluator roles. Search engine evaluation is one of the best AI Community rating jobs where you can work online remotely.
Below is a table that better illustrates the most popular search engine evaluator jobs at Appen and TELUS International:
| Client | Search Engine Service | Company | Project Name / Job Title |
| Web Search | Appen | Project Yukon | |
| Web Search | TELUS AI | Personalized Internet Assessor | |
| Ads | Appen | Project Arrow | |
| Ads | TELUS AI | Personalized Internet Ads Assessor | |
| Apple | Maps | Appen | Project Shasta |
| Apple | Maps | TELUS AI | Online Maps Quality Analyst |
| Bing | Web Search | TELUS AI | Web Content Assessor |
| Bing | Web Search | TELUS AI | AI Community Surfer |
There are still other names that can be used to refer to search engine evaluators. On the search quality rating guidelines for its web search service, Google calls them search quality raters. This is the same terminology used by the SEO community to refer to these raters, sometimes called Google raters. Another term that is sometimes used, but has been losing popularity, is web search evaluator.
Search Engine Evaluation Tasks
Besides rating the quality of webpages and the relevance of search results, there are hundreds of other different types of tasks that can be performed by search evaluators, such as:
- rate the helpfulness of autocompletion, related searches and other Google suggestions
- evaluate if an automated voice sounds natural or not
- give an opinion on which set of search results is more relevant
- classify topics that represent the content of a web page
- classify search queries by different types of predefined search intents
- evaluate whether a business belongs to a certain category
- etc.
Example of a typical search evaluation task
Imagine a user going to Google and typing in the query <wfh>.
What results would be relevant to show users for this query? How does the relevance of each result compare to each other? These are the questions that search engine evaluators help to answer.
Rating Process
The first step in the rating process is to understand the query. Dominant interpretation is that WFH is the acronym for "work from home", so some possible user intents for this query are:
- to look for the meaning of this acronym
- to look for work from home jobs
- to look for information about what work from home is and entails
- to look for images that illustrate the topic 'work from home'
- to look for news about working from home
- etc.
Considering the likelihood of possible search intents for the given query, the rater needs to assign a relevance rating for each result that is presented to them on the rating tool. A result can be a web page or a Special Content Result Block (i.e. a featured snippet that is displayed directly in the SERP, a knowledge panel, an image block, etc.).
If you were a rater, which ratings would you assign for the following results 1, 2 and 3, considering the query <wfh>?
What about results 4 and 5?
To assign a relevance rating, the rater needs to consider how well the result possibly meets the needs of the user. For that, they use a scale with 9 different rating levels and must adhere to the instructions provided on the rating guidelines.
Rating Guidelines
For each type of task, there are always instructions or guidelines that must be followed. These can be a few paragraphs of text printed directly on the task screen, or lengthy PDF documents with over 100 pages.
These instructions are usually confidential. The exception is Google’s search quality evaluator guidelines (download PDF here), which were made publicly available for the first time a couple of years ago. Today, Google even encourages SEOs and website owners to read the rater guidelines so that they can have a better understanding of what great content should look like.
Search Engine Evaluator Salary
The current pay rate for search engine evaluators ranges from $3/hour to $20/hour. The rate varies depending on the rater’s country and the company for which they are working. In the US, it also varies by state.
On average, the rate in more developed countries is between $10/h and $14/h. Considering an average task availability of about 20 hours per week, raters on search evaluation projects can expect to earn between around 800 USD and 1200 USD per month in these countries.
Payment Method
Search engine evaluators earn by the hour and are paid on a monthly basis, no matter their locale or position type.
At Appen, payment is done through Payoneer, which is also available as a payment method for some independent contractors at TELUS International. To earn a $25 reward from Payoneer as per their terms and conditions, you can register here with our referral link. Sign up today to take advantage of this campaign while it’s available.
TELUS International also pays via PayPal or direct deposit.
Application Process
The search evaluation program is outsourced by Google and Bing to a handful of companies that contract with evaluators and are responsible for managing them. Currently, the two most popular ones are Appen and TELUS International AI.
At Appen, first, you need to register as an Internet Analyst on Appen Connect. To go directly to the application page, click here.
Once your registration is approved, look for one of the search engine evaluation projects on the ‘All Projects’ tab of the Appen Connect dashboard. Examples of these projects are Yukon (Google Web Search), Shasta (Apple Maps), and Arrow (Google Ads). There may be other search evaluation projects as well as new ones are added to the list on an ongoing basis.
If you don’t see any of these projects listed on that page, it means that they are not available for you at this time. This may be due to the projects being full in your country, so make sure to regularly check the project page for when new vacancies become available. For more information on how to work from home with Appen, make sure to read this post.
At TELUS International's AI Community, you must look for the following job titles on its AI Community jobs page: Personalized Internet Assessor (Google Web Search), Online Map Quality Analyst (Apple Maps), Personalized Internet Ads Assessor (Google Ads), and Web Content Assessor (Bing Web Search). The title of the job may slightly vary or change from time to time.
If your profile matches the requirements for the job, these companies will get back to you. Please note that this process can take weeks or even months as it will depend on the number of vacancies that they have available in your country.
Search Engine Evaluator Exam
The search engine evaluator exam will vary depending on the project for which you are applying. Here, we will focus on the qualification exam for Appen's Yukon project, which is known by the job title Personalized Internet Assessor at TELUS International. This is the search evaluation program for Google's web search service, and it's the only one for which Google publicly discloses the rating guidelines.
Yukon / Internet Assessor Exam
How It Works
The qualification exam for Project Yukon (Appen) / Internet Assessor (TELUS International) is divided into three parts.
The first part of the exam will test your theoretical understanding of the guidelines, with 32 multiple-choice questions. This is an “open book” type of test as candidates have the chance to research through the document while they answer the questions. For that reason, this is considered the easiest of the three tests by most candidates.
The second and third parts consist of practical questions.
In the second test, you will be requested to rate the quality of webpages, while in the third test, you will have to assign relevance ratings for search results. The relevance rating of a search result is always determined in relation to a search query, which will be provided in the task.
The third part of the qualification exam is where most candidates fail. If you fail in any of the three parts, you will usually be given a second chance—unless you’ve failed miserably.
Exam Answers
The only section of the Yukon / Internet Assessor exam that is always the same is part 1, so if you are looking for the exam answers, you are probably referring to those 32 multiple-choice questions. It doesn’t make sense to look for the answers to parts 2 and 3 because those tests are frequently updated.
Disclosing Appen’s or TELUS International's exam answers is not part of our proposal. Firstly, doing so would be unethical. Secondly, it wouldn’t be beneficial to you, since you would probably fail the next two parts of the exam without mastering the concepts provided in that document.
Most importantly, cheating on the exam would likely result in you leaving the project only a few weeks or months after receiving an offer. The performance of newly hired evaluators is closely monitored by the quality team of the search engine evaluator companies.
Exam Preparation
You can prepare for the Yukon (Appen) / Internet Assessor (TELUS International) Exam by reading the general guidelines and enrolling in our courses at the Search Evaluator Academy.
Part 1
The General Guidelines Training course has a section titled Quality Theoretical Content Training, which can be useful to help you prepare for questions that involve Quality in part 1 of the exam.
However, please note that the focus of that course is not to help with part 1. The best approach for nailing this test is to take your time and carefully consult the guidelines while answering the questions.
Part 2
To prepare for part 2, there is a section in the General Guidelines Training course called Quality Training, which is composed of quizzes from the theoretical content and rating examples on Quality. All quizzes were created from the content and examples taken directly from the guidelines, so if you are looking for examples other than those provided in that document, then this course may not be a good fit.
The SEA Model - Master Search Evaluation course has a lesson titled 2.1.3 C-Attributes Convincing - Lesson 2 that some students find helpful for that part of the exam.
Part 3
To prepare for part 3, there is a section in the General Guidelines Training course called Needs Met Training, which is composed of quizzes from rating examples on Needs Met taken directly from the guidelines.
Our search engine evaluator course called SEA Model - Master Search Evaluation is useful for this part. It teaches an alternative approach for assessing the relevance of search results, and these techniques can be used as a complementary resource to Google's guidelines.
If you are on a budget, a plausible preparation strategy would be to enroll in the SEA Model course only if and after you've failed your first attempt at the exam -- considering that many applicants who fail are given a second chance.
Other Search Engine Evaluator Exams
Our courses are NOT useful for those who are applying for other search engine evaluator projects like Appen Arrow or Shasta, which are known, respectively, as Internet Ads Assessor and Online Maps Quality Analyst at TELUS International. They also DON’T work for Bing’s web search evaluator program, which is sometimes called Web Content Assessor at TELUS.
If you want to prepare for these projects, make sure to read their guidelines slowly and thoroughly, ideally twice before taking the exam.
For more information about these other types of search evaluator exams, please refer to the following pages:
For maps projects
For ads projects
For Bing web search project
Is Search Engine Evaluation an AI Training Job?
Yes, search engine evaluation can be considered a kind of AI training job.
The data collected from search engine evaluators is used to train and improve the underlying algorithms and AI models that power search engines. By providing this human-generated feedback, evaluators help the AI systems better understand user intent, improve search result relevance, and identify areas where the algorithms might need improvement.
In this sense, search engine evaluation plays a crucial role in training AI models to provide more accurate and useful search results for users.
Conclusion
Search engine evaluators play a pivotal role in enhancing the performance and accuracy of search engines. By becoming a search engine evaluator, you will not only be contributing to the ongoing development and refinement of AI-driven search systems but also become part of a community dedicated to shaping the future of online search.
If you are someone who enjoys the challenge of navigating the ever-evolving digital landscape and making a meaningful impact on the way people access information online, then becoming a search engine evaluator could be a great career choice for you.