Appen Shasta: What Is This Project All About? (+ Exam Tips)

Project Shasta is a search engine evaluator job at Appen centered on map analysis. The rater will evaluate location relationships displayed on a mobile search platform.

The goal of this online, work-from-home project is to improve the content, user relevance, and geographic accuracy of the service provided by Apple Maps. The production task itself is intricate and requires critical thinking with an ability to grasp navigational concepts. Consequently, the qualification process is also quite extensive and may be perceived as overwhelming at first to some.

At TELUS International AI (formerly Lionbridge AI), this project is known by the job title Online Map Quality Analyst.

Since you are looking for information on Appen projects, you may also like to register on these other websites that offer work from home opportunities.

Getting Started

Project Shasta is one of Appen's many flexible work opportunities, and you can learn more by first applying on Appen Connect. Once registered and approved, you will gain access to this portal which serves as a comprehensive support and information platform. 

Depending on project status, Shasta may be displayed on the 'All Projects' page with the option to apply, or you could receive a direct invite from project management if you are determined to be a good fit based on your application. 

Shasta is classified as an on-going project with the potential for long-term assignment, possibly for years. Posts on the Appen Connect social page do substantiate this claim, though work availability and new onboarding are subject to fluctuation.

To apply for the project, you will need reliable, high-speed internet and a computer with an operating system of Windows Vista or higher, or Mac OS. No phones or tablets are acceptable.  You should also have a reputable, subscription anti-virus, the latest version of Google Chrome, Adobe Acrobat Reader, Microsoft Edge, or any free PDF reader

Appen Shasta Pay

Reports vary, but you can expect to be compensated in the $9 - 14/hr range, depending on where you live. Payments are released once per month, via Payoneer.

Candidates are typically asked to commit to a minimum of 15 hours per week and must consistently meet quality and quantity metrics in order to remain on the roster.

If you intend to apply for Project Shasta but do not yet have a Payoneer account, you can sign up with our link to earn a $50 bonus under their terms. Register today to take advantage of this campaign while it is available.

Shasta Guidelines & Preparation Tools

The Guidelines are a nearly 250-page document, broken down into many sections and steps. Diagrams, examples, tables, and various visual aids often accompany rules and concepts.

Familiarity with the sections and the ability to find information and navigate through its many parts is essential. You will reference this guide recurrently throughout the qualification process and in production.

You will be given access to an Interactive Study guide, which is separate from the Guidelines. This may be the best place to start when beginning the qualification process. It is more simplified than the guidelines, and has a user-friendly format. It familiarizes you with the key concepts and procedures that are critical to accurate and timely judgments. There are interactive activities and multiple practice quizzes to test your knowledge.

As part of the qualification you may be selected to participate in a Jeopardy-style concept trivia game in which you compete with other project applicants using speed and accuracy metrics. A $50 reward is given to the top ten high scorers.

You will also likely be provided with at least one short video which provides a visual of the rating process with narration and explanation.

Project Shasta Exam

The Shasta qualification exam is broken down into two parts which are both ‘open-book.’

In Phase 1, a theoretical understanding of the rating concepts is assessed by 20 multiple-choice questions. The representations below are of a similar nature.

This often tricky portion of the exam also gauges your ability to search for and interpret information found within the guidelines. A passing score is required in order to proceed. You will be notified immediately if you passed, and move into the next phase.

The second part consists of practical tasks, similar to what you will encounter in production.

You are provided with a sample exam with sets to rate, and these resemble the final exam format. It is critical that you complete these sets and go over the provided correct answers thoroughly.

The practical application questions are extensive, often with many components, and require a high level of attention and analytical thinking. Refer to the task section of this article for more information

on the format you can expect to see. The final qualification exam is to be completed seven days from the date you received the project invite email.

A time commitment of up to 10-12 hours total can be expected in the qualification process. This reflects estimated study and learning pace, as well as time spent on the exam itself.

Candidates are typically allowed two attempts to achieve a passing score.

Exam Tips and Strategies

• Carefully study the examples found throughout the guidelines. These will familiarize you with the query types and provide direction on how to proceed. Drawing parallels between the case illustrations and the sets on the exam is a useful strategy. Test questions may seem to be 'different versions' of guideline examples (i.e. "McDonald's near me" vs "Burger King near me"). You may be able to arrive at the correct answer by taking similar steps as laid out in the guidelines.
•  Optimize your search. You will constantly be using Control + F /Search features to find solutions in the guidelines for both test parts. Knowing what keywords to look for and minimizing your search to specific terms will help you navigate more quickly and effectively.
• Take notes on the practice exam sample questions. Corrected answers will be shown. Go over your mistakes and your process, and understand how to adjust them.  Being able to reference where you went wrong (or right) is advantageous to doing well on the actual exam. The results you receive in practice will help you to gauge what you need to review before attempting the final qualification.
• Schedule the exam over several days. It is highly recommended that you space the tasks in order to remain fresh. You may exit and return to the exam at your discretion during this time.
•  DO READ the guidelines. This is not a test that you can 'wing.' Absolute memorization is not realistic, but you have to have a good grasp on the information.
• Engage in the activities found in the Interactive Study Guide. Go through them multiple times. These help cultivate a better intuition on to approach each scenario type. And some are even rather entertaining as they test your knowledge.

The Rating Process

The task is based on a sample query that a user may have entered while using the mobile maps platform.  They might have been searching for a business, category, address, or point of interest.

The tool displays the user 'viewport', which represents the map area the user was looking at when they performed the search. This image is representative of what the tool looks like:

The rating process will generally follow these steps:

1. User Intent.  Determine what the user was intending to find when they entered the query.
   This is done using information provided in the tool, or through web resources and general knowledge. Examples of an inquiry might be Starbucks in San Francisco, London Bridge, 123 Main St. Boston MA, fast food near me, etc.
2. Identify if a navigational result exists that would satisfy the user's intent. Is there a location in the real world that corresponds with the search, and the user would be inclined to go there?
3. Establish the relevance of the result using multiple steps and contexts like prominence and distance. For example, you might rate 'Excellent' for Starbucks locations that are in close proximity to the user as shown by the viewport, if that was the nature of the query. Other less desirable results would be demoted accordingly.
4. Authenticate data accuracy using accepted resources. You will check the name, address, and pin location of the result, provide proof of such, and assign accuracy ratings to each element. Examples of verification resources would be official business websites, social media sites, street/satellite imagery, reviews, postal authorities, and government sites.

There are many deviations and peripherals to consider in the rating process. More options will appear as you go through the menus.

In production, you will be required to meet speed and accuracy metrics. The tool measures your RPH (rate per hour) or JPH (judgements per hour) and quality "test questions" called real-time audits (RTA's) are performed which have known/correct answers, though you typically have the ability to appeal in cases with which you disagree. Failure to meet base metric requirements may result in a requalification procedure or removal from the project pending review.

Final Notes

There is a substantial amount of information that needs to be learned and applied in regard to Shasta. However, it is an established project that may provide more stability and possibly higher pay than some of Appen’s other options. Many report overall satisfaction with the assignment, making it a highly sought-after Appen project.

If you are interested in this type of work, make sure to read our 7-step guide on working from home with Appen. Best of luck!