Exclusive: Google’s Gemini is forcing contractors to rate AI responses outside their expertise

Generative AI may look like magic, but behind the development of these systems are armies of employees at companies like Google, OpenAI, and others, known as “prompt engineers” and analysts, who rate the accuracy of chatbots’ outputs to improve their AI.

But a new internal guideline passed down from Google to contractors working on Gemini, seen by Read, has led to concerns that Gemini could be more prone to spouting out inaccurate information on highly sensitive topics, like healthcare, to regular people.

To improve Gemini, contractors working with GlobalLogic, an outsourcing firm owned by Hitachiare routinely asked to evaluate AI-generated responses according to factors like “truthfulness.”

These contractors were until recently able to “skip” certain prompts, and thus opt out of evaluating various AI-written responses to those prompts, if the prompt was way outside their domain expertise. For example, a contractor could skip a prompt that was asking a niche question about cardiology because the contractor had no scientific background.

But last week, GlobalLogic announced a change from Google that contractors are no longer allowed to skip such prompts, regardless of their own expertise.

Internal correspondence seen by Read shows that previously, the guidelines read: “If you do not have critical expertise (e.g. coding, math) to rate this prompt, please skip this task.”

But now the guidelines read: “You should not skip prompts that require specialized domain knowledge.” Instead, contractors are being told to “rate the parts of the prompt you understand” and include a note that they don’t have domain knowledge.

This has led to direct concerns about Gemini’s accuracy on certain topics, as contractors are sometimes tasked with evaluating highly technical AI responses about issues like rare diseases that they have no background in.

“I thought the point of skipping was to increase accuracy by giving it to someone better?” one contractor noted in internal correspondence, seen by Read.

Contractors can now only skip prompts in two cases: if they’re “completely missing information” like the full prompt or response, or if they contain harmful content that requires special consent forms to evaluate, the new guidelines show.

Google did not respond to Read’s requests for comment by press time.