How Uber’s Scaled Solutions tests and evaluates LLM and AI models

How well does the model understand and follow the instructions it’s given? This is critical for applications like customer service chatbots or AI assistants.

In scenarios where creativity is needed, such as content generation, we test the model’s ability to deliver engaging and innovative responses while remaining relevant.

This area focuses on whether the model avoids generating harmful content, including biases, toxicity, and misinformation.

We evaluate the model’s ability to process complex information and provide sound, logical outputs.

Factuality is essential for AI systems providing information. Our tests ensure that models generate truthful and accurate content.

This involves periodic evaluations by human experts and automated systems. The goal is to regularly check a model's performance across the 5 areas mentioned earlier. Key activities include:

• Version control and regression testing: We compare different model versions to track improvements or identify any regressions
• Exploratory evaluation: At major development milestones, we conduct in-depth evaluations of the model’s strengths and weaknesses, culminating in a comprehensive report

Even after deployment, continuous monitoring ensures that AI models remain aligned with performance expectations. Our automated systems flag any problematic outputs, which are then reviewed by human experts to correct issues and update training datasets.

In this stage, Uber employs a team of human experts who specifically attempt to expose the model's vulnerabilities. This process is designed to catch any harmful behaviors, such as spreading misinformation or generating inappropriate content. Once identified, these issues are cataloged and addressed through further model training and fine-tuning.