Snorkel AI

Snorkel AI is a research-led data development company that partners with frontier AI labs and enterprise teams to build specialized training data, evaluation environments, and benchmarks for advanced AI models and agents. Founded out of the Stanford AI Lab, the company has been shaping and benchmarking frontier AI for nearly a decade, with over 200 peer-reviewed papers and collaborations with organizations including Google, Anthropic, OpenAI, Microsoft, Amazon Web Services, and Mistral AI.

How It Works

Snorkel AI follows a three-step iterative process. First, it evaluates model behavior against task-specific benchmarks inside realistic environments with programmatically defined pass/fail criteria. Second, it curates data through rubric-guided pipelines with calibrated domain experts in the loop, including environment construction with tools, documents, and verifiable reward signals. Third, it refines the system by analyzing disagreements, tracing failures, and mapping coverage gaps, updating rubrics and expanding benchmarks for underperforming slices.

Key Capabilities

Snorkel AI provides two primary ways to access data. The Snorkel Data Series offers curriculum-structured datasets for task areas that frontier models are pushing hardest, including agentic coding, terminal tasks, enterprise reinforcement learning environments, multimodal STEM reasoning, and specialized computer use agents. Each series includes rubrics, reviewer guidance, difficulty tiers, and evaluation slices built in. For gaps that off-the-shelf coverage cannot reach, Snorkel AI offers custom data development engagements that start with the model's failure surface and build bespoke datasets, evaluation environments, and benchmark expansions to close those gaps.

Data types produced include expert demonstrations and reasoning traces, preference labels and rankings, rubrics and verifiable outcomes, as well as standard and custom environments such as repository and CLI tools, browser and GUI harnesses, multi-step stateful workflows, and simulated environments. The platform supports SOC 2 and HIPAA compliance.

Data Development Process

Snorkel AI's proprietary process emphasizes design decisions that determine whether training data drives model improvement. Tasks are scoped to actual model failure modes with target distributions, acceptance criteria, and verifier definitions written before data work begins. Expert reviewers are calibrated against gold sets authored by Snorkel researchers, scored for agreement and bias, and recalibrated per task. Fine-tuned evaluator models and programmatic graders work alongside human spot-checks, with rubrics co-designed by Snorkel researchers and domain experts. Every label passes through an author, multi-reviewer, and final-adjudicator pipeline with full audit trails, ensuring every data point is traceable to who decided what, when, and on what evidence.

Specialized Agents

Beyond data development, Snorkel AI builds custom specialized agents grounded in expert data. These agents are evaluated against task-specific rubrics and programmatic checks rather than generic benchmarks, and are refined through the same adjudication and provenance practices used in production model development. They are designed for specialized workflows and high-consequence decisions rather than generic copilots.

Intended Users

Snorkel AI serves frontier AI labs, enterprise AI teams, and research institutions that need to close distributional gaps in specialized domains, address benchmark blind spots, and solve failure modes that only surface at scale. The company maintains an expert contributor community of over 1,000 domain experts who participate in data development projects.