Eval My AI

Eval My AI is an automated testing service designed to verify and evaluate answers generated by large language models and retrieval-augmented generation applications. Developed by Profinit, an Amdocs company with over 27 years of IT expertise and 650+ IT professionals across Europe, this cloud-based SaaS tool helps developers and QA teams replace manual review of AI outputs with programmatic semantic evaluation using its proprietary C3-score metric.

How It Works

Eval My AI compares AI-generated answers against expected correct answers to determine semantic equivalence. Users submit questions paired with both a ground-truth answer and the AI-produced answer via REST API or Python client library. The service analyzes the response across three dimensions and returns a composite C3-score along with detailed reasoning about why an answer passed or failed each dimension. It integrates directly into CI/CD pipelines and supports popular ML frameworks such as LangChain, making it a practical tool for automated quality assurance in AI development lifecycles.

C3-Score Metric

The C3-score is a balanced qualitative metric consisting of three components. Completeness checks whether any facts are missing from the AI answer compared to the expected answer. Correctness verifies that the answer contains no extra or fabricated information, effectively detecting hallucinations. Contradiction ensures there is no logical inconsistency within the answer. Each component contributes to an overall score that expresses how semantically equivalent the AI output is to the expected answer, with clear severity levels to help teams prioritize issues.

Key Capabilities

• REST API integration enables seamless embedding into development workflows and automated CI/CD testing pipelines for continuous validation.
• Python client library simplifies the evaluation process for Python-based projects with a straightforward evaluator interface requiring only authentication and data inputs.
• Cloud-based SaaS architecture scales automatically based on model count, test frequency, and question set size without infrastructure management.
• Customizable Sem-Score parameters allow testers to adjust evaluation context based on risk profiles and specific application requirements.
• Dedicated technical customer support provides guidance for developers integrating the service into existing systems.

Use Cases

• Developers building RAG applications who need to verify that retrieved content is accurately represented in generated answers during development and after release.
• QA teams testing LLM-based products who require consistent, repeatable evaluation across regression test suites as models are updated.
• Organizations deploying generative AI in production who need automated monitoring of answer quality and hallucination rates without manual spot-checking.

Pricing

Eval My AI offers an Early Adopters package with 10 million free tokens to evaluate the service. Additional tokens are available in recharge packs at 5 USD per 1 million tokens, providing a usage-based pricing model that scales with testing volume. This approach makes the service accessible for teams of all sizes to incorporate automated AI answer verification into their workflows.