Skip to content
Agentmart
Submit a tool

DeepEval

Open-source LLM evaluation framework, similar to Pytest, for unit testing LLM apps and agents

Observability tool by Confident AI

IndexedApache-2.0★ 19,081
Write a review

Chalk Labs

Need help deploying DeepEval?

Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.

Book a call

Install

Pick your client. Read the repository before you give any tool access to your data.

pip install deepeval

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

We use your email only for this newsletter. Privacy

About

DeepEval evaluates LLM apps end to end, complete agent trajectories and individual agent steps such as LLM calls, tool use and retrieval. Metrics such as G-Eval, task completion, answer relevancy and hallucination use an LLM of your choice as judge or local NLP models. Confident AI adds production tracing and failure monitoring on top.

Use cases: llm-evaluation, testing

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Fact sheet

Category
Evals · open source
Self-host
Yesconfident-ai.com
Open source
Yesraw.githubusercontent.com
Compliance (vendor says)
SOC 2, HIPAAconfident-ai.com
Regions
Canada, Australia, Japanconfident-ai.com
Works with
langchainraw.githubusercontent.com

Plans

Free$0forever; 2 user seats; 1 project; 5 test runs per week; 1 GB-month of trace spans
Starter$200/monthunlimited user seats; 5 projects; 5 GB-months of trace spans then $1 per GB-month
Team$2000/monthunlimited seats and projects; 75 GB-months of trace spans; SOC2; SSO; dedicated support channel
EnterpriseContact salescustom pricing; dedicated on-prem deployment; HIPAA; custom data residency; 24x7 support

From the vendor's pricing page, read Oct 24, 2026.

Facts read from the vendor's own pages by claude-haiku-5-5 on Oct 24, 2026; a fact is shown only with the page it came from.

Signals

Stars
19,081
Forks
2,131
Open issues
332
Last commit
Oct 23, 2026 (7 days ago)
Last release
Sep 22, 2026 (1 month ago)

Details

Packages
pypi deepeval
License
Apache-2.0
Repository
github.com/confident-ai/deepeval
Website
deepeval.com
Docs
deepeval.com/docs/getting-started
First seen
Oct 24, 2026
Last checked
Oct 24, 2026
Source
READMEWebsiteDocsPricing page

More from Confident AI

Done for you

Rather not set it up yourself?

Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.

See how we build AI agents →Read the case studies

Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.

launches shipped
50+
launches shipped
media placements
500+
media placements
founders on every account
2
founders on every account

Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block

Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.

Get DeepEval running in your business

Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.

Goes to the Chalk Labs founders, not the newsletter. Privacy

Need help deploying DeepEval?

CallGet help
Cite this page

DeepEval. Agent Mart. https://agentmart.dev/observability/deepeval (last updated 2026-10-24)