Promptfoo

Open-source CLI and library for evaluating and red-teaming LLM apps and agents

Observability tool by Promptfoo (OpenAI)

IndexedMIT★ 26,378

Chalk Labs

Need help deploying Promptfoo?

Chalk Labs sets up agents like this inside real teams: connected to your tools, tested on your data, with a human in the loop where it matters.

Book a call

Install

Pick your client. Read the repository before you give any tool access to your data.

npm install promptfoo

Get the weekly diff

The weekly diff: one email a week with new agent tools, notable updates and grade changes, plus security alerts for tools you follow. Double opt-in, one-click unsubscribe.

About

Promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps. It compares prompts and models side by side, automates checks in CI/CD, scans pull requests for LLM security and compliance issues, and runs evals locally so prompts never leave the machine. The project is MIT licensed and is now part of OpenAI.

Use cases: llm-evaluation, security-scanning

Reviews

Write a review

No reviews yet. Installed it? Reviews come from verified installers only. How reviews work

Fact sheet

Category: Evals · open source
Self-host: Yespromptfoo.dev
Open source: Yesraw.githubusercontent.com

Plans

Community	$0	free forever; open-source tool; 10k red-team probes per month; run locally or self-host
Enterprise	Contact sales	custom; team sharing, SSO, managed cloud deployment, priority support and SLA guarantees
On-Premise	Contact sales	custom; deployment on your own infrastructure; complete data isolation; dedicated runner

From the vendor's pricing page, read Oct 24, 2026.

Facts read from the vendor's own pages by claude-haiku-5-5 on Oct 24, 2026; a fact is shown only with the page it came from.

Signals

Stars: 26,378
Forks: 2,567
Open issues: 152
Last commit: Oct 23, 2026 (7 days ago)
Last release: Oct 23, 2026 (7 days ago)

From GitHub (fetched Oct 24, 2026).

Details

Packages: npm promptfoo
License: MIT
Repository: github.com/promptfoo/promptfoo
Website: www.promptfoo.dev
Docs: www.promptfoo.dev/docs/intro/
First seen: Oct 24, 2026
Last checked: Oct 24, 2026
Source: Website Docs Pricing page README github-graphql

Done for you

Rather not set it up yourself?

Chalk Labs scopes, builds and runs agents for your team: the task audit, the eval harness, the integrations and the weekly ops report. Fixed scope, priced before we start.

See how we build AI agents →Read the case studies

Chalk Labs is a two-founder AI and growth studio. We build AI agents, CRMs and MVPs for Web3 and AI companies, and have shipped 50+ product and token launches.

launches shipped: 50+; launches shipped
media placements: 500+; media placements
founders on every account: 2; founders on every account

Press we've placed: Forbes · CoinDesk · Decrypt · Cointelegraph · The Block

Rahil Jain, Co-founder, Chalk Labs. Ex-Goldman Sachs engineer. You talk to the person who builds it.

Get Promptfoo running in your business

Tell us what you want to automate. A founder replies within one working day with a plan, not a pitch deck.

Need help deploying Promptfoo?

Call Get help

Cite this page

Promptfoo. Agent Mart. https://agentmart.dev/observability/promptfoo (last updated 2026-10-24)