AGENTS.md · open-metadata/OpenMetadata

Sections

1. About OpenMetadata
2. Stack at a glance
3. Environment setup (every session)
4. Repository layout
5. Hard cross-cutting constraints (apply to every session, all languages)
6. Pointer index — when to reach for what
7. Git commit conventions

First lines (shown under the repo's Apache-2.0 license)

# AGENTS.md

Always-loaded guidance for every session, whatever the harness — **the only instruction file:** every `CLAUDE.md` and
`.github/copilot-instructions.md` is a symlink to the `AGENTS.md` beside it
(ADR:2026-10-09-agents-md-is-the-one-instruction-file), so edit this file, never a copy. **Language- and path-specific
rules live in `.claude/rules/*.md` (Claude Code auto-loads them on matching files; other harnesses read the matching
one before editing); procedures live in skills (loaded on invoke).** This file is the map — see the pointer index at
the bottom. Read [ARCHITECTURE.md](ARCHITECTURE.md) for the **system map** (modules, the request/ingestion/search
paths, the invariants that hold); read [DEVELOPER.md](DEVELOPER.md) for **how to build, test, and add an entity or
connector** (deep dives + end-to-end checklists). Consult [docs/index.md](docs/index.md) — the **knowledge index** —
to **find existing design, plan, and reference docs** for whatever area you're working on.

## About OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, observability, and governance — a
multi-module project with a Java backend, a React/TypeScript frontend, a Python ingestion framework,
and Docker infrastructure.

## Stack at a glance

- **Backend**: Java 21 + Dropwizard, multi-module Maven.
- **Frontend**: React + TypeScript, built with **Vite** (dev server on :3000); component library
  `openmetadata-ui-core-components` — the **UntitledUI + Tailwind v4** (`tw:` prefix, react-aria-components)
  go-forward design system; legacy stack is Ant Design + Less (deprecated). Machine-readable design-system
  specs: `openmetadata-ui/src/main/resources/ui/specs/` (read `specs/README.md` before UI work).
- **Ingestion**: Python (`>=3.10`, no pinned ceiling; **CI runs 3.10**) + Pydantic 2.x, 75+ connectors.
- **Database**: MySQL (default) or PostgreSQL. **Search**: Elasticsearch 7.17+ or OpenSearch 2.6+.
- **Infrastructure**: Apache Airflow for ingestion orchestration.

## Environment setup (every session)

- **Local Python commands synchronize locked profiles; application profiles reuse `env/`:**
  ```bash
  source env/bin/activate          # first time: python3.11 -m venv env
  python --version                 # expect 3.10.x/3.11.x
  ```
  `make install_test` bootstraps pinned uv and Nox; sessions in `ingestion/noxfile.py`
  synchronize dependencies from the lock before execution. Style, generation and coverage
  tools use separate `ingestion/.nox/` environments. `make install_dev_env` remains an alias
  for `make install_test`; published extras remain available. Use
  `make generate ARGS=--python-only` for Python-only generation and
  `python3 scripts/check_generated_models.py --check` to validate freshness.
  In a Claude Code **worktree** the venv is NOT copied — create one
  (`python3.11 -m venv env && source env/bin/activate && cd ingestion && make install_dev`) or
  symlink the main repo's (`ln -s /path/to/main-repo/env env`).
- **One-call setup (macOS + Linux)**: `make dev_setup` (or `./scripts/dev_setup.sh`) does everything
  below — toolchain, venv, generation, UI deps, pre-commit — and is idempotent. `make dev_check`
  diagnoses an existing checkout without changing it. See the `dev-setup` skill.
- **First-time bootstrap** (from the **repo root** — `make generate` is a root-only target; it does
  not exist under `ingestion/`):
  ```bash
  make prerequisites
  source env/bin/activate && cd ingestion && make install_dev_env && cd ..
  make generate                    # regenerate models after any schema change
  make yarn_install_cache
  make install_test precommit_install   # activate the commit-time format/license gate (pre-commit)
  ```
  The last line installs the `pre-commit` hooks (`.pre-commit-config.yaml`): on `git commit` they run
  Java format (spotless), Python format (ruff), UI format (prettier), design-token, and Apache-2.0
  license checks on your changed files, matching CI. Do not skip them with `--no-verify`.
- **Java**: Java 21; use `mvn`. **Frontend**: use `yarn` (never `npm`); frontend root is `openmetadata-ui/src/main/resources/ui/`.
- **Docker dev services**: `docker compose -f docker/development/docker-compose.yml up -d`.

## Repository layout

Maven modules (reactor order is computed from the graph, not this list):

- `openmetadata-spec/` — JSON Schemas + generated POJOs; the schema-first source of truth
- `openmetadata-sdk/` — Java client SDK
- `common/` — shared utilities (`CommonUtil`, etc.)
- `openmetadata-shaded-deps/` — ES/OS clients relocated behind `es.*`/`os.*` (do not edit — see rules)
- `openmetadata-service/` — core Java backend, REST APIs, repositories, migrations runner
- `openmetadata-k8s-operator/` — Kubernetes operator
- `openmetadata-integration-tests/` — backend API integration tests (`*IT.java`)
- `openmetadata-mcp/` — MCP server
- `openmetadata-ui-core-components/` — canonical React component library
- `openmetadata-ui/src/main/resources/ui/` — React frontend application
- `openmetadata-dist/` — packaging/distribution
- `openmetadata-clients/` — client artifacts

Other key trees: `ingestion/` (Python framework + connectors), `bootstrap/sql/` (DB migrations), `conf/` (configuration), `docker/` (local + prod deployment).

## Hard cross-cutting constraints (apply to every session, all languages)

**Secrets & security.** Never commit secrets — use environment variables or a secrets manager.
Auth is JWT with OAuth2/SAML; RBAC lives in Java entities; config in `conf/openmetadata.yaml`.
**Do not modify `.github/workflows/**` on your own** — CI workflows are a supply-chain surface; a
`PreToolUse` hook blocks edits there unless the user explicitly authorizes them (by setting
`CLAUDE_ALLOW_WORKFLOW_EDITS=1`). Ask first.

**All caches MUST be bounded.** Never use a bare `dict` / `HashMap` / `Map` as a cache without an
explicit size cap — they grow with input and OOM on large catalogs/ingestions (only exception: the
user explicitly asks for unbounded). Pick a sane default (100–1000 entries); if unsure, ask.
Python: `collections.OrderedDict` + `popitem(last=False)`, `@functools.lru_cache(maxsize=N)`, or
`cachetools.LRUCache` (cache hits **and** misses). Java: Caffeine/Guava `maximumSize(N)`.
TypeScript: `lru-cache`. Before adding a cache, check it isn't already cached a layer down (e.g.
`OpenMetadata._search_es_entity` is already `@lru_cache(maxsize=512)`).

**Comments explain *why*, never restate code.** Do NOT add comments that describe what obvious code
does (`// Create user` before `createUser()`). Only comment complex business logic, non-obvious
algorithms/workarounds, public-API JavaDoc, or `TODO/FIXME` with a ticket reference. If code needs a
comment to be understood, refactor it to be clearer instead.

**Testing philosophy.** Test real behavior, not mock wiring — if a test mocks 3+ of your own classes
to verify a method call, it tests the wrong thing. Prefer integration tests over heavily-mocked unit
tests (this project has real ITs: `OpenMetadataApplicationTest`, Docker, real OpenSearch). Mocks are
for boundaries (HTTP clients, third-party APIs), not internals. Ask "what breaks if this test passes
but the code is wrong?" — if the answer is "nothing", rewrite it. Assert on observable outcomes
(API responses, DB state), not internal `verify()` calls.

**License headers are per-module — copy one from a sibling file, never assume Apache.** UI TS/TSX:
Apache-2.0 (`yarn license-header-fix`). Python: `ingestion/` is **Collate Community License 1.0** (`ingestion/LICENSE`); `openmetadata-airflow-apis/` Python files use the same Collate header template.
Java: Apache-2.0, most files carry none.
Only the UI is enforced — the `ui-license-header` pre-commit hook and CI `ui-checkstyle`; spotless
and `py_format_check` never look at headers, so a wrong Python or Java header ships silently.

**Schema-first.** JSON Schemas in `openmetadata-spec/` are the single source of truth; all generated
code (Java POJOs, Pydantic models, TS types) is derived. **Edit the schema, then regenerate — never
hand-edit generated output.** Details in `.claude/rules/schema-first.md`.

**Output style.** Clean code blocks, no unnecessary explanation; assume an experienced reader; focus
on functionality over education. Do not add unnecessary blank lines between prose and code blocks.

**No wrappers around core-ui components.** If a wrapper component exists that wraps a core-ui component, skip it — use the `openmetadata-ui-core-components` component directly at the call site. Only create a wrapper if it adds genuine, non-trivial behaviour (custom hook logic, composed sub-components, domain-specific state). A wrapper that only passes props through is a maintenance liability: it hides which core component is in use and drifts from upstream updates.

**Icons and assets: always go through the design-system layer.** Import icons from `@openmetadata/ui-core-components/icons`, never directly from `@untitledui/icons` — the barrel ships only the design team's own curated SVG set, so add a missing glyph there via `yarn icons:generate` (see the core-components `ICONS.md`) rather than reaching for `@untitledui/icons`. For the same reason, do not import SVG icons directly from `assets/` paths; use the designated abstraction instead.

## Pointer index — when to reach for what

### Path-scoped rules (`.claude/rules/*.md`, auto-load on matching files)

| Rule file | Reach for it when you are editing… |
|---|---|
| `java.md` / `java-impact-map.md` | any `**/*.java` — style, spotless, no-wildcard, Kafka-grade method/class rules, ITs; own new code and ITs in `.github/java-tests/impact-map.json` before the PR |
| `frontend-react.md` | UI `*.{ts,tsx}` — components, hooks, state, types, and the CI lint code-rules |
| `frontend-styling.md` | UI `*.{ts,tsx,less,css}` — `tw:` prefix, design tokens, ring→border, token-audit |
| `component-library.md` | UI `*.{ts,tsx}` — prefer `ui-core-components`, do not add Ant Design for new work |
| `frontend-performance.md` | UI `*.{ts,tsx}` — waterfalls, barrel imports, re-renders, bundle discipline |
| `frontend-permissions.md` | UI `*.{ts,tsx}` — `useEntityPermissions` and derived flags; never read a permission key directly |
| `frontend-a11y.md` | UI `*.{ts,tsx}` — semantics over `div`+`role`, keyboard, focus, contrast, targets |
| `i18n.md` | UI `*.{ts,tsx}` + `src/locale/**` — no string literals, `yarn i18n`, translate placeholders |
| `frontend-playwright.md` | UI `playwright/**` — E2E test constraints |
| `python-ingestion.md` | `ingestion/src/**/*.py` — pytest style, connector-specific-file rule, `model_str()` |
| `schema-first.md` | `openmetadata-spec/.../schema/**` and any `generated/**` — regen, never hand-edit generated |
| `migrations.md` | `bootstrap/sql/**` — append-only, native path, MySQL+Postgres, idempotent |

### Repo coding conventions (read before writing non-trivial code)

- `docs/design-patterns.md` — the design patterns this codebase uses idiomatically (Template Method
  for repositories, Factory/Registry for dispatch, Strategy/Adapter/Observer, the ingestion
  Source→Sink pipeline, …) with the canonical class to copy each from. Extend the established pattern
  rather than inventing a parallel one.
- `openmetadata-ui/src/main/resources/ui/DEVELOPER_HANDBOOK.md` — **the UI folder structure and file
  naming spec.** Read before creating any new file under `openmetadata-ui/.../ui/src/`. Layers stay
  top-level (`components/`, `pages/`, `rest/`, `utils/`, `hooks/`) and are grouped inside by
  `domain/feature/`; new files use one stem with a role suffix (`GlossaryList.tsx`, `.types.ts`,
  `.utils.ts`, `.test.tsx`). Legacy `.component.tsx`/`.interface.ts` files stay as they are.
- `REVIEW.md` — **the review bar every PR is held to**: the logic defects, test gaps and flaky-test
  shapes reviewers keep finding here. Read it before writing code; `pr-checklist` self-reviews against it.
- `docs/decisions/` — dated records of the choices that shape the code. **When a choice looks