AGENTS.md · sqlfluff/sqlfluff

Sections

1. Project Overview
2. Repository Structure
3. Universal Conventions
4. Core Commands
5. Architecture Principles
6. Development Workflows
7. Component-Specific Instructions
8. Common Pitfalls
9. Configuration
10. Plugin System
11. Quick Reference

First lines (shown under the repo's MIT license)

# SQLFluff AI Coding Assistant Instructions

## Project Overview

SQLFluff is a dialect-flexible SQL linter and auto-fixer supporting 25+ SQL dialects including T-SQL, PostgreSQL, BigQuery, MySQL, Snowflake, and more. The project is written primarily in **Python** with an experimental **Rust** component for performance optimization.

**Core Architecture:**
- **Parser-first design**: SQL is lexed → parsed into segment trees → linted by rules → optionally auto-fixed
- **Dialect inheritance**: Each dialect extends ANSI base grammar using `.replace()` to override specific segments
- **Segment-based AST**: Everything is a `BaseSegment` subclass forming a recursive tree structure
- **Rule crawlers**: Rules traverse segment trees to find violations and generate `LintFix` objects

## Repository Structure

```
/
├── src/sqlfluff/              # Main Python package (see src/sqlfluff/AGENTS.md)
│   ├── dialects/              # SQL dialect definitions (see src/sqlfluff/dialects/AGENTS.md)
│   ├── rules/                 # Linting rules by category
│   ├── core/                  # Parser, lexer, config infrastructure
│   ├── cli/                   # Command-line interface
│   └── api/                   # Public Python API
├── sqlfluffrs/                # Experimental Rust components (see sqlfluffrs/AGENTS.md)
├── test/                      # Test suite (see test/AGENTS.md)
│   ├── fixtures/              # Test data (SQL files, YAML expected outputs)
│   ├── dialects/              # Dialect parsing tests
│   └── rules/                 # Rule testing infrastructure
├── docsv/                     # VitePress documentation (see docsv/README.md)
├── plugins/                   # Pluggable extensions (dbt templater, examples)
├── utils/                     # Build and development utilities
└── examples/                  # API usage examples
```

## Universal Conventions

### Language Support
- **Python**: 3.10 minimum, 3.13 recommended for development, 3.14 supported
- **Rust**: Experimental, used for performance-critical lexing/parsing

### Code Quality Standards
- **Formatting**: Black (Python), rustfmt (Rust)
- **Linting**: Ruff + Flake8 (Python), clippy (Rust)
- **Type Checking**: Mypy strict mode (Python)
- **Pre-commit hooks**: Run before all commits via `.venv/bin/pre-commit run --all-files`

### Testing Philosophy
- All tests must pass before merging
- Test coverage should reach 100%
- Use YAML fixtures for dialect/rule tests
- Mirror source structure in test directories

### Commit Messages
- Keep messages clear and descriptive
- Reference issue numbers when applicable
- Use conventional commit style when appropriate

## Core Commands

### Environment Setup
```bash
# Create development environment (first time)
tox -e py312 --devenv .venv
source .venv/bin/activate

# Always activate before working in a new terminal
source .venv/bin/activate
```

### Testing
```bash
# Run full test suite
tox

# Run specific Python version tests
tox -e py312

# Run with coverage
tox -e cov-init,py312,cov-report,linting,mypy

# Quick dialect test (after adding SQL fixtures)
python test/generate_parse_fixture_yml.py -d tsql
```

### Quality Checks
```bash
# Run all pre-commit checks (format, lint, type check)
.venv/bin/pre-commit run --all-files

# Individual checks
ruff format src/ test/          # Format
ruff check src/ test/           # Lint
mypy src/sqlfluff/              # Type check
```

### Building
```bash
# Install package in editable mode (done automatically by tox --devenv)
pip install -e .

# Install plugins
pip install -e plugins/sqlfluff-templater-dbt/
```

## Architecture Principles

### Layer Separation
The codebase enforces strict architectural boundaries (via `importlinter` in `pyproject.toml`):
- `core` layer cannot import `api`, `cli`, `dialects`, `rules`, or `utils`
- `api` layer cannot import `cli`
- Dependencies flow: `linter` → `rules` → `parser` → `errors`/`types` → `helpers`

### Immutability
- Segments are immutable - never modify directly
- Use `.copy()` or `LintFix` mechanisms for changes
- Parser creates fresh tree structures

### Lazy Loading
- Dialects loaded via `dialect_selector()` or `load_raw_dialect()`
- Never import dialect modules directly
- Supports dynamic dialect discovery

## Development Workflows

### Adding Dialect Features
1. Create `.sql` test files in `test/fixtures/dialects/<dialect>/`
2. Run `python test/generate_parse_fixture_yml.py -d <dialect>` to generate expected `.yml` outputs
3. Implement grammar in `src/sqlfluff/dialects/dialect_<name>.py`
4. Use `dialect.replace()` to override inherited ANSI segments
5. Verify: `tox -e generate-fixture-yml -- -d <dialect>`

See `src/sqlfluff/dialects/AGENTS.md` for detailed dialect development guide.

### Grammar is Permissive, Not a Validator

SQLFluff's parser is a **loose, allowable grammar** — not a SQL validator. It is
deliberately more permissive than any real engine, and parsing SQL that a given
database would reject is **not** a bug.

So, when adding or reviewing dialect grammar:

- ❌ Don't add (or ask contributors to add) "negative tests" asserting that
  invalid SQL fails to parse — no `unparsable` fixtures, no
  `pytest.raises(SQLParseError)` for "this should be rejected".
- ✅ Only add positive fixtures: valid SQL that must parse, plus its expected
  `.yml` tree.
- ❌ Don't tighten a grammar purely to reject something. Rejection is the
  engine's job; over-tightening breaks dialect variants, vendor extensions and
  templated SQL.
- ✅ Keep clause ordering, optionality and combinations loose unless a *valid*
  statement would otherwise mis-parse.
- ❌ **"`<engine>`-invalid SQL such as `...` parses successfully"** is not a
  finding on its own. Neither is "this accepts X despite the docs/docstring
  saying only Y". Both describe intended behaviour.
- ✅ **Parse time is not execution time.** A statement the server accepts
  syntactically and then rejects at runtime (semantic errors, privilege
  errors, "unknown column", MySQL error 1396, …) must still parse cleanly.

A finding here is only real if it names **valid SQL that fails to parse**, or
valid SQL that parses into the wrong tree. "Invalid SQL is accepted" is working
as designed.