ambient-healthcare-agent-with-nemotron-voice-agent

Files (42)

• BENCHMARK.md
• SKILL.md
• EVAL.md · evals/
• config.yml · evals/
• evals.json · evals/
• grader.py · evals/
• __init__.py · references/appointment-making/database/
• db.py · references/appointment-making/database/
• schema.sql · references/appointment-making/database/
• seed.py · references/appointment-making/database/
• appointment_types.jsonl · references/appointment-making/database/seed_data/
• doctors.jsonl · references/appointment-making/database/seed_data/
• expected-conversation.yaml · references/appointment-making/
• prompt.yaml · references/appointment-making/
• test-ambient-agent-tools.py · references/appointment-making/
• tool-implementation.py · references/appointment-making/
• tools.yaml · references/appointment-making/
• expected-conversation-template.yaml · references/custom/
• generic-implementation.md · references/custom/
• guide.md · references/custom/
• prompt-template.yaml · references/custom/
• testing-protocol.md · references/custom/
• tool-implementation-template.py · references/custom/
• tools-template.yaml · references/custom/
• deployment-modes.md · references/
• example-design-choices.md · references/
• expected-conversation.yaml · references/patient-intake/
• patient-intake-speech-guard.py · references/patient-intake/
• prompt.yaml · references/patient-intake/
• test-ambient-agent-tools.py · references/patient-intake/
• tool-implementation.py · references/patient-intake/
• tools.yaml · references/patient-intake/
• user-experience-flowchart.md · references/
• apply_generic_agent_template.py · scripts/
• inspect_nva_generic_defaults.py · scripts/
• nva_generic_defaults.py · scripts/
• run_expected_conversation.py · scripts/
• verify_docker_compose_access.py · scripts/
• skill-card.md
• skill.oms.sig
• test_apply_template_safety.py · tests/
• test_eval_contract.py · tests/

SKILL.md (first lines)

# Ambient Healthcare Agent with Nemotron Voice Agent

## Purpose

Customize `src/examples/generic/` in a user-selected NVIDIA Nemotron Voice Agent (NVA) checkout. Use this skill for appointment making, five-field patient intake, or a developer-defined healthcare workflow that fits one Pipecat pipeline with an LLM prompt, OpenAI-style tool schemas, and Python handlers.

This skill is owned by Healthcare TME. It is not maintained or endorsed by the NVA team, and it does not live in the NVA repository. For ordinary NVA deployment or non-healthcare voice work, follow the public NVA documentation instead.

Read [the user-experience flowchart](references/user-experience-flowchart.md) when a visual overview of the gated workflow or bundled scenario state machines would help.

## Requirements

- A compatible checkout of `https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent`
- Python 3.10+ with PyYAML, Git, and Docker Compose
- Network access and credentials for the user-selected LLM, ASR, and TTS services
- Permission to write only to the NVA checkout path the user explicitly supplies
- Human review before sending fictional healthcare-style histories to any model endpoint

Treat the skill loader's installed directory as `SKILL_DIR`. Resolve bundled `scripts/` and `references/` from that directory, not from the current working directory.

## Required Welcome

Begin a new positive workflow with this phrase verbatim:

```text
Welcome to the NVIDIA Nemotron Voice Agent (NVA for short). We will customize NVA for creating ambient healthcare agents.
Now I will make a fresh clone of the Nemotron Voice Agent repository, and this will be the directory we work out of. Where would you like me to clone the repo to? Please provide a path.
If you already have a clone of the repository somewhere, please point me to the path.
```

Then stop. Do not inspect files, search for clones, reuse a path from earlier context, run tools, or choose a default. Set `NVA_ROOT` only from a path the user provides or confirms after this welcome.

## Instructions

### 1. Resolve and validate the NVA checkout

If the user requests a fresh clone, clone only to their exact destination and only with network and write permission:

```bash
git clone https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent.git "$NVA_ROOT"
```

If cloning fails, report the observed reason and ask the user to either grant the session the required access via `/permissions` and request a retry, or manually clone the NVA repository and provide its path. Do not retry until the user grants access or supplies a checkout path. For either a fresh or existing checkout, require these markers:

```bash
test -f "$NVA_ROOT/docker-compose.yml" && \
test -f "$NVA_ROOT/examples_registry.yaml" && \
test -d "$NVA_ROOT/src/examples/generic"
```

An invalid path is a hard stop. Do not modify any repository before this check passes.

After the markers pass, create `$NVA_ROOT/.env` by copying `$NVA_ROOT/.env.example` when the target does not already exist. Preserve an existing `.env` and never print its contents. If `.env` is absent and the template is missing, report that setup failure and stop. Do this immediately after validating a fresh clone or an existing checkout, before asking the user to choose hosted services.

### 2. Inspect compatibility and service defaults

Run:

```bash
python3 "$SKILL_DIR/scripts/inspect_nva_generic_defaults.py" \
  --nva-root "$NVA_ROOT" --check-compatibility
```

Stop on compatibility errors. Report the inspected prompt plus the LLM, ASR, and TTS key, display name, model/server, base URL when present, and catalog source. Never substitute release-specific defaults from memory.

The preflight checks Python structure and required capabilities rather than an NVA version number or one exact source string. It also discovers deployment skills from both `skills/*/SKILL.md` and `.agents/skills/*/SKILL.md`. A passing check is not permission to guess through an unknown layout: stop when syntax or a required semantic hook is ambiguous.

### 3. Select the runtime and verify setup access

Explain that the default is the compatible public NVIDIA AI Endpoint entries in the NVA cloud catalog. Before any inference, let the user choose public NVIDIA endpoints, NVA-managed local NIMs, existing NIM endpoints, or a mixed layout. Never silently fall back to public endpoints after opt-out. In the same message as these choices, give the user the actual absolute path to `$NVA_ROOT/.env` and tell them to fill in its `NVIDIA_API_KEY=` entry if they choose public NVIDIA AI Endpoints. Explain that the key authenticates access to those endpoints. Direct the user to the public NVA deployment documentation for credential setup; never ask them to paste a secret into chat or display the file contents. State:

```text
The NVIDIA_API_KEY is required to utilize public NVIDIA AI Endpoints. With this key configured, I will be running live tests while customizing and standing up a Nemotron Voice Agent application.
```

Before applying a healthcare overlay:

1. Run `python3 "$SKILL_DIR/scripts/verify_docker_compose_access.py"` and require all checks to pass. If Docker Compose access is blocked by session permissions, report the observed failure, ask the user to grant the required access via `/permissions`, and stop until the user requests a retry.
2. Use the public NVA deployment instructions to identify one recipe and the selected runtime's credential and endpoint requirements. Do not start the unmodified Generic recipe or run inference at this stage.

If repository compatibility or Docker Compose access fails, report the exact failed gate and stop before applying an overlay. Missing credentials or unavailable endpoints do not prevent overlay and static validation, but they block the later authenticated service checks, live validation, and handoff. Do not change runtime modes silently or claim the app is ready.

### 4. Obtain informed scenario selection

After checkout compatibility and Docker Compose access pass, restate the resolved runtime and services, then ask exactly:

```text
What type of voice agent application would you like to create? We have two example default use cases, appointment making and patient intake, or you could tell me your own use case.
```

Offer `appointment-making example`, `patient-intake example`, and `customize your own use case`. In the same message, replace the destination placeholder below with the actual resolved LLM display name and base URL or host:

```text
If you choose either example, I will apply its customization and send its bundled fictional conversation histories to <resolved destination> for live validation during setup and before handoff. The appointment example includes the fictional patient Jordan Patel, date of birth 1979-09-24, and appointment details. The patient-intake example includes the fictional patient Maya Chen, date of birth 1988-04-12, symptoms, current medications, and pharmacy details.
```

Wait for selection. Selecting a preset after this disclosure authorizes only the disclosed fixture and destination. Ask again if either changes.

### 5. Apply a preset

For a selected preset, apply the bundled overlay without additional design questions:

```bash
python3 "$SKILL_DIR/scripts/apply_generic_agent_template.py" \
  --nva-root "$NVA_ROOT" \
  --scenario-dir "$SKILL_DIR/references/appointment-making"
```

Use `references/patient-intake` for patient intake. The applier must preserve current LLM/ASR/TTS defaults, set the scenario prompt as the Generic default, patch only supported insertion points, and remain idempotent.

The applier reruns compatibility with the selected scenario before writing. To inspect that gate separately, pass `--scenario appointment-making`, `--scenario patient-intake`, or `--scenario custom` together with `--check-compatibility`. Scenario checks must cover every scenario-specific hook, including deterministic session startup for both examples.

Appointment making installs SQLite support, initializes `data/appointment-making/appointment_schedule.sqlite`, and creates or merges `docker-compose.override.yml`; it must not edit the base Compose file. It also queues the exact fixed opening greeting once at session start and suppresses the model-generated intro. Patient intake collects name, date of birth, symptoms, current medications, and preferred pharmacy, and installs deterministic turn/speech guards plus generated unit tests. Preserve its shared conversation state, earliest-missing-field question, exactly-once direct tool responses, one-time welcome, silent empty tool transitions, interruption forwarding, and interruptible welcome. These are code-backed safety invariants, not prompt-only suggestions. Read `references/example-design-choices.md` when implementation detail is needed.

### 6. Design a custom workflow

For `customize your own use case`, first ask what the conversation should accomplish; which fields are required, optional, or sensitive; what must be known and confirmed before each tool call; what each tool should read, write, and return; and which representative histories demonstrate message content and tool timing.

Use `references/custom/guide.md` and its templates. Present the proposed expected-conversation artifact, resolved destination, and data fields to the user. Do not implement or transmit it until the user approves that exact artifact and destination.

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills