# Manual Invoice

Codex plugin framework for auditable manual billing using dynamic read-only BigQuery, O11Y metrics, and O11Y logs. Future integrations can be added as MCP-backed tools without changing the case and calculation contracts.

The plugin does not require one SQL template or query-plan file per question. It discovers allowed BigQuery schemas at runtime, generates parameterized GoogleSQL, and runs validation, dry-run estimation, and execution through one command while saving exact JSONL results and query evidence.

Architecture overview: [Manual Invoice Plugin: Architecture and Execution Flow](docs/architecture-and-execution-flow.md).

## Structure

```text
.codex-plugin/plugin.json     Plugin manifest
skills/manual-invoice/       Reusable Codex workflow
cases/                       Versioned billing rules, queries, schemas, calculations
runs/                        Local generated run artifacts
../manual-invoice-output/    Private reconciliation bundles (default when run from workspace root)
```

## Install in Codex

The repository is a self-contained Codex plugin marketplace. Install it from GitHub with:

```bash
codex plugin marketplace add git@github.com:tidbcloud/manual-invoice-kit.git --ref main
codex plugin add manual-invoice@manual-invoice-kit
```

For local plugin development, run these commands from the repository root:

```bash
codex plugin marketplace add "$PWD"
codex plugin add manual-invoice@manual-invoice-kit
```

After pulling or publishing a new plugin version, refresh and reinstall it:

```bash
codex plugin marketplace upgrade manual-invoice-kit
codex plugin add manual-invoice@manual-invoice-kit
```

Verify the installation with `codex plugin list`, then start a new Codex task so the updated Skill is loaded. Run `$manual-invoice configure` in that new task to configure BigQuery, O11Y, and optional system tools.

## Nutshell/O11Y authentication prerequisite

Before running `$manual-invoice configure` for O11Y metrics or logs, create a GitHub Personal Access Token at [github.com/settings/tokens/new](https://github.com/settings/tokens/new) with these permissions:

- `read:user`
- `read:org`

If your GitHub organization requires SSO authorization, authorize the token for the organization as well. Keep the token private; it will be stored as `GITHUB_PAT` in the protected Manual Invoice configuration. Connect through an Office IP or Feilian VPN when configure performs the O11Y authentication check.

## First-use configuration

For normal use, start a new Codex task and invoke:

```text
$manual-invoice configure
```

You normally only need to provide two values:

| Setting | Required | Purpose |
|---|---|---|
| `GOOGLE_APPLICATION_CREDENTIALS` | When ambient Google ADC is unavailable | Absolute path to the BigQuery Service Account JSON file. Store the file outside this repository. |
| `GITHUB_PAT` | When using O11Y metrics or logs | GitHub PAT with `read:user` and `read:org`; Skill installation also needs access to `tidbcloud/nutshell-skills`. |

You can pass both values in the configure request:

```text
$manual-invoice configure GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/bigquery-service-account.json GITHUB_PAT=<github-pat>
```

This configures the project without asking for those two values again. However, putting a real PAT in the request stores it in the Codex task history. The recommended form is therefore:

```text
$manual-invoice configure GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/bigquery-service-account.json
```

The workflow then requests `GITHUB_PAT` through the hidden secret prompt. Do not paste the PAT into README files, committed configuration, or shared task messages.

Press Enter to leave `BQ_JOB_PROJECT` and `BQ_LOCATION` empty. The plugin infers them from the credentials and referenced BigQuery tables. The default query byte limit is also sufficient for initial use.

`$manual-invoice configure` handles the rest:

- creates the private `.manual-invoice.env` configuration;
- creates a persistent Python environment and installs `requirements.txt`;
- installs the approved Clinic/O11Y Skills;
- checks BigQuery and O11Y connectivity;
- detects required `curl` and `jq` commands and asks before installing them if missing.

Optional tools such as `gh`, `aws`, `gcloud`, `bq`, and `lark-cli` are not required for basic BigQuery and O11Y queries. O11Y access still requires an Office IP or Feilian VPN. The Service Account must already have the necessary BigQuery query and table permissions.

For manual or advanced configuration, see [configuration.md](skills/manual-invoice/references/configuration.md) and [.env.example](.env.example).

## Data-source behavior

All billing-source queries are file-backed. BigQuery runs save SQL, parameters, raw Job metadata, and JSONL results; Loki runs save LogQL, exact time bounds, raw paginated responses, normalized JSONL logs, and redacted metadata through `query_o11y_loki.py`.

After a Markdown invoice report is rendered, the workflow checks for `lark-cli`. When available, it asks whether to append the report to an existing Feishu/Lark document and requires the user to provide the target document link. Publication never happens implicitly. A redacted receipt containing the report hash, document URL, mode, and outcome is saved under run evidence before the reconciliation bundle is built.

For O11Y, callers no longer need to know provider, region, or deploy type up front. Given a cluster ID, `resolve_o11y_cluster.py` uses the installed nutshell `clinic-api` Skill when its required authentication is already available, then resolves the O11Y target through `o11y-data` with a bounded GS fallback. It validates an optional tenant ID and persists a token-free cluster context. `query_o11y_metrics.py` performs that resolution automatically before running file-backed PromQL and saving the raw response, normalized samples, and metadata.

```bash
<python-from-bootstrap> skills/manual-invoice/scripts/query_o11y_metrics.py \
  --cluster-id 10658480506726873193 \
  --tenant-id 1372813089209327038 \
  --query-file runs/example/evidence/query.promql \
  --start 2026-06-01T00:00:00Z \
  --end-exclusive 2026-07-01T00:00:00Z \
  --cluster-output runs/example/evidence/cluster.json \
  --raw-output runs/example/evidence/metrics-raw.json \
  --output runs/example/normalized/metrics.jsonl \
  --metadata-output runs/example/evidence/metrics-job.json
```

Provider, region, and deploy type remain optional hints for validation or for retrying an uncommon region after the bounded automatic scan fails.

Bootstrap dependencies outside the plugin cache:

```bash
python skills/manual-invoice/scripts/bootstrap.py
```

The JSON output contains the persistent Python path. Validate BigQuery using that interpreter:

```bash
<python-from-bootstrap> skills/manual-invoice/scripts/test_bigquery.py \
  --dataset gcp-digital-bi.metadb_new
```

Test the runner without submitting a query:

```bash
<python-from-bootstrap> skills/manual-invoice/scripts/query_bigquery.py --help
```

Discover any table in an allowed Dataset:

```bash
<python-from-bootstrap> skills/manual-invoice/scripts/bq_catalog.py \
  list-tables --dataset gcp-digital-bi.metadb_new

<python-from-bootstrap> skills/manual-invoice/scripts/bq_catalog.py \
  describe-table gcp-digital-bi.metadb_new.billing2_invoices
```

After generating parameterized SQL, query it with one command:

```bash
<python-from-bootstrap> skills/manual-invoice/scripts/query_bigquery.py \
  --sql-file runs/example/evidence/query.sql \
  --params-file runs/example/input/params.json \
  --output runs/example/normalized/result.jsonl \
  --metadata-output runs/example/evidence/job.json \
  --auto-execute
```

`--auto-execute` performs SQL policy validation and a dry-run estimate first, then executes only when the estimate is within `BQ_MAX_BYTES_BILLED`.

The bundled semantic model currently allows `gcp-digital-bi.metadb_new.*`. `billing2_invoices` is only the first reviewed entity mapping for the word “invoice”; it is not the only queryable table. To allow another Dataset, create `.manual-invoice/semantic-model.yaml` in the project, copy the bundled model, and add a reviewed `project.dataset.*` pattern under `policy.allowed_tables`.

For an auditable billing workflow, copy `cases/example-basic/` to a new case directory and replace the example project, dataset, rule, and schema values. For an ad hoc query, invoke `$manual-invoice` with the natural-language request; the Skill will inspect the needed schemas, generate SQL, execute it, and save evidence under a new `runs/` directory.

## Reconciliation output

Every completed billing calculation ends with a private, self-contained directory containing `report.md`, exact results, raw query evidence, normalized inputs, calculation code, validation/lineage, and a SHA-256 `manifest.json`. By default, running from the workspace root creates:

```text
manual-invoice-output/<run-id>/
```

Build a bundle from a completed run with:

```bash
<python-from-bootstrap> skills/manual-invoice/scripts/build_reconciliation_bundle.py \
  --run-dir runs/<run-id> \
  --report-source runs/<run-id>/manual-invoice.md
```

Set `MANUAL_INVOICE_OUTPUT_DIR` in `.manual-invoice.env` to choose a persistent output root, or pass `--output-dir` for one run. Existing bundles are not overwritten unless `--force` is explicit. Approved runs should always use a new run ID instead.