# Analysis Readiness Sample Evaluation

Date: 2026-03-29

## Goal

Evaluate whether the new sample artifact:

- `knowledge/repos/tidb-management-service/analysis-readiness.json`

actually improves analysis quality, instead of merely adding another file.

This evaluation is intentionally narrow.
It only tests **repo-local readiness questions** for `tidb-management-service`.
It does **not** claim that cross-repo join quality has improved yet.

## Compared Modes

### Baseline

Allowed files:

- `knowledge/repos/tidb-management-service/meta.json`
- `knowledge/repos/tidb-management-service/deep-scan-state.json`

### Treatment

Preferred file:

- `knowledge/repos/tidb-management-service/analysis-readiness.json`

Fallback files were not needed during this evaluation.

## Evaluation Questions

The same five questions were asked in both modes.

1. Is this repo currently suitable for runtime logical analysis?
2. Is the conclusion fresh, and what is the basis?
3. Which binaries are the primary runtime services?
4. What is the canonical runtime service ID for `dedicated-global-service`?
5. What confidence level and current limitations should be attached to the repo-level readiness judgment?

## Metrics

The evaluation uses four metrics.

### 1. files_opened

How many KB files must be opened to answer the question.

### 2. fully_answerable

Whether the question can be answered directly and explicitly from the allowed files.

### 3. answer_fields_complete

Whether the answer can include the expected structured fields instead of inference.

Expected fields by question:

- Q1: `analysis_readiness`, `scope`, `review_status`
- Q2: `freshness_status`, `source_commit`, `freshness_checked_at`, `basis`
- Q3: explicit primary runtime-service list
- Q4: canonical runtime service ID
- Q5: `confidence`, `conflict_status`, `known_gaps`

### 4. raw_source_needed

Whether raw source trees outside the KB would be needed.

## Results

| Q | Baseline files_opened | Treatment files_opened | Baseline fully_answerable | Treatment fully_answerable | Main difference |
|---|------------------------|------------------------|---------------------------|----------------------------|-----------------|
| 1 | 2 | 1 | partial | yes | treatment makes the readiness judgment explicit |
| 2 | 2 | 1 | no | yes | treatment adds freshness metadata and basis |
| 3 | 1 | 1 | yes | yes | no major gain except unified access path |
| 4 | 2 | 1 | no | yes | treatment adds canonical runtime service ID |
| 5 | 2 | 1 | no | yes | treatment adds confidence and known gaps |

## Structured Scoring

### Baseline

- files opened across 5 questions: `9`
- fully answerable questions: `1 / 5`
- questions answerable without structured inference: `1 / 5`
- raw source needed to fully complete all 5 answers: `yes`

### Treatment

- files opened across 5 questions: `5`
- fully answerable questions: `5 / 5`
- questions answerable without structured inference: `5 / 5`
- raw source needed to fully complete all 5 answers: `no`

## What Improved

### A. Lower lookup cost

The sample reduces file-open cost from `9` to `5` across the same question set.

More importantly, the answer path becomes stable:

- baseline requires remembering which fields live in which file
- treatment consolidates repo-local readiness into one artifact

### B. Less inference, more explicit fields

The biggest improvement is not coverage.
It is explicitness.

Before the sample:

- freshness could not be stated explicitly
- canonical runtime-service IDs did not exist
- confidence and known gaps were mostly narrative inference

After the sample:

- those fields are directly tracked

### C. Better repeatability

The treatment makes two repeated-answer risks smaller:

- forgetting to mention review status or freshness
- answering with different naming forms for the same runtime service

## What Did Not Improve Yet

This sample does **not** yet prove improvement for:

- deployment joins
- config joins
- cluster/namespace reverse lookups
- cross-repo blast-radius analysis

Reason:

the current sample artifact is repo-local and runtime-focused.
It is not yet connected to reverse indexes or deployment/config artifacts.

## Verdict

Verdict: `improvement proven for repo-local readiness questions`

The improvement is sufficient to justify expanding this pattern to one more already-scanned runtime repo.

The improvement is **not** sufficient to justify broad claims about end-to-end cross-repo analysis yet.

## Recommended Next Step

The next validation should be:

1. apply the same artifact pattern to one more already-scanned runtime repo
2. rerun the same 5-question evaluation
3. only then start the `by-service` reverse-index pilot

Why:

- step 1 validates whether the schema is reusable
- step 2 checks whether the benefit is repeatable
- step 3 extends the benefit from repo-local reasoning toward cross-repo reasoning