# ISS-0084: Agent 在工具返回结果中不存在的数据上产生幻觉 ID

## Background

**Workspace**: ws-1779066858883-0e03b954e (dev environment)  
**Source GitHub issue**: #547 — "LLM 的幻觉太严重了"  
**User**: nizheming@pingcap.com  
**Date observed**: 2026-05-18

The user asked the agent to find cluster IDs for "wetech saas clusters 01~05" given a known tenant.
The agent called the DevOps skill to search for wetech tenants.

## Observed Behavior

### What DevOps actually returned

The DevOps search returned **exactly three** SKA/related orgs:

```
id: 1372813089209061633  alias: Wetech  name: Buffet's Org       plan: on_demand  segmentation: SKA
id: 1372813089209301464  alias: Wetech  name: aliyun's Org       plan: on_demand  segmentation: SKA
id: 1372813089209061799  alias: Zhuo Fanl  name: Wetech Trading  plan: free_trial segmentation: Non-KA
```

### What the agent fabricated

In the first assistant turn, the agent wrote:

> "两个 tenant 都可能是 wetech SKA 客户（`1372813089196917698` 和 `1372813089209301464`）"

The ID `1372813089196917698` **does not appear anywhere** in the DevOps response. Later the agent
also used `1372813089196917633` (another fabricated variant).

When the user pointed out the discrepancy, the agent confirmed:

> "这个 ID 是我**凭空捏造的**，不存在于任何返回数据中。这是严重的幻觉错误。"

### Tool call sequence

1. `[completed]` Search wetech tenant → returned 3 real IDs
2. Agent immediately fabricated a 4th ID `1372813089196917698` in its text
3. Subsequent tool calls used the fabricated ID to search for clusters — all failed
4. 22 tool calls total wasted on a phantom tenant ID

## Root Cause Analysis

This is a **numeric ID transcription hallucination** — the model modified digits in a long numeric
string (18+ digits) between reading it from tool output and writing it into its own reasoning.
This is a known LLM failure mode with long numeric identifiers.

The problem is distinct from ISS-0073 (ignoring user corrections) — here the agent never received
a user correction; it fabricated data unprompted in the first response.

**Contributing factor**: The clinic-cli lacked a `search clusters --tenant-id` command at the time
(fixed in commit 07bfb3c "支持 clinic-cli 按 tenantID 搜索集群"). The agent's repeated failed
tool calls to search clusters may have induced confabulation as the model tried to proceed despite
hitting API limitations.

## Fix Plan

### Short-term (SKILL.md instruction)

Add a **self-verification rule** to the relevant skills (tidb-clinic-diagnosis, clinic-cli,
devops SKILL.md):

> **ID Verification Rule**: When referencing any numeric or alphanumeric identifier (tenant ID,
> cluster ID, org ID, ticket ID) in reasoning or tool calls, you MUST copy it verbatim from the
> most recent tool output. Never reconstruct, modify, or guess an identifier from memory.
> If you are unsure which ID to use, re-read the tool output before proceeding.

### Long-term

1. **Structured data extraction**: After DevOps/API calls return ID lists, have the agent
   immediately produce a structured summary (e.g. "Found N tenant IDs: [list]") and reference
   only that summary in subsequent steps — reducing the distance between source and use.

2. **Cross-check before use**: Before passing any ID to a downstream tool call, verify the ID
   appears literally in the preceding tool output. If not found, re-query rather than proceed.

## Verification

A regression test should:
1. Feed a DevOps response with known tenant IDs
2. Verify the agent only uses IDs present in the response — never fabricated variants
3. Check that subsequent tool calls use verbatim IDs from the API output

## Related Issues

- [[ISS-0073]]: Agent sticks to wrong analysis after user correction (different: user must correct;
  here agent self-fabricates without external correction)
- clinic-cli commit 07bfb3c now supports `search clusters --tenant-id`, which removes the
  immediate trigger for this confabulation (search failures → agent fills gaps with guesses)