# ISS-0094: IOPS Metrics Without `group_right` Aggregate at Project Level Instead of Cluster Level

## Background

Discovered in workspace ws-1779849323231-f013bf8a9 (prod, `investibook.com`, created by
nizheming@pingcap.com). The workspace title is "wetech 用户 saas 集群组 01 ~ 05 的 tikv block
cache hit". The user reported via `/issue`:

> IOPS 相关的分析如果不加 group right 默认都是 project 级别，而不是 cluster 级别，
> 而且没法知道具体的 tikv 信息

Source: GitHub issue #601.

## Observed Problem

When `collect_evidence.py` queries IOPS metrics via the Clinic metrics API (or Prometheus-
compatible endpoint), the PromQL expressions used for EBS IOPS analysis do not include
`group_right()` (or equivalent label-preserving join/aggregation). As a result:

- The aggregated IOPS value is scoped to the entire **project** rather than the specific
  **cluster**.
- Per-instance (per-TiKV node) IOPS breakdown is unavailable — the user cannot identify
  which specific TiKV node is hitting IOPS limits.

This is especially impactful for Premium/Dedicated clusters where multiple clusters share
a project, and for cases where a single TiKV node is the hotspot (ISS-0007 related).

## Root Cause

The IOPS PromQL expressions in `collect_evidence.py` aggregate without preserving the
`cluster_id` or `instance` label, e.g.:

```promql
# Incorrect — project-level aggregation
sum(rate(tikv_store_size_bytes[5m]))

# Correct — cluster + per-instance level
sum by (instance) (rate(tikv_store_size_bytes[5m]))
```

The `group_right()` issue arises when joining metric series from different label sets and
the join drops per-instance labels. For Clinic API queries using `--query`, the caller must
ensure the PromQL includes per-instance selectors or uses the correct `by (instance,
cluster_id)` clause.

## Fix Plan

1. **Audit all IOPS/EBS-related PromQL in `collect_evidence.py` A1/A2 segments:**
   - Find all `sum(...)`, `avg(...)`, `max(...)` expressions used for IOPS metrics.
   - Add `by (instance)` (or `by (instance, cluster_id)`) to preserve per-TiKV granularity.

2. **Use `cluster_id` filter in Clinic API calls:**
   - Ensure all Clinic metrics API calls include the cluster-scoped selector
     `{cluster_id="<ID>"}` in every PromQL expression to avoid cross-cluster aggregation.

3. **Validate per-TiKV output in `evidence_summary_for_agent.txt`:**
   - The IOPS section should show a row per TiKV instance, not a single aggregate value.
   - Add a check: if all IOPS values are identical across instances, warn that aggregation
     may be project-level.

4. **Update SKILL.md Node C guidance:**
   - Add note: "IOPS metrics must be queried with `by (instance)` aggregation to identify
     per-TiKV hotspots. A single aggregate IOPS value is project-scoped and insufficient."

## Verification

Run a diagnosis on a cluster with 3+ TiKV nodes where IOPS varies across nodes. The
evidence_summary should show distinct IOPS values per instance (e.g., db-tikv-1: 2000
IOPS, db-tikv-2: 800 IOPS). If all values are identical, the aggregation is still project-
level.