--- name: wetech-op-grafana description: Plan bounded read-only Wetech OP Grafana and TiDB Dashboard evidence batches for the auto-oncall Kubernetes runtime. Use only when the canonical Account Alias is Wetech or apac-test and the runner supplies one exact target and finite incident window. --- # Wetech OP Grafana Use the separately installed `evidence-query-planning` Skill for shared metric scope, source-cardinality, and selector-provenance requirements. This Provider Skill owns supported operations, actual label mappings, and transport only in that planning boundary; its existing product-capability guidance still applies. Under Agent9, call `managed_auto_oncall_wetech` as the only provider entrypoint; its Tool Sandbox owns credentials and transport. Under the legacy Kubernetes runtime described below, the Evidence Agent writes a plan and the runner-owned Provider Executor invokes the bundled helper. Never mix the two paths in one evidence round. Agent9 Managed Tool operations are: - `metrics_query_range`: exact target, absolute start/end, 15-300 second step, neutral name and unit, and one bounded Agent-authored PromQL expression; - `topsql`: exact metric-selected TiDB instance and a range no wider than 900 seconds. Default to five entries; request more only for a specific evidence question, with a top count no greater than 20. The Tool fixes `instance_type=tidb`, `group_by=sql_digest`, and `window=10s`; - `statements`: bounded TiDB Dashboard Statements aggregates; - `slow_query`: bounded TiDB Dashboard Slow Query diagnostic fields with optional 6-to-64-hex digest or prefix, order, direction, and a limit no greater than 20. Omit `orderBy` to use `query_time`; a non-default order must exactly match a field returned by the target Dashboard's `available_fields` endpoint. With a selected Digest or prefix, the Tool includes query text, decoded plans and structured operator facts for up to three executions, paired with each execution's identity, interval and numeric timings. No `includePlan` argument or separate diagnostic refinement is needed to request them. Broad discovery returns compact execution summaries without plans. Encoded `binary_plan` stays inside the Provider; when needed the Tool asks Dashboard to decode it and returns the decoded plan with structured operator facts. Missing or undecodable plans remain explicit per-execution gaps. Truncated plans are incomplete evidence. For a target SQL Slow Query, use a supplied full Digest or the Digest prefix from a prior Provider result. Returned Digests retain the full API value; reuse it unchanged for exact follow-up queries. The Tool converts historical prefixes to server-side text filtering before the row limit, then checks returned Digest matches. Opaque `entity-*` references are not Digests. Never request an already supplied Digest again from the customer. TopSQL and Slow Query `limit` is a JSON integer from 1 through 20; For TopSQL, explicitly pass `limit: 5` by default, including ordinary workload ranking and the first query on a selected instance. Do not start with 10 or 20 merely to improve coverage; request more only when a specific evidence objective cannot be answered with five entries. Do not use a larger discovery limit such as 100. If the Tool returns `dashboard_query_parameter_invalid`, use its `field` and `expected` details to correct that same objective in the current Turn, retaining the query ID and authorized target/window. Do not repeat unchanged arguments. The Agent-visible Managed Tool JSON Schema is the argument contract. Every call uses a unique stable semantic `queryId`; reuse that ID only for a changed same-Turn correction of the same failed objective. `unit` is a short plain string label of at most 32 characters, such as `cores`, `seconds`, `bytes`, `ratio`, or `count`, not an object or a conversion expression. For Managed Tool TopSQL, `instance` is required: copy an available exact TiDB selector as a non-empty string of at most 256 characters, without surrounding whitespace. Omission does not mean cluster-wide TopSQL. Opaque `entity-*` references are not selectors, and the legacy helper's `instanceFromQueryId` is not a Managed Tool argument. If no exact selector is available, preserve that TopSQL gap or use authorized Slow Query for the same evidence objective; do not invent an instance or repeatedly submit a missing selector. Within one Evidence Agent Turn, keep at most two Dashboard operations (`topsql`, `statements`, and `slow_query`) in flight. After issuing two Dashboard calls, wait for at least one Tool result before issuing another Dashboard call. Keep this concurrency control inside the same Turn; do not start another Evidence Agent Turn or Scenario refinement to schedule Dashboard work. `metrics_query_range` does not consume a Dashboard slot and may be issued independently. Treat the Managed Tool as an ordinary read-only Provider program. Inspect a primary error's `errorClass`, `field`, `expected`, `httpStatus`, `errorMessage`, `upstreamErrorCode`, `requestDigest`, and `repair`. When `repair.eligible=true` and one bounded caller-owned correction exists, call the Tool again in the current Evidence Agent Turn with the same `queryId`, frozen target and window, and at least one changed rejected argument. Make at most four correction attempts, stop earlier when no bounded correction exists, and never repeat the same `requestDigest` or unchanged arguments. Do not correct a supplemental Statements result. Same-Turn query repair is part of the current acquisition round; it is not a Scenario Diagnostic refinement. Authentication, shared Provider-session bootstrap, readiness, authorization, and browser/runtime availability are Provider-wide prerequisites rather than query errors. If the Tool reports `failureScope=provider_session`, `repair.action=stop_provider_calls`, or an equivalent failure before a Provider request, preserve that first failure and stop all remaining Provider calls in the current Turn. Do not correct arguments, repeat login, or start another objective against the unavailable shared session. TopSQL and Slow Query broad lists default to `includeSqlText=false`: retain full API-returned Digests, original instance selectors and key metrics, not SQL text or detailed plans. For single-query target matching, explicitly set `includeSqlText=true` only when candidate SQL text is needed; a text match remains a candidate for the Scenario Diagnostic Agent to interpret. Digest-selected Slow Query defaults to retaining SQL text with up to three execution-paired detailed plans. Copy a returned full Digest for that focused lookup. Overall cluster-query-latency and workload-hotspot lists stay text-free; do not fetch every candidate's SQL. `sqlTextOmitted` is deliberate evidence selection, not redaction, unsupported capability or failed acquisition. The Tool validates the APID ticket shape, reviewed cluster, operation, and absolute window before starting the browser. Installation policy is not a current-ticket input. This package is the legacy Kubernetes Provider Executor contract. The Evidence Agent plans queries; the runner invokes `scripts/wetech_op_grafana.py batch`. The Evidence Agent must never invoke provider operations itself. ## Security boundary - Never inspect or print credentials, browser state, cookies, tokens, TOTP seeds, or one-time codes. Bounded Provider results retain SQL, PromQL, full Digests, addresses and diagnostic specifications; these are not credentials. Do not bypass result-size limits to read unbounded raw payloads. - The package contains no credential value or TOTP artifact. The runtime reads `WETECH_P81_USERNAME`, `WETECH_P81_PASSWORD`, `WETECH_P81_TOTP_SECRET`, `WETECH_GRAFANA_USERNAME`, `WETECH_GRAFANA_PASSWORD`, `WETECH_DASHBOARD_USERNAME`, and `WETECH_DASHBOARD_PASSWORD` only from Kubernetes Secret-backed environment variables. - Keep the exact runner-owned account, target and finite time window. Never switch cluster, region, datasource, provider, or time semantics. - Do not query component logs. This package exposes no log operation. ## Batch plan contract When runner-owned `comparisonWindows` is present for a `workload-hotspot` refinement, plan exactly seven independent `browser-query-range` entries. Each entry must use one exact daily `start`/`end` pair from that list with identical metric semantics, entity grouping, unit, aggregation, and step. One batch is a transport container only; never replace the entries with one multi-day range. Write one `auto-oncall-op-query-batch-plan-v1` JSON object with exactly one bounded plan before execution: - top level: `schemaVersion`, `provider="wetech"`, exact `scenarioId`, exact `timeWindow`, `targetReference`, and `queries`; - copy `auth.targetReference` from the runner context into the top-level `targetReference` unchanged; - each query has a unique semantic `queryId`, one `operation`, and `params`; - use at most 12 selected queries; the limit is a ceiling, not a target; - the helper executes independent metric queries with a per-service-instance limit of five and independent TopSQL, Slow Query, and Statements Dashboard queries with a separate per-service-instance limit of two; a query using `instanceFromQueryId` or `windowFromQueryId` waits for its source metric; - eligible retries stay inside the operation that owns them, and the helper atomically refreshes one `auto-oncall-op-query-batch-v1` result after every completed query. Supported agent-facing operations: - `metric`: `params` requires `name` and PromQL `query`; it may contain `step`, `unit`, `metricGroupId`, `start`, and `end`. Apply `evidence-query-planning` for source-work limits and selector provenance. The helper rejects Prometheus responses above 8 MiB before JSON decoding with `grafana_query_result_too_large`. - `slow-query`: `params` may contain `limit`, `previewLimit`, `fields`, `text`, `digest`, `orderBy`, `desc`, `start`, and `end`; the query may also contain an ordered `handoffFields` array with at most 12 safe field names. - `statements`: `params` may contain `limit`, `previewLimit`, `fields`, `text`, `start`, and `end`. Use it only as supplemental context after TopSQL or Slow Query has established the workload candidate. Cluster-group sampling means its rows cannot establish rank, prevalence, aggregate contribution, a representative baseline, or absence. - `topsql`: requires either an exact `instance` with an explicit absolute `start`/`end` no wider than 900 seconds, or both `instanceFromQueryId` and `windowFromQueryId` referencing an earlier instance-labelled metric in the same batch. It may set `hotspotWindowSeconds` up to 900 and `limit`. `instanceFromQueryId` is the only helper-owned cross-operation selector dependency. It extracts the exact `instance` label from the referenced metric query. There is no generic metric-to-metric label dependency: when a second metric family is needed, use its native bounded grouping rather than assuming that `instance`, `store`, `address`, or another label has the same vocabulary. For a TiKV CPU hotspot or CPU-capacity question, start with TiKV process CPU per instance from `rate(process_cpu_seconds_total[2m])`, aggregated by the metric family's proven `instance` label. Compare it with the actual TiKV quota from `tikv_server_cpu_cores_quota` by that family's proven instance label. Add only TiKV-scope matchers already established by the current datasource contract or compact result; do not invent a component, job, pod, namespace, or cluster label. "Native grouping" means grouping by labels exposed by the metric family being queried. It does not mean switching to Kubernetes pod or container metrics. Do not replace TiKV process CPU and quota with `container_cpu_usage_seconds_total`, `kube_pod_container_resource_*`, or `container_cpu_cfs_*` merely because the objective mentions CPU or capacity. Select those Kubernetes families only when the supplied evidence question explicitly requires verification of a Kubernetes quota, throttle, or cgroup capacity boundary and their pod/namespace/cluster label scope is independently known from the same family or the current provider contract. Never derive a pod selector from a TiKV instance value or the reverse. Query fields may be nested under `params`; do not use provider CLI operation names in the plan. `fields` is a comma-separated non-empty string. The helper also normalizes a non-empty string array from older or imperfect plans to that wire format before validation. All query windows must stay inside the runner-owned window. Descriptions remain neutral and factual. For performance and resource incidents, plan toward an observable acquisition result rather than a successful command count: align the affected entity and window, then obtain a stable workload candidate from TopSQL or Slow Query. Statements may only corroborate that established candidate. Put already-known alternative surfaces in the same batch; do not wait for a Digest that the stable surfaces can discover. An empty or failed load-bearing query is a gap, not completion; an empty or failed Statements query is a visible supplemental gap that does not make the batch partial or trigger repair. The Provider Executor always materializes an explicit Slow Query order. An explicit plan value wins; otherwise `memory-oom` uses `memory_max DESC`, and other scenarios use `query_time DESC`. It also adds identity, total latency, resource, scan, and the active order field to any explicit `fields` projection. Use `handoffFields` to tell the Provider Executor which non-mandatory facts are most useful to the current Scenario Diagnostic Agent question. Do not list every field. Identity, digest prefix, instance, finish timestamp, derived execution interval/overlap, and `query_time` are always preserved. Prefer mechanism-discriminating groups: `stats`; Cop max/address pairs; `memory_max` and `disk_max`; RU/resource-control delay; scan/RocksDB counters; lock/backoff; or optimizer-phase timing. Keep a value and its address together. The helper validates this allowlist, adds selected fields to the provider projection, and preserves their order in the compact handoff. Before writing a Slow Query `handoffFields` array, read `references/slow-query-handoff-fields.json`. It is the small, exhaustive agent-facing field contract and also maps each diagnostic scenario's semantic question to the useful field groups. The Scenario Diagnostic Agent supplies a semantic evidence draft; the Evidence Agent owns translating synonyms such as "write phase", "Cop wait", "scan amplification", or "optimizer time" into the exact supported fields during initial planning and same-round query repair. Select only names present there, keep paired value/address fields together, and replace an unsupported optional projection with the closest supported field group that answers the same semantic question. Do not send field syntax back to the Scenario Diagnostic Agent, infer a new field from TiDB knowledge, or inspect helper source. If no supported equivalent exists, preserve that phase as a gap. The Provider Executor omits any unsupported optional name, records it in the materialized query result, and continues the remaining valid plan. For a narrow exact-Digest request that includes `binary_plan`, the Provider Executor uses the authenticated TiDB Dashboard `POST /api/visualplan/generate` surface to decode up to three incident-overlapping plans before compaction. It passes only the decoded operator summary to the Scenario Diagnostic Agent and never exposes the encoded plan or decoded raw plan. A decode rejection remains `encoded_plan_only`; it does not invalidate the otherwise usable Slow Query row. TiDB Slow Query `Time`/`timestamp` is the log print time after execution, not the statement start. For every Slow Query request, the Provider Executor keeps the runner-owned incident window canonical, extends only its internal finish-time candidate query by 15 minutes, fetches at least 100 candidates, derives `start_time = finish_time - query_time`, and retains only rows whose execution interval overlaps the original incident window. Use the derived `start_time`, `finish_time`, `overlaps_incident_window`, and `incident_overlap_seconds`; never reinterpret raw `timestamp` as onset. An overlap is temporal correlation only. A same-row `memory_max` is direct query-level peak-memory evidence, but does not by itself prove process RSS growth or the cluster-level root cause. The helper retries one transient Dashboard HTTP 500/502/503/504 response. For Slow Query `common.bad_request`, it makes one changed, minimal-projection request while retaining the effective order and direction. These are Provider Executor repairs, not new Evidence Agent rounds. Other executed-query failures are returned to the same Evidence Agent thread, which may rewrite the failed load-bearing objective up to four times in the same acquisition round. It may change PromQL, aggregation, labels, window, step, ordering, projection, or supported surface, but must preserve the exact target and diagnostic question. Stop instead of forcing a repair when the query is non-load-bearing, unsupported, already answered, or has no bounded changed plan. Never execute an unchanged plan. Statements is always non-load-bearing: never retry or refine it in a later Evidence Agent plan after an empty or failed result. For every metric plan, set `emptyResultExpectation` to `expected_non_empty`, `sparse_allowed`, or `unknown`. Use `expected_non_empty` only for a common baseline that this skill establishes for the exact target/component and window. Use `sparse_allowed` only for a legitimately sparse event/condition metric whose exact selector/label contract is established here. Otherwise use `unknown`. A successful expected-baseline empty gets one same-round, same-family, minimal selector/label validation. If it remains empty, preserve an unexplained gap and never use it as counter-evidence. A sparse empty is verified only under the trusted contract above. A rejected or nonexistent metric family remains failed. When the target inventory supplies an exact `datasource_uid`, direct Prometheus queries use that UID without enumerating `/api/datasources`. For a Grafana origin that denies datasource listing and individual lookup APIs, the inventory may instead supply a live-verified origin-local `datasource_id`; direct queries use that numeric proxy ID without enumeration. A numeric ID must not be reused on an unverified standby origin. These selectors are runner-owned target configuration, and the query response remains the readiness check. Datasource enumeration and exact-name fallback are used only when neither selector is configured. ## Runtime helper The runner invokes: ```bash python3 scripts/wetech_op_grafana.py batch \ evidence/scenarios/query-execution/op-query-context.json \ evidence/scenarios/query-execution/op-query-plan.json \ evidence/scenarios/query-execution/op-query-batch.json ``` Read `references/op-automation-contract.md` only when the plan needs a precise operation field not covered above. Do not read helper source during normal planning.