# Meta-server Cluster Event Scalability ## Current Behavior Meta-server maintains a process-local cluster event window with one latest event per cluster. The retained-event safety scan loads the latest event for each cluster with cluster-name keyset pagination. `GET /api/v1/events` still returns the complete latest-event snapshot in one response. The scan no longer transfers every retained historical event to meta-server, but its aggregation cost still grows with retained rows, while the API response size grows with the number of clusters. ## Required Invariants Any change must preserve the following behavior: - A full reload uses one consistent MySQL transaction snapshot for clusters and events. - The latest committed event for every cluster is retained. - The global high watermark remains the maximum observed event ID. - Late commits with lower event IDs remain detectable by the retained-event safety scan. - Clients can detect an epoch change and rebuild their local cache. - No event list is silently truncated. ## Proposed Stages ### Stage 1: Return Only the Latest Event Per Cluster Status: implemented as the retained-event full-scan path. The safety-scan query selects `MAX(event_id)` for each cluster and joins it back to `cluster_events` to obtain the event payload. It uses cluster-name keyset pagination and keeps the fast incremental query based on `event_id > high_watermark`. The current TMS schema provides both relevant composite indexes: ```sql (kubernetes, name, event_id) (kubernetes, event_id, name) ``` Each compact page groups the next cluster-name range, using `(kubernetes, name, event_id)`, and returns at most one row per cluster. Meta-server sorts the completed compact result by Event ID before publishing it. This avoids repeatedly aggregating hot-cluster history across Event-ID pages while reducing rows transferred to meta-server from all retained event history to at most one row per cluster. The subquery uses `FORCE INDEX (idx_namespace_name)`, sourced from `inframysql.ClusterEventACKIndex`, because MySQL can otherwise prefer the Event-ID-first index and rescan retained history on every page. A MySQL 8.0.36 `EXPLAIN FORMAT=JSON` test verifies that the subquery uses this index with `range` access. Production-like `EXPLAIN ANALYZE` remains useful for tracking rows examined and latency. Meta-server treats all-empty RDS connection fields as MySQL disabled. Partial MySQL configuration fails startup with the missing fields reported, while complete MySQL configuration also requires a non-empty Kubernetes cluster name. All cluster and Event queries include that Kubernetes predicate, so the forced composite index always has its leading column constrained. ### Stage 2: Add a Paginated Event Snapshot API Introduce a versioned endpoint that returns a bounded page: ```text GET /api/v2/events?epoch=&snapshot=&cursor=&limit= ``` Each response should include: - epoch - immutable snapshot generation - high watermark - rewind lower bound - events - next cursor - completion flag A client must finish one immutable snapshot generation before applying it. If the epoch or snapshot generation changes during pagination, the client restarts from the first page. The existing v1 endpoint remains unchanged during migration. This stage bounds response size and encoding latency. It can initially page over the current immutable in-memory slice, so it does not need to change event reconciliation at the same time. ### Stage 3: Materialize Latest Events If the number of clusters itself becomes too large, maintain a producer-side table with one latest event row per cluster, updated in the same transaction as cluster mutations. Meta-server can then load the compacted table and use the append-only event table only for incremental catch-up and audit history. This is the most invasive option because it changes the writer contract and requires backfill, dual-write validation, and rollback planning. It should only follow measurements from stages 1 and 2. ## Additional Metrics The implementation exposes: - `metaserver_cluster_event_full_scan_duration_seconds` - `metaserver_cluster_event_full_scan_returned_rows` - `metaserver_cluster_event_full_scan_pages` - `metaserver_cluster_event_snapshot_entries` `returned_rows` measures rows delivered to meta-server, not MySQL rows examined. Storage-level scan cost must be validated with MySQL observability and `EXPLAIN ANALYZE`. Future API pagination work should add serialized response bytes, endpoint encoding duration, and client restart counts caused by epoch or snapshot changes. ## Approaches To Avoid - Increasing the existing batch size only shifts memory and latency pressure. - `OFFSET` pagination becomes slower as the offset grows. - Returning only the first N events breaks cache correctness. - Removing the retained-event safety scan reintroduces the lower-ID late-commit gap.