# MaxCompute Parquet Predicate Pushdown

## Background

Clinic Data Service executes Alibaba Cloud warehouse queries through MaxCompute
SQL tasks. The current task construction does not provide query settings. For
OSS-backed Parquet external tables, a narrow timestamp predicate can therefore
scan the same date partition as a much wider timestamp range, and ordered wide
projections spend most of their latency inside the MaxCompute job.

Production measurements through the canonical Data Proxy SQL API showed that
reducing a timestamp window from three hours to thirty minutes did not reduce
MaxCompute input bytes. This plan addresses that engine behavior directly. It
does not depend on, or change, the Slow Log Explorer API.

## Goal

Add a reversible MaxCompute engine setting that enables the native Parquet
reader and Parquet predicate pushdown only for eligible slow-query Parquet
reads. Keep the setting disabled unless deployment configuration explicitly
enables it, then validate it with controlled staging A/B queries before
production rollout.

## Current Behavior

- `MaxComputeEngine.Execute` creates `clinic_query` with nil task hints.
- Query predicates are preserved in SQL, but the engine receives no Parquet
  native-reader or predicate-pushdown settings from Clinic.
- DDL, table existence checks, partition repair, and health checks use separate
  SQL tasks.

## Target Contract

- Add `alicloud.maxcompute.enable_slow_query_parquet_predicate_pushdown` and a
  required `slow_query_parquet_predicate_pushdown_regions` rollout allowlist.
- The default is `false`, preserving existing behavior for configurations that
  omit the field.
- When enabled, a `MaxComputeEngine.Execute` task receives the settings only if
  all of the following fail-closed checks pass:
  - the data source is Alibaba Cloud OSS;
  - the query region is in the explicit allowlist (an empty list keeps the
    optimization disabled);
  - strict AST parsing proves the statement is a SELECT or set operation;
  - every referenced table is exactly `slow_query_logs` or
    `slow_query_logs_<numeric cluster id>`;
  - every qualified database is the current region's MaxCompute project.
- Eligible tasks receive:
  - `odps.ext.parquet.native=true`
  - `odps.sql.parquet.use.predicate.pushdown=true`
- Parse errors, tableless queries, DDL/DML, TopSQL, views, mixed-table queries,
  cross-project queries, and unsupported data sources keep nil hints.
- DDL/DML maintenance tasks, table checks, health checks, API request/response
  contracts, SQL rewriting, and all non-MaxCompute engines remain unchanged.
- The setting is deployment-owned. Canonical and legacy API callers cannot
  select query hints or override provider placement.

## Affected Modules and Routes

- Go module: `clinic_data_service`
- Packages: `pkg/config`, `pkg/engine`, `pkg/engine_router`
- Entrypoint: `clinic_data_service/cmd/api/main.go`
- Runtime image: `clinic/data-service`
- API paths using the engine include canonical
  `/data-proxy/api/v1/orgs/{orgID}/clusters/{clusterID}/sql-queries` and their
  existing legacy compatibility route.

No module ownership, executable entrypoint, image name, workflow, or
app-delivery mapping changes, so `docs/clinic-module-overview.md` does not need
an update.

## Implementation

1. Add the default-disabled MaxCompute configuration field and required region
   allowlist, and document them in the sample Data Proxy configuration.
2. Pass the field through `engine_router` into `MaxComputeEngineConfig`.
3. Use the existing strict AST table-reference extractor to prove slow-query
   eligibility, build a fresh hints map for each eligible task, and pass nil for
   every disabled or ineligible case.
4. Include the enabled state and region allowlist in the existing safe startup
   log so a rollout can confirm the effective configuration without logging SQL.
5. Add focused positive and negative tests for scope, SDK settings
   serialization, region gating, and config loading/default behavior.

## Explicit Exclusions

- No SQL projection or `ORDER BY` rewrite.
- No change to `MSCK REPAIR TABLE`, cache TTL, table creation, or project
  provisioning.
- No MaxCompute job cancellation or timeout change.
- No query-priority, quota, result-tunnel, or storage-layout change.
- No Slow Log Explorer code or UI change.

## Compatibility and Risk

- Disabled behavior is byte-for-byte equivalent at task construction because
  the SDK continues to receive nil hints.
- Enabling the setting changes only eligible MaxCompute slow-query execution
  planning. Result correctness must be checked before rollout because Parquet
  reader behavior may depend on file encoding, row-group statistics, and data
  types. Wide projections over many small files may also increase OSS requests.
- Rollback is a configuration change back to `false`, followed by the normal
  GitOps rollout.

## Test Plan

Run from `clinic_data_service`:

```text
go test -count=1 ./pkg/config ./pkg/dialect ./pkg/engine ./pkg/engine_router
go test -race ./pkg/engine
```

Focused tests must verify:

- omitted configuration defaults to disabled;
- explicit true and false values load correctly;
- enabled configuration requires an explicit, nonempty region allowlist;
- disabled and ineligible query hints are nil;
- eligible query hints contain exactly the two expected keys and values;
- AST scope rejects non-SELECT, malformed, TopSQL, view, mixed-table,
  cross-project, wrong-vendor, wrong-scheme, and excluded-region inputs;
- logical/physical slow-query tables, CTEs, and all-slow UNIONs are accepted;
- returned hint maps cannot mutate later query task settings.

## Staging Verification

1. Deploy the feature branch with Clinic CD using environment `staging` and
   component `clinic-data-service`.
2. Establish an off-state baseline through the canonical Data Proxy API.
3. Enable the setting for the benchmark region in the staging
   `clinic-data-service` app-delivery configuration and wait for the normal
   GitOps rollout.
4. Repeat identical two-column ordered queries over thirty-minute and
   three-hour windows, with multiple warm runs.
5. Compare result columns/rows (or a stable result digest), Data Proxy latency,
   MaxCompute instance ID, input bytes, queue/run/total time, CU, and memory.
6. Accept only if results remain identical and the narrow-window input bytes
   and warm execution latency materially decrease. Roll back the setting on any
   correctness error, new MaxCompute failure, or latency regression.

## Deployment and Release Boundary

Affected source modules:

- `clinic_data_service/pkg/config`
- `clinic_data_service/pkg/engine`
- `clinic_data_service/pkg/engine_router`
- `clinic_data_service/cmd/api`
- image `clinic/data-service`

Required deployment modules:

- Clinic CD component: `clinic-data-service`
- app-delivery application: `clinic-data-service`
- Kubernetes workload: Clinic Data Service Deployment
- staging first; production `prod-cloud` only after successful A/B and explicit
  production-release authorization

The deployment needs both an image bump and an explicit configuration change
to enable the optimization. Merging or deploying the Clinic source PR alone
does not enable the setting.

## Local Validation Results

- `go test -count=1 ./pkg/config ./pkg/dialect ./pkg/engine ./pkg/engine_router`:
  passed.
- `go test -race ./pkg/engine`: passed; the local macOS linker emitted a
  non-fatal `LC_DYSYMTAB` warning.
- `go vet ./pkg/config ./pkg/dialect ./pkg/engine ./pkg/engine_router`: passed.
- `go test ./...`: attempted. All changed-package tests passed, but the module's
  existing root test does not compile because `cluster_compare_test.go` calls
  `ClusterSqlDiff` with an outdated signature. The separate `test/` package also
  expects local HTTP services, external Feishu resources, and `/data` fixtures.
  Those failures are unrelated to this diff and are intentionally not changed.