# Issue 890 Picklist Option Extensions

Issue: https://github.com/pingcap-inc/NexusCRM/issues/890
Status: implementation-ready
Date: 2026-05-16

## Purpose

Issue #890 is the L0 slice split out of issue #843 and closed PR #853. It
adds only the generic storage/preservation substrate for option-scoped picklist
metadata. The platform must keep the extension payload opaque, must not know
about CRM keys such as `defaultProbability`, and must not publish Salesforce
record-type-scoped picklist-values resources before the record-type value-set
substrate exists.

The goal is preservation, not interpretation.

## Contract

- `models.PicklistOption` carries an optional opaque extension map whose values
  are raw JSON.
- Legacy picklist JSON arrays such as `["Open","Closed"]` continue to parse as
  `value == label` options with no tags and no extensions.
- Structured picklist JSON entries preserve `value`, `label`, `tags`, and
  `extensions` through JSON round trip.
- Structured picklist JSON must fail closed when the option shape is malformed,
  including non-object `extensions` payloads. The compatibility fallback is only
  for valid legacy string arrays, not for invalid structured metadata.
- Pack object specs parse option extensions through the existing
  `FieldSpec.Options` path. No app-specific key is added to pack schemas.
- `_System_Field.Options` remains the storage source of truth. Existing JSON
  marshal/unmarshal persistence, metadata scan, schema cache, and field update
  paths carry the full `PicklistOption` structure.
- Spec drift treats option-extension changes as drift for locked/default pack
  verification because apply writes the full option payload.
- REST describe keeps its explicit Salesforce-style picklist entry projection
  and does not expose `extensions` or `defaultProbability`.
- Broad metadata schema read endpoints keep returning regular picklist options,
  but project out opaque option extensions before serializing
  `models.ObjectMetadata`.
- UI API object-info does not expose option extensions or picklist option
  values. Hidden picklist fields therefore cannot leak option data through
  object-info, because this slice does not add `fields[].picklistValues`.
- This issue does not add record-type-scoped picklist-values routes.

## Non-Goals

- No record-type scoped picklist value-set metadata. That is #891.
- No `/ui-api/object-info/{objectApiName}/picklist-values/{recordTypeId}`
  endpoints. Those belong to #892 after #891 exists.
- No first-class `defaultProbability` field on platform picklist options.
- No CRM L1 app-pack payload, source-pack hash bump, or app-pack blessing.
- No runtime Opportunity `StageName -> Probability` behavior.
- No revival of PR #853; it is reference material only.

## Implementation Shape

Shared model:

- Add `Extensions map[string]json.RawMessage` to `PicklistOption` with
  `json:"extensions,omitempty"`.
- Keep the existing custom `PicklistOptions.UnmarshalJSON` ordering: try the
  legacy string-array form first, then structured options. This protects
  existing specs because unmarshalling `["A"]` into `[]PicklistOption` would
  otherwise silently produce zero-value structs.
- Let `PicklistOptions.MarshalJSON` continue to marshal the typed option slice;
  `json.RawMessage` preserves extension values without normalizing them at the
  model boundary.
- If the legacy string-array parse fails, return structured-option unmarshal
  errors instead of treating malformed structured metadata as an empty option
  list.

Pack/load/store/read:

- Reuse existing `FieldSpec.Options models.PicklistOptions`, domain
  `ColumnDefinition.Options`, `FieldMetadata.Options`, and `_System_Field.Options`
  plumbing.
- Add focused tests proving pack object-spec parsing and field persistence keep
  the extension payload. No parser branch should inspect extension keys.
- Leave validation focused on generic option value shape. Extension contents
  are opaque and are not validated by L0 beyond being valid JSON in the parent
  option entry and being carried under an object-shaped extension map.

Spec drift:

- Compare option extension maps by key and by JSON value.
- Treat nil and empty extension maps as equivalent.
- Compare JSON semantically when values parse successfully, so formatting and
  numeric spelling differences such as `60` versus `60.0` do not create false
  drift.
- Report extension mismatches per option value, matching the current label/tag
  diagnostics style.

Public APIs:

- Do not add routes.
- Do not add frontend API helpers.
- Do not add picklist-values to UI API object-info. Add a regression test with
  an FLS-hidden picklist field whose stored options contain extensions, and
  assert that the object-info JSON contains no `picklistValues`, `extensions`,
  or `defaultProbability` payload.
- Keep REST describe's picklist mapping as a manually projected response. The
  new `Extensions` field is never copied into that response.
- Add a projection helper for `/metadata/objects` and
  `/metadata/objects/{apiName}` so those broad schema reads cannot serialize
  opaque extensions even when returning otherwise unfiltered schema metadata.

## Self-Review Iterations

Iteration 1 traced the closed PR #853 blockers. The unsafe work was the new
record-type-scoped UI API endpoint and the object-info picklist-values exposure,
not the opaque storage field itself. This design removes the public endpoint
work entirely.

Iteration 2 traced the storage path. Pack specs, column definitions,
`FieldMetadata`, `_System_Field.Options`, metadata scan, and schema cache all
already use `models.PicklistOptions`. Adding the field to `PicklistOption`
therefore preserves extensions through one existing authoritative path instead
of creating a parallel registry.

Iteration 3 checked API leakage. REST describe does not marshal
`PicklistOption` directly; it builds each picklist value entry by hand. UI API
object-info currently has no `PicklistValues` field in its `FieldInfo` model,
so even an FLS-hidden picklist field cannot leak option values there. The
implementation only adds tests around those negative contracts.

Iteration 4 checked boundary risk. The platform stores raw JSON under an
opaque `extensions` namespace and never switches on keys such as
`defaultProbability`. A Jira-like app pack could store its own option-scoped
metadata through the same contract without changing L0 code.

Iteration 5 checked compatibility. Legacy string-array options still unmarshal,
structured options without tags/extensions still produce nil optional fields,
and existing tag behavior remains intact. No endpoint or route shape changes.

Iteration 6 checked drift behavior. Because apply writes the full option JSON,
verify must compare the full option JSON surface. The design adds semantic raw
JSON map comparison so harmless JSON formatting differences do not look like
drift while real extension changes do.

Iteration 7 checked fail-closed and broad-read behavior after gate review. The
legacy fallback remains only for valid `[]string`; malformed structured option
entries now reject at decode time, and broad metadata schema read endpoints
strip opaque extension maps before serializing raw `ObjectMetadata`.

Final confidence check: I am 100% confident in this narrow design because every
acceptance criterion maps to one existing generic path, the public API changes
from PR #853 are deliberately absent, and no CRM-specific option key becomes
platform behavior.

## Verification

Focused checks:

- `cd shared && go test ./pkg/models`
- `cd backend && go test ./internal/controlplane/packs ./internal/controlplane/specdrift ./internal/infrastructure/persistence ./internal/interfaces/rest ./internal/features/uiapi`

Baseline checks:

- `cd backend && go test ./...`
- `cd shared && go test ./...`
- `cd frontend && npm run build`
- `git diff --check`

`npm run build` rewrites `frontend/public/version.json`; revert that generated
change unless it is intentionally part of the task.