# ssh-proxy (web-console-ssh) — Onboarding & Maintenance Guide

This directory (`ssh-proxy/`) is a self-contained Go module that builds a single service: an HTTP server with a WebSocket endpoint that proxies an interactive SSH shell to a browser.

Primary entrypoint: `cmd/web-console-ssh/main.go`.

## Naming

- Module: `ssh-proxy` (this folder; has its own `go.mod`).
- Binary: `web-console-ssh` (under `cmd/web-console-ssh/`).
- “Web console session”: one WebSocket connection bridged to one SSH `session` + interactive shell.

## What This Service Does

At a high level:

- A browser opens `GET /ssh-server/web?auth=<JWT>` and upgrades to WebSocket.
- The server verifies the JWT (Ed25519 / EdDSA).
- JWT claims define the SSH target (host/user/port) and the “region” used to select an SSH identity file.
- WebSocket frames are bridged to/from SSH streams (stdin/stdout/stderr).

### End-to-end data flow (ASCII)

```
  +-------------------+     WebSocket (auth=JWT)      +---------------------+        SSH (key auth)        +-------------------+
  | Browser frontend  |  <--------------------------> | web-console-ssh      |  <------------------------> | Target sshd host  |
  | (xterm.js)        |                               | /ssh-server/web      |                            | (interactive shell)|
  +-------------------+                               +---------------------+                            +-------------------+
```

## Key Concepts (Glossary)

- Control plane: external service used when `debug: false` to register this SSH server and receive periodic status reports (`internal/controlplane/client.go`).
- Region: a string in JWT claims used as a key into `ssh_identity_files` to pick the outbound SSH private key (`internal/web_console/http_handler.go:406`–`internal/web_console/http_handler.go:410`).
- JWT signing key pair:
  - Server needs the **public key** (PEM, PKIX) to verify tokens: `jwt_public_key_pem` (`internal/config/config.go:28`–`internal/config/config.go:37`).
  - Token issuer (typically control plane) needs the **private key** to sign `EdDSA` JWTs.
- SSH identity file (per region): a private key file path used by this service to authenticate to the target SSHD (`ssh_identity_files`, `internal/config/config.go:36`).

## Session Lifecycle (Happy Path)

Sequence (one browser session):

1) HTTP request hits `GET /ssh-server/web?auth=<JWT>` (`internal/web_console/http.go:34`).
2) Server verifies JWT and upgrades to WebSocket (`sshHandler.webSocket`, `internal/web_console/http_handler.go:398`–`internal/web_console/http_handler.go:437`).
3) Server waits for the first terminal size control message (Text, `"1:"+JSON`) before doing any SSH work (`sshClient.getWindowSize`, `internal/web_console/http_handler.go:110`–`internal/web_console/http_handler.go:135`).
4) Server dials SSH, requests a PTY, and starts an interactive shell (`internal/web_console/http_handler.go:306`–`internal/web_console/http_handler.go:367`).
5) Server bridges data with three goroutines (WS→stdin, stdout→WS, stderr→WS) (`internal/web_console/http_handler.go:371`–`internal/web_console/http_handler.go:387`).
6) When either side closes, the bridge tears down and decrements the active session counter (`internal/web_console/http_handler.go:431`–`internal/web_console/http_handler.go:436`).

Minimal sequence diagram:

```
Browser                 web-console-ssh                           Target sshd
  | GET /ssh-server/web?auth=...  |                                     |
  |------------------------------>| verifyJwt + Upgrade                 |
  |<------------------------------| 101 Switching Protocols             |
  | Text: 1:{size}                |                                     |
  |------------------------------>| RequestPty + Shell                  |
  | Binary: keystrokes            |                                     |
  |------------------------------>| stdin                               |
  |<------------------------------| Binary: stdout/stderr               |
```

## Directory Map (Read This First)

- `cmd/web-console-ssh/main.go`: loads config, initializes logging, optionally registers to control plane, starts HTTP server.
- `internal/config/config.go`: YAML loading and Ed25519 public key parsing (`jwt_public_key_pem`).
- `internal/web_console/http.go`: HTTP routing (`/health`, `/ssh-server/web`, optional static frontend in debug mode).
- `internal/web_console/http_handler.go`: WebSocket upgrade, JWT verification, SSH dial + PTY + shell, and IO bridging.
- `internal/web_console/status.go`: exported getter for active session count (used by control plane reporter).
- `internal/controlplane/client.go`: register/report APIs to the Labs control plane (non-debug mode only).
- `internal/logger/logger.go`: initializes PingCAP `log` global logger (zap).
- `frontend/` (optional; gitignored by default, `ssh-proxy/.gitignore:30`): a minimal debug UI using xterm.js (served only when `debug: true`).

## Service Modes

`cmd/web-console-ssh/main.go` toggles behavior via config `debug` (`cmd/web-console-ssh/main.go:31`):

- `debug: true`
  - Skips control plane registration/reporting.
  - Serves `./frontend/` at `/` for manual testing (`internal/web_console/http.go:31`–`internal/web_console/http.go:33`).
- `debug: false`
  - Registers once to the control plane, then periodically reports status (`cmd/web-console-ssh/main.go:31`–`cmd/web-console-ssh/main.go:40`).

## Public HTTP Surface

Routes are registered in `internal/web_console/http.go`:

- `GET /health` → `200` JSON `{"message":"I'm alive"}` (`internal/web_console/http.go:35`–`internal/web_console/http.go:39`).
- `GET /ssh-server/web?auth=<JWT>` → WebSocket upgrade and SSH bridge (`internal/web_console/http.go:34`–`internal/web_console/http.go:35`).
- (debug only) `GET /` → static file server rooted at `./frontend/` (`internal/web_console/http.go:31`–`internal/web_console/http.go:33`).

Server timeouts:

- `ReadHeaderTimeout = 5s`, `IdleTimeout = 60s` (`internal/web_console/http.go:41`–`internal/web_console/http.go:46`).

## WebSocket Protocol (Client Contract)

The server uses a small custom protocol on top of WebSocket frames. The contract is defined by `internal/web_console/http_handler.go` and locked by tests under `internal/web_console/*_test.go`. Some developer environments also include an optional reference frontend under `frontend/` (gitignored by default).

### 1) Initial terminal size message (required)

Right after the WebSocket is established, the client must send a Text frame within `messageWait=10s` (`internal/web_console/http_handler.go:26`, `internal/web_console/http_handler.go:110`–`internal/web_console/http_handler.go:135`):

- Frame type: Text
- Payload: `"1:" + JSON`
- JSON schema: `{"high": <rows>, "width": <cols>}` (`internal/web_console/http_handler.go:65`–`internal/web_console/http_handler.go:68`)

The server validates the size and rejects invalid values (`internal/web_console/http_handler.go:181`–`internal/web_console/http_handler.go:189`).

If you use the optional `frontend/` debug UI, it sends this on WebSocket open and on window resize.

### 2) Terminal input

- Frame type: Binary
- Payload: raw bytes written to SSH session stdin (`internal/web_console/http_handler.go:258`–`internal/web_console/http_handler.go:263`)

If you use the optional `frontend/` debug UI, it encodes user keystrokes as UTF-8 bytes and sends them as binary frames.

### 3) Resizes

- Frame type: Text
- Same `"1:"+JSON` format as the initial size message.
- Mapped to SSH `window-change` (`internal/web_console/http_handler.go:271`–`internal/web_console/http_handler.go:281`).

### 4) Server output

- SSH `stdout` and `stderr` are both forwarded as WebSocket Binary frames (`internal/web_console/http_handler.go:191`–`internal/web_console/http_handler.go:243`).
- Some setup failures are sent as a WebSocket Text message (e.g. `failed to dial: ...`, `internal/web_console/http_handler.go:313`–`internal/web_console/http_handler.go:316`).

### Message size limits

On upgrade, the server sets `conn.SetReadLimit(maxWSReadLimit)` to cap each WebSocket message at `64 KiB` (`internal/web_console/http_handler.go:417`).

Practical implication: very large “paste” operations sent as a single WebSocket message can be rejected with a close frame (see `internal/web_console/http_handler_test.go:53`–`internal/web_console/http_handler_test.go:136`).

## Authentication Model (JWT)

The auth token is passed as a query parameter:

- Query key: `auth` (`internal/web_console/http_handler.go:450`)

JWT verification rules (see `sshHandler.verifyJwt`, `internal/web_console/http_handler.go:449`–`internal/web_console/http_handler.go:483`):

- Algorithm must be `EdDSA` (`internal/web_console/http_handler.go:453`–`internal/web_console/http_handler.go:456`).
- Signature is verified using the configured Ed25519 public key (`internal/config/config.go:51`–`internal/config/config.go:77`).
- `exp` is checked against current time (`internal/web_console/http_handler.go:476`–`internal/web_console/http_handler.go:480`).

Claims are decoded via JSON marshal/unmarshal into `RemoteSshConfig` (`internal/web_console/http_handler.go:439`–`internal/web_console/http_handler.go:447`).

Required/used claims:

- `region` (string): selects `ssh_identity_files[region]` (`internal/web_console/http_handler.go:406`–`internal/web_console/http_handler.go:410`).
- `m_ip` (string): SSH target IP (`internal/web_console/http_handler.go:425`–`internal/web_console/http_handler.go:426`).
- `m_user` (string): SSH username (`internal/web_console/http_handler.go:426`).
- `m_port` (int, optional): SSH port; defaults to `22` if omitted/0 (`internal/web_console/http_handler.go:419`–`internal/web_console/http_handler.go:422`).
- `exp` (int): JWT expiry (unix seconds) (`internal/web_console/http_handler.go:446`–`internal/web_console/http_handler.go:480`).

Note: the full claims JSON is currently logged at `INFO` (`internal/web_console/http_handler.go:471`).

## Outbound SSH Behavior

Connection setup happens in `sshClient.bridgeWSAndSSH()` (`internal/web_console/http_handler.go:285`–`internal/web_console/http_handler.go:390`):

- Private key is loaded from `ssh_identity_files[region]` (`internal/web_console/http_handler.go:294`–`internal/web_console/http_handler.go:304`).
  - The key must be unencrypted (no passphrase), because the code uses `ssh.ParsePrivateKey` (`internal/web_console/http_handler.go:299`).
- SSH dial target: `net.JoinHostPort(m_ip, m_port)` (`internal/web_console/http_handler.go:425`–`internal/web_console/http_handler.go:426`).
- Host key verification is disabled (`ssh.InsecureIgnoreHostKey()`, `internal/web_console/http_handler.go:309`).
- PTY is requested with term `"xterm"` (`internal/web_console/http_handler.go:354`) and a fixed `terminalModes` map (`internal/web_console/http_handler.go:43`–`internal/web_console/http_handler.go:47`).
- A login shell is started via `sess.Shell()` (`internal/web_console/http_handler.go:361`).

## IO Bridging Implementation (Concurrency Model)

Once SSH is established, `bridgeWSAndSSH()` starts three goroutines (`internal/web_console/http_handler.go:371`–`internal/web_console/http_handler.go:387`):

```
            +--------------------+
            |  WebSocket Conn    |
            +--------------------+
              ^     ^       |
    wsWrite() |     |       | wsRead()
    stdout -->+     |       +--> stdin
                    |
    wsWriteStderr() |
    stderr ---------+
```

Key details:

- Gorilla WebSocket requires single-writer semantics; writes are serialized with `sshClient.writeMu` (`internal/web_console/http_handler.go:80`–`internal/web_console/http_handler.go:108`).
- The bridge avoids closing early until both stdout and stderr streams are done (`signalOutputDone`, `internal/web_console/http_handler.go:95`–`internal/web_console/http_handler.go:101`; tested in `internal/web_console/bridge_close_test.go:12`–`internal/web_console/bridge_close_test.go:64`).

## Active Session Tracking (for reporting)

- `activeWebConsoleSessions` is a process-wide atomic counter (`internal/web_console/http_handler.go:57`–`internal/web_console/http_handler.go:63`).
- Increment per accepted WebSocket connection (`internal/web_console/http_handler.go:431`–`internal/web_console/http_handler.go:436`).
- Exported getter `GetActiveWebConsoleSessions()` is in `internal/web_console/status.go:5`–`internal/web_console/status.go:8`.

## Control Plane Integration (non-debug only)

`internal/controlplane/client.go` implements two POST APIs:

- Register: `POST <lab_base_url>/api/v2/ssh-servers/registration` (`internal/controlplane/client.go:17`–`internal/controlplane/client.go:20`, `internal/controlplane/client.go:74`–`internal/controlplane/client.go:102`)
  - Payload: `{domain, regions}` (`internal/controlplane/client.go:75`–`internal/controlplane/client.go:82`)
  - Response: `{id}` stored as `sshServerId` (`internal/controlplane/client.go:90`–`internal/controlplane/client.go:101`)
- Report: `POST <lab_base_url>/api/v2/ssh-servers/{id}/report` (`internal/controlplane/client.go:19`–`internal/controlplane/client.go:20`, `internal/controlplane/client.go:104`–`internal/controlplane/client.go:114`)
  - Payload: `{score, active_session}` (`internal/controlplane/client.go:31`–`internal/controlplane/client.go:35`)

Auth:

- If `lab_api_key` is set, it is sent as `Authorization: Bearer <key>` (`internal/controlplane/client.go:64`–`internal/controlplane/client.go:66`).

Reporter:

- Started in `cmd/web-console-ssh/main.go:39`.
- Default interval is 5 seconds (`cmd/web-console-ssh/main.go:39`); each report uses a 3-second timeout (`internal/controlplane/client.go:139`–`internal/controlplane/client.go:141`).

## Configuration (YAML)

Config is loaded by `internal/config.InitConfig()` (`internal/config/config.go:39`–`internal/config/config.go:57`) using `sigs.k8s.io/yaml` and the struct’s `json` tags.

Important implications:

- YAML keys must match the `json` tags (snake_case), e.g. `http_port`, `jwt_public_key_pem`, `preferred_regions` (`internal/config/config.go:14`–`internal/config/config.go:37`).
- `jwt_public_key_pem` is required; startup fails if it can’t be parsed as a PKIX Ed25519 public key (`internal/config/config.go:51`–`internal/config/config.go:77`).

### Config fields (current)

See `internal/config/config.go:14`–`internal/config/config.go:37`:

- `debug` (bool)
- `log` (object): PingCAP log config (`github.com/pingcap/log`)
- `domain` (string): used for control plane registration
- `preferred_regions` ([]string): used for control plane registration
- `http_port` (int): HTTP listen port
- `jwt_public_key_pem` (string): Ed25519 public key PEM (PKIX)
- `lab_base_url` (string): control plane base URL
- `lab_api_key` (string): bearer token for control plane APIs
- `ssh_identity_files` (map[string]string): `region` → SSH private key path

### Template and secrets

- `config.yaml.template` shows a minimal example.
- Avoid committing secrets. `ssh-proxy/.gitignore` ignores `config.yaml` (`ssh-proxy/.gitignore:29`).

## Frontend (Debug UI)

This directory is intended as a convenience UI to validate the backend protocol. It is optional and gitignored by default (`ssh-proxy/.gitignore:30`).

Behavior (high level):

- Reads JWT from `/?auth=<JWT>`.
- Opens WebSocket `ws(s)://<host>/ssh-server/web?auth=<JWT>`.
- Sends size control message `"1:"+JSON` on open and resize.
- Sends terminal input as binary.
- Writes output to xterm (binary preferred; also handles Text errors).

Build (optional):

1) `cd frontend`
2) `npm install`
3) `npm run build`

Run the server from the `ssh-proxy/` directory with `debug: true` so `./frontend/` is found (`internal/web_console/http.go:31`–`internal/web_console/http.go:33`).

## Development Workflow

### Build / run

Run from the module root (`ssh-proxy/`) so relative paths work:

- Build: `go build ./cmd/web-console-ssh`
- Run: `go run ./cmd/web-console-ssh -c ./config.local.yaml`

### Unit tests

- `go test ./...`

### E2E tests (real WS + real SSH handshake)

There is an end-to-end test behind a build tag (`internal/web_console/e2e_web_console_test.go:1`):

- `go test -tags=e2e ./...`

## Local Smoke Test (localhost, debug mode)

This is the quickest manual verification loop without the control plane:

1) Prepare SSH access to the target machine (localhost works):
   - Ensure `sshd` is running and you can SSH to it.
   - Generate an SSH private key **without** passphrase (the server uses `ssh.ParsePrivateKey`, `internal/web_console/http_handler.go:299`) and add its public key to the target user’s `authorized_keys`.
2) Generate a JWT Ed25519 key pair:
   - Server needs the public key PEM (PKIX) in `jwt_public_key_pem` (`internal/config/config.go:59`–`internal/config/config.go:77`).
3) Create a local config (example keys):
   - `debug: true`
   - `http_port: 8080`
   - `ssh_identity_files: { local: /path/to/private_key }`
4) Run `web-console-ssh` from `ssh-proxy/` so `./frontend/` can be served (`internal/web_console/http.go:31`–`internal/web_console/http.go:33`).
5) Sign a JWT with claims `{region, m_ip, m_user, m_port, exp}` (see `RemoteSshConfig`, `internal/web_console/http_handler.go:439`–`internal/web_console/http_handler.go:447`).
6) Open `http://127.0.0.1:<http_port>/?auth=<JWT>` and verify you get an interactive shell.

## Troubleshooting (Most Common)

- WebSocket handshake fails with `401`
  - JWT invalid/expired/wrong alg (`internal/web_console/http_handler.go:399`–`internal/web_console/http_handler.go:403`, `internal/web_console/http_handler.go:453`–`internal/web_console/http_handler.go:480`).
- WebSocket handshake fails when `region` has no key configured
  - `getSshIdentityFile` returns error and handler returns without upgrading (`internal/web_console/http_handler.go:406`–`internal/web_console/http_handler.go:410`).
- WebSocket connects but you see `failed to dial: ...`
  - SSH dial to `m_ip:m_port` failed (`internal/web_console/http_handler.go:312`–`internal/web_console/http_handler.go:316`).
- WebSocket connects then silently closes during setup
  - SSH key file unreadable or encrypted (setup logs only; key read/parse at `internal/web_console/http_handler.go:294`–`internal/web_console/http_handler.go:303`).
- Connection closes ~10 seconds after connect
  - Client did not send initial size control message (`internal/web_console/http_handler.go:110`–`internal/web_console/http_handler.go:135`).

## Production Hardening Notes (Read Before Shipping)

These are current behaviors visible in code; treat them as a checklist for production readiness:

- Origin checks are disabled: `Upgrader.CheckOrigin` always returns `true` (`internal/web_console/http_handler.go:49`–`internal/web_console/http_handler.go:55`).
- SSH host key verification is disabled: `ssh.InsecureIgnoreHostKey()` (`internal/web_console/http_handler.go:309`).
- Auth token is in the URL query string (`internal/web_console/http_handler.go:450`), which can be logged by proxies and browsers.
- JWT claims are logged at `INFO` (`internal/web_console/http_handler.go:471`), potentially leaking host/user metadata.
- No explicit HTTP status is written when identity file is missing (no upgrade, but not a clear error) (`internal/web_console/http_handler.go:406`–`internal/web_console/http_handler.go:410`).
- No bandwidth/rate limiting; a single session can produce large stdout/stderr and consume significant resources (hot paths: `internal/web_console/http_handler.go:191`–`internal/web_console/http_handler.go:283`).

## Making Changes Safely

Guiding principles for this module:

- Treat the WebSocket message contract as a stable interface unless intentionally versioned (client and tests rely on it).
- When touching `internal/web_console/http_handler.go`, keep goroutine lifecycle and Gorilla WebSocket single-writer rules in mind (`writeMu` exists for a reason).
- If you add or change config fields:
  - Use snake_case keys via `json` tags (`internal/config/config.go:14`–`internal/config/config.go:37`).
  - Update `config.yaml.template` to match.
  - Add/adjust tests where the behavior is contract-like (see `internal/web_console/*_test.go`).