# Security Pingkai

Security Pingkai is a Go service for security-operations workflows. It receives internal task requests and selected callbacks, queries MySQL and Redis, integrates with external systems, and creates security alerts or operational records.

This README is the handoff guide for an engineer who is new to the service. After reading it, that engineer should be able to build a release candidate, deploy **only** the Pingkai service, verify the runtime, recover safely, and understand which scheduled workflows own each notification.

## Service Boundary

The production host runs two independent services. Treat them as separate deployment targets.

| Logical service | systemd unit | Port | Deployment directory | Scope |
| --- | --- | ---: | --- | --- |
| Security Pingkai | `security-feishu.service` | `8867` | `/root/security-new` | This repository and this runbook |
| Security Pingcap | `security-lark.service` | `8868` | `/root/security-old` | Out of scope for Pingkai changes |

`security-feishu.service` is the Pingkai unit even though its systemd name does not contain `pingkai`. It uses `Restart=always` and is enabled for boot. Before making a production change, verify this mapping on the target host rather than relying on historical names.

```bash
sudo systemctl show security-feishu.service \
  -p ActiveState -p SubState -p MainPID -p WorkingDirectory -p ExecStart
sudo systemctl is-enabled security-feishu.service
sudo ss -ltnp | grep ':8867'
```

Do not restart, replace, enable, or otherwise modify the port `8868` service when deploying Pingkai.

## Architecture

```mermaid
flowchart LR
    callers["Cron jobs, callbacks, and internal callers"] --> routes["Gin HTTP routes"]
    routes --> controllers["Controllers"]
    controllers --> services["Workflow services"]
    services --> stores["MySQL stores"]
    services --> redis[("Redis")]
    services --> integrations["Lark, Google, Feilian, D&B, port scanning, and other integrations"]
```

The command layer starts the application and loads configuration. Controllers own HTTP request handling, services coordinate business workflows, stores own MySQL persistence, and models define shared data structures. Keep new behavior in these layers; do not put persistence or third-party orchestration directly in a controller.

## Prerequisites

- Go `1.18` or a compatible toolchain.
- Reachable MySQL and Redis instances.
- Managed credentials and configuration for the integrations that are enabled in the target environment.
- A Linux `amd64` build artifact for the current production host.
- Access to the Pingkai systemd unit and its managed configuration.

Install Go dependencies:

```bash
go mod download
```

Inspect supported flags without starting the server:

```bash
go run ./cmd/alert-server --help
```

## Configuration and Security

The binary accepts a configuration file with `-c` or `--config`. YAML, JSON, TOML, HCL, and Java properties formats are supported. Always pass an explicit configuration file in non-development environments.

The production unit reads its managed configuration from its `ExecStart` command. That configuration is not a repository artifact and must not be copied into this repository, tickets, terminal recordings, or documentation.

Configuration contains at least these categories of values:

- HTTP server, logging, MySQL, and Redis settings.
- Internal-task and external-health bearer tokens.
- Credentials and endpoints for enabled integrations.

Use the organization-approved secret-management and file-permission process. Do not commit passwords, tokens, service-account files, or production URLs. Treat application logs as sensitive operational data: restrict access, redact before sharing, and never assume an integration response is safe to paste into a ticket.

## Local Development

Run with a controlled local configuration:

```bash
go run ./cmd/alert-server -c /secure/path/alert-server.local.yaml
```

Run the test suite and build a local binary:

```bash
go test ./...
go build -o bin/alert-server ./cmd/alert-server
./bin/alert-server -c /secure/path/alert-server.local.yaml
```

The default insecure listener is `0.0.0.0:8867`. For local development, prefer `127.0.0.1` unless a controlled test requires a wider bind address.

## Health and Authentication

| Endpoint group | Authentication | Purpose |
| --- | --- | --- |
| `GET /healthz` | None | Local readiness endpoint used by the server and deployment verification. |
| `GET /health` | External bearer token | Protected health endpoint. |
| `POST /inter/*` | Internal bearer token | Scheduled and internal security workflows. |
| `POST /lark/company` | Callback boundary | Lark company-related callback. |

Verify local readiness after startup:

```bash
curl --fail http://127.0.0.1:8867/healthz
```

Readiness proves that the HTTP server is reachable. It does **not** prove that a database write, an external API call, an approval, or a notification was delivered.

## Scheduled Pingkai Workflows

The production scheduler is the host root crontab. It sends authenticated localhost requests to port `8867`; the bearer token must stay in managed configuration and must never be copied into this document.

| Schedule | Internal route | Effect and current notification routing |
| --- | --- | --- |
| Daily at 00:00 | `check-ports` | Port and vulnerability alerts; Liu Yin receives the static alert and approval ownership. |
| Hourly at minute 30 | `feilian-alert` | Feilian data-loss-prevention events; Liu Yin receives the approval. |
| Hourly at minute 01 | `external-user` | External-member events; Liu Yin receives the approval. |
| Monthly, day 1 at 06:00 | `monthly-report` | Monthly report email to Liu Yin, Jason Yu, and Leo Zhuang. |
| Every 10 minutes | `check-downloads` | See the download escalation policy below. |
| Daily at 10:00 and 18:00 | `company` | Company-search processing; Liu Yin receives failure notifications. |

### Download Escalation Policy

`check-downloads` uses the following recipients. The service removes duplicates when a person occupies more than one role.

| Alert stage | Recipients |
| --- | --- |
| Immediate L3/L4 sensitive-file alert | Downloader and Leo Zhuang |
| First threshold: more than 3 exports in 10 minutes | Downloader and Leo Zhuang |
| Second threshold: more than 10 exports in 60 minutes | Direct manager, Liu Yin, and Leo Zhuang |
| Third threshold: more than 20 exports in 24 hours | Direct manager, Liu Yin, and Leo Zhuang |
| No internal user record is available | Liu Yin fallback |

Do not use a synthetic security event as a deployment check. Keep a redacted receipt from the next eligible real workflow if terminal notification delivery must be proven.

## Production Deployment

The current production runtime is systemd, not a container deployment. A Dockerfile exists for image workflows, but do not switch the runtime model as part of an ordinary Pingkai release.

### 1. Prepare an Exact Release Candidate

Start from the intended `main` revision and keep the commit SHA with the deployment record.

```bash
git fetch --prune origin
git switch main
git pull --ff-only origin main
go test ./...

REV="$(git rev-parse --short=12 HEAD)"
ARTIFACT="dist/security-${REV}-linux-amd64"
mkdir -p dist
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 \
  go build -trimpath -buildvcs=false -o "$ARTIFACT" ./cmd/alert-server
shasum -a 256 "$ARTIFACT"
file "$ARTIFACT"
```

The artifact must be a statically linked Linux x86-64 executable. Do not deploy a macOS binary to the production host.

### 2. Preflight the Target

Set the host through your approved access channel. Confirm the unit, port, configuration binding, and current health before replacing anything.

```bash
export DEPLOY_HOST='<managed-host>'

ssh "$DEPLOY_HOST" '
  systemctl show security-feishu.service \
    -p ActiveState -p SubState -p MainPID -p WorkingDirectory -p ExecStart --no-pager
  systemctl is-enabled security-feishu.service
  ss -ltnp | grep ":8867"
  curl --fail --max-time 5 http://127.0.0.1:8867/healthz
'
```

Stop if the unit is not `security-feishu.service`, the working directory is not the Pingkai deployment directory, the listener is not port `8867`, or the service is unhealthy. Those are evidence failures, not reasons to continue with a broad restart.

### 3. Transfer and Verify the Candidate

Copy to a new candidate filename; never overwrite the running binary during transfer.

```bash
REMOTE_CANDIDATE="/root/security-new/security.candidate-${REV}"
scp "$ARTIFACT" "$DEPLOY_HOST:$REMOTE_CANDIDATE"

ssh "$DEPLOY_HOST" "
  chmod 0755 '$REMOTE_CANDIDATE'
  file '$REMOTE_CANDIDATE'
  sha256sum '$REMOTE_CANDIDATE'
"
```

Compare the remote SHA-256 with the local checksum before proceeding.

### 4. Replace, Restart, and Preserve a Backup

Run the following only after the candidate checksum matches. It keeps the previous binary as a rollback target.

```bash
ssh "$DEPLOY_HOST" "REV='$REV' bash -s" <<'REMOTE_SCRIPT'
set -euo pipefail
SERVICE='security-feishu.service'
DEPLOY_DIR='/root/security-new'
REV="${REV:?missing verified revision}"
TARGET="$DEPLOY_DIR/security"
CANDIDATE="$DEPLOY_DIR/security.candidate-$REV"
BACKUP="$DEPLOY_DIR/security.backup-pre-$REV-$(date +%Y%m%dT%H%M%S)"
FAILED="$DEPLOY_DIR/security.failed-$REV"

test -x "$TARGET"
test -x "$CANDIDATE"
mv "$TARGET" "$BACKUP"
mv "$CANDIDATE" "$TARGET"

rollback() {
  mv "$TARGET" "$FAILED"
  mv "$BACKUP" "$TARGET"
  systemctl restart "$SERVICE"
}

if ! systemctl restart "$SERVICE"; then
  rollback
  echo "Deployment failed and was rolled back. Failed candidate: $FAILED"
  exit 1
fi

for attempt in $(seq 1 15); do
  if curl -fsS --max-time 2 -o /dev/null http://127.0.0.1:8867/healthz; then
    echo "Deployment succeeded. Backup retained at: $BACKUP"
    exit 0
  fi
  sleep 1
done

rollback
echo "Deployment failed health verification and was rolled back. Failed candidate: $FAILED"
exit 1
REMOTE_SCRIPT
```

The command passes the already verified local `REV` value to the remote shell. Do not run this procedure for `security-lark.service` or against port `8868`.

### 5. Verify the Runtime

```bash
ssh "$DEPLOY_HOST" '
  systemctl is-enabled security-feishu.service
  systemctl show security-feishu.service -p ActiveState -p SubState -p MainPID --no-pager
  ss -ltnp | grep ":8867"
  sha256sum /root/security-new/security
  curl --fail --max-time 5 -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8867/healthz
'
```

Acceptance requires all of the following:

1. `systemctl is-enabled` returns `enabled`.
2. The unit is `active` and `running` with a new main PID after the restart.
3. The process listens on port `8867`.
4. The running binary SHA-256 matches the verified candidate.
5. `/healthz` returns `200`.

Then separately observe the next real eligible workflow if notification delivery itself needs operational acceptance.

### Rollback

If the unit cannot restart or `/healthz` does not recover within the agreed release window, restore the backup immediately.

```bash
sudo systemctl stop security-feishu.service
sudo mv /root/security-new/security /root/security-new/security.failed-<revision>
sudo mv /root/security-new/security.backup-pre-<revision>-<timestamp> /root/security-new/security
sudo systemctl start security-feishu.service
curl --fail http://127.0.0.1:8867/healthz
```

Record the failed candidate SHA, the restored backup SHA, the systemd status, and the health-check result. Do not delete the failed binary or the backup until incident review and retention requirements are satisfied.

## Troubleshooting

| Symptom | First checks |
| --- | --- |
| Service does not start | `systemctl status security-feishu.service --no-pager`, configuration readability, listener conflicts, and runtime permissions. Redact logs before sharing. |
| Port 8867 is not listening | Confirm the Pingkai unit and `ExecStart`, then check whether another process owns the port. Do not inspect or restart the port 8868 service. |
| `/healthz` fails | Check the unit state, bind address, bind port, and local firewall. A healthy process may take a short time to bind after restart. |
| `/health` or `/inter/*` returns an authentication error | Use the matching managed bearer token and the exact `Authorization: Bearer <token>` format. |
| A workflow fails while health is green | Check the workflow's database, Redis, integration configuration, and durable result. HTTP readiness is not business-delivery proof. |

## Change and Handoff Checklist

Before handing this service to another engineer:

- Confirm the repository remote is `github.com/pingcap-inc/security-pingkai`.
- Confirm Pingkai is the enabled `security-feishu.service` on port `8867`.
- Confirm the intended source revision, artifact SHA-256, running-binary SHA-256, and retained rollback backup.
- Keep production configuration and credentials in managed storage, outside the repository.
- Preserve a redacted terminal result for a real workflow after a notification-routing release.
- Keep the paired Pingcap service on port `8868` out of Pingkai deployment scope.

## Verification Before Commit

```bash
gofmt -w <changed-go-files>
go test ./...
go run ./cmd/alert-server --help
```

Do not claim a production notification is delivered solely because the build, service restart, or health endpoint succeeded.