# dataflow-failed-job-analyze

Generate a stable daily dataflow failed job report, run OpenClaw analysis through the running Gateway, write the result to Feishu Wiki, and send a deterministic Feishu card.

## Overview

This cronjob separates stable orchestration from AI analysis:

1. Query yesterday's failed Import/DM/Backup jobs from Feishu Bitable
2. Build a deterministic input payload and fixed output format
3. Call `openclaw agent` through the running Gateway for root-cause analysis
4. Write the full report to Feishu Wiki
5. Send a fixed interactive card to one or more Feishu targets

The scheduler, output format, and Feishu delivery are owned by Python. OpenClaw is only responsible for the analysis step.

## Architecture

```text
import-dm-failed-job-analyze/
├── README.md
├── main.py    # Stable orchestration + Gateway analysis wrapper
└── base.py    # Feishu/Bitable helpers + direct Feishu sender
```

## Execution Flow

```text
[1/5] Fetch failed jobs from Bitable
       ↓
[2/5] Build deterministic analysis input
       ↓
[3/5] Run OpenClaw analysis via Gateway (`openclaw agent`)
       ↓
[4/5] Write markdown report to Feishu Wiki
       ↓
[5/5] Send fixed Feishu card to configured targets
```

## Schedule

Daily at 03:00 UTC.

```bash
0 3 * * * cd /home/openclaw/github/data-service-infra-ai-configurations && python3 cronjobs/team-workflow/import-dm-failed-job-analyze/main.py >> /tmp/import-dm-failed-job-analyze.log 2>&1
```

## Gateway Requirement

This job expects the OpenClaw Gateway to already be running. It does **not** use `--local` agent mode.

Quick check:

```bash
openclaw gateway health --json
```

Expected: `"ok": true`

## Data Source

- Feishu Bitable
- App Token: `FAILED_JOBS_APP_TOKEN`
- Table ID: `FAILED_JOBS_TABLE_ID`
- Fields used: `Job ID`, `Component`, `Failed Tasks`, `Failed Reason`, `Cluster`, `Time`

## Outputs

1. Feishu Wiki page
   - Title: `Dataflow Failed Jobs Report - YYYY-MM-DD`
2. Deterministic JSON stdout summary
3. Feishu interactive card

Example stdout:

```json
{
  "analysis_status": "ok",
  "date": "2026-03-20",
  "backup_failed": 2,
  "dm_failed": 3,
  "import_failed": 1,
  "notification_status": "sent",
  "overview": "1 Import, 3 DM, and 2 Backup jobs failed on 2026-03-20.",
  "risk_level": "medium",
  "wiki_url": "https://pingcap.feishu.cn/wiki/..."
}
```

## Notification Targets

`ONCALL_NOTIFY_CHAT_ID` now supports multiple targets.
Use comma, semicolon, or whitespace as separators.

Supported target formats:

- `chat:oc_xxx`
- `oc_xxx`
- `user:ou_xxx`
- `ou_xxx`

Example:

```bash
export ONCALL_NOTIFY_CHAT_ID="user:ou_xxx,chat:oc_xxx oc_yyy"
```

## Usage

Run for yesterday:

```bash
python3 cronjobs/team-workflow/import-dm-failed-job-analyze/main.py
```

Run for a specific date:

```bash
python3 cronjobs/team-workflow/import-dm-failed-job-analyze/main.py 20260320
```

Send only to one personal target:

```bash
ONCALL_NOTIFY_CHAT_ID="user:ou_xxx" python3 cronjobs/team-workflow/import-dm-failed-job-analyze/main.py 20260320
```

## Environment Variables

| Variable | Description |
|----------|-------------|
| `CRONJOB_TEAM_WORKFLOW_APP_ID` | Feishu app id |
| `CRONJOB_TEAM_WORKFLOW_APP_SECRET` | Feishu app secret |
| `FAILED_JOBS_APP_TOKEN` | Failed-jobs bitable app token |
| `FAILED_JOBS_TABLE_ID` | Failed-jobs bitable table id |
| `ONCALL_NOTIFY_CHAT_ID` | One or more Feishu notification targets |
| `OPENCLAW_BIN` | Optional OpenClaw CLI path. If unset, auto-detect from `PATH` and common install locations |
| `OPENCLAW_ANALYSIS_TIMEOUT` | Analysis timeout in seconds. Default: `600` |
| `SKIP_SEND_NOTIFICATION` | Skip Feishu card send when set to true |
| `SKIP_OPENCLAW_ANALYSIS` | Skip AI analysis and use fallback summary |
| `AI_CONFIG_REPO` | Repo root override |

## Troubleshooting

1. Gateway not running

```bash
openclaw gateway health --json
```

2. Validate direct agent call

```bash
openclaw agent --agent main --json --thinking minimal --timeout 60 --message 'Reply with exactly the JSON string {"ok":true} and nothing else.'
```

3. Run one-day test to personal target

```bash
ONCALL_NOTIFY_CHAT_ID="user:ou_xxx" python3 cronjobs/team-workflow/import-dm-failed-job-analyze/main.py 20260320
```

4. Check recent log

```bash
tail -50 /tmp/import-dm-failed-job-analyze.log
```