# Inference Service

A simple inference service for hosted embeddings.

## Features

- **OpenAI-compatible API**: Compatible with OpenAI Embedding API format
- **Simple HTTP Integration**: Direct HTTP calls to AWS Bedrock using bearer token authentication
- **Usage Tracking**: Tracks usage by tenant and model with local JSON persistence
- **Quota Management**: Configurable monthly usage limits per tenant
- **Metrics**: Global Prometheus metrics for monitoring usage and API performance
- **Simple Configuration**: YAML config with environment variable overrides
- **API Documentation**: Auto-generated Swagger/OpenAPI documentation

## Supported Models

All supported models and their pricing are defined in [`internal/config/config.go`](internal/config/config.go).

## API Endpoints

### GET /swagger/index.html

Access the interactive API documentation (Swagger UI).

### POST /api/v1/inference/embeddings/{tenantId}

Create embeddings for the given texts.

**Request Body:**

```json
{
  "model": "amazon-titan",
  "texts": ["text1", "text2"],
  "encoding_format": "float"
}
```

**Response:**

```json
{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [0.1, 0.2, ...],
      "index": 0
    }
  ],
  "model": "amazon-titan",
  "usage": {
    "prompt_tokens": 10,
    "total_tokens": 10
  }
}
```

### GET /api/v1/inference/quota_usage/{tenantId}

Query the usage quota for a specific tenant.

**Response:**

```json
{
  "tenant_id": "tenant123",
  "current_usage": 15.5,
  "quota_limit": 25.0,
  "reset_date": "2024-08-01T00:00:00Z"
}
```

### GET /api/v1/inference/quota_usage

Query usage quota for all tenants since app start.

**Response:**

```json
{
  "tenants": [
    {
      "tenant_id": "tenant123",
      "current_usage": 15.5,
      "quota_limit": 25.0,
      "reset_date": "2024-08-01T00:00:00Z"
    }
  ]
}
```

## Configuration

### Single Source of Truth

All default configuration (models, pricing, server settings) is defined in [`internal/config/config.go`](internal/config/config.go).

YAML configuration files should only contain **overrides** for specific deployments:

```yaml
# Example: only override what's different from defaults
quotas:
  specialLimits:
    tenant123: 100.00
    tenant456: 50.00

storage:
  # Local storage (default)
  # usageDataDir: "./data"

  # OR S3 storage
  # usageDataDir: "s3://my-bucket/usage-data"
```

### Storage Options

The service supports two storage backends for usage data:

- **Local Storage** (default): Stores usage data in local JSON files. Specify a local directory path in `storage.usageDataDir`.
- **S3 Storage**: Stores usage data in AWS S3. Use the format `s3://bucket-name/prefix` in `storage.usageDataDir`.

For S3 storage, ensure your environment has proper AWS credentials configured (via AWS CLI, IAM roles, or environment variables).

## Development

### Prerequisites

- Go 1.24 or later
- AWS Bedrock bearer token

### Local Development

1. **Clone and setup:**

   ```bash
   git clone https://github.com/tidbcloud/inference-service
   cd inference
   ```

2. **Configure environment:**

   ```bash
   echo "AWS_BEARER_TOKEN_BEDROCK=your-token-here" > .env
   ```

3. **Run locally:**

   ```bash
   make run
   ```

## Deployment

Staging:

```shell
aws sso login --profile=dbaas-sso
cd app-delivery/staging/inference-service
kubectl -n staging-ms diff -k env/staging
kubectl -n staging-ms apply -k env/staging
```

Note: must refresh k8s secret manually if secret manager value is changed (in `app-delivery/staging/inference-service`):

```bash
#!/usr/bin/env bash
set -euo pipefail

NS=staging-ms
ES=inference-service-secret
DEPLOY=inference-service
KUSTOMIZE_DIR=env/staging

old_refresh="$(kubectl -n "$NS" get externalsecret "$ES" -o jsonpath='{.status.refreshTime}')"

kubectl -n "$NS" annotate externalsecret "$ES" force-sync="$(date +%s)" --overwrite

for _ in $(seq 1 60); do
  if kubectl -n "$NS" get externalsecret "$ES" -o json \
    | jq -e --arg old "$old_refresh" \
      '(.status.refreshTime // "") != $old and any(.status.conditions[]?; .type=="Ready" and .status=="True")' >/dev/null;
then
    break
  fi
  sleep 2
done

new_refresh="$(kubectl -n "$NS" get externalsecret "$ES" -o jsonpath='{.status.refreshTime}')"
if [[ "$new_refresh" == "$old_refresh" ]]; then
  echo "ExternalSecret did not refresh in time" >&2
  exit 1
fi

kubectl -n "$NS" apply -k "$KUSTOMIZE_DIR"
kubectl -n "$NS" rollout restart "deployment/$DEPLOY"
kubectl -n "$NS" rollout status "deployment/$DEPLOY" --timeout=180s
```