# Ops Server with FluxCD Self-Management

This Helm chart deploys an observability ops-server with embedded FluxCD controllers for self-managed GitOps operations.

## Architecture

The deployment consists of three containers running as sidecars in a single pod:

1. **ops-server**: The main O11y operations API server
2. **source-controller**: FluxCD source controller for Git/OCI repository management
3. **helm-controller**: FluxCD Helm controller for Helm release lifecycle management

## Features

- 🚀 **Self-Management**: After bootstrap, the ops-server manages itself via GitOps
- 🔐 **IRSA Support**: AWS IAM Roles for Service Accounts for ECR access
- 📦 **Multi-Container**: FluxCD controllers as sidecars
- 🎯 **O11y Operations**: API for managing observability components
- 🔄 **GitOps Workflow**: Declarative configuration management

## Prerequisites

- Kubernetes 1.19+
- Helm 3.0+
- AWS EKS cluster with OIDC provider configured
- FluxCD CRDs installed
- ECR access for container images

## Quick Start

### 1. Bootstrap Deployment

```bash
# Clone the repository and navigate to the chart
cd deployments/ops-server

# Bootstrap with cluster name and region
./bootstrap.sh my-cluster us-west-2

# Or with self-management enabled
./bootstrap.sh my-cluster us-west-2 https://github.com/your-org/your-repo.git main
```

### 2. Manual Installation

```bash
# Install FluxCD CRDs (if not already installed)
kubectl apply -f https://github.com/fluxcd/flux2/releases/latest/download/install.yaml

# Set up IRSA role (see IRSA Setup section)
# ... IAM role setup commands ...

# Install the chart
helm install ops-server . \
  --namespace ops-system \
  --create-namespace \
  --set global.clusterName=my-cluster \
  --set global.region=us-west-2 \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ops-server-irsa-role
```

## Configuration

### Key Values

| Parameter | Description | Default |
|-----------|-------------|---------|
| `global.registry` | Container registry URL | `552185537300.dkr.ecr.us-west-2.amazonaws.com` |
| `global.region` | AWS region | `us-west-2` |
| `global.accountId` | AWS account ID | `552185537300` |
| `global.clusterName` | EKS cluster name | `""` |
| `selfManagement.enabled` | Enable GitOps self-management | `true` |
| `selfManagement.gitRepository.url` | Git repository URL | `""` |

### Container Configuration

#### Ops Server
```yaml
opsServer:
  image:
    repository: ops-server
    tag: "latest"
  config:
    port: 8180
    logLevel: info
    vendor: aws
```

#### FluxCD Controllers
```yaml
sourceController:
  image:
    repository: fluxcd/source-controller
    tag: "v1.7.4"
  config:
    concurrent: 20

helmController:
  image:
    repository: fluxcd/helm-controller
    tag: "v1.5.4"
  config:
    concurrent: 8
```

## IRSA Setup

The ops-server uses AWS IAM Roles for Service Accounts (IRSA) for ECR access:

### 1. Create IAM Role

```bash
# Get OIDC issuer
OIDC_ID=$(aws eks describe-cluster --name CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

# Create trust policy
cat > trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.REGION.amazonaws.com/id/$OIDC_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.REGION.amazonaws.com/id/$OIDC_ID:sub": "system:serviceaccount:ops-system:ops-server"
        }
      }
    }
  ]
}
EOF

# Create role
aws iam create-role \
  --role-name ops-server-irsa-role \
  --assume-role-policy-document file://trust-policy.json
```

### 2. Attach ECR Policy

```bash
cat > ecr-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name ops-server-irsa-role \
  --policy-name ECRAccess \
  --policy-document file://ecr-policy.json
```

## Self-Management

When self-management is enabled, the ops-server creates:

1. **GitRepository**: Points to your Git repository containing the chart
2. **HelmRelease**: Manages the ops-server deployment itself

### Git Repository Structure

```
your-repo/
├── deployments/
│   └── ops-server/          # This Helm chart
│       ├── Chart.yaml
│       ├── values.yaml
│       ├── templates/
│       └── ...
└── ...
```

### Git Credentials

Create a secret with Git credentials:

```bash
# For SSH key
kubectl create secret generic ops-server-git-credentials \
  --namespace=ops-system \
  --from-file=identity=/path/to/private-key \
  --from-file=identity.pub=/path/to/public-key \
  --from-file=known_hosts=/path/to/known_hosts

# For HTTPS token
kubectl create secret generic ops-server-git-credentials \
  --namespace=ops-system \
  --from-literal=username=your-username \
  --from-literal=password=your-token
```

## Monitoring

The deployment includes monitoring capabilities:

- **Metrics**: Exposed on port 8181 for ops-server
- **Health Checks**: Liveness and readiness probes
- **ServiceMonitor**: Optional Prometheus ServiceMonitor

## API Usage

Once deployed, the ops-server API is available:

```bash
# Port forward to access the API
kubectl port-forward service/ops-server 8180:8180 -n ops-system

# Test the API
curl http://localhost:8180/api/v1/observability-ops/vmclusters
```

## Troubleshooting

### Check Pod Status
```bash
kubectl get pods -n ops-system
kubectl logs -f deployment/ops-server -n ops-system -c ops-server
kubectl logs -f deployment/ops-server -n ops-system -c source-controller
kubectl logs -f deployment/ops-server -n ops-system -c helm-controller
```

### Check FluxCD Resources
```bash
kubectl get gitrepository,helmrelease -n ops-system
kubectl describe helmrelease ops-server -n ops-system
```

### Check IRSA
```bash
kubectl describe serviceaccount ops-server -n ops-system
aws sts get-caller-identity  # When exec'd into pod
```

### Common Issues

1. **ECR Access Denied**: Verify IRSA role and ECR permissions
2. **Git Repository Access**: Check Git credentials secret
3. **FluxCD CRDs Missing**: Install FluxCD CRDs first
4. **Controller Startup**: Check container logs for specific errors

## Development

### Building Custom Images

```bash
# Build ops-server image
docker build -t 552185537300.dkr.ecr.us-west-2.amazonaws.com/ops-server:latest .

# Push to ECR
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 552185537300.dkr.ecr.us-west-2.amazonaws.com
docker push 552185537300.dkr.ecr.us-west-2.amazonaws.com/ops-server:latest
```

### Local Testing

```bash
# Render templates
helm template ops-server . --values values.yaml

# Validate
helm lint .

# Install in test mode
helm install --dry-run --debug ops-server .
```

## Security Considerations

- **RBAC**: Minimal required permissions for FluxCD controllers
- **Security Context**: Non-root containers with read-only root filesystem
- **IRSA**: No stored AWS credentials in cluster
- **Network Policies**: Consider implementing network policies for isolation

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes and test
4. Submit a pull request

## License

This project is licensed under the Apache License 2.0.