harb/docs/ci-pipeline.md

# Woodpecker CI

## Infrastructure
- **Server**: Woodpecker 3.10.0 runs as a **systemd service** (`woodpecker-server.service`), NOT a Docker container. Binary at `/usr/local/bin/woodpecker-server`.
- **Host**: `https://ci.niovi.voyage` (port 8000 locally at `http://127.0.0.1:8000`)
- **Forge**: Codeberg (Gitea-compatible) — repo `johba/harb`, forge remote ID `800173`
- **Database**: PostgreSQL at `127.0.0.1:5432`, database `woodpecker`, user `woodpecker`
- **Config**: `/etc/woodpecker/server.env` (contains secrets — agent secret, Gitea OAuth secret, DB credentials)
- **CLI**: Downloaded to `/tmp/woodpecker-cli` (v3.10.0). Requires `WOODPECKER_SERVER` and `WOODPECKER_TOKEN` env vars.
- **Logs**: `journalctl -u woodpecker-server -f` (NOT `docker logs`)

## Pipeline Configs
- `.woodpecker/build-ci-images.yml` — Builds Docker CI images using unified `docker/Dockerfile.service-ci`. Triggers on **push** to `master` or `feature/ci` when files in `docker/`, `.woodpecker/`, `containers/`, `kraiken-lib/`, `onchain/`, `services/`, `web-app/`, or `landing/` change.
- `.woodpecker/e2e.yml` — Runs Playwright E2E tests. Bootstrap step sources `scripts/bootstrap-common.sh` for shared deploy/seed logic. Health checks use `scripts/wait-for-service.sh`. Triggers on **pull_request** to `master`.
- Pipeline numbering: even = build-ci-images (push events), odd = E2E (pull_request events). This is not guaranteed but was the observed pattern.

## Monitoring Pipelines via DB
Since the Woodpecker API requires authentication (tokens are cached in server memory; DB-only token changes don't work without a server restart), monitor pipelines directly via PostgreSQL:
```bash
# Latest pipelines
PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
  "SELECT number, status, branch, event, commit FROM pipelines
   WHERE repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb')
   ORDER BY number DESC LIMIT 5;"

# Step details for a specific pipeline
PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
  "SELECT s.name, s.state,
    CASE WHEN s.finished > 0 AND s.started > 0 THEN (s.finished - s.started)::int::text || 's'
    ELSE '-' END as duration, s.exit_code
   FROM steps s WHERE s.pipeline_id = (
     SELECT id FROM pipelines WHERE number = <N>
     AND repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb'))
   ORDER BY s.started NULLS LAST;"
```

## Triggering Pipelines
- **Normal flow**: Push to Codeberg → Codeberg fires webhook to `https://ci.niovi.voyage/api/hook` → Woodpecker creates pipeline.
- **Known issue**: Codeberg webhooks can stop firing if `ci.niovi.voyage` becomes unreachable (DNS/connectivity). Check Codeberg repo settings → Webhooks to verify delivery history and re-trigger.
- **Manual trigger via API** (requires valid token — see known issues):
  ```bash
  WOODPECKER_SERVER=http://127.0.0.1:8000 WOODPECKER_TOKEN=<token> \
    /tmp/woodpecker-cli pipeline create --branch feature/ci johba/harb
  ```
- **API auth limitation**: The server caches user token hashes in memory. Inserting a token directly into the DB does not work without restarting the server (`sudo systemctl restart woodpecker-server`).

## CI Docker Images
- `docker/Dockerfile.service-ci` — Unified parameterized Dockerfile for all service CI images (ponder, webapp, landing, txnBot). Uses `--build-arg` for service-specific configuration (SERVICE_DIR, SERVICE_PORT, ENTRYPOINT_SCRIPT, NEEDS_SYMLINKS, etc.).
  - **sync-tax-rates**: Builder stage runs `scripts/sync-tax-rates.mjs` to sync tax rates from `Stake.sol` into kraiken-lib before TypeScript compilation.
  - **Symlinks fix** (webapp only, `NEEDS_SYMLINKS=true`): Creates `/web-app`, `/kraiken-lib`, `/onchain` symlinks to work around Vite's `removeBase()` stripping `/app/` prefix from filesystem paths.
  - **CI env detection** (`CI=true`): Disables Vue DevTools plugin in `vite.config.ts` to prevent 500 errors caused by path resolution issues with `/app/` base path.
  - **HEALTHCHECK**: Configurable via build args; webapp uses `--retries=84 --interval=5s` = 420s (7 min), aligned with `wait-for-stack` step timeout.
- **Shared entrypoints**: Each service uses a unified entrypoint script (`containers/<service>-entrypoint.sh`) that branches on `CI=true` env var for CI vs local dev paths. Common helpers in `containers/entrypoint-common.sh`.
- **Shared bootstrap**: `scripts/bootstrap-common.sh` contains shared contract deployment, seeding, and funding functions used by both `containers/bootstrap.sh` (local dev) and `.woodpecker/e2e.yml` (CI).
- CI images are tagged with git SHA and `latest`, pushed to a local registry.

## CI Agent & Registry Auth
- **Agent**: Runs as user `ci` (uid 1001) on `harb-staging`, same host as the dev environment. Binary at `/usr/local/bin/woodpecker-agent`.
- **Registry credentials**: The `ci` user must have Docker auth configured at `/home/ci/.docker/config.json` to pull private images from `registry.niovi.voyage`. If images fail to pull with "no basic auth credentials", fix with:
  ```bash
  sudo mkdir -p /home/ci/.docker
  sudo cp /home/debian/.docker/config.json /home/ci/.docker/config.json
  sudo chown -R ci:ci /home/ci/.docker
  sudo chmod 600 /home/ci/.docker/config.json
  ```
- **Shared Docker daemon**: The `ci` and `debian` users share the same Docker daemon. Running `docker system prune` as `debian` removes images cached for CI pipelines. If CI image pulls fail after a prune, either fix registry auth (above) or pre-pull images as `debian`: `docker pull registry.niovi.voyage/harb/ponder-ci:latest` etc.

## Debugging Tips
- If pipelines aren't being created after a push, check Codeberg webhook delivery logs first.
- The Woodpecker server needs `sudo` to restart. Without it, you cannot: refresh API tokens, clear cached state, or recover from webhook auth issues.
- E2E pipeline failures often come from `wait-for-stack` timing out. Check the webapp HEALTHCHECK alignment and Ponder indexing time.
- The `web-app/vite.config.ts` `allowedHosts` array must include container hostnames (`webapp`, `caddy`) for health checks to succeed inside Docker networks.
- **Never use `bash -lc`** in Woodpecker pipeline commands — login shell resets PATH via `/etc/profile`, losing Foundry and other tools set by Docker ENV. Use `bash -c` instead.