harb/docs/ci-pipeline.md

Woodpecker CI

Infrastructure

• Server: Woodpecker 3.10.0 runs as a systemd service (woodpecker-server.service), NOT a Docker container. Binary at /usr/local/bin/woodpecker-server.
• Host: https://ci.niovi.voyage (port 8000 locally at http://127.0.0.1:8000)
• Forge: Codeberg (Gitea-compatible) — repo johba/harb, forge remote ID 800173
• Database: PostgreSQL at 127.0.0.1:5432, database woodpecker, user woodpecker
• Config: /etc/woodpecker/server.env (contains secrets — agent secret, Gitea OAuth secret, DB credentials)
• CLI: Downloaded to /tmp/woodpecker-cli (v3.10.0). Requires WOODPECKER_SERVER and WOODPECKER_TOKEN env vars.
• Logs: journalctl -u woodpecker-server -f (NOT docker logs)

Pipeline Configs

• .woodpecker/build-ci-images.yml — Builds Docker CI images using unified docker/Dockerfile.service-ci. Triggers on push to master or feature/ci when files in docker/, .woodpecker/, containers/, kraiken-lib/, onchain/, services/, web-app/, or landing/ change.
• .woodpecker/e2e.yml — Runs Playwright E2E tests. Bootstrap step sources scripts/bootstrap-common.sh for shared deploy/seed logic. Health checks use scripts/wait-for-service.sh. Triggers on pull_request to master.
• Pipeline numbering: even = build-ci-images (push events), odd = E2E (pull_request events). This is not guaranteed but was the observed pattern.

Monitoring Pipelines via DB

Since the Woodpecker API requires authentication (tokens are cached in server memory; DB-only token changes don't work without a server restart), monitor pipelines directly via PostgreSQL:

# Latest pipelines
PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
  "SELECT number, status, branch, event, commit FROM pipelines
   WHERE repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb')
   ORDER BY number DESC LIMIT 5;"

# Step details for a specific pipeline
PGPASSWORD='<db_password>' psql -h 127.0.0.1 -U woodpecker -d woodpecker -c \
  "SELECT s.name, s.state,
    CASE WHEN s.finished > 0 AND s.started > 0 THEN (s.finished - s.started)::int::text || 's'
    ELSE '-' END as duration, s.exit_code
   FROM steps s WHERE s.pipeline_id = (
     SELECT id FROM pipelines WHERE number = <N>
     AND repo_id = (SELECT id FROM repos WHERE full_name = 'johba/harb'))
   ORDER BY s.started NULLS LAST;"

Triggering Pipelines

• Normal flow: Push to Codeberg → Codeberg fires webhook to https://ci.niovi.voyage/api/hook → Woodpecker creates pipeline.
• Known issue: Codeberg webhooks can stop firing if ci.niovi.voyage becomes unreachable (DNS/connectivity). Check Codeberg repo settings → Webhooks to verify delivery history and re-trigger.
• Manual trigger via API (requires valid token — see known issues):

  WOODPECKER_SERVER=http://127.0.0.1:8000 WOODPECKER_TOKEN=<token> \
    /tmp/woodpecker-cli pipeline create --branch feature/ci johba/harb
  

• API auth limitation: The server caches user token hashes in memory. Inserting a token directly into the DB does not work without restarting the server (sudo systemctl restart woodpecker-server).

CI Docker Images

• docker/Dockerfile.service-ci — Unified parameterized Dockerfile for all service CI images (ponder, webapp, landing, txnBot). Uses --build-arg for service-specific configuration (SERVICE_DIR, SERVICE_PORT, ENTRYPOINT_SCRIPT, NEEDS_SYMLINKS, etc.).
  • sync-tax-rates: Builder stage runs scripts/sync-tax-rates.mjs to sync tax rates from Stake.sol into kraiken-lib before TypeScript compilation.
  • Symlinks fix (webapp only, NEEDS_SYMLINKS=true): Creates /web-app, /kraiken-lib, /onchain symlinks to work around Vite's removeBase() stripping /app/ prefix from filesystem paths.
  • CI env detection (CI=true): Disables Vue DevTools plugin in vite.config.ts to prevent 500 errors caused by path resolution issues with /app/ base path.
  • HEALTHCHECK: Configurable via build args; webapp uses --retries=84 --interval=5s = 420s (7 min), aligned with wait-for-stack step timeout.
• Shared entrypoints: Each service uses a unified entrypoint script (containers/<service>-entrypoint.sh) that branches on CI=true env var for CI vs local dev paths. Common helpers in containers/entrypoint-common.sh.
• Shared bootstrap: scripts/bootstrap-common.sh contains shared contract deployment, seeding, and funding functions used by both containers/bootstrap.sh (local dev) and .woodpecker/e2e.yml (CI).
• CI images are tagged with git SHA and latest, pushed to a local registry.

CI Agent & Registry Auth

• Agent: Runs as user ci (uid 1001) on harb-staging, same host as the dev environment. Binary at /usr/local/bin/woodpecker-agent.
• Registry credentials: The ci user must have Docker auth configured at /home/ci/.docker/config.json to pull private images from registry.niovi.voyage. If images fail to pull with "no basic auth credentials", fix with:

  sudo mkdir -p /home/ci/.docker
  sudo cp /home/debian/.docker/config.json /home/ci/.docker/config.json
  sudo chown -R ci:ci /home/ci/.docker
  sudo chmod 600 /home/ci/.docker/config.json
  

• Shared Docker daemon: The ci and debian users share the same Docker daemon. Running docker system prune as debian removes images cached for CI pipelines. If CI image pulls fail after a prune, either fix registry auth (above) or pre-pull images as debian: docker pull registry.niovi.voyage/harb/ponder-ci:latest etc.

Debugging Tips

• If pipelines aren't being created after a push, check Codeberg webhook delivery logs first.
• The Woodpecker server needs sudo to restart. Without it, you cannot: refresh API tokens, clear cached state, or recover from webhook auth issues.
• E2E pipeline failures often come from wait-for-stack timing out. Check the webapp HEALTHCHECK alignment and Ponder indexing time.
• The web-app/vite.config.ts allowedHosts array must include container hostnames (webapp, caddy) for health checks to succeed inside Docker networks.
• Never use bash -lc in Woodpecker pipeline commands — login shell resets PATH via /etc/profile, losing Foundry and other tools set by Docker ENV. Use bash -c instead.