CI/CD & GitHub Actions
Trigger an existing Observyze regression set from CI and fail the check when the run fails, times out, or reports failed cases.
Quick Start: GitHub Actions
This workflow calls the implemented regression API directly. Store a project API key in GitHub Actions secrets and replace the example regression set ID.
name: AI Regression Tests
on:
pull_request:
branches: ["main"]
jobs:
observyze-eval:
runs-on: ubuntu-latest
env:
OBSERVYZE_API_URL: https://api.observyze.com
OBSERVYZE_API_KEY: ${{ secrets.OBSERVYZE_API_KEY }}
REGRESSION_SET_ID: rs_replace_with_your_set_id
steps:
- name: Trigger and verify regression run
shell: bash
run: |
set -euo pipefail
response="$(curl --fail-with-body --silent --show-error \
--request POST \
--header "x-api-key: $OBSERVYZE_API_KEY" \
"$OBSERVYZE_API_URL/api/v1/regression/$REGRESSION_SET_ID/run")"
run_id="$(jq -er '.run_id' <<<"$response")"
echo "Started regression run $run_id"
final_payload=""
for attempt in $(seq 1 60); do
final_payload="$(curl --fail-with-body --silent --show-error \
--header "x-api-key: $OBSERVYZE_API_KEY" \
"$OBSERVYZE_API_URL/api/v1/regression/runs/$run_id")"
status="$(jq -r '.run.status' <<<"$final_payload")"
if [[ "$status" == "completed" ]]; then break; fi
if [[ "$status" == "failed" ]]; then
jq '.run' <<<"$final_payload"
exit 1
fi
sleep 5
done
status="$(jq -r '.run.status' <<<"$final_payload")"
[[ "$status" == "completed" ]] || {
echo "Regression run timed out with status: $status"
exit 1
}
failed_cases="$(jq -r '.run.failed_cases // 0' <<<"$final_payload")"
jq '.run' <<<"$final_payload"
[[ "$failed_cases" -eq 0 ]]How it Works
1. Queue a Durable Run
The authenticated API creates a run record and queues the selected regression set.
2. Execute & Evaluate
The worker calls your configured replay webhook, then applies safety and optional hallucination evaluation.
3. Fail Closed in CI
The check fails on API errors, worker failure, timeout, or any failed regression case.
Prerequisite: the Redis-backed regression worker and your application-owned replay webhook must be configured and reachable. The run endpoint reports service unavailability instead of fabricating a result.