CI/CD
Designing a conformance regression pipeline
How to structure smoke, regression, nightly, and certification tiers so conformance feedback arrives in minutes without burning lane budget.
7 min read
The mistake teams make when conformance suddenly becomes cheap is running everything on every commit. Tier the work instead: fast feedback on the cases that catch most regressions, deep coverage on a schedule.
| Tier | Scope | Trigger | Target time | Blocking |
|---|---|---|---|---|
| smoke | 12 cases, 1 lane | every push | under 20 min | yes |
| regression | 180 cases, 4 lanes | merge to main | under 3 h | no |
| nightly | 647 applicable, 8-12 lanes | 02:00 schedule | overnight | no |
| certification | full suite | release candidate | 2-4 days | release gate |
Choosing the smoke set
- Every case that failed in the last 90 days.
- One case per protocol domain (NAS, RRC, PDCP, RLC, MAC).
- Initial registration and PDU session establishment — these break first.
- Any case touching code changed in the diff, if you maintain a mapping.
Automate smoke-set selection
bashbash
aeon exec list --state verdict --since 90d --json \ | jq -r '.[].cases[] | select(.verdict=="fail") | .id' \ | sort -u > .aeon/flaky-and-failing.txt aeon exec run --cases-from .aeon/flaky-and-failing.txt --build "$GIT_SHA" --wait --exit-code
noteReview the smoke set monthly. A gate nobody trusts gets bypassed, and a gate that takes an hour gets disabled.
Run this against a real SDR lane
The tester is a service, not a box. Push a build, reserve a lane, get a verdict.