Autonomous CI/CD: Letting an Agent Triage the Build Without Letting It Ship

Mobile release engineering has a specific texture that anyone who has done it recognises instantly. The build breaks. You open a 4,000-line log. Somewhere in it is one line that matters. You fix a version number, a certificate, a Gradle property. It builds. Two weeks later, a different one of the same six problems.

It is not hard work. It is interrupt-driven work, which is worse — it arrives at the wrong moment, costs half an hour of context switch, and teaches you nothing the second time.

This is close to an ideal task for an agent. It is also a place where "let the AI handle it" goes wrong in expensive, public ways, because the end of this pipeline is a binary going to millions of devices. So the interesting engineering is not "can an agent fix a build" — it plainly can. It is where the boundary goes.

Classify before you fix

The mistake is to hand the agent the log and the repository and say "make it green". That produces an agent that disables a failing test, because disabling a failing test does make it green.

Insert a classification step first, and make the class determine what is allowed to happen next.

The critical row is the last one. An agent that patches genuine regressions is an agent that hides them, and the second-order effect is worse than the build failure: the team stops trusting green, which was the only thing CI was for.

Classification is also the cheap part. It is one model call against a filtered log and a diff, and it is right far more often than the fix would be.

Filter the log before the model sees it

A 4,000-line build log in a prompt is mostly noise, and — as the previous article in this series covered — noise is not neutral, it actively degrades the answer.

# Keep the error region and a little context, not the whole build.
grep -nE "error:|FAILURE:|FAILED|\*\* BUILD FAILED|Exception|fatal" build.log \
  | head -40 > errors.txt

# Plus the 30 lines around the first error, which is usually the only real one.
first=$(grep -nE "error:|FAILURE:" build.log | head -1 | cut -d: -f1)
sed -n "$((first > 15 ? first - 15 : 1)),$((first + 15))p" build.log > context.txt

Native toolchains also cascade: one real error produces forty downstream ones. Prioritising the first error, not the last, is a small heuristic that dramatically improves diagnosis quality.

The trust ladder

Do not start at the top. Move up one rung only when the rung below has been boring for a month.

Rung 1 — Explain. The agent reads the failure and comments on the PR: what broke, where, why, and the suggested fix. It changes nothing. Even here the value is real: the person who broke the build stops needing to open the log.

Rung 2 — Propose. The agent opens a PR against the failing branch with the fix. A human reviews and merges. You now have a measurable success rate, which is what tells you whether rung 3 is safe.

Rung 3 — Auto-fix the safe classes. Environmental retries and a narrow allowlist of configuration changes apply automatically. Everything else stays at rung 2. The allowlist is a literal list of file paths and change shapes, not a judgement call the model makes.

Rung 4 — Autonomous internal release. Build, sign, upload to TestFlight or an internal Play track, run smoke tests, post the result. Still no public release.

Rung 5 — Public release. Do not. Not because it is technically hard, but because a staged rollout to production is a business decision with a rollback cost measured in days of review. This rung belongs to a human pressing a button.

Most teams should live at rung 3 and visit rung 4 for internal builds. That captures nearly all the time saved, with almost none of the risk.

What automation the agent should be driving

The agent should not be inventing release mechanics. It should be operating the mechanics you already have, which means those mechanics need to be scripted, idempotent and inspectable before an agent touches them.

# fastlane/Fastfile — the shape that makes automation safe.
platform :ios do
  desc "Internal TestFlight build"
  lane :internal do
    # Deterministic: same inputs, same output, every time.
    setup_ci if ENV["CI"]

    # Match keeps signing identities in a repository rather than in tribal
    # knowledge. This is the single biggest source of "works on my machine".
    sync_code_signing(type: "appstore", readonly: true)

    increment_build_number(build_number: ENV.fetch("GITHUB_RUN_NUMBER"))

    build_app(
      scheme: "App",
      export_method: "app-store",
      # No interactive prompts, ever. An automation that can block on a
      # dialogue is an automation that will block at 2am.
      xcargs: "-allowProvisioningUpdates"
    )

    upload_to_testflight(
      skip_waiting_for_build_processing: true,
      distribute_external: false
    )
  end
end

Three properties make this agent-operable, and they are worth having regardless:

1. Idempotent. Re-running is safe. An agent will re-run.
2. Non-interactive. No prompt can ever appear. Use App Store Connect API keys, not an Apple ID with two-factor.
3. Narrow. internal cannot publish to the App Store. The capability is absent, not merely unused — which means no prompt, however confused, can reach it.

That third point is the general principle: prefer absent capabilities to forbidden ones. An agent that cannot reach production because the credential does not exist in its environment is safe in a way that an agent instructed not to touch production is not.

Flaky tests need an owner and a deadline

Quarantining flaky tests is the feature that most obviously pays for itself and most reliably rots.

Rules that keep it honest:

• Detection is statistical, not a single re-run. A test that fails once is not flaky, it is failing. Flag it when it fails and passes on the same commit, or when its failure rate across recent runs is between a few percent and most-of-the-time.
• Quarantine creates a ticket with an owner and an expiry. Thirty days. Not fixed by then, the test is deleted, and its absence is visible in coverage.
• Quarantined tests still run, with their results reported separately. A quarantined test that starts passing consistently should come back automatically.
• Cap the quarantine. More than a small percentage of the suite quarantined means the suite is the problem, and the pipeline should fail loudly on that condition.

Without the deadline and the cap, quarantine becomes a place where coverage goes to die quietly, and you find out during an incident.

Wire it with a hook, not a bot account

The integration that works is a failure-triggered one: CI fails, a webhook fires, the agent runs in a sandbox with a read-only checkout and the filtered log, and it posts a comment or opens a PR.

# .github/workflows/triage.yml
on:
  workflow_run:
    workflows: ["Build"]
    types: [completed]

jobs:
  triage:
    if: github.event.workflow_run.conclusion == 'failure'
    runs-on: ubuntu-latest
    permissions:
      contents: read        # read the code
      pull-requests: write  # comment or open a PR
      # No packages: write, no deployments, no secrets beyond the model key.
    steps:
      - uses: actions/checkout@v5
      - name: Classify and report
        run: node scripts/triage.mjs --run-id ${{ github.event.workflow_run.id }}

Note the permissions block. It is the most important part of the file. The agent can read the code and write a comment. It cannot push to a protected branch, publish a package, or deploy — and that is enforced by the token, not by the prompt.

The number that tells you if it is working

Not "how many builds did the agent fix". That number rewards patching symptoms.

Measure mean time from red to a human understanding why. That is the thing that was actually costing you: not the fix, but the half hour of log archaeology before the fix.

An agent at rung 1 — explaining, changing nothing — moves that number more than most teams expect. If rungs 3 and 4 never happen, you have already taken the win.

The line to hold

An agent should be able to tell you why the build broke, retry what deserves retrying, and propose the boring fix.

A human should be the one who decides that a binary goes to users.

Keep those two sentences separate and autonomous CI/CD is a straightforwardly good trade. Blur them and you have automated the one decision in your pipeline that was never expensive to make and always expensive to get wrong.

Part of a series on the highest-friction problems in modern software engineering.