The Process in Practice
Our previous blog outlined our approach to designing and implementing an AI-native SDLC. In this post, we’ll take a deeper dive into how we follow it, the challenges we’ve faced, and what’s coming next.
We created a list of tenets to follow, premised on the condition that they could change if we identified better ones. These tenets guide how we follow our process and where we focus on improvements.
- Never touch the code directly. Work backward from where the solution to the issue could reasonably be defined and start there.
- Issues that can be identified in review are flagged as early as possible and automated.
- Developers can use any tool of their choice and reliably follow our SDLC.
Before we continue, let’s take a quick look at our tech stack.
Tech Stack
Our tech stack is primarily composed of Python for the back end, with a sprinkling of Go and TypeScript for the front end. Infrastructure is managed using Terraform, and we use GitHub Actions for CI on PRs and CD post-merge. Our services can be deployed to AWS, Azure, and GCP.
For tooling, we use Notion for collaborative documents, Linear for managing work, and Slack for internal communications and chat. We also use a combination of Antigravity, ChatGPT/Codex, Claude, Cursor, Pi, and a variety of open-source models. We support multiple models and multiple agent harnesses because no single combination performs the best for every task. Locking into one vendor also means we would miss out whenever a new model or harness update is made by another vendor.
Product Design
Any work we do must start from a product-centric perspective. What is the problem we want to solve, who is it for, and how do we want the product to work? To answer these questions, we begin with feedback from customers and the field. AI-generated transcripts and notes from these sessions are the starting point of our process. The person defining the feature combines those source materials with their own product judgment, using an agent to draft the product requirements document (PRD).
The requirements themselves are written in Easy Approach to Requirements Syntax (EARS). The goal is to strike a balance between writing something that a person can understand and giving agents explicit triggers, actors, conditions, and expected behaviors. Each requirement also includes the intended persona, such as an insurance underwriter or business admin. We use a skill to write these, and run a test after each version of the PRD is drafted.
For example, assume we are adding support for an integration engineer to bring in an externally defined agent. The requirement would be:
EXT-10: When an integration engineer publishes an external agent through the application programming interface, the system shall validate the agent configuration against the active registry schema before changing the agent’s status to published
A person reviews the PRD, iterates it, and eventually approves it, making it ready for the next phase.
System Design
The system design spec covers how we build the system and requires a PRD so we know what build. We develop the design through an iterative, Socratic interview in which the agent challenges assumptions, identifies missing decisions, and tests the design against the approved PRD. Plugins like superpowers and grill-with-docs support that process. We also use a skill to test the design. The test checks for a reference to a PRD, ensures the requirements cited actually exist, and provides a readability score for people reviewing the spec.
Continuing with the same external agent example, the design spec would include an API, the validation it will perform, and a reference to the requirement from the PRD.
As with the PRD, a person reviews the spec, iterates on it, and eventually approves it. A person decides whether the spec should satisfy all requirements or if the feature is too large and must be divided into subset specs.
Implementation and Review
After people approve the requirements and design, agents can perform much of the implementation without continuous supervision. The design is broken down into tasks by the agent, and each task starts by following test-driven development. The agent writes the tests first to establish expectations for the implementation, verifies that they fail, writes the application code, and then reruns the tests to verify the application meets expectations. The completed change requires human approval before merge. Our definition of “complete” depends on passing specific checks. We also have guardrails that run during implementation – we want to shift checks as far left as possible.
Our guardrails are triggered through hooks, specifically on post-edit and stop events. For example, if a Python file has been edited, the hook will run a script that runs ruff for code linting. If it’s a Terraform module, we run a checkov scan with defined custom policies. Stop events run a series of release checks including pytest and vitest with coverage, checkov again, and a custom license check for all 3P dependencies that were pulled in to ensure compliance with our 3P policy. Our 3P policy specifies which licenses are allowed and checks for the presence of reported CVEs and their severity. The hooks are configured to return an error code, blocking the agent from continuing until the error has been resolved. We centralized all hooks into scripts so every harness we use can just reference the script without needing to redefine it.
A PR is created once the implementation has been completed, but it’s still not ready for a person to review. We’ve configured CI to run on PR create and synchronize events. They include the same set of checks on agent stop events (in case the hook was misconfigured and didn’t execute) along with some additional checks.
One check is for PRD references. Continuing with the same external agent example, an integration test would invoke the API and require a negative test case that fails validation. The test would refer to the specific requirement it is checking for.
Model Ensemble Review
Another of the checks is a custom AI code reviewer that we’ve built. The reviewer takes in the main model provider that was used to implement the code, labeled on the PR, and uses that to select an ensemble of models that review the code. For example, if Claude was the author, then GPT and Gemini would be the reviewers. In our experience, an ensemble of reviewers combined with an adversarial approach provided more comprehensive feedback than any one individual model. We use models from providers that are different from the author to avoid potential blind spots where a miss in the implementation is likely to be missed by the reviewer. Asking Claude to review work done by Claude brings diminishing returns.
The review consists of three phases:
- Each reviewer independently goes through the PR and surfaces potential findings with a severity and category. Each finding includes justification for why it’s an issue.
- Each reviewer gets to see what the other reviewer found and decide whether to challenge or accept each finding. They can also introduce new findings, given new evidence.
- Each reviewer can see responses from their partners and decide whether to withdraw or commit their initial findings.
All committed findings are synthesized into a review on the PR, and each comment must be resolved before the PR can be merged. Over the past three months, AI reviewers examined 678 PRs and reported at least one finding in 477 of them and a total of 2,358 findings, 299 of which were classified as high findings. Of these, 1,344 were accepted automatically and fixed by the agent in the same session. The remaining findings were referred to human feedback, with 132 marked as intentional based on human feedback. For example, there was a finding where a request could fail to be processed if a config file wasn’t present. This error wasn’t intentionally guarded against, because the config was provided as a local file in the same repo and there was a unit test verifying the file was accessible at run-time.
When to Involve a Human
The agents can monitor the status of CI on the PR and address findings. The change proceeds to human review after CI passes with no open findings. Human judgment is important, but so is their time. We don’t want people spending time reviewing code until we know it has passed all of our checks. If reviewers find an issue that should’ve been caught in an automated check, they go back and update the check. For example, they might add a new ruff rule for magic values, PLR2004.
After the PR has been approved and merged, CD takes over. First, we create the runtime bundle containing the code and specs. The bundle is deployed to beta, tested, deployed to a shadow production environment, tested again, and finally tagged for distribution to customers as a versioned artifact.
Challenges
We’ve encountered a few challenges in our journey as we strive to continuously improve. Here are a few specific ones to highlight.
Test Quality
Unit tests are one of the easiest ways to verify behavior, but it’s also easy to game the system when they are combined with a check on line coverage. We found a number of our unit tests were quite hollow – they were exercising lines of code, but not actually verifying all of the behavior. Mutation testing helped us identify these tests and acted as another check to guard against them in the future. Source code is mutated in a variety of ways, and the unit tests are rerun. If no unit test switches from pass to fail due to the mutation, it is recorded and flagged as a potential issue. Below is some sample code that determines whether a request should be blocked based on the IP and its properties.
No tests failed when changing the or to and because the test cases were setting both is_loopback and is_private to the same value and were never testing when only one of them was true.
This is a problem because a future change could alter the validation behavior without anyone noticing. The tests would still pass, even though the behavior changed in a major way. We set our line coverage target to 80%, and we were sitting nicely at a little more than 90%. A manual audit of the mutation testing survivors found that our actual coverage for lines with verified behavior was closer to 70%, more than 20 percentage points lower.
Intent of User Experience
User experience was another challenge. Our process starts with a product requirements spec and eventually results in working code. However, we realized we were failing to specify and review customer journeys when they interact with the system. We specified functionality, and it worked once everything was configured by the user, but the act of configuring the system through the UI was convoluted. The gap in specification meant the agent came up with its own choices, and those choices couldn’t be cross-verified against the spec to identify gaps or misalignment. This became clear once multiple features had shipped, and we started trying to use them in combination through the UI.
This was a major gap that we plugged in all phases of the SDLC. Our product requirement specs include a section on customer journeys. We check each journey to confirm three things: every capability it mentions has a defined requirement, the steps in the journey fit together, and a customer can get something running quickly. The implementation is verified against the customer journeys for alignment.
Process Consistency
One final challenge to highlight: increasing the floor for everyone in the organization. Our SDLC has undergone revisions over time, some of which were automatic for all, such as running mutation tests to check the quality of unit tests. But others were changes that individuals must adopt to see gains in output quality from the agents. The challenge was getting adoption of those changes across the organization so that the learnings of one person benefitted everyone.
We’ve created a marketplace of plugins to assist in document writing, such as the PRD. This includes skills and testing scripts. We’ve started distributing these through internal marketplaces supported by the providers. We’ve also modified our code repositories to reference these and other open-source plugins directly in our agent documentation files so the agent can check for local installation and prompt the user for permission or instructions to install plugins so that they can benefit without having to keep up with the latest and greatest.
What’s Next
The rapid pace of development in capabilities of the models, harnesses, and solutions from the community create a continuously evolving landscape. We’re happy with the gains in productivity while maintaining a high bar for quality and operations, but we know there’s more we can do.
Reviews continue to be a bottleneck, and there are a few things we want to do specifically to alleviate the delays. Sometimes, there will be an incremental change to a feature, or even the creation of a new feature that functions very similarly to an existing one. Those can be the easiest to review when you are certain there are no regressions and that it actually is following a well-known path. For example, it’s a common practice to have a design system for UI elements, such as a feedback form. Any time someone wants to add a form to collect feedback, they use that design element, and they know it will have consistent behavior and mesh well with the rest of the UI. We want to do the same for our architecture and define a design system for our architecture, integrations, and interfaces. Any changes that don’t follow the design system draw more scrutiny during review.
We also want to invest in our own benchmarking setup based on sampled changes we’ve previously made. This setup will give us confidence in harness or process changes, whether they are our own or pulled in from providers or the community. The public benchmarking results are either too general and may not completely apply to how we work and what we work on, or they are hidden and it becomes difficult to trust the claims. We can’t reliably tell how relevant they are to us. Using our own setup with our own tasks that we’ve verified means we have a quantitative way to measure the impact of changes. That impact could be in the form of token efficiency or result quality and task acceptance.
Finally, there is one major piece of the SDLC we haven’t even scratched the surface of: dev ops. The runtime bundle of specs shipped with every deployment means we already have the foundations of a knowledge base that a dev ops agent can use. If we include the location of metrics and logs, the agent has a significant head start debugging issues without needing to start a blind search. The agent could identify the root cause, propose a fix, and verify the fix is aligned to the specs before sending it out for human review.
Conclusion
We’re still learning and shaping our process as we go. An AI-native SDLC isn’t just about making software more quickly. It’s about capturing human intent, compounding it across the entire lifecycle, and maintaining that chain of intent past software delivery. Any miss we find is feedback to the process.
