About


Hey hey, I’m Eric – a life long engineer, leader, (published) author, military combat veteran, and martial artist who enjoys helping people and systems perform at their best. Across my work in technology, consulting, and team leadership, I’ve learned that strong organizations are built through communication, structure, and thoughtful habits. I share those lessons through my writing, management training videos, and my podcast, “Beyond the Belt”, where I talk with other BJJ Black Belts who’ve pursued excellence in their passions and lives.

Whether it’s building software, coaching managers, or exploring the mindset behind high-level performance, I care about helping people grow in a way that feels real, practical, and sustainable.

Reach out to me if you have questions or come take a BJJ class with me at Fenriz Gym in Berlin, Germany.

If you want my posts delivered as they go live with no fluff, subscribe to the newsletter or follow me on Substack.

Latest Writing


All Writing

Retrofitting: Maintaining Software Everyone Can Build On

At one of our Builder’s Unscripted events, where we encourage people at Mapp to share what they’re working on in whatever state it’s in, I was explaining how we use retrofits to keep our Mapp Labs applications up to date. I wanted the builders responsible for that maintenance to better understand how the process works and why we’re asking them to follow it.

But a few people who aren’t engineers pushed back on the process itself.

It usually went something like, “Sometimes this takes my agent a full day or two, and I can’t do other work on the application in the meantime.”

They’re not wrong. Some of the necessary retrofits can be substantial. Depending on how far an application has diverged from the original template, the agent might spend a considerable amount of time investigating the existing code, making changes, and testing them to ensure the outcomes are met and nothing is broken.

My response was basically, “Tough shit. This is part of software maintenance and you need to do this.”

I explained to the non-Engineers in the room that owning software can be a bit like owning a car. You can drive it into the ground, or you can maintain it and keep it running longer. You don’t necessarily need to know how to replace your spark plugs, but you do need to make sure the work gets done. You either learn to do it yourself or take the car to a mechanic.

Building software is just one of those things that comes with maintenance obligations.

AI has made it possible for people who aren’t engineers to build and operate applications. It hasn’t removed the engineering work required to keep those applications running.

And there’s an important distinction here: you can delegate the work of maintaining software to an agent. You can’t delegate away the responsibility for maintaining it.

Fortunately, that particular conversation was well-received. A whole bunch more retrofits got done pretty quickly after that meeting.

I’ve already written about Mapp Labs and the engineering foundation behind it. We’ve now had more than sixty applications built through Labs, by people across Engineering, Product, Customer Success, Marketing, Compliance, and other parts of the company.

The challenge isn’t just helping those people build software on a sturdy foundation. It’s maintaining an increasingly diverse collection of applications without making Tech responsible for every one of them.

When it comes to Labs apps, that change in organizational strategy means we also needed to adjust how our engineers are spending their time. Instead of maintaining every application ourselves, we now have to define what those applications need to guarantee, make those requirements available to their builders, and figure out whether or not those requirements are actually being met.

As we’ve discovered, defining the right requirements can be harder than actually implementing them.

But we’ve been solving that problem pretty effectively through behavioral retrofits, agent-readable compliance requirements, and fleet tracking systems.

One of my favorite parts of all this is that the system itself keeps improving as we learn from maintaining the applications.

The Template Is Only the Starting Point

Every Labs application begins as a copy of the same template, but it doesn’t remain like that for long.

To me, it’s a feature, not a bug, that builders have independent repositories and can change the architecture however they need. Some applications remain fairly close to the template. Others have separate frontends and backends, different infrastructure, substantial data models, and their own deployment requirements.

But when those applications diverge, updating the original template doesn’t magically update any of them.

If we change the way authentication works, fix an infrastructure assumption, or introduce a security requirement, I can’t simply copy the change into every application. Some builders may already have solved the problem in their own way. Others may no longer have the relevant functionality in their app. And some may have diverged enough from the foundation that copying the template’s code verbatim would break something.

We use shared libraries, components, and architectural conventions wherever possible in the template. But a shared library can’t fix every infrastructure, deployment, or application-specific requirement.

Builders who aren’t engineers don’t necessarily know to tell their agent to reuse existing components instead of writing something new. I also can’t expect them to independently investigate every template change and figure out which parts apply to their application.

So I need a way to tell their coding agent what has changed, what problem we’re trying to solve, and what needs to be true afterward.

That’s what our retrofit system does.

Retrofits are About the Outcome, Not the Code

Our first retrofits were fairly prescriptive. They described which files to modify, the code to add, and the steps to follow.

That worked reasonably well when applications still resembled the template.

Then I used those same retrofits to catch up the Commerce Graph Studio (CGS), one of the applications I (primarily) maintain myself and which has grown into a substantial part of our Mapp Fashion infrastructure.

CGS had diverged significantly from the original template it was built on. It had different deployment scripts, authorization conventions, data models, infrastructure, and functionality that didn’t exist in the base template.

I asked its coding agent to catch up with the latest template, and it went through more than a dozen retrofits ranging from library upgrades to data model changes. It completed the work, but the instructions derived from the app template often required additional investigation and my input on what to do. Sometimes the files named in the retrofit didn’t exist anymore. Sometimes CGS had implemented the same capability differently. And sometimes I preferred the solution I had already worked out in CGS to the approach used in the template.

After the retrofits were finished, I asked the CGS’s build agent what could have been written differently in the retrofit documentation to make the work easier and reduce mistakes. That feedback eventually became a more explicit standard for writing retrofits.

The biggest change was putting the required behavior ahead of the implementation instructions.

Now, a retrofit defines the rules that must hold when completed, how to detect whether the application already satisfies them, what decisions require the owner’s input, and how to verify and record the result. It still provides an example implementation, but that implementation isn’t necessarily what the application should adopt.

For example, we have a preview-as-customer capability that allows an authorized Mapp employee to view an application how a specific customer would see it.

Here’s an abbreviated version of the rules from its retrofit:


1
2
3
4
5
6
7
8
9
10
11
12
Rules — what must be true afterwards

1. Only an active, authorized employee can start or continue a preview.
2. Authorization is rechecked on every request.
3. A preview session cannot write customer data.
4. Customer API routes cannot accept the preview session.
5. Preview activity is audited.
6. The preview must expire.

The template's implementation is one way to satisfy these rules.

If the application already satisfies them differently, preserve its implementation and record the deviation assuming there is no concession to security.

The full retrofit contains additional instructions, tests, and documentation, but the example should make the goal(s) pretty clear.

The template’s implementation is an example. The rules are the contract.

The agent starts by assessing the actual application, which it should be mostly familiar with anyway. Detection focuses on behavior first, then properties of the implementation, and finally code structure.

A test demonstrating that a preview session cannot modify customer data is more useful than checking whether a particular authorization helper exists under a specific filename.

If the implementation already meets the requirement in the rules, the agent can record the retrofit as applied without making any changes at all. If only some rules are satisfied, the agent makes the necessary changes. If a capability doesn’t exist in that app or isn’t relevant, it records that the retrofit isn’t applicable. And if the correct implementation depends on a builder’s decision, the agent asks for the relevant input.

I think this is a much more useful way to distribute engineering requirements than telling every application to adopt the latest reference implementation. Software engineering is an art and not every person or build agent will do it the same way.

This also (intentionally) puts more responsibility on whoever creates the retrofit. This person should be an experienced engineer and needs to distinguish between the requirements that every application must satisfy and the implementation choices we happen to make in the template.

The agent can investigate the application and ask its owner questions, but deciding what should be a shared requirement still needs experienced engineering judgment.

The Best Fixes Don’t Always Start in the Template

The preview-as-customer capability is an interesting example because CGS implemented it before it became part of the template.

I built the functionality in CGS, then generalized it into the shared foundation so other applications could use it. It was pretty obvious to me that others would want this capability in their applications. And I didn’t want non-engineers to try to implement authentication workarounds without understanding the risks and potential pitfalls.

By the time CGS went through the corresponding retrofit, the two implementations had diverged quite a bit. CGS used its own authorization components, account model, and administrative interface. Its audit log records also used CGS-specific entities.

The feature already worked, but when CGS’s agent evaluated it against the retrofit, it found gaps around authorization, read-only behavior, and how preview-as-customer interacted with the rest of the application.

Some requirements from the retrofit were already satisfied from the initial implementation. Others needed more work than I’m comfortable admitting, especially since I guided and checked the original build myself.

The agent ended up hardening CGS’s implementation rather than replacing it.

A few days later, we added another capability that let customers modify data instead of leaving the customer-facing side as read-only. In other words, we properly added a customer-facing component to this app. Under normal circumstances, the write by a customer needed to work. But when an employee was previewing the application as a customer, that same operation had to be blocked.

The tests introduced during the retrofit meant we had to account for that distinction when adding the new functionality.

That made the original maintenance requirement useful beyond the initial correction. It became an expectation for future development.

Interestingly, the exchange of learnings also works in the opposite direction.

I encountered an audit-logging problem while working on CGS, corrected it there, and then brought that correction back into the template as a retrofit for all the other applications and builders to take advantage of. CGS obviously didn’t urgently need to reapply that particular retrofit because it was where the problem had originally been fixed.

Labs applications aren’t just consumers of the foundation. They’re also where we discover what the foundation is missing. That could be something I (or another builder) finds while working on CGS or another builder’s agent flags something back to the team in Slack.

Not every fix should become a requirement for the entire fleet. So which of those lessons or flagged issues should become foundational requirements?

To decide that, I first look at whether the problem came from the shared foundation, whether it introduces security, privacy, or compliance risks, or whether other applications are likely to encounter it. If it’s specific to one application’s circumstances, I’ll probably leave it with that application.

But I also have a pretty good idea of what other people are building (again because of fleet tracking) and where we’re trying to take the company strategically. Sometimes I see a capability in one application and know that several other builders are going to want it once they find out it exists. If it’s difficult to implement, or particularly easy to get wrong, I’d rather bring it into the foundation than have everyone try to figure it out independently.

Sometimes that means adding a capability other builders can choose to use (or ignore). Other times it means defining a requirement every applicable application must satisfy. Making that distinction comes down to experience, engineering judgment, and understanding what the company is trying to accomplish.

Once we’ve made something a shared requirement, we need to know what happened when each application evaluated it.

Recording What Happened

Each application records its retrofit status in a .retrofit-state.json file in the repo.

The file isn’t meant to contain every command the agent ran or the entire investigation behind its conclusions.

In fact, we initially tried keeping much more detail in that state file. It grew way too quickly, and most of the extra information wasn’t particularly useful day to day.

But the experiment helped us figure out what was worth keeping, even though generating it and then pruning the rest was a massive waste of tokens.

The current approach is to keep the state reasonably small. The code, tests, commits, and further agent investigation are all there as possibilities if and when we need more evidence.

We’ve also made the results more structured as we’ve started using them in various ways across the fleet.

For example, here’s an abbreviated historical record showing two different outcomes from CGS:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
  "retrofits": {
    "2026-10-14-fleet-identity": {
      "status": "partial",
      "templateVersion": "0.0.49",
      "remaining": [
        {
          "kind": "ops-step",
          "what": "Deploy updated Repository and Kind AWS tags"
        }
      ]
    },
    "2026-10-14-public-api-audience": {
      "status": "applied",
      "notes": "Existing implementation satisfies the requirement.",
      "deviations": [
        "Application-specific OpenAPI registry retained."
      ]
    }
  }
}

Shortened and sanitized from the records captured during the fleet-identity rollout.

The second entry was applied because CGS already met the requirements. The first was considered partial because the application changes were complete, but an infrastructure deployment still needed to happen.

Earlier, an agent might have marked that first retrofit as applied and buried the deployment step in the notes. Now a partial retrofit records what remains, including whether it’s an operational action or an additional verification.

We also intentionally distinguish between skipped and not-applicable as valid results.

One of our retrofits implementing a maintenance-deployment system exposed why we needed this distinction. An application that had diverged quite a bit from the base template already had its own process for handling the same risk. Its owner declined to adopt the template’s scripts, so the agent recorded the retrofit as skipped.

That matched the instructions, but it misrepresented the result. The application wasn’t refusing to do maintenance in the retrofit. It had just already handled the risk differently.

The problem wasn’t the application. We’d written the retrofit around the template’s implementation rather than the outcome we needed. The resulting record made an architectural difference look like unfinished maintenance work.

An application that doesn’t use a specific capability can record the retrofit as not-applicable. An owner who deliberately declines an applicable requirement records skipped. And an application that already satisfies the requirement through a different implementation can record applied, along with the deviation.

Those statuses are all similar, but they aren’t interchangeable. At fleet scale, confusing them means we’ll either chase maintenance that isn’t necessary or miss work that actually needs doing.

And as the fleet keeps growing and every application evolves independently, there’s no realistic way to track all of that by hand. We need tooling that can tell us what’s happening across the entire fleet.

When the Inventory Didn’t Add Up

To do that, we’ve been building an application called Labs Overview to give us an inventory view of the applications and infrastructure we’re operating.

I’ve been managing fleets of application servers in various forms for over 20 years, and the questions I start with haven’t changed much in that time. I always want to know, What’s running? Where is it running? Which versions of which apps are deployed? Who owns what? When were they last changed?

The Overview app was started with those basics in mind. Then questions specific to Labs were added like: which template versions applications had been evaluated against, which retrofits were outstanding, and what maintenance still needed to be done.

Next, it was expanded to collect information across our AWS infrastructure and GitHub organizations.

That first broader inventory was particularly painful. I expected to find a few inconsistencies, but nowhere near as many as we did. What bothered me the most was knowing how many of these issues could have been prevented with a better foundation.

Fortunately, the issues all smelled kind of the same. Some apps had repository names that didn’t match their application names. Some shared infrastructure was being classified as applications. Owner information appeared in different formats. Some applications had no retrofit state file or app config info.

And the retrofit records themselves, when they were present, were often inconsistent. Agents had invented names for retrofits that didn’t have obvious identifiers. The same maintenance change appeared under several different names depending on the application. The first inventory of roughly 30 apps at the time, found 117 distinct identifiers that hadn’t been standardized against the template’s retrofit guides.

Even when a record included a template version, we couldn’t reliably tell which applications had applied a particular retrofit because the identifiers didn’t match.

There were also cases where an agent recorded skipped simply because an application didn’t use a particular capability, making irrelevant work appear to be maintenance debt.

It’s possible that the Overview app (fleet tracker) could have worked around those problems with some increasingly complicated matching rules. But I didn’t want to go that route when there was clearly a better way.

Ultimately, we learned that the hard part of tracking the fleet wasn’t collecting the data. It was getting every application to mean the same thing when it reported something.

So we made a lot of changes to the Labs app template.

We introduced a machine-readable retrofit index with official identifiers, applicability information, criticality, and aliases for historical record keeping. We clarified the status rules and required partial retrofits to identify their outstanding work.

We added consistent repository and application identifiers to app.config.json, along with Repository and Kind infrastructure tags in AWS so the Overview could associate deployed applications with their repositories without guessing from names.

New Labs applications now also get a retrofit JSON state file when they’re created, rather than waiting until their first maintenance catch-up. And we added a lightweight validator to help agents and builders identify malformed or inconsistent records. The validator deliberately only warns rather than blocking work from progressing to avoid getting in the builder’s way.

As a result of all this prep work, the Overview app only has to read the contract (which it has to adhere to itself anyway). When the contract isn’t good enough, we improve it at the source and push it out, rather than having the Overview invent increasingly elaborate interpretation rules.

This gave us a different kind of feedback loop. Individual applications had already been showing us problems in the template. Now the Overview app was showing us problems in how the entire fleet reported its state.

We ended up turning those findings into new requirements and then sent them back out through the same retrofit system that had helped create the mess in the first place.

Consistent Records Can Still Be Wrong

After running the newer fleet tracking for a little while, we encountered another important distinction we needed to deal with.

An application’s recorded template version doesn’t necessarily tell us whether its maintenance is complete.

We found applications claiming to have been evaluated against the latest template while still recording security retrofits as pending. This is annoyingly specific, but it isn’t actually contradictory. The version identifies what the agent evaluated the application against, not whether it completed every requirement.

So the Overview app now separates out those pieces of information. It shows the evaluated version, but computes the maintenance status from individual retrofit entries, their applicability, and criticality.

It can distinguish an application that’s up to date from one that’s been assessed against the latest requirements but still has work to do.

Of course, those records are still statements made by agents. A builder could tell an agent to mark something as completed even when it isn’t if they don’t want to wait for the agent to do all the work.

But as an operating model, I’m choosing to trust people as much as I can rather than design the entire system around preventing dishonesty. That doesn’t mean I assume an agent’s reported status is always correct. We have other review processes for applications where the risks warrant checking things further. And if something doesn’t look right, as a final fallback, I can always pull down the repository and investigate it myself.

We also found another problem when we started examining the outstanding work recorded in partial retrofits.

Several internal applications were reporting outstanding security-related deployment steps. Their agents had requirements to deploy the retrofitted changes to production and record the completion of that task. Except those applications didn’t have production environments.

Some of our internal tools deliberately run only in staging even though they are effectively the live applications for their users. Their owners don’t need a second copy running in production, and requiring one would add cost and maintenance overhead without providing any additional value.

Personally, I’d prefer those tools to run in production only instead of a staging environment; mostly because of our networking rules and access to other systems. But moving them isn’t free. It often requires help from someone more experienced than the builder, and for an internal tool that’s already working, the change often isn’t worth the effort.

We also have other applications that are temporarily staging-only while they are in research-mode and will eventually move to production. The initial set of retrofits didn’t account for that distinction either.

This meant that the JSON was valid and the fleet tracking system correctly identified that the recorded step remained outstanding. But the step itself was wrong for those specific applications.

Getting every application to report consistently doesn’t mean we’ve written the right requirements. A record can be accurate about a requirement that’s wrong.

This feels obvious in hindsight, but it wasn’t when we wrote the retrofits. We had to make the maintenance records comparable before we could see where our requirements made incorrect or insufficient assumptions about particular applications. Schema validation wasn’t going to catch those mistakes. Comparing the results across the fleet did.

That’s been a recurring pattern with this fleet of applications: fixing one problem gives us enough visibility to discover the next one (or next set). Then we still have to decide whether it’s the application or the shared requirement that needs to change.

Compliance Can’t Review What It Can’t See

I’ve written before about the compliance problems created when software no longer passes through the traditional software development lifecycle (SDLC). The SDLC wasn’t just where compliance requirements reached software. It was also one of the ways Compliance teams learned that software even existed.

We don’t want every Labs application going back through Engineering for these kinds of checks. Instead, experienced engineers and compliance professionals have already put the common requirements for authentication, authorization, sensitive data handling, and audit logs into the template’s agent instructions and checklists.

The retrofits carry new or changed requirements into existing labs applications. Then we record the results alongside each application’s maintenance state in .retrofit-state.json.

With the common requirements already covered by the template and retrofits, the agent can now focus more on what the builder has added or changed. It also needs to ask how those capabilities are actually being used, since that isn’t always something it can easily determine from the code.

The questions themselves are pretty standard: Who uses it? Does it handle customer data? Is there personal information? Does it use an LLM? Will customers have access to it?

When I ran some of the newer retrofits myself against the apps I maintain, the agent explicitly asked me questions like these when it couldn’t figure it out on its own. It didn’t assume it could infer every important fact from the code.

We’ve now extended the checklist to include a structured risk assessment against our internal AI Builder Standard. This is the policy document that our compliance team wrote to ensure that we stay ISO 27001 compliant while letting non-Engineers build software.

That assessment records the application’s risk tier, lifecycle, relevant triggers, data flows, and operational information. This can include important but not necessarily obvious things like, how the application can be stopped during an incident.

The owner (again) confirms the facts the agent can’t reliably determine from the code. And identifying a risk characteristic isn’t the same as finding a compliance violation. An application handling customer data, for example, might need additional controls or a closer review. That doesn’t automatically mean the application has done something wrong.

The assessment records that information so the compliance team can decide what needs further attention. The record itself lives alongside the application’s other maintenance information.

Here’s a shortened example of the structure as it sits in the retrofit state json file:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "compliance": {
    "checklistVersion": "3",
    "type": "full",
    "result": "pass",
    "risk": {
      "tier": "team-tool",
      "lifecycle": "internal",
      "confirmedBy": "<owner>",
      "factNotes": {
        "customerDataInTesting": "No production stack; staging is the operating environment."
      }
    }
  }
}

An abbreviated, sanitized example from an internal application’s assessment.

The full record has much more detail, and the Overview app renders it as a readable risk assessment. It also flags things that might need another look, such as outdated assessments, expired exceptions, relevant code changes, or recorded compliance gaps.

Recently, I went through those assessments in Overview with our Protect (compliance) team. We discussed the findings and identified changes that would make the information both more complete and easier to visually interpret.

But a few of the applications were entirely new to the Protect team. The apps hadn’t passed through the traditional SDLC or even appeared in the Slack announcements yet. That alone made the review of the collected data worthwhile.

The retrofit process is the delivery system for how we get compliance requirements into software built outside the old SDLC. The Overview app gives the Protect team another way to discover that software, assess the risks around it, and decide which applications need further review.

The Protect team now works directly from these structured records collected from the repos instead of requiring a separate Full Review document with the same information from the builders. Sometimes the assessment is enough for them to decide no further investigation is needed. Other times, particularly when sensitive information is involved, they need to speak with someone who understands the application, aka the builder.

When it comes to putting some of these applications in the hands of customers, I also still personally review the select few Labs applications that fall into this class. Some of the problems I’ve found during those reviews have also become new retrofits for the entire fleet.

None of this means we automatically trust every conclusion an agent writes down. We’re planning more automated reviews and alerting. But ultimately those assessments still need to be evaluated by people who understand the risks.

Applying the Same Model to MCP Servers

We’ve also extended the retrofit model to our enterprise MCP reference architecture.

Different teams are building MCP servers using different programming languages and deployment approaches. Some are standalone services, while others are part of Labs applications. I don’t want to dictate every detail about how teams implement things. But I do want a consistent way to track what each server does, who owns it, and which requirements it’s been assessed against.

We made the Labs app template and MCP reference architecture agree on the structure of .retrofit-state.json, while keeping their respective requirements separate. Labs owns the template and retrofits sections, its compliance checks own compliance, and the MCP reference owns mcp.

Getting those two architectures to agree on a structure took some work on both sides. They still maintain their own releases and requirements, but the Overview app now knows how to interpret each system’s records without treating them as if they were the same thing.

But those changes exposed an interesting problem with the inventory. We had live MCP servers that hadn’t gone through the conformance process yet. This is an agent skill we use to ensure that the MCP servers are ready to be used and follow our required conventions. They had no .retrofit-state.json, so they weren’t appearing in our MCP inventory.

The Overview app’s GitHub collector now looks through the repositories for dependencies in package.json (or whatever is relevant for that language) that suggest something might be an MCP server. It’s not definitive, but it gives us a way to flag potential servers that aren’t reporting their maintenance state yet. I can then contact the owner and ask them to apply the relevant retrofits. Once they’ve done that, the next collection has a state file to work with and can properly report on itself.

There’s also a second path. Anyone building an MCP server that needs to connect to our Context Forge router, which is all of them, has to run the conformance skill first. Part of that process checks that the required retrofits have been completed, which also gets the state file in place and the application up to date.

We also introduced a deferred status for assessments that can’t yet be completed; which can be true for any number of reasons. That’s separate from discovering the server’s existence in the first place. Not assessed is different from failed, and we shouldn’t mark a server as behind on requirements that either don’t exist yet or it’s not ready to fulfill.

The Overview app also goes a step further and collects the tools and intended audience reported by each MCP server. We added that capability during one of the retrofits when we were working through why some of our tools were confusing agents during the tool selection process. That extra step gave us a central inventory when we’re reviewing tool coverage, improving declarations, and running our golden-prompt evaluations.

The first MCP retrofit we attempted demonstrated why we’ve been emphasizing behavioral requirements over specific implementation steps. The Python reference implementation handled MCP endpoint routing and authorization one way, while CGS’s TypeScript/Express server already satisfied the same routing and authorization requirements differently.

Ultimately, the agent verified the behavior and recorded the retrofit as applied without replacing the existing code.

That’s the exact behavior we wanted: different applications and reference architectures meeting their applicable requirements in different ways, while still reporting maintenance status in a consistent format across the fleet.

The Maintenance System Needs Maintenance Too

There’s another benefit to building the Overview app on the same Labs foundation we’re asking everyone else to use: we have to maintain it ourselves.

The Overview app receives the same retrofits, runs the same compliance and risk checks, and records its maintenance status just like every other Labs application in the fleet. We’re dogfooding the foundation we build on top of.

That is an important step because the people building the foundation also have to live with the requirements put into it. If the template makes assumptions that don’t hold for an actual application, we’re more likely to encounter those problems ourselves.

During one template update, we realized that the Overview’s validator couldn’t understand a newer JSON Schema feature. The agent caught that before the next production refresh and corrected it. If we hadn’t caught it, the application responsible for checking maintenance records across the fleet would have stopped correctly validating records using the newer schema.

We also found a problem with a “quickly stop everything related to this app in the case of an incident” retrofit. It handled shutting down and restarting the main services, but in a more complex application it missed restoring some of the scheduled processes. That would have left an application online without everything it normally needed running alongside it. A subsequent agent review with a human asking some specific questions caught that gap as well.

We can validate records, compare deployments, and add tests, but we still can’t prove that an agent understood every requirement or caught every problem. And I don’t want to solve that by requiring exhaustive evidence for every retrofit, sending every deployment through Compliance, or routing every change back through Engineering. We’d be recreating the bottleneck and lack of autonomy that Labs was meant to avoid.

Instead, we correct requirements as we learn, add tests where they help, and use the fleet information to identify what deserves more investigation.

Some problems belong to individual applications. Others reveal assumptions in the shared foundation that need correcting. Deciding which is which still requires experienced engineering judgment.

And while the fleet tracking application makes all this information easier to see, there’s still a decision the system can’t make for us: which applications actually need attention from their owners.

The Fleet Doesn’t Maintain Itself

By this point, we have enough information in the Overview app to know which applications haven’t been updated recently, whether people are using them, and which security or compliance work remains outstanding.

That’s where my team and I come in. We have to decide which applications need attention and follow up with their owners.

We’ve contacted people and said things like, “Hey, I noticed you haven’t deployed your app in a few weeks and there haven’t been any logins. Are you still using it?”

I’ve contacted others to tell them that they’re missing a security retrofit and need to bring their application up to date given the volume of usage.

When I reach out about a missing retrofit, I usually try to include a prompt the builder can hand straight to their coding agent.

That’s been more helpful than I originally expected. Especially with people doing this kind of maintenance for the first time. Instead of telling them their application is behind on retrofits and leaving them to interpret the requirements, I’m giving them a way to quickly get started.

Generally, I ask the builders to check for retrofits at least once a week. They don’t always do it, which is what I expect from maintenance tasks. We also don’t necessarily publish new retrofits every week either. But the ones who leave it for a month or more, are usually the ones who end up spending multiple days playing catch up.

There’s also a practical limit on how much work their agents can do. People are using AI subscriptions with usage limits and timed resets, and they’re already using those subscriptions for their regular jobs. A long retrofit can eat into the capacity they were planning to use for something else. Giving them both Claude and ChatGPT helped, but it doesn’t make that constraint disappear.

There’s another interesting side effect to this. Some of the people who used to ask Product and Engineering why their feature wasn’t getting done now have to make some of those same decisions themselves. Do they spend their limited time, headspace, and agent usage maintaining what they’ve already built, or working on the next feature? I hope it gives them a better sense of why Engineering can’t fit in everything we’re asked to do.

Of course, I don’t push every retrofit equally hard. I’m much more concerned about CVEs and important permission, privacy, or compliance issues. I also push a little harder for customer-facing applications. But either way, the builders still have to make time, answer their agents’ questions, and get the changes deployed.

When someone gets stuck or their agent asks something they don’t understand, they come to our Slack support channel. Either I or another engineer helps them figure out what to do, and they continue working with their agent.

We’ve tackled some fairly complicated retrofits this way, and so far, nobody has been unable to complete one. The builders don’t need to become engineers, but they do need to take responsibility for getting the work done.

The obligation belongs to the builder. The technical expertise doesn’t have to.

The Overview app doesn’t send automated warnings to everyone, block deployments, or shut down applications (yet). It does give me and my team enough information to follow up when it’s necessary. Sometimes that involves reaching out to the builder’s manager. Sometimes it’s escalated to me and I have to be a little more stern about maintenance being a requirement, not a suggestion.

I have also told some people that if they didn’t bring their application up to date, I’d have to shut it down. And fortunately, sometimes I get told that nobody is using it anymore and I can kill it. This is also one of the desired outcomes.

The objective isn’t to keep every application alive forever. It’s to ensure that the applications we choose to operate are actually being properly maintained.

Maintaining the Freedom to Build

People who build Labs applications don’t need to know how to fix everything. Experienced engineers don’t always know how to fix everything either. But the builders do need to take responsibility for getting problems resolved, whether that means working with their agent, asking Engineering for help, or retiring something that’s no longer worth running.

Engineers still build Labs applications too, myself included. We go through the same retrofits and compliance checks as everyone else. What’s changed is that Engineering also has to decide which behaviors need to be consistent across applications we don’t maintain, write those requirements so agents can apply them, and correct them when we discover our assumptions were wrong.

I’m comfortable with applications having different architectures and solving problems in different ways. That was deliberate. What matters to me is knowing who owns each application, what maintenance needs attention, and whether that work is actually getting done.

The people using Labs have their own problems to solve. I want them spending as much of their time and headspace as possible building applications that solve those problems, not becoming experts in authentication, authorization, deployments, or dependency management. That’s why we’re putting so much engineering effort into the foundation, retrofits, and fleet tracking. Without that work, we’d eventually end up putting Engineering back into every release.

The system doesn’t take away their responsibility for maintaining what they build. It gives them a way to meet that responsibility without turning every builder into an engineering team.

Latest Manager Trainings Videos


All Training Videos

Managing In All Directions

In this episode, Eric explores how managers can work effectively in every direction—upward with leaders, across peers, and diagonally across functions—by treating communication and relationship-building as intentional leadership practices. He explains how to clarify the purpose of each relationship, build trust through consistent and transparent actions, and lead through influence rather than relying on formal authority. Eric shares practical tactics for keeping stakeholders aligned, tailoring updates to the audience, handling communication barriers and difficult relationships, and using informal channels responsibly. The episode closes with a simple “subscription levels” system for maintaining regular check-ins with direct reports, peers, stakeholders, and collaborators, helping leaders make relationship maintenance a deliberate habit.

Motivation

In this episode, Eric explores motivation as a core leadership skill—what it is, why it matters, and how it shows up (or disappears) in day-to-day work. He breaks motivation down into intrinsic and extrinsic drivers, then walks through common causes and observable signs of demotivation, emphasizing the importance of knowing your people well enough to spot meaningful changes in behavior. Eric also draws a clear distinction between demotivation and clinical depression, outlining what a manager should watch for and how to respond with empathy and appropriate support rather than trying to “fix” it. The episode closes with practical strategies for re-engaging individuals and teams—through purpose, clarity, feedback, recognition, autonomy, and healthier working environments—plus question prompts leaders can use in 1:1s to uncover what genuinely motivates each person.

Latest Beyond the Belt Episodes


All Episodes

Don’t Buy My Book, It’s Old

Straight to Your Inbox

Videos

Manager Training

Beyond the Belt

Writing Archives

contact