bitlog.com

Jake Voytko's personal log of bits

Blog posts

I built my own Google Docs for my blog

I recently noticed that my blog has fewer than 30 posts, but I've been maintaining it for almost a decade. I'm sure you can guess my next thought:

"This is because I don't have my own hyper-personalized version of Google Docs to use as an editor."[0]

So I set out to fix that. And I have a major advantage: from 2010-2015, I was a Google employee that worked on Google Docs. It's been a while, but I still remember a few things about what makes the ol' girl tick.

It took an 8-hour Q&A with Codex. It took four ChatGPT resets on my $200/mo plan to get it done with Astra on xhigh. But it's done. This is my first post produced entirely by my own custom Google Docs "clone". It powers my blog's editor. It has its own custom operational transform stack. It's slow as shit. I love it to death.

A screenshot of an earlier draft of this post, being edited within the new blog editor

Here is my own personal Google Docs, in all of its glory and horror, from an earlier draft of this post.

To be clear, you're not reading directly from the Google Docs editor's software. The public-facing site and the editor are decoupled. The public site is just static content. The editor accumulates changes and snapshots while I work, tracking everything in a Postgres database. A job is started each time I publish, republish, unpublish, or delete. That job regenerates the static site. That is what you are reading now.

The Q&A from hell

I planned this with GPT 5.6 Sol on xhigh. Astra came out right when I was finishing up the Q&A. Otherwise I would have used it for the Q&A too.

I started the same way everyone starts a big task: I wrote out a huge prompt and hit send.

But I also had one more request: Codex was not allowed to leave any inconsistency or ambiguity in the spec. It had to keep asking me questions until the spec was fully resolved, starting with the most important unanswered question.

It mulled over my initial prompt and cached about 380 questions in a Markdown file. Many of my answers induced Codex to ask several follow-up questions. I didn't do it in a single block. This is just a side project and I have a family and a job. So I did the interview over 4 or 5 days.

Some of these questions were grueling and required long explanations of major pieces of architecture that I wanted. Some required a lot of thought. Some of them were crucial for correctness. Some of them were implementation details that I happen to know about implementing word processors. I settled on a flow that let me gallop through questions pretty quickly.

  • I used voice input and told it "I'm using voice input which might transcribe incorrectly; work harder to try to guess what I originally said."

  • I asked it to ask between 1 and 3 questions at the same time, but it could only ask multiple questions if they were related.

  • And if it could think of an obvious default, it would explain that option and then ask, "agree?" So if all three were okay I could say "agree to all." Usually I needed a hybrid approach like "one: agree. two: do this instead. three: yes but with a small modification."

By the time I got to the last 50 or so questions, I was basically just saying "Agree. Agree. Agree." to each triplet because all of the decisions were inconsequential.

What I asked for

I wanted the static site to have feature parity with my existing blog. For historical context, my blog was WordPress for a long time. In February, I converted it to a Golang server that I host on DigitalOcean. The existing blog needed only very basic features for full parity: headings, text styles, links, lists, images, inline code, block code, quotes, footnotes, embedded YouTube content. Nothing too crazy.

I also wanted my favorite feature from Google Docs: magic redo.

"What is magic redo?"

I'm glad you asked! Go to doc.new and copy/paste this post into the editor. Select some text and bold it. Now scroll down and select some other text (without doing anything else in the meantime!) and type the Redo keyboard shortcut (of undo/redo fame). The text in the new location is now bold. Select some more text and type redo. That text is also now bold. And that is magic redo! It executes the last action as if it was done with the current selection model. It's useful when you want to do the same thing a lot, like adding a few columns to a table. Magic redo has some implications on the backend design, because it means that everything the editor does needs to act on f(selection).

Where I compromised

The biggest compromise was in the view layer. When I worked on Google Docs, it did a lot of clever things to try to make the editor as efficient as possible. They also measured every change to guard against regressions. After I left, they later switched rendering to canvas, which I understand was a huge performance win.

I did not do any of this. I just have it update the DOM node of the paragraph it's editing. I'm not doing any clever tricks. I don't have performance guardrails yet.

I also probably could have just pointed an LLM at its obfuscated source and said, "reverse engineer every feature this editor has" and wound up with a fairly complete feature list. But then it would have generated an editor that did not suit the needs of my blog.

I don't have any accessibility or right-to-left text support. If I ever go blind or move to Jordan I'll yell at Claudex until it makes it happen.

I don't have offline support. No suggested changes.

When I joined Google, Docs was missing what I now consider an essential OT command: "move text". A lot of operations in Docs were implemented with "delete the text here and re-add it here," which could transform poorly against incoming commands and blow them away (since they'd get applied to the range that got deleted). I am not using it at the moment, but I have plans for an "outline" feature that allows me to reorder lists very quickly, and I plan to implement that with move.

"But Jake, you're the only editor. Why do you need to care about how stuff OTs?" Buddy, I implemented my own Google Docs instead of editing a Markdown file. We're beyond logic here.

I've also been experimenting with a "proof of human" feature, which is effectively a read-only Etherpad that I'd link from my posts. Since I have the full command history, every essay could have a complete playback to show that I wrote it. I don't feel completely comfortable launching it, because my writing is VERY exploratory. I write a lot of things that I don't mean which I later delete. I don't want people to be able to screenshot it out of context.

The implementation

I implemented the project using GPT-6 Astra on xhigh.

I have a $200/mo ChatGPT account for my side project coding, and needed four resets to implement it. But I had one advantage: the week I implemented it was an incredible week for resets. So huge shoutout to Tibo. I started right after my weekly quota refresh and already had three banked resets available for use. And there was one more unbanked reset. The initial implementation used 4 full weekly quotas and almost finished the fifth before it finally finished.

I made some basic mistakes in prompting that came back to bite me. A lot of time was wasted on testing and verification. I don't remember exactly how I prompted the model since I did the Q&A a month ago. But I'm sure that I said that it needed to verify everything, that it needed to test all cases, and stuff like that. I think I also suggested that it try fuzz testing on the editor to test out the OT stack.

I made the mistake of acting like I was describing a design to a senior engineer. But no, Astra is not human and does not have a shared human understanding of what I mean when I speak inexactly. So it handled these in the most robotic way possible. "Fuzz testing" turned into a series of 10,000+ case integration tests that needed over five hours to run each. Each day I'd tell it to delete a massive test and replace it with the minimum number of test cases to achieve equivalent case coverage, and each night it would birth another massive 10,000 case integration test that took 5 hours to run. I'm sure this burned a massive number of tokens. Eventually I banned it from generating tests with more than a few hundred cases. It still mentions this fact in completely unrelated situations. This always makes me wonder, "..... but were you actually thinking you might do that?"

Telling it to "verify" things was also a mistake. The app was unusably slow at first. It added runtime checks that choked the app to death.

For background, when you have a full operational transform stack, many features need to be transformed against incoming changes. If you have an undo and redo stack, everything in the undo and redo stack needs to be modified so that they can still apply cleanly. If a command bolds the letter 'b' and a letter is added before it, the indexes referenced in the command need to shift forward so that 'b' is still bolded if it gets applied.

In this case, I requested that grapheme clusters always had to be valid when edits were applied. To me, this means "after application of the edit but before sending the commands to the server, check the surrounding model context until a safe boundary to ensure that the operation did not result in an invalid character sequence." But the generated app would "verify" the undo and redo stack on every single character press by making a copy of the document, applying the edit, and then running a full-document grapheme cluster check. So typing a few characters into the editor was enough for it to stall and stutter.

I've done a few rounds of performance work at various granularities, like "walk me through everything that happens when I press a key" and "optimize typing 1000 characters", and I think I've eradicated most of the unnecessary work. Typing is relatively smooth, which is nice given that I did not optimize the design for performance.

The cutover

Since the project involved migrating from the static site to the hybrid site (dynamic backend, static frontend), the cutover became an Epoch event of religious significance to Astra. Astra was obsessed with the cutover. My Makefile and Ansible implementations became littered with a ton of extra commands and checks. Not only support for the old version and the new version, but tons of verification checks to ensure complete site sanity at every intermediate step of the cutover. And Astra needed Verification, which it got by hashing lots of different things and calling it Proof. So the entire thing was Verified with Proof, which was very comforting.

Since my site was approximately 30 Markdown files that were backed up on GitHub, the stakes could not be lower. I decided to just try Astra's cutover. If it failed I was just going to recreate the droplet and have it start fresh. But it did a fine job. Migration successful. The site works great.

But then the cutover continued to exist. Every change needed to be tested for compatibility with the pre-cutover and post-cutover and intermediate steps. It still asked me about how I'd like to handle the legacy Markdown site when describing new features.

So obviously I eradicated the cutover. I tasked Sol with the cleanup. It deleted the old Markdown static site as well as the Go code for serving it. It also eliminated all of the LLMslop guardrails: the cutover mentions, verification checks, all Proof and Evidence of Verification. All of it gone. And as a bonus, I haven't seen it mention the cutover in a while. Codex has truly forgotten about the cutover.

How do I edit it now that it's live?

First, I have a bunch of skills that manage each stage of the process. File a ticket (and give me a Q&A over any ambiguities), start a demo server, clean up and submit, etc.

I have a vibecoded workspace switcher, lovingly called ngmi, which manages all of my agents and how they interact with the project lifecycle. When it picks up a ticket, it automatically creates an agent, blocks on unmerged dependencies, has a merge queue to serialize the submission of finished tickets, etc. It runs on a separate DigitalOcean droplet, and can handle 3 or 4 agents running at once without struggling too much. This would be too limiting for my day job, but is fine for a side project. I run it on a DigitalOcean droplet since I work from both my Windows gaming PC and my MacBook M1.

ngmi can spawn its own non-task agents and has its own MCP, so I can also command it to do things from within the agents managed within ngmi.

Since I love taking my side projects too far, I have my own ticket tracking program with its own MCP at bitlogwork.com. Whenever I want to change something, I just file a ticket and ngmi automatically picks it up. It detects what part of the stack I'm working on (ticket tracking, analytics, the blog) and starts a demo server when it completes the task. It can generate and serve explainer HTML pages to highlight different aspects of the change that I'm curious about. I can switch to the agent and ask for changes.

I've been doing performance work and adding features over the past month, as I have energy to do side project work. It's finally in a position where it's not so painful to write a post, so here it is! The inaugural blog post coming from my own personal Google Docs clone.

Lesson: reality has an infinite amount of detail

One-shotting is for losers. Find ways to uncover all of the hidden spec that matters without getting bogged down in tangents that don't matter.

If you tell an LLM "Make a modal dialog with options OK/Cancel that asks if you're sure you want to delete this before executing the delete action," you may think that you very exactly specified what it does. You can probably already picture what it looks like. But the LLM will guess an insane amount of spec behind the scenes. Where is it positioned? Do the corners have border-radius? Does it have a border? Does it have a drop shadow? What is the padding? Does it need a separate title or does the raw question text suffice? What order are the buttons? What color are the buttons? What is the exact copy? It just goes on and on. There is no limit to how detailed the spec can be.

But when you're building something and you care about the output, there are an infinite number of possible details that matter. There are a lot of types of implementation where you can just shut off your brain and allow the LLM to pick something for you. But if you really want to produce a great product, you still need to have vision and taste and take control of the output. Take it from me, the guy who does not have vision or taste but really wanted to take control of the output.

Lesson: software isn't a moat

Any of your old employees can recreate large swaths of your product in a few days of motivated work (or a month or two if they're kinda just winging it on the side). It's all about ecosystem, partnerships, and all of those other intangibles that actually require hitting the pavement, working the phones, hunting through your network, and doing all of the other grueling work of building up your place in the world brick by brick.

Footnotes

[0]

You might say, "But Jake, you do have your own copy of Google Docs. Google Docs is free. You could simply use it and copy-paste the content." But I turn that back around to you, and ask, "Is it really mine? Do I have the freedom to make the real Google Docs this underfeatured and unperformant?"

How to cure AI psychosis

To any afflicted CEO in my readership: I want to cure your AI psychosis.

⚠️ First, a disclaimer ⚠️ This post does not discuss legitimate medical issues; I'm not a doctor and none of this is medical advice. Talk to a real medical doctor if your mental health is in question.

As a quick aside, I'm experimenting with recording my blog posts for YouTube. So if you'd rather see that, here it is:

View on YouTube

Instead, let's talk about the fun kind of AI psychosis. The kind where people see LLMs as a genie-in-the-bottle, able to grant any wish.

Aaron Levie, the CEO of Box, recently tweeted that CEOs are uniquely prone to this kind of AI psychosis. He says that CEOs do not perform the "last mile" of work. They never see the effort required to coax LLMs to perform useful work. They just see the happy path. "Here ya go boss! I made this prototype with an LLM prompt, just like you asked."

If you're a CEO and you're working daily with LLMs and you have a good sense of their limitations: I hope you're having a good day! You can close the tab. But for the rest of you, buckle up, because we need to talk.

Anyone riding the NYC subway these days can observe this themselves. The ads plastering the trains are preposterous. They make promises like, "make an entire presentation with a single prompt." Obviously this is disingenuous. But there must be some subway riders influenced by this, at least enough to visit the website. And those people must somehow be responsible for inking SaaS contracts. And you might wonder, "Why can't it? Why can't it make a whole presentation with a single prompt?" Let's save that question for the end of this post. We can decide together whether it's just "a single prompt" or there is an incredible amount of hidden work.

The answer has to do with two questions.

  1. How easily can the LLM supplement its context?

  1. How quickly can the LLM verify its output?

But we'll talk about that later.

Let's go back to the beginning.

At release, ChatGPT could do things that no computer had ever done before. You could type any request you wanted, and it could do it. Need to write an email to a business? ChatGPT can do it. A poem about your friend in the style of Walker, Texas Ranger? Giddyup partner.

So why didn't the entire world lose their jobs at this point? Because LLMs were absolutely useless for real work. They confidently lied, over and over. They fabricated citations. They became legitimately unhinged if a conversation extended more than a few rounds of back-and-forth. They couldn't follow simple step-by-step instructions without getting lost. It generated code that wouldn't even compile, or used imaginary APIs on imaginary libraries. They couldn't count the letters in a word. They failed at trivial reasoning exercises. Sometimes their rudimentary safety checks prevented you from doing mundane tasks, but other times the LLM urged people to kill themselves. And that's an incomplete list of their shortcomings. They had some utility, but the outputs were largely a novelty.

But LLMs have improved so quickly over the past few years. Each of the above problems have better mitigations, and LLMs have gained new capabilities. They got chain-of-thought reasoning strengthened with reinforcement learning. They can call tools. There are now protocols that allow LLMs to interact with your data in any application. Every few months they get model upgrades that supercharge its abilities.

And they are world beaters. They are solving novel math problems that humans have been trying to solve for decades. They can perform complicated research tasks and produce convincing summaries of the results. They beat software engineers in coding competitions. And by all accounts, they can write college-level essays.

So I don't blame anyone for drawing the trend line from these capabilities and saying, "if an LLM can perform high-level mathematics, or implement a complicated step-by-step pull request, then surely it can do any job."

But, I have to level with you: in all that time, with all of those improvements, they have gone from "useless for real work" to "somewhat useful for real work. They cannot produce a final product unsupervised if the quality matters at all.

They will save some time if you are skilled at using LLMs. But LLM use is a skill, and it's a skill with a low skill floor and a high skill ceiling. An unskilled person will take longer to produce a professional-level output than if they had done it themselves. You can also be a LLM veteran and do something that worked before, and it just produces a worse result this time. Unlucky.

Panel 1: How CEOs think LLMs work - almost all time is saved except the part the LLM runs. Panel 2: How LLMs actually work. You have to assemble context, run the LLM, and verify its output, and you do save a little bit of time but it's not almost the whole task.

This diagram isn't meant to be pixel perfect. But I stand by the idea; for each LLM prompt, there is some intrinsic amount of context building and some intrinsic amount of verification. And if the output is too far from the desired result, then you're likely losing time.

If you want to break outside of the tech sphere, go talk to a competent lawyer about how useful they find LLMs. They're great, they save so much time! You can summarize really complex documents and generate briefs and do all of these other things! But you need to check every single thing that it outputs. You don't want to build your argument on a hallucinated case or an incorrect summary. You don't want to email an opposing council LLM slop.

What the hell? Why can LLMs solve math problems, but you can't rely on them to build a legal case without obsessively double-checking everything they do? Why can an LLM beat software engineers at a coding competition, but it can't build a simple presentation, a task that many people might delegate to an intern?

Let's revisit the two questions I wrote above. They explain the whole story.

  1. How easily can the LLM supplement its context?

  1. How quickly can the LLM verify its output?

And let's look at this through the lens of "Build a presentation with a single prompt." Let's really consider everything that you would need to build a successful presentation.

How easily can the LLM supplement its context?

Ok, what context is required to build a presentation in a business setting? Let's say that you're building a pitch deck for a new product offering, and you want potential customers to buy. Your customers might want to know…

  1. What problem they are currently experiencing.

  1. What value the new product offering delivers.

  1. A high-level overview of how the product works.

  1. Data that reinforces your story.

  1. Industry research explaining how similar companies solve this problem.

  1. Social proof from someone that tried the product.

  1. The story about how they successfully migrate to the project.

  1. Any offers, trials, concierge services, etc that make it easier to perform a pilot project for the service.

To build a successful presentation, you might need to additionally consider…

  1. How long is the scheduled time?

  1. Is there anything that would piss off your investors?

  1. Is there anything that would piss off key stakeholders?

  1. Do you know something about the audience that should be incorporated into the presentation, like their tech stack, stated preferences, etc?

  1. Do you have the ability to request that new art assets be made, or are you stuck working from some pre-canned set?

You also might not want them to know a few things.

  1. That you'd accept a steeper discount if they said they were trying a competitor.

  1. There's a feature that you're pitching that is still technically a roadmap feature and not code complete.

  1. Everything is a giant shitshow and we're just keeping it together for public appearances.

And the list goes on. So can the LLM easily supplement its context? Obviously not! The LLM might need to snake its tendrils into every single data source that your company has, including the brains of people who work there. And at a large organization, you will have so much data that it cannot fit into the context window of an LLM. So now the LLM needs to solve the open research question of "how do I find the exact right context that I need to answer all of these questions?"

How quickly can the LLM verify its output?

How do you need to verify a presentation that you didn't write?

First, you need to flip through the presentation. Look for any weird outputs. Are there overlapping regions? Things that are a little too close? When you put two images next to each other, do they visually clash in some way? Some of this could be handled by the LLM but some of it is just the human experience.

Then you need to consider the structure. Is this a compelling pitch? Is this ordered correctly? Should any slides be omitted? Is anything missing?

Then you need to nitpick the content. Are all of the sentences correct? Is it using good word choices throughout the presentation? Did it misunderstand any of the source material and needs to be reworked for accuracy?

And finally, you'll want to actually give the presentation slide-by-slide. Does it all flow naturally when spoken out loud? Are there any unnatural moments? Does anything need to be restructured or reworked?

So the LLM cannot really verify its output at all, so it can't iterate and recurse on the problem. But there are aspects that the LLM might be able to assist with, like finding source documents, answering questions about data that it can access, and generating the slides. So it's very likely that you could build the presentation faster, but there's also a tremendous amount of hidden work involved: uncovering the full context of your problem, what you need to present, and being a strong advocate for the audience in mapping the LLM output into a convincing presentation.

The cure for LLM psychosis

The cure is simply the ability to find the hidden work.

If the LLM can add to its own context via simple searches, or even perform its own logical thinking like building its own lemmas and theorems when investigating a math problem, then the problem is in the LLM's sweet spot. It doesn't need people at all, it can just churn by itself! However, if it needs to sort through more information than it can access within its own context, then the LLM is going to need extra assistance and there is hidden work.

If the LLM is working on a programming problem and can simply write its own unit tests and run the program on sample inputs to verify its own outputs, then you are in the LLM's sweet spot. However, once verification starts to have complex causes like "how do human beings feel when they view the output?" and "Does this otherwise-correct change introduce unwanted emergent behavior when run in the production system?", then there starts to be a lot of extra hidden work for getting the LLM to exist within an ecosystem.

Why are we getting worse at software engineering?

Software quality is obviously getting worse. I'm not talking about LLM slop features nobody asked for. I'm not talking about services collapsing under unprecedented LLM-powered demand. Companies are obviously shipping user-visible bugs at an accelerating rate. And consequently, the software we are using is getting worse and worse.

As a quick aside, I'm experimenting with recording my blog posts for YouTube. So if you'd rather see that, here it is:

View on YouTube

Back to our regulary scheduled blogging!

You can thank GitHub for this post. I recently commented on a deleted line in GitHub and hit submit. An error dialogue appeared saying that some block of client-side code couldn't find the line ID. GitHub has had almost two decades to perfect "comment on code." Yet it regressed. And hilariously, I just triggered a Google Docs copy/paste bug while typing this[0]. And heaven help any heavy Claude Desktop users. Everything's getting worse around us.

I blame three factors for this.

  1. As you write code faster, the acceptable error rate drops.

  1. Implementation time is becoming decoupled from competence.

  1. The value of implementation becomes so high that slack time will trend to zero

I have faith that we can overcome these as a discipline, but we can't do it by applying our old bag of tricks. Our old bag of tricks got us our old error rate. We need to evolve.

As you write code faster, the acceptable error rate drops[1]

I think this is obvious but it's worth stating. If you're shipping code faster, and you have a certain regression rate per change, then obviously you are shipping more regressions. It's just math.

So there are 2 pieces of this: are we shipping faster, and how do our users experience that?

Why does this matter? Let's say that you flip a switch and can immediately double your code production. Nothing else changes. Just twice as much code as before. There are a few consequences:

First, you ship 2x more user-visible regressions over any time period. This makes sense, right? Each change has about the same chance to introduce a regression as it did before. So on average, double the code means double the regressions.

The problem with user-visible regressions is that users encounter them. After you flip this "double code production" switch, the number of active issues in your product will trend towards 2x the baseline. If users are lucky, you notice the issues they encounter in some way, and you can go and fix the issue yourself with your magical new implementation rate. But often, users have to tell you that you fucked up. "This menu is broken in this configuration and I can't even click this option anymore." And that's slow. You need to aggregate the reports, try to reproduce, prioritize the fix, etc. And your best users, your power users? They're the poor saps that trigger all of your new bugs, over and over again.

I'm sure somebody wants to counterargue "LLMs can detect all of these kinds of issues and automatically fix them," and I don't want to hear it. Software is obviously degrading all around us! Whatever LLMs are currently doing isn't enough to resolve it. And if you have some magic technique that the rest of us aren't applying, please scream it from the rooftops. Even better, try to get it integrated into the official Claude Code harness the official way: by randomly Tweeting at members of the team on the off chance one of them notices.

So when you hear someone say "we're shipping 100x faster" and they can't explain how they ship 1% the number of errors that they did before, run away from their software before it explodes.

Implementation time is becoming decoupled from competence

In the old days[2], If you didn't know how to do something, then it took you a long time. But if you put in the reps, you'd develop expertise and get faster and faster, and eventually it became a natural part of your workflow.

This slowness was a blessing. The slowness was learning. You allowed the problem to impress itself on your brain. You weren't just reading the theory. You were actually developing the muscle memory for execution. You were learning every wrinkle, every pitfall, every exception, and you learned how to handle each of them.

Man, that went out the window, didn't it? Now you can be as clueless as you want to be. When I set up my blog recently, I spent weeks hammering the codebase into the right shape. Even though I generated it with Claude Code, I still have a good idea of how the code is organized and what each piece does. I chose to understand the project.

And when it came time to actually deploy my blog to Digital Ocean, I didn't want to understand. I didn't want to remember how to use Ansible and look up guides for hardening VPS instances. I didn't want to spend days tracking down the cause of obscure error messages. I just have one hour per day of side project time. I don't want to waste it. I told Claude everything I wanted: the Makefile command names, Ansible deployments, hardening, etc. It finished within 20 minutes. And sure, I checked over everything to make sure it wasn't leaking API tokens or anything. But I just read the generated code. I never truly allowed it to flow through my brain.

Did it do a good job? Not any worse than I would have done setting up my first VPS with Ansible in 6 years.

Did I learn anything? Absolutely not! I'm running Caddy in production now, and I have no idea how it's different from Nginx beyond automatically setting up HTTPS.

And that is my central point here. Implementation time is becoming decoupled from competence. Whether I knew how to set up a droplet or not, it would have taken about 20 minutes either way. Sure, an expert might have added bells and whistles, set up some extra monitoring, got Tailscale going, and whatever else experts do with their VPSes. But I did 3 days of 1-hour-a-day side project time in like 20 minutes. And that's bad! I shouldn't be the type of person that can set up a VPS in 20 minutes. It should actually take me a long time because I don't know what I'm doing. In some ways, it's actually dangerous that I can do this.

And that's what we're seeing throughout the industry. People can accelerate tasks outside of their own expertise. So review and expertise are becoming an increasingly important part of the job. Just because someone sent you a change no longer means that they have the competence level required to get it working. I sure didn't when I deployed my blog.

I'm not impressed anymore when somebody says that they pointed Claude at a ticket with its MCP or CLI, implemented the code, wrote tests, and pushed the result to a GitHub pull request with the Github CLI[3]. That's where the work starts now: actually evaluating the prompts and output for correctness, for scalability, for maintainability. For removing all of the little quirks that LLMs introduce. I'm impressed when engineers say, "I found this problem that I wouldn't have otherwise" or "it tuned this better than I could tune it myself" or "I had this insight I never would have had by myself."

Does this LLM speed boost lead to software correctness? If anything, you can now ship code faster if you're clueless because you're unconstrained by reality. You're not pouring over the code looking for API keys it's leaking, looking for obvious scaling bottlenecks, looking for unnecessary bundles you're sending the client. You've merged, and you're already looking for the next ticket. And the number of active regressions in your product just ticked up a bit.

The value of implementation becomes so high that slack time will trend to zero

Slack[4] is an important concept in system resilience. It's the amount of time that an entity is unallocated. It is your tolerance to deviations from the norm. In manufacturing, slack might be the amount of time that a factory is not utilized. At zero slack (i.e. the factory has to run every hour to meet its demand), then even a single hour of downtime needs to be made up. That's when the system comes under pressure. This is when accidents and errors creep in.

For software engineers, slack looks like unscheduled time. From the top down, unscheduled time sounds bad. This is time where engineers don't have guaranteed outcomes. But in reality, a little bit of slack can be some of the most important time that they spend. This is the time that they delete dead code, that they build that observability dashboard that everyone has been putting off, that they say, "This weird thing has been bothering me for a few weeks, I need to look into it. Oh fuck!". It's when you have a chance to say to your teammates, "Why does this part of the codebase feel wrong? What can we do about it?" and whiteboard for a week and come up with the architecture that powers you for the next 5 years.

Obviously no leader says that they want to suffer Knight Capital's fate, or that they wished they didn't have that dashboard that noticed that launch regression, or that they wished the subtle data loss bug was still live in prod, etc. But how do you make it a repeatable business outcome when it comes as the result of unallocated time? You can't.

You might say, "All of these things should obviously be part of any project." But that just misses the point of how software is made. I wrote more about it here, but the tl;dr is that in most methodologies, you set an objective like "Make widget Foo", you set the launch date along with the initial scoping of the project, and then repeatedly negotiate the scope until you have Foo on the launch date (or maybe pushed back a week or two). When your schedule starts to slip, do you know what gets descoped first? Your nice-to-have dashboards, the code cleanup tickets from the last project, etc. All the slack work goes right out the window.

I expect the industry to operate with less and less slack in the future. If we can really accelerate feature implementation, then there's some rate of implementation where it doesn't make sense to give your engineers unallocated time anymore. It just becomes too valuable to perform feature work. "So you're telling me that it used to take my team 3 months of work to figure out whether we might get +/- 2% from an engineering change? And now we can turn it around in 6 weeks?" Your quarter just got 6 weeks back, and you can bet that you are not spending it on anything but implementing more product features. Every implementation hour just got twice as valuable. That downtime between projects you used to have? Now you're spending it writing the specs for the next project.

And it leads to worse software outcomes. Without slack you can't even handle minor bumps in the road. What happens if a project suddenly needs more headcount? There's nowhere to borrow it from; everybody's already allocated. Something needs to get bumped, but you already burned a bunch of engineering time on it. You're already starting to lose the gains. Who's going to go back and delete all of the branches of the old experiments? Nobody. Who's going to spend a week whiteboarding to determine the future architecture of your company? Nobody. Who's going to investigate that weird thing before it becomes a huge problem? Nobody.

Hope for the future

I don't want to just be Doom and Gloom about the future. We can do something about this. Are we going to reduce the error rate by 100x? Probably not. But increased implementation speed applies to everything. It means that we don't have to be stuck in the old paradigm where we pair every single implementation change with a unit test and say that the test means that we verified that our software works.

I can imagine a future where a LLM with the right skill could actually verify that every single line of code has a test that fails if it regresses.

I can imagine a future where it becomes so cheap to produce integration tests that we default to integration tests over unit tests. Anyone who's worked with me knows that I hate mocking frameworks and thinks they lead to worse engineering outcomes, so this could become the perfect axe for me to grind.

I can imagine a world where we get so good at writing integration tests (because we exercise the muscle so much more) that they don't flake all that much.

And it's a bit dangerous, right? It's dangerous for me to assign LLMs these magical capabilities. Just because they can be given a prompt and produce an output doesn't mean it's correct. That doesn't mean it would help.

But the good news is that software engineering is more verifiable than it's ever been. It's never been easier to just take a change and open up one worktree or one checkout and do one technique there and then run a second implementation in parallel and compare the two outputs. What is different? Do you like one more than the other? Did one have more errors on the other?

But I'm not trying to prescribe Exact Solutions. We can imagine a world beyond what software engineering was in 2023. It doesn't have to just be an implementation paired with a new set of unit tests until you retire. We can work together and share our results, share what's working.

Footnotes

[0]

They fixed it since I started writing this, but here was the repro: triple-click a single-line paragraph. Type Ctrl-C (Cmd-C on Mac). Triple click a headline. Type Ctrl-Shift-v (Cmd-Shift-v) to paste without formatting. The line is replaced with the paste content plus three newlines in a row. Even assuming that it needed to keep both the source and destination paragraph's newlines, where did the third one come from?

[1]

While I was writing this post, I read a different treatment of this subject here that views this through the lens of maintenance costs. If you feel like you'd be more swayed by a devex argument, check this out!

[2]

Before 2025.

[3]

I mean, I'm very impressed with the Claude Code and Codex teams that they made an agent where this is even possible. Holy cow. What a time to be alive.

[4]

I am absolutely not referring to the SaaS product here.