Hacker news

  • Top
  • New
  • Past
  • Ask
  • Show
  • Jobs

Ask HN: If AI tools are so good, why isn't your bug backlog empty?

103 points by ex-aws-dude 4 days ago | 117 comments | View on ycombinator

frostiness 4 days ago |

I don't know if I'm doing something wrong but bug fixing with AI has always been a crapshoot. You can absolutely tell an agent to fix an issue, and it will fix that issue, but the vast majority of the time that solution will be by adding code to cover the specific symptom in the bug report. It rarely considers or recommends solutions that have tradeoffs that may change behavior but will eliminate an entire class of similar bugs.

This is mostly a problem of agents not having the autonomy of managing an entire project and having the ability to make tradeoffs to increase simplicity or maintainability, but when given that autonomy no AI model I've used uses that privilege to write simpler, more maintainable code.

td521534f12 4 days ago |

All my solo project backlogs _are_ empty.

For my team projects, I still need my teammates to review code, so that bottlenecks how quickly we can burn down the backlog. If I didn't need code review I would have the backlog burned down in weeks, but I'd create a bunch of outages in the process (we do catch things in code review!)

Also, some of my solo projects are gamedev-related, and gamedev is also bottlenecked on actually playing your game to see if its fun :) but it is great to get out of a 2 hour playtest session with notes and have a new build in like 15 minutes with the feedback incorporated.

fishtoaster 4 days ago |

It pretty much is? Every easily-described bug has been fixed. The only things that remain are "this isn't wrong per se, it just violates customer expectations" and someone needs to figure out what the product should do in this case.

We burnt through our backlog of "obviously we should do this when we have time" stuff in the spring. It was pretty fun to kill pretty much all of my multi-year refactor and cleanup list.

Now we are pretty much bottlenecked on the business side.

yokuze 4 days ago |

> The bug backlog should be empty

I don't follow.

Maybe it's just me, but I've never worked at a company and on any substantial product that had an empty bug backlog. If a bug queue is empty, the product is either not used much (or not a priority), not tested well enough, or not monitored enough.

I don't know that a world where AI has mostly replaced programmers is a world with an empty bug queue.

While AI is reducing the _quantity_ of programmers needed in many cases, it is also an output multiplier for the remaining programmers, while often raising the bug-to-code ratio.

AI has led to more output, more projects, and more bugs.

GuB-42 3 days ago |

Because AI is better at finding bugs than at fixing them.

It is not good at writing bug-free code either.

AI and humans don't make the same bugs. So AI may find bugs that are easily overlooked by humans while at the same time being incredibly stupid on some other parts.

undefined 4 days ago |

undefined

mattbrewsbytes 3 days ago |

If your team and product had a large backlog before thats not going to change. AI tools amplify whatever it is you do so if you've been ignoring bugs and focusing most of the time on new features, AI isn't going to magically change what a team is working on.

On a technical level, take your average NullPointerException or the equivalent in whatever language you use. Ask AI to go fix it and it will sanely wrap or check for nulls in that piece of code. Ask a lead or senior engineer to fix it and they will (hopefully) start with "why is this data null? it shouldn't be" and then trace back through logs or data, try to recreate with a real scenario and not only fix that bug but find there's an unhandled scenario somewhere else in the code base. Someone could prompt the AI to expand further and do a root cause analysis but do you trust it to find things and fix them properly?

nzach 4 days ago |

AI only can improve velocity if it mostly works unsupervised. But most people don't trust their agents to run alone.

And to build that trust will take some time, people will need to learn how to use AI effectively.

I think the PS3 hardware could be a good analogy. It was really powerful, but nobody really knew how to use it. After a few years people got really good at it . And you can view this evolution by the improvement of graphical fidelity we got over the years.

fritzo 4 days ago |

Parkinson's Law:

"Work expands so as to fill the time available for its completion"

https://en.wikipedia.org/wiki/Parkinson%27s_Law

davesmylie 4 days ago |

I'm in devops, not programming, so slightly different.

I can say our backlog of absolutely essential jobs that we haven't been able to get around to for the last 5 years is absolutely being smashed out one by one. A lot of this is unattended server migrations running over night being validated the next day.

The backlog as a whole though is growing as we have AI continually probing the infrastructure and various code bases looking for new security concerns or ways to improve performance and save money. Directly as a result of this, our cloud costs dropped massively in the last 6 months with only very minimal impact on performance.

I absolutely see a time in the not too distant future where all our infrastructure tech debt has been cleaned up, the security situation is as good as we can get it, the backlog has completely emptied and some of my team will be surplus to requirements.

g051051 4 days ago |

Because just like without LLMs, focus is on delivering new products and features, not on paying down technical debt.

cmadar 3 days ago |

I think building new code up to a functional, code and security standard will get easier and easier with AI and we may reach a point where the list of bugs is shorter. However, based on what I have seen building my software factory (7 agents, each with their own roles) and iterating on it, an agent will only have a certain amount of context and scrutiny it can dedicate. So I found that segregation of resonsibilities between agents (which one does which checks) + automated checks and tests are really important. However, I don't know yet to what extent. It feels like my factory is getting better and better, but there are still things being sent back, and still there is an occasional helper that is getting re-written a bit differently (even if I have a tech lead agent who is looking at the whole code). So I can only imagine the complexity for a legacy app or a huge app where even multiple agents cannot have enough context to write code that considers code and functions next to the feature it is working on. I don't think that agents at this time can have the capability of a senior dev who really knows the system. The best thing we can do is to distill how a senior dev looks at the system from all different angles and write instructions as clear and short as possible. Then iterate and test continuously with measuring the outcomes. At least that's what I'm trying to do.

jjice 1 day ago |

My dad (older software engineer) runs a small, niche software company (five employees). I finally got him to give Claude Code a go like three weeks ago and it's changed everything for them. He's expecting to clear his bug backlog by the end of the year. It's been incredible for them. He's improved stability, improved performance, and reduced memory footprint of his Windows Forms application.

For him, it really is helping him crush his backlog. It's a scale we forget about sometimes where you have hundreds of users, and not millions.

lordnacho 4 days ago |

Last year during the summer I had a bunch of things I wanted to do to improve on the trading system I was working on. A bunch of things that were mostly fiddly, which I had estimated would take me 6 months.

Someone told me to try out Claude Code during my holiday.

I learned the backstory of Sauron with a friend while getting the entire backlog cleaned up. Things worked, better and faster and more maintainable.

Did I run out of things to do? Well, the firm I was with didn't know what they wanted built, so it was a lot of not getting anything done while waiting for them. But the things they had previously wanted, yes, they were done.

aynyc 4 days ago |

Mine is. I've been telling my senior managements that it's cheaper to rewrite stuff from scratch with AI then fix OLD products. They ate that up and everything we have now never reach v1.0 status.

saulpw 4 days ago |

The VisiData backlog is basically empty: https://github.com/saulpw/visidata/issues

muzani 3 days ago |

The perfect app is like crafting dorodango. You make smaller and smaller changes, and use finer tools.

AI is not a fine tool. It's one of the earlier stages of the crafting process. It's like a block of marble, roughly in the form of your sculpture, but you must continue to remove from it.

But if you're making rough bricks instead of fine art, then it's a perfectly decent tool.

pickleglitch 3 days ago |

I'm not a big fan of AI, but we are expected to use it at my job. In my position, it wouldn't really matter how good AI is, because when we find a bug it will often take an hour long meeting, or even multiple meetings, just to figure out what a given behavior should actually be before we can even attempt to fix it.

cadamsdotcom 4 days ago |

If you or your company has nothing to build it means they're out of ideas. And if they're out of ideas, they have too many people.

That's not a place you want to get to.

jfrisby 2 days ago |

For my personal projects:

4 have zero backlog of any sort.

1 has a fleet of agents working through some product improvements that I postponed until the very end of the project. It's a very rote "rebuild this game on a modern engine" project to resurrect a game I developed 20 years ago, and there's a few small improvements I want to make to the game that deviate from the original. Today is day 5 of the rebuild, so it's not exactly a long-lived project per se -- but the bots did identify a bunch of defects in the original game and addressed them as part of the conversion project.

1 has:

- A few flaky tests to clean up. - A couple heisenbugs that I can't pin down until I get a couple PCs for testing (3D graphics stuff is Fun<tm>). - A whole lot of features that consist of one sentence I wrote down to avoid losing track of an idea, and which I may or may not ultimately choose to implement at all. - A couple issues that can't proceed until a human gets involved. I.E. I need to go through the hassle of setting up a corporation and setting up accounts with the various gate keepers (Apple, Google, Valve).

The biggest bottlenecks for me, at this moment are -- in descending order:

1. Running the linters / test suites / build processes for two of my projects completely slam my machine (load average hit 290 earlier today with 3 agents running things across the 2 game dev projects; CircleCI / Github Actions would be waaaay too expensive so I'm running it all locally until I can set up a CI runner node). 2. My bandwidth for acceptance testing, and other human-only work (read: game balance / play experience) 3. Weekly usage caps for fixed-price plans from the three services I use.

Setting aside my relatively unusual circumstances: At any business, you're going to have a problem of information rot when it comes to a backlog. The person who recorded a defect or feature request may not remember enough context to make it actionable (or they may not be at the org anymore). The assumptions underpinning the issue may or may not hold anymore (and it may not be clear which situation you're in). Etc.

Plus, there's going to be a not-unreasonable risk aversion to making changes that are in peripheral areas of the system that don't get a lot of attention -- lest fixing one issue cause some unexpected but undesirable outcome.

A lot of it mostly boils down to either "institutional knowledge gap" or "acceptance testing is a bottleneck".

adamtaylor_13 1 day ago |

All of my clients' software has an empty backlog and bugs typically get solved same-day.

What gives is most people work on decades old crusty crap, and they don't get to build from the ground up with AI.

You can't unstuck a system as fast as you can build it right the first time.

AnimalMuppet 3 days ago |

I recall reading that when you fixed a bug, the odds that you introduced a new bug were 30 to 50 percent. (That was decades ago - anyone with better, more recent numbers, please supply them. Especially if you have actual data for the odds when you let AI fix the bug.)

There is no guarantee that the new bug is not worse than the old bug.

For this reason, not every bug should be fixed. Especially, not every bug should be fixed and immediately pushed to production.

But maybe that wasn't the question. The question said "bug backlog". You could argue that bugs that we have chosen not to fix are not part of the backlog. Even so, fixing 20 bugs and pushing them to production may mean pushing 7 to 10 new bugs to production. Maybe you don't want to do them all right away.

hawkice 4 days ago |

It is empty, for me and my team (I'm an engineering manager). We still have a support rotation but it's just so someone is aware they're supposed to respond to any alerts that come up.

Edit: product doesn't have a deep backlog either. Usually we can only plan somewhere between a couple days and a month in advance. Major recent product launch so we are longest-horizon we have been for a while.

KronisLV 4 days ago |

> The bug backlog should be empty

Yup, I actually achieved that for a smaller system at work.

Tracked down and resolved every single logged bug there, alongside additional correctness improvements - after migrating the system from Oracle to PostgreSQL (it had a narrow enough domain where that was possible, made iterating way easier due to how easy it is to work with PostgreSQL and tune its setup for a specific workload), adding Testcontainers and pushing the test coverage to the high 90s percent, as well as migrating the front end from Vue 2 to Vue 3. Also wrote a custom load testing tool and improved the performance by up to 20x (though there were some really bad querying and N+1 issues at play).

It wasn't completely smooth and did create a few bugs along the way which also needed to be fixed, but overall was pretty solid - mostly due to the fact that the system had been developed with support for mock data from day 1 where you could spin up an instance and play around with pretty much anything you wanted. Hell, one could probably do a Rust rewrite of it if they didn't have anything better to do (not a Rust shop though).

That approach wouldn't really work for larger systems as is (e.g. there are some that are like around half a million SLoC), underdocumented systems, or systems where testing is contrived and difficult, as well as some stacks that just suck (e.g. bringing down a shared dev DB instance preventing N people from doing their work). Oh and also it took about an order of magnitude more tokens than would normally be given to your average dev.

Right now in my circumstances the bottleneck doesn't actually seem to be creating changes/features, but rather deployment and shipping as well as configuration management (assuming you have clients that need release instructions). Oh and being able to sell new work packages. Either way, in general you need more tests and specs than you think you do.

skyberrys 1 day ago |

It's because engineering is generally taking one big problem and turning it into two smaller problems. So if you fix one bug you likely just created two new problems you haven't registered as bugs yet.

undefined 3 days ago |

undefined

PeterStuer 4 days ago |

On the project I am working on atm, I think I am averaging a net addition of around 50-200 production LoC per day. The system produces around 10x as much test-harness code (feature + regression). Planning/exploring takes about 1/4th of the time, coding 1/20th, documentation 1/10th and testing all the rest. Backlog has no known bugs, quite a few de-prioritized very niche edge cases that in theory could arise but never showed up in production, but still could be nice to haves, and plenty of new desired functionality. I'm sure I could go 10x faster, but I choose not to as this pace nicely balances cost, progress and keeping it under control for me.

iotage 2 days ago |

They (currently) just aren’t good enough to guarantee complex bug fixes, particularly in large complex systems.

This is exacerbated by giving a narrow prompt to fix a bug, will not magically result in a model being tasked to evaluate all the consequences and side-effects of changing code.

drltaidev 3 days ago |

There's a chicken and egg story here as well: AI speeds up everything so the amount of code written is much higher, which does still contain bugs, even if AI can mostly solve them as well, new code will keep being written as well... That doesn't mean AI tools aren't good, but obviously they aren't perfect either. Luckily...

efficax 4 days ago |

I feel like you could only ask this question if you don't really understand software development. If you're building features, you're building bugs. Nobody ever said AI produced bug free code. But it does make it possible to evaluate code extensively to find bugs before you ship it. I believe the code I'm shipping now has far fewer bugs than ever before, but bug free? I don't even know if that makes sense. it all depends on what ends up counting as a bug, in the end.

johnfn 4 days ago |

Because some bugs are underspecified, and the specification is the remaining work. For instance, "I can add the same meal to the meal catalog twice". That's a bug. But how do you solve it? Do you merge them? Do you error on the second attempt? Maybe you allow for two variations of the same meal? Once you specify the bug then you can toss it into an LLM and it's done. If you toss it into an LLM without doing that it might solve it in a way you don't like.

swader999 4 days ago |

Performance, Defect rate, feature velocity (5x), cycle time on defects to prod are all at the best they've been on our 7 year LOB 1.5 million loc system. 3 devs/2 qa, 2 sme's. 200 paying customers, moves money so it has to work.

First time in 3 decades I don't have any tech debt that I NEED to tackle. Customers can't keep up with us. We are considering new products/adjacent markets. Strangely, our competitors (about 5 serious ones) don't seem to be matching us.

teo_zero 3 days ago |

Isn't it because of the halting problem? Just like you can't ask an algorithm to determine if a routine will reach its end (because the algorithm itself may run forever), you can't have an algorithm tell if a piece of code is bug-free (because the algorithm itself may contain bugs).

So humans end up taking the final decisions and remain the bottleneck.

undefined 4 days ago |

undefined

0xbadcafebee 2 days ago |

Jevons paradox. Improvements in efficiency cause the total consumption of a resource to rise, not fall. If you can crush more bugs, you have more time to find more bugs, write more features, and make more tickets.

d--b 3 days ago |

Yeah in my company the bug backlogs are empty.

Doesn't mean we're getting unemployed, there is still a lot of things to do. Longer term projects, nice-to-haves we'd never have done before, experiments, experience improvements.

What is certain though is that we're not increasing headcount anytime soon.

getfacl 4 days ago |

I'll go out on a limb here and say that if your bug backlog is currently deep it might be because of general disorganization. We've all seen shops where there are no standards other than ship something "on time" and the result is a spaghetti mess that only gets worse as time goes on. This is a management problem that AI doesn't fix.

seizethecheese 4 days ago |

A hidden assumption in your question is that the ai tools themselves don't create bugs or new backlog items. This is false. Even with perfect AI tools, bugs would slip in because of imprecise specifications.

There are many universes where the number of bugs grows faster with better agents.

theybash 3 days ago |

One of the reasons I think about a lot is that code is a list of micro decisions and sometimes, inevitably agents didn't have the same mental model as me and also lazy rules

francisofascii 4 days ago |

Our users don't have enough feature requests at this point. I have to be diligent in generating new ideas, features, improvements, etc. for the projects I work on. Otherwise we would run out of work.

sjrd 4 days ago |

My bug backlog has been empty [1] at each release for 5 years in a row. We did not need AI to get there.

[1] https://github.com/scala-js/scala-js/issues?q=is%3Aissue+sta...

johnshahbazian 4 days ago |

It's empty for all the projects I'm working on. Is your team just not putting in the time to fix them?

emscb 3 days ago |

I've a question for those who say their backlog is empty. How many of the tickets you closed were re-opened later? How do you check that a bug fixed by AI is really fixed?

koliber 3 days ago |

Anecdotally this is exactly what is happening in my tiny software development organization as well as with the client I advise that has a 5 person dev team.

I no longer have a backlog. Every implementation idea that passes muster goes into development. Each sentry bug is investigated. I proactively do lots of “best practice” compliance things.

The client I consult noticed similar things. Development is now constrained around the pre development product tasks and post-deployment education and adoption of new features.

Aeseir 3 days ago |

Best way i can describe it:

Developing with AI is like 2 steps forwards and 1 step back in a blink of a eye.

The goal is to keep gaining a lot and losing a little.

rozenmd 3 days ago |

My bug backlog _is_ empty. At least of the bugs that o11y has seen in prod.

My feature backlog is now 10x bigger than it used to be, though.

Theophane_Rame 3 days ago |

In many cases, it just moves the bottleneck from writing code to understanding the problem and validating the solution.

wannabe44 3 days ago |

Lack of test suite or even documentation on how to test the changes. Agent ends up breaking something else somewhere.

Kuyawa 4 days ago |

Bug tracker is empty but todo list is full with things to work on and new features to add

I haven't seen AI make a single mistake yet while coding my apps

guessmyname 4 days ago |

My bug backlog is empty. My task backlog is not.

tete 3 days ago |

The same reason why AI hasn't brought us fusion power, cures for cancers, etc. yet.

undefined 4 days ago |

undefined

bearjaws 4 days ago |

"Competitor X is working on Y, we need to beat them to it!"

Thus Y is prioritized over bugs.

mock-possum 3 days ago |

Speak for yourself, my backlog has never been lighter.

SAI_Peregrinus 4 days ago |

Some of my solo projects can't be done with AI. I just built a rack to hold some kayaks, I need to paint it. AI can't (currently) paint anything.

Most of the rest of my solo projects are things where I want to learn something. Having an AI build the thing won't teach me what building the thing myself would. So AI is not particularly useful here either. Sometimes it can be good as a search tool or to provide hints, but it's mostly counterproductive.

The remaining two projects from my backlog where I just needed a tool specific to my own needs, I did with AI. They're OK, not perfect, definitely buggy, but they worked for what I wanted when I made them and none of the imperfections interfere with my use.

For my employer's projects, the backlogs are much bigger and the projects are bigger and still a lot of it can't be done solely by AI. And of course approvals become a bottleneck, but they also catch bugs. My job has legal compliance products, so we have an extensive QA & testing process which a "YOLO ship slop" approach won't work for.

rurban 3 days ago |

It is. Only features left. Exciting months

world2vec 3 days ago |

But my bug backlog at work went down by a lot?

fnoef 3 days ago |

mine is empty. i just ship features that people love and fix bugs within an hour after they are reported.

HoldOnAMinute 4 days ago |

Even AI breaks things when trying to fix bugs.

david-green 3 days ago |

It depends how long the backlog is and the features Product are asking for - tech debt might be easier to fix, but its just as hard to prioritise in a lot of places.

I'd also add that AI tends to find a way to churn out 1000-line changes when a judicious 3-line change would have served two years ago... i.e. teams risk drowning in PRs for features nobody cares about.

andrewstuart 4 days ago |

Scope creep scales, that’s why.

sailorganymede 4 days ago |

Standards are super low, so slop gets approved and no one is checking. We get graded on how many Jiras we close. A bug gets a Jira. So we are doing great!

bob1029 3 days ago |

Politics and ideological BS in the case of both of my clients.

We could be in double digit issues completely blocked on the customer right now. I don't understand how some people think coveting a deep backlog of shit that isn't getting done is somehow a good thing.

brodydung about 8 hours ago |

[flagged]

samratkar about 6 hours ago |

[flagged]

jatin_builds about 22 hours ago |

[flagged]

colleentaylor10 about 20 hours ago |

[flagged]

darshilbabel 1 day ago |

[flagged]

nortiztech about 21 hours ago |

[dead]

hzz4444433333 2 days ago |

[dead]

inshopgroup 3 days ago |

[flagged]

haukebri 4 days ago |

[flagged]

esdg556 3 days ago |

[flagged]

betamatch 2 days ago |

[dead]

abapforagents 3 days ago |

[flagged]

vaibhavtaprico 3 days ago |

[flagged]

singh_abinashi 3 days ago |

[flagged]

cby821555203 2 days ago |

[dead]

jheriko 4 days ago |

[dead]

ameynathe304 3 days ago |

[flagged]