Yes, and: the most important time to use your brain isn't in the loop.

That _is_ valuable, and will remain so, but the temptation to turn it off is there because the loop is good and getting better at what it does. Once you're in the loop, with the rare high-value exception of catching total mistakes nad redirecting, you're mostly choosing between similar-yet-reasonable options. This isn't so much right-vs-wrong as relativistic optimization. Letting the loop do the work will get you a mediocre result quickly, and that's usually fine.

The most important time to use your brain is _before the prompt_. Once you engage with your LLM and agent, you start biasing yourself, and its reasonable suggestions constrain your visibility into other options, other worlds.

First, use your brain. Then write the prompt.

It doesn't help that this is being forced from above. I agree you can't turn off your brain, but any time I argue that AI isn't the right tool for something management gets angry. There is a constant push to fully automate things, regardless of what those things are.

Even if I agree that you have to use your brain, when my boss is literally telling me to stop questioning whether AI automation is the right answer in team meetings/chats I'm just going to shut up and give them what they want.

Even pre-LLM: I've found a lot of colleagues (mystifyingly to me, both seniors and juniors) who really want to be able to turn their brain off when doing investigations into complicated cross-system behavior (feedback loops abound), and they don't like hearing that it can't be collapsed into a runbook. If a drinking bird (https://en.wikipedia.org/wiki/Drinking_bird) on an Enter key could do it, I could rig something up so I wouldn't need you!

I think a lot of people are not able to reason to this point and then adjust behaviour accordingly. They probably reason to the point when pressed for it, but cannot resist the low effort and low resistance route of asking AI, and then probably make up reasons why in their case it's fine (typical behaviour to address cognitive dissonance).

Then again, for many people their job security doesn't depend on what work they do, it depends on how well they are embedded in the organisation. Before AI you also already had plenty of people doing their job at a questionable level, yet never seem to get fired.

You are a photographer from the olden age. Taking a photo back then took forever. You are standing on a hill trying to get the perfect picture of a sunset. You measure and sort and calculate angles and light. A tourist stands next to you with an high-speed iphone auto-settings camera. They spin around and get thousands of photos from different angles while you are thinking and measuring your one shot. That’s what AI does. It gives you many shots at the same problem. The thinking guy with the one approach will likely lose a photo competition to one of the accidentally better shots of the person who tried a lot more. Make and filter now may be a better approach than overthink and try only once.

Airline pilots and train conductors are basically meat proxies 95% of the time and they are still employed. Still necessary for when things really go wrong.

An AI system cannot replace professional people. It doesn’t replicate or do the work that they do. That’s not what it is for and it is incapable of performing these tasks.

If you are using these systems you still need to know what you want and how to specify it. You can use agents to automate much of the work but it can’t tell you that your ideas are bad, that there’s a better way to accomplish your goals; it can’t tell you what you want. And if you can’t understand what it is outputting you’re failing in your primary task as a developer: understanding the problem.

I agree with the title. I just don’t think these systems will ever be capable of replacing humans. As much as product managers and executives would love it and not have to deal with us; they’ll just be shooting themselves in the foot if they try to replace professional developers with AI harnesses and fire all the people with the institutional knowledge and skill.

"What reason is there for the company to employ the meat proxy?" Maybe some form of liability, accountability...

New concept: the AI patsy

     Let's say that happens. What reason is there for the company to employ the meat proxy? The company can just run the LLM in a loop and lay off the employee. There's no point at which this methodology will work for the employee

I would argue that there's still an important role for human employees as compliance ablative armor.

The human's purpose is to keep an eye on the LLM, occasionally intervene/interfere with its work, and most importantly, incur social/legal liability for anything that the LLM does wrong.

Kind of like the backup drivers used by self-driving cars.

The version of "meat proxy" I've heard is the person who forwards a machine-answer to someone else who didn't ask for a machine answer, and could just as easily made one themselves.

Calling someone a "meat proxy" for being the human in the loop is wild.

Thats not the problem of LLM/AI.

The problem is, that digital work, the new frontier of employment for a lot of epeople, got and will continue to get A LOT cheaper, faster and more accessable.

The PO who always complains about someone? They will just vibe stuff. It might even be shit but it will work good enough until LLMs/AI/AGI/ASI is good enough or better anyway than an avg developer.

Man i have seen so so much shitty code even in software companies.

You know when your HR software takes seconds to load? Yeah an LLM can do that today too

I've noticed some tasks (80-90%) can be accomplished by quickly typing out a feature/bug fix, accepting plans with ~10% brain power, and moving on. But there are other tasks where the LLMs struggle, and it requires significantly more time and mental effort to think through the architecture and requirements before it's to a point where the LLMs can process it.

What's difficult, is I now spend most of my time on these high-throughput tasks, where my brain isn't exactly off, but focused on quantity over quality. It's often difficult to identify when a task requires "real" brain-power, and when it does, it adds a lot of friction to dig into the code and orient myself. So it's hard to know when it's the right time to do that.

Turning your brain off is usually a defense mechanism to protect yourself against something bad. dissociate during trauma etc. lighter dissociation points to the interaction trending toward badness, not goodness. Seems like the mind and body is telling us something that we shouldn't ignore

That's the smallest dan luu post I've seen.

Yeah, I experience this a lot. I call it “zombie mode.” It’s like pulling a slot machine, though. Sometimes AI can go a few turns just with encouragement. But when I don’t maintain conceptual engagement, drift is inevitable.

My solution is saying “gnite” and coming back in a new terminal window with a clear restatement of my purpose. Or “what are you working on and why” can help.

Terminal windows are not great for keeping context; waiting for better design. It is possible.

Turning off your brain with LLMs is how you get fired.

Even with small things, you need to be at the wheel. The other day I had to stop copilot from literally deleting the file I was working on. Why did it choose to go down that route? I don't know, but I nixed and reported the chat once it did that.

LLMs are dangerous and if for some reason you are stuck with using them, do NOT ever turn your brain off.

Ideally, you should not be using LLMs at all.

The more I see posts like this, the more I realize that the problems people have with AI are no different than the problems we had with outsourcing (development to low-cost countries) 20 years ago. The results and challenges are the same; except now the feedback loop is within minutes instead of overnight.

Agents are still a bit to eager to please, so they are still missing cases when the better answer is to push back, change course or refactor.

But when the human suggests changing the course, the LLMs will often agree the new direction is better and start heading there...

I think this is the dissonance that has been making me feel insecure / depressed around the state of AI coding. On the one hand I am way more productive. But on the other hand it is almost requiring more of my brain than before, because I am having to determine what is good output and what is slop. But that's not the part of my brain that I particularly like engaging — I like understanding something deeply. But in this case, that is just not possible. I like to think that as the human-in-the-loop, I am supplying the requisite _taste_ to the design and overall structure of the program. But at what point does my taste get taken over by the LLM as models improve?

Another problem is that when something breaks then I am very limited in my ability to reason through the breakage. It's necessary to go back to the LLM and ask it why it broke. At which point, I'm able to follow the reasoning — but not in any durable manner where I'll be able to remember it. To me, it's sort of like the difference between memorizing a passage and having a ready reference at hand at all time. We had this same problem with information before — now it's come for our reasoning.

I feel that AI's biggest threat is making us lose the intuition and judgment we need to correct it.

> What reason is there for the company to employ the meat proxy? The company can just run the LLM in a loop and lay off the employee. There's no point at which this methodology will work for the employee.

Company is pleased! Cost savings!

Alas, the joy doesn't last long as pretty soon after the company realizes that after an LLM replaces almost all their employees, the much easier task of replacing the entire company begins, because the company itself has just become a much thinner meat proxy for getting an LLM to do whatever it is the customer relies on the company for.

Customer is pleased! Cost savings!

Alas, the joy doesn't last long as whatever savings they have from replacing the company with their own LLM loop is offset by whatever it is they do to earn income being similarly impacted.

All of the moats degrade together though they may fully give out at slightly different speeds.

> Let's say that happens. What reason is there for the company to employ the meat proxy? The company can just run the LLM in a loop and lay off the employee.

Why do you think management is so gung-ho about LLMs? Employees are expensive, and laying them off is therefore extremely appealing.

The entire, explicitly stated goal of the AI movement is to produce an ASI, which is better across every metric than a human. Even if you refuse to turn off your brain, the economics of paying you to keep the lights on upstairs aren't going to work out.

This was never going to work out well for the human.

I'm not sure what else to say. You listened to these people speak, what the heck did you expect?

This guy footnotes.

There are a lot of programmers who are working with chatbots and watching YouTube simultaneously for the first time, simply matching the rest of the economy for the nature of work.

If you can do your job while watching TV your brain is turned off.

Maybe, but the parts of my brain I'm still using are a lot easier to use.

I was dreaming to be promoted to a cushy management role where I could talk bs for a living.

I managed to attain 'hands-on team lead' and later an ill-fated 'bootstrapped cryptocurrency CTO' role... Nobody wanted to pay me to talk as a career. Now they do!

So I'm very happy about that.

I feel contempt for all the people who had it easy and 'chose' to be engineers voluntarily.

It's an interesting idea. I'm not sure I agree.

Here's the thing I'm finding - LLMs allow me to do more coding when my brain isn't firing on all six cylinders. It means I get more code written because I don't have to be "in the zone" to exhaustively hunt down every piece of API and dependency and have my flow continuously broken up by missing information. The code written in that state is lower quality, but I can patch up five-cylinder-brain code faster than I can synthesize code from a blank page outside of my flow state.

When throughput is an issue, you have to think judiciously. I'm definitely not saying you shouldn't think but you have to apply human oversight effectively. For me that means front-loading through dialectic planning, and maximizing guardrails.

> Let's say that happens. What reason is there for the company to employ the meat proxy? The company can just run the LLM in a loop and lay off the employee. There's no point at which this methodology will work for the employee.

If this is true, turning off the brain... is completely rational. Why burn out the brain for something happening either way? It's not like being fully engaged is going to result in anything that much better for the employee. In a world where code is this disposable, memorizing a codebase that won't recognizably exist in 3 years is also stupid.

trends like these remind me very much of how true some of the predictions in Douglas Hofstadter's "Anti-Intellectualism in American Life" were: https://ia802903.us.archive.org/6/items/richard-hofstadter-a...

he was writing in reaction to the McCarthy era which, arguably, is very similar to kind of bombastic nativism we're seeing now from our political elites. his argument essentially being that there are cultural and institutionalized forces that are driving people to, well, turn their brains off - specifically to no longer be skeptical of what they consume, to not want to refine and improve their toolbox of analysis, to not want to understand how things work for the sake of knowing

and that these acts were deliberate - a populace is so much easier to control when it's complacent and unknowing, who believes, unskeptically, that Oceania has always been at war with Eastasia, unquestioning about who was writing the PR missives that feed the news cycle (see also: https://www.commondreams.org/opinion/what-is-copaganda), how verbatim they are, and what the real intent of the authors were

like, I get that you don't want to put in effort for a job you don't like. but that effort also makes you better at your work. it makes it easier on you, it lets you create less incidents, it empowers you to be more agent in your own life, to, who knows, have the guts to start a consulting gig because your skill-sets actually are better than the average

there's so many benefits from just being more diligent, longterm, proactive benefits. but that's not everybody's extrinsic motivator and it's so much easier to just be Skinner-boxed by social media companies into doomscrolling, I suppose

I still don't get who is this guy and why anyone should care.

Turning my brain off actually works really well when I'm doing something more creative, and while I know this isn't exactly what he's talking about, I wonder if we are just offloading things we shouldn't have to think about to have more space to think about other things.

Of course, this becomes a lot more difficult when you're unemployed.

The obligatory XKCD, panel 2: https://xkcd.com/568/

There's some software that really doesn't require my brain. I gave Astra an ssh key for root on a little Pi Zero with a microphone attached and asked it to make recordings of a siren going off in our neighbourhood, based on a couple of sirens. This would have been a great deal of work a couple of years ago, now it's a few tokens, and honestly, it doesn't really matter if it's shitty code, I just want the timestamps.

Then there's the software that I write for a living, and almost my entire day is now spent making hard decisions. /grilling gives me difficult technical design decisions endlessly and I am forever doing product management on features from agents. My brain is back in CTO mode, which isn't what I'd planned for myself after I'd left that particular cursed path, but fine. I am doing the opposite of turning my brain off, because I no longer can just chill out and mindlessly write code.

Both these states can exist simultaneously. But my brain work has shifted into something different, while getting no less intense.

I would like to argue opposite.

Developers like to "turn their brain off" during "super hard debuging" sessions. Mental flow state is a well documented as pleasurable experience. It is like a drug!

Auditing and reviewing autogenerated commits, is hard gruelling mental labour, compared to that. You do not even get a credit as an author! You can not get into groove to get your fix!

[dead]

Um, some data:

”Nowhere is this clearer than in entry-level software engineering—though the picture depends on whom you ask. Brynjolfsson discovered that employment among developers aged 22 to 25 has fallen nearly 20% from its late-2022 peak. The online job site Indeed also paints a stark picture: software development job postings have fallen 53% from the same starting point.”

https://insights.som.yale.edu/insights/the-real-job-destruct...

They aren’t firing the meat proxies to that degree yet, but they absolutely have stopped hiring the meat proxies. It’s going to be a very dirty year on Wallstreet when you’ll see massive sneaky layoffs of the existing meat proxies.

You can turn your brain off with these models. It is what it is.

All is possible with good blackbox testing.