Most of What We Call Progress

Most of What We Call Progress

Most of what we call progress in software is just motion. New tools, new frameworks, same problems. Maybe fancier logos. Our industry always has this collective thrill that a new fancy method, framework, process will make things infinitely better. Perhaps, perhaps, perhaps.

I’ve watched that excitement enough times to recognize its cycles. Years ago, a colleague was setting up Apache Spark for a modest analytics job. The dataset was small. He knew the requirements intimately. In all honesty, the complexity was completely unnecessary. I asked why. His answer was what I’ve heard countless times since: 

Because it scales.

That moment stayed with me. Not because Spark was wrong, but because I realized how easily engineers, myself included, confuse capability with necessity. After years in management, I still feel like an engineer. The illusion of progress often looks exactly like progress: more systems, more layers, more abstraction. It feels like forward motion until it is not. That’s why I wrote a chapter arguing you can do most analytics with a few scripts and Postgres. And damn, I love Postgres.

With time, you start to notice a deeper pattern. The same optimism that drives innovation also fuels waste. Every cool modern stack eventually becomes tomorrow’s cautionary tale. I talked about it here but there’s a bigger lesson to be taught. Every abstraction that promises freedom adds a little more friction. Every clever idea you once loved becomes something you later apologize for maintaining. 

The difference is not about tools. It’s about perspective. What changes with experience is not your stack. Generally speaking, it’s your sense of what matters. You start seeing that progress depends less on what you build and more on how you think, how you decide, how you work with others.

After years of building, breaking, and leading teams, I’ve changed my mind on many things. It is not out of cynicism, but because experience has a way of sanding down certainty. These are a few of the lessons that stayed.

Clarity Over Cleverness

Early in your career, cleverness feels like proof of skill. The shorter the code, the more elegant the abstraction, the more you feel you’ve outsmarted the problem and maybe, a little, everyone else. I used to believe that too. The best engineers, I thought, wrote code you had to study twice to understand. 

Unfortunately, clever code doesn’t scale. Not across time, not across people. What feels ingenious in the moment becomes a small trap for whoever inherits it next including your future self. Someone will wholeheartedly say what the fuck is going on here? When you’re the one debugging a late-night incident and reading your own comments from two years ago, clever loses its charm fast. The real production code isn’t a leetcode problem to show off.

At some point, you realize that simplicity is not the absence of sophistication as Da Vinci said. It’s the evidence of mastery. Writing clear code, building transparent systems, communicating intent precisely. Because clarity compounds like everything else. One clear decision enables ten more.

When I teach at university, I tell them this: if your code requires an explanation, it probably needs a rewrite. It’s not about intelligence; it’s about empathy. You’re not writing for the machine. You’re writing for the person who comes next. Now that I think about it, we have AI generated code. Add that to the fire!

And the irony is, the longer you work in this field, the more you admire the opposite of what once impressed you. The truly rockstar engineers make complexity manageable. Companies keep trying to automate apprenticeship with tooling and documentation. But no static check or framework can replace the slow transfer of judgment that happens face to face.

Apprenticeship Beats Abstraction

Software is craftsmanship. You can learn it from books or courses but not all. You can also learn it by building alongside someone who has a different experience than you. That doesn’t necessarily mean number of years but sometimes it’s just the stack they worked with. You watch how they think, how they name things, how they stay calm when the system crashes. Those are the kind of things that make it memorable. 

I remember pairing a staff engineer with a mid-senior once. Honestly, I didn’t expect them to bond so well. It became something else entirely. They started finishing each other’s sentences in code reviews. They challenged each other’s assumptions, refined each other’s instincts. Together, they built clean and durable services that I was proud of.

When I had to split the team, one of them moved to another group. They both hated it. Not out of resistance, but because they’d built something deeper than cracking the coding bottlenecks. They kept talking anyway. They became the bridge between the teams. So, I was happy that I did. It got two teams closer…

That’s what apprenticeship really is. It’s continuity. It’s the invisible thread between people who build things the same way not because they were told to, but because they learned to see through each other’s eyes. That’s something that stays longer than projects, teams or companies. Those are people you refer to hiring. 

Abstraction teaches structure. Apprenticeship teaches care. And good systems, like good people, survive on both.

The Humbling of Process

Good ideas start out pure. Then someone turns them into a process, and that’s when they start to rot.

Agile was born from rebellion, a handful of engineers tired of bureaucracy from unfit waterfall. TDD was the same. It was a call to slow down and think. Both were tools for clarity. But somewhere along the way, they became economies. Certifications, coaches, metrics. Agile died the day it became a job title. I decided I didn’t want to be a scrum master.

You can follow every Scrum ritual and still move nowhere.
You can hit 100% test coverage and still 0% confidence or understanding.
You can follow every process to the letter and still turn smart people into spectators.

The more you manage, the clearer this becomes. Process doesn’t save bad culture; it amplifies it. The point of the process is to make communication cheaper, not to replace it. You can have a dozen rituals, but if people aren’t trusted to think, all you’re doing is project management theater.

And that’s where the real danger lies when the process turns into religion. When people stop asking why and start quoting how. Dogma starts when we mistake the tool for the truth. When the ritual matters more than the result. When following the “right” method feels safer than thinking for yourself.

After years in leadership, I’ve learned that the best process is invisible when it works. It has its tenets. And the team is guided by vision, not ceremony. I’m not saying you shouldn’t have status updates. I’m just saying that’s not the goal. The goal is actual delivery. When you establish that, you end up with self-managing teams. Small, autonomous, trusted. 

Process has its place. It creates safety, consistency, and rhythm. But it should bend to fit people, not the other way around. The moment you start defending the framework instead of the purpose, you’ve already lost the plot.

Experience humbles you. You start evolving your processes and focusing on the right people. The rest follows.

The Myth of Scale

Every engineer eventually overbuilds something. You think you’re being smart. You’re thinking ahead, building for growth and before you know it, you’ve created a system ten times heavier than your actual problem. That’s the trap. We keep designing for imaginary futures for scale that may never come and call it engineering. But it’s not engineering. It’s over-engineering.

The industry rewards it too. Nobody gets promoted for keeping things small and sane. You get promoted for complexity. For showing initiative. For saying microservices and distributed. Over-engineering doesn’t come from stupidity. It comes from ambition and fear. Ambition that wants to prove mastery. Fear that wants to look prepared. Nonetheless, the result is the same.

Experience burns that instinct out of you. You stop trying to impress the invisible audience. You start building for the one that’s actually there.

The Boring Stuff That Matters

When you’re young, you want freedom. Understandable. Dynamic languages, no process, no linting rules. Need for speed. Yay! You want to move fast and prove you’re clever enough to live without the guardrails. I was like that too. I remember arguing why I’ve to put the final keyword before all the constant variables. My colleague pointed me to a convention doc and asked me to read it. If I’ve better idea next time, I could have come up and defended it. Lucky for me, I did not. I quickly realized I was just too lazy to type “final”. My arguments were silly. 

Then you grow up, and you start cleaning up after people like you.

I saw many people hate typed languages. Extra ceremony. Slowing me down. Blah blah. Now they also see them as early warning systems. The same goes for linters, tests, even code reviews. Most of the guardrails you once resisted linters, type checks, static analysis aren’t bureaucracy. They’re collective defense mechanisms. They make sure the system stays consistent when people come and go. The older I get, the more I see stability not as friction, but as freedom from chaos. They don’t exist to restrict you. They exist to stop you from drowning in your own chaos.

Engineering principles, coding standards, templates, even naming conventions. Now I see it for what it is: collective memory. The small agreements that let ten people act like one. The bigger the system gets, the more those small things matter.

Experience humbles your taste. You stop craving freedom and start craving stability. You realize the boring stuff such as type safety, documentation, process is what lets you take real risks somewhere else.

It’s not about rules. It’s about focus.

Leverage Over Control 

For a long time, I questioned why PMs existed. Both product and project. I used to see them as blockers, a layer between engineers and the work. Another meeting. Another update. Another person who didn’t write code telling me what to do. Yeah, I was an engineer too once. 

Then I led teams. And I realized how much of my day vanished into coordination, context-switching, and glue work that nobody saw. That’s when it clicked: a good PM isn’t there to control engineers. They’re there to protect them. They buy them time. They filter noise so they can think. They ease up the cognitive load.

Many PMs don’t do that. They live in Jira, chase timelines, and mistake movement for progress. They measure velocity, not value.

But the good ones are different. They translate business goals into product decisions. They connect dots others don’t see. They make sure projects move forward when multiple teams are involved. They know why something matters, not just when it’s due. And they have answers to many questions. Why are we building this? Who’s blocked? What’s the trade-off? What’s the priority? 

They absorb chaos so the rest of the team can stay calm. The bad ones make noise. The good ones build silence. And that’s the irony: for years, engineers chase autonomy, freedom from meetings, managers, and oversight only to realize that the right kind of structure is what gives you freedom back. A good PM is not overhead. They’re leverage.

The Real Work Is Social

You can spend years thinking software is about code, tools, or architecture. Then one day you realize it’s not. It’s about people. Treat them well to be treated well. Prisoner’s Dilemma.

Every system that failed on my watch didn’t break because of missing tests or weak design. It broke because someone didn’t talk to someone else. Misaligned assumptions. Undefined ownership. A decision left hanging too long. And sometimes edge cases. 

All the things I once dismissed. Be it documentation, announcements, even meetings turned out to be social scaffolding. Not process for process’s sake, but structure for understanding. The older you get, the more you notice how fragile clarity is. You can build the best architecture in the world, but if two teams interpret one requirement differently, it’s over. The code follows the conversation. If the conversation breaks, so does the code.

The real work isn’t technical. It’s relational. You need different rules to win.
Likability, trust, alignment, consistency. They are invisible but they make the difference a lot.
You can patch systems, but you can’t patch silence.

Experience strips away the illusion that software is a technical craft. It’s human coordination disguised as code. The best engineers I know don’t just fix bugs. They fix misunderstandings before they become them.

It’s the same lesson I’ve learned in hiring. You can interview for cleverness all day and still end up with the wrong person if you don’t pay attention to red flags. The best hires aren’t the ones who ace puzzles. Surprisingly, they’re the ones who reduce entropy. You remember them not for what they built, but for how calmly they helped others build.

The Quiet Maturity Curve

In the beginning, you want to prove yourself. You want your code to stand out, your process to be the smartest, your architecture to be the cleanest. You want people to notice. Then you spend enough time in the industry, and you start to see what actually lasts. It’s not clever abstractions. It’s not a big launch. It’s not the tech stack you once defended like religion. It’s the simple things that keep working long after the hype dies.

Experience gradually makes you quieter. You stop trying to win every argument. You start letting small things slide because you’ve seen how little they matter. You stop chasing perfection and start chasing flow that state where people, systems, and ideas move together without friction.

The engineers I respect most barely talk about technology anymore. They talk about trust, focus, and momentum. They still love building, but the thrill is different now. It’s not the rush of release; it’s the satisfaction of knowing things won’t fall apart when you’re gone. Or refactoring untested code and make it great again! That doesn’t mean they don’t care about quality. They do. They just highlight it as a different problem. If you don’t promote a culture of quality, you won’t have quality even if you scream for it. If you end up in a good engineering org with high bandwidth and quality, there’s little chance that you’ll mess things up.

I used to think maturity was about mastery. Now I think it’s about knowing when to walk away. Knowing when “good enough” really is. Knowing which fights are worth it and which ones are just noise.

You spend years trying to make software bend to your will. Then you learn to bend with it. That’s the quiet curve nobody tells you about.  The one where progress stops being about moving fast and starts being about moving right. I really like my colleague’s one liner: “I write distributed systems that won’t wake you up often.” You stop idolizing tools. You start valuing ecosystems that stay up while you’re asleep. You choose PostgreSQL over the shiny new datastore, Bash over the latest task runner, and typed APIs over clever hacks, not because they’re exciting, but because they keep working.