The Great Crew Change – AI and the new generation of oilworkers

These were the early days of the American shale revolution, before we truly understood what horizontal drilling and hydraulic fracturing would mean for the country.

The work was brutal: 12-hour shifts, covered in drilling mud, wrestling pipe in 110-degree heat. But even then, you could feel something shifting. Rigs were popping up across the Permian Basin. Engineers were figuring out how to crack open formations and produce oil from rock that had been written off as uneconomic for decades.

I watched that technological revolution unfold from the rig floor. Over the next decade, American shale transformed us from an energy importer to an energy exporter, and changed the balance of power between nations.

I believe we’re standing at a similar inflection point today. This time, AI is the technological revolution. It’s coming in the nick of time: a whole generation of oilworkers is about to retire, without an obvious replacement for their labor or for their hard-earned tribal knowledge. We have to get this transition right, and I’m going to tell you why.

The technological paradox of oil & gas Oil and gas is simultaneously the most advanced and most antiquated industry when it comes to technology.

Most people have never thought about what it takes to drill an oil well. The engineering that goes into drilling a modern horizontal well (the materials science, the directional drilling, the real-time telemetry from miles underground) is genuinely pushing the bounds of what’s physically possible.

Miles beneath the surface, drill bits navigate through complex geological formations with tight precision, hitting impossibly precise targets under extreme temperatures and pressures, working with materials designed to withstand conditions that would destroy almost anything else. Engineers model subsurface reservoirs that no human will ever see, making decisions based on data transmitted from miles underground. It’s applied physics at a scale and complexity that few industries can match, and it’s why oil & gas continues to deserve its longstanding reputation as a truly great “engineering culture”.

The software systems running most oil and gas operations were built in the 1990s and early 2000s and the majority of work is done in cumbersome excel spreadsheets. This industry skipped the entire SaaS revolution. While Silicon Valley was building elegant tools for marketers and salespeople, oil and gas was left to fend for itself.

As a result, petroleum engineers today spend roughly 40% of their time just searching for information, digging through legacy software, PDFs, and endless CSV files. Another 40% goes to generating documents and reports. That leaves precious little time for actual engineering work.

Meanwhile, the energy industry generates a staggering amount of data but utilizes only about 2% of it. Compare this to consumer tech: platforms like Instagram, YouTube, and TikTok analyze every single data point across their ecosystems, using that intelligence to serve ads. Yet here we have industries critical to human civilization, industries that quite literally keep the lights on, leaving 98% of their data on the table.

Meanwhile, the energy industry is experiencing what insiders call “The Great Crew Change” and it’s a bigger threat than most realize.

Look at the age distribution of the energy workforce and you’ll see something striking: a bimodal distribution with boomers on one end and millennials on the other. Gen X is almost entirely absent.

That’s because the oil and gas downturn of the 80s drove an entire generation away from the industry. And the consequences are now coming due. The boomer generation is retiring, and they’re taking decades of tribal knowledge with them. They have an intuitive understanding of downhole systems and subsurface geological formations that can’t simply be replicated.

I saw this firsthand on the rigs. The best company men and drillers had a feel for the formation that bordered on instinct. They could read the mud returns, sense a kick coming, know when something was off before the instruments caught it. That kind of knowledge takes decades to develop and it’s walking out the door.

Ask any oil and gas executive how much of their workforce’s tribal knowledge is actually documented in any system, and you’ll get a sobering answer: somewhere between 75% and 90% of it lives exclusively in people’s heads. Not in databases, not in manuals, and not in software.

That alone should be alarming because this isn’t just a matter of efficiency, it’s a matter of life-and-death. Oil and gas is a dangerous industry. When you make the wrong decision on a rig floor or in a control room, people die.

I learned that my second month on the job. I was a twenty-year-old kid on the rig when our motor man had his fingers ripped off his hand by the drill line. He came running up to me in shock, hands mangled. We picked his fingers up off the ground, threw them in a pickup truck, and sent him to the hospital. That’s the cold reality of this work.

The mechanism for transferring safety knowledge has always been deeply personal: experienced field hands teaching the next generation through years of proximity, through stories of what went wrong and why, through instinct that comes from having seen bad things happen in real-life situations.

But as an industry, we’re terrible at institutionalizing that knowledge. After an incident, someone writes a report, it gets filed away on a server, and it’s never leveraged again to prevent future accidents. We make the same mistakes in the field over and over again.

The generation that carries decades of safety intuition is leaving behind a younger generation that will have no choice but to learn through their own incidents causing events such as blowouts like Deepwater Horizon, pipeline spills, and refinery explosions. The environmental and human toll of getting this wrong is enormous. This may be the single most urgent reason to use AI to capture and operationalize tribal knowledge: not just to optimize production and operations, but to keep people alive and prevent environmental catastrophe.

Here’s the second half of the crisis: young people aren’t coming in to replace them.

The industry has an image problem. Decades of cultural messaging have painted oil and gas as the villain, and talented young engineers are choosing other paths.

And the talent crisis is worse than the 1980s gap alone would suggest, because millennials who did enter the industry have been battered by repeated boom-bust cycles. The 2008 financial crisis. The 2014-2016 oil price collapse. Then 2020 delivered the unthinkable: negative $40 oil and a global pandemic.

Each downturn triggered mass layoffs, and each round of layoffs drove more young professionals out of the industry for good: not because they couldn’t do the work, but because they were looking for more stability in their careers.

I’ve watched it happen. Data scientists leaving oil and gas companies to work logistics at the grocery store chain, HEB. Engineers jumping to Amazon and Microsoft. Elon Musk recruiting offshore engineers straight off platforms for SpaceX. There’s a pervasive, pessimistic undertone among young energy professionals that you simply can’t build a stable career in an industry with this much volatility. And honestly? It’s hard to blame them. But that makes the workforce gap even more urgent to address.

The institutional knowledge is leaving at the top and a dry pipeline of talent is at the bottom. The workforce gap is widening from both directions. This is a big problem, and we need to fix it as an industry.

You don’t want to contemplate life where fossil fuels are scarce. (But if you insist: one billion people on Earth still live in energy poverty, with no access to electricity or clean fuels. They’re forced to cook and heat their homes with wood and animal dung, leading to severe health and environmental problems.)

Here’s an uncomfortable truth: every energy demand forecast you’ve ever seen has been wrong. Not only will energy demand continue to increase as humans use more electricity and compute, but energy demand has the potential to grow infinitely.

There’s a direct, undeniable correlation between energy consumption and human prosperity. The more energy a society consumes, the higher its standard of living, life expectancy, and economic output. Petrochemicals power materials and manufacturing, the fuel that moves global supply chains, the natural gas that powers electrical grids. The secular drivers pushing energy demand forward aren’t slowing down, they’re accelerating:

So it’s really important that we don’t mess up this moment. Energy has reinvented itself before; I saw one of those times firsthand. Now we have to get this one right, too.

The energy industry spends over $3 trillion annually on capital expenditures: drilling wells, building infrastructure, transporting and refining hydrocarbons, building power plants and electric transmission lines, etc. On top of that, over $1 trillion goes to G&A, the operational overhead of running these massive enterprises.

AI has the potential to materially increase revenues while decreasing capex and other costs.

AI systems can capture and operationalize the tribal knowledge of retiring experts before it disappears, preserving decades of hard-won intuition in systems that can train the next generation.

Natural language interfaces can eliminate the 40% of time wasted searching for information. Machine learning models will actually utilize the 98% of data currently sitting idle, optimizing everything from drilling operations to equipment maintenance schedules to reservoir management.

The downstream effects are significant. Lower break-even costs on a barrel of oil. Capex reduction across extraction, transportation, and refining. Marginal wells becoming economical. More energy produced, more efficiently, at lower cost.

Beyond operational efficiency, some of the biggest unsolved challenges in oil and gas are ripe for AI-driven breakthroughs.

Consider recovery rates. In American shale, we recover only about 5% of the oil in a reservoir. We leave 95% in the ground. Let that sink in: the shale revolution that transformed America’s energy position, that reshaped global geopolitics, that I watched unfold from the drill floor, all of that came from accessing just 5% of what’s actually down there.

The potential unlock from improving recovery rates is almost incomprehensible. Smart engineers armed with AI will develop new methodologies, better reservoir modeling, optimized completion designs, enhanced recovery techniques, to access more of what we’ve already found.

Then there’s the water problem. Oil and gas wells produce enormous volumes of water alongside hydrocarbons, and managing that produced water is one of the industry’s biggest operational and environmental challenges. Much of it gets injected back into formations for disposal and many believe this injection is causing the increased seismic activity we’re seeing in oil fields.

The Permian Basin has become the epicenter of water recycling technology, with operators developing new methods to treat and reuse produced water rather than dispose of it underground. AI will accelerate these iterations dramatically, helping engineers optimize treatment processes, predict water production, and develop new recycling methodologies faster than traditional R&D cycles would allow.

These are fundamental problems that, if solved, reshape the entire economics and environmental footprint of the industry.

Here’s something important to understand about energy: humanity has never transitioned away from an energy source. We don’t replace; we add.

We burn more wood today than at any point in human history. Coal consumption is still massive globally. Every new energy source has been additive, layered on top of what came before. The future of energy is not a transition but a diversified mix of sources. Natural gas, solar, battery storage, nuclear, and technologies we haven’t yet scaled will all play critical roles.

And it’s worth noting: natural gas was the primary driver of CO2 emission reductions in the United States over the past two decades. Not solar. Not wind. Natural gas, displacing coal in power generation, did more to reduce American emissions than any other single factor.

But here’s what most people miss: all of these adjacent energy verticals are deeply dependent on oil and gas expertise, wisdom, and resources. The operational knowledge required to build and maintain energy infrastructure at scale? That lives in oil and gas. The engineering talent that understands how to manage complex, distributed energy systems? Largely in oil and gas. The capital and project management experience to deploy at the scale the world needs? Oil and gas.

The AI tools being built for oil and gas today won’t stay siloed. The solutions for optimizing drilling operations will extrapolate to geothermal. The systems for managing pipeline networks will inform battery storage logistics. The knowledge capture platforms preserving petroleum engineering expertise will train the next generation of energy engineers across every vertical. And eventually, as humanity expands beyond Earth, the lessons we’re learning about energy systems here on this planet will become the foundation for powering civilization in space.

I’ve seen what a technological revolution can do to the energy industry. Shale changed everything, not just for oil and gas companies, but for American foreign policy, manufacturing, and the global economy.

Here’s the cycle I see emerging: AI will bring unprecedented prosperity to humanity, but that AI runs on energy. Oil and gas will power that future, including the data centers and compute infrastructure the AI revolution demands. And in turn, AI will transform oil and gas itself, making it more efficient, more economical, and more capable of meeting the world’s growing energy needs.

That energy abundance unlocks more prosperity, which drives more demand for AI, which drives more demand for energy. It’s a virtuous cycle and oil and gas is at the center of it.

I saw what happened when we cracked the code on shale, now it’s time to do it again.

Message from Jim Robinson:

Dear FRiends,

We need your continuing support to keep FR funded. Your donations are our sole source of funding. No sugar daddies, no advertisers, no paid memberships, no commercial sales, no gimmicks, no tax subsidies. No spam, no pop-ups, no ad trackers.

If you enjoy using FR and agree it's a worthwhile endeavor, please consider making a contribution today:

Click here: to donate by Credit Card

Or here: to donate by PayPal

Or by mail to: Free Republic, LLC - PO Box 9771 - Fresno, CA 93794

Thank you very much and God bless you,

Jim

This article is from the newsletter of Andreessen Horowitz.

Andreessen Horowitz (a16z) is a leading Silicon Valley venture capital firm founded in 2009 by Marc Andreessen and Ben Horowitz. It manages over $90 billion and invests across stages (seed to growth) in tech sectors like AI, bio/healthcare, consumer/enterprise software, crypto, fintech, games, and American dynamism (defense, space, manufacturing). Known for backing transformative companies (e.g., Instagram, GitHub), a16z provides not just capital but also talent networks, policy advocacy, and operator expertise from former founders and executives. Website: a16z.com.

Sounds like a great consulting retirement teaching the AI for somebody who knows a lot.

I have had varying interests and have been on location for dozens of horizontal wells, in Oklahoma, North Dakota, and West Texas, And I can say that the technology is incredible. To drill vertically to 12,000 feet, turn and then drill as many as a dozen 2,500’ lateral is awesome. The control room looks like a cockpit of s Moon Shot, and might even be more complicated. The US will never run out of gas and oil, although it will continue to be more expensive.

This guy seems to be more on the Drilling side rather than Completions. Frac guys have been using machine analytics for awhile now. Same can be said on the Production side as well

He’s right though about the boom bust nature of the business and how it affects personnel.

Boom-bust cycles where everyone is laid off, company yards closed and equipment sold off.

I was recruited out off college by Haliburton, worked 80+ hours a week; then the bust of the early 80’s came and everyone was laid off.

That is when one understands the oil field is not a career.

Oil and gas should be used primarily as feedstock for petrochemical products like plastics, fertilizers and for vehicle fuel.

Correct. Battery powered cars are too dangerous (particularly in accidents). Solar/battery systems are just as bad, particularly dealing the end-of-useful-life considerations. We do have safety problems with power distribution lines, but those are mostly due to catastrophically poor forest management.

You see the same bimodal distribution of workers in many similar fields. Imagine how many young engineers want to burn coal as a career? All the experienced hands that invented supercritical steam power cycles and grew boiler sizes to 1,200 MW are retiring or have retired. Same with the people that operate and maintain the units. That institutional knowledge is dying or dead.

Why is that? Because we let the ultra-greeniacs get the upper hand and they drove the myth about how CO2 is going to end earth as we know it. The politicians dutifully fell in line and scuttled the oil, gas, coal, and auto industries with their stifling regulations.

The bigger problem is the cyclic nature of humans. Technology continues to grow and advance while political waves come and go. People choose careers based on which way the political winds are blowing when they come of age and graduate from college.

Will the Trump resurrection of domestic manufacturing and the renaissance of gas, oil and coal last? Or will the industries get whipsawed back to being the bad boys the next time “D” wins and election.

My money is on continued whip-sawing such that business planning is difficult to impossible and talented people won’t go near industries that keep getting the lash by government.

The only area I'd push back against is this. O&G companies have, for many years, compartmentalized all that data and knowledge into separate, bespoke teams who were often reticent to share the data or help others in the org understand why and what.

It's hard for AI to solve a bad data architecture issue. So yes, many many TB of data. Much of it orphaned and siloed without any context. Only domain experts can turn it into something useful.

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.