Six gates that move an AI pilot to production

Episode 14 - Six gates that move an AI pilot to production - Transcript

Matt: Welcome to Data Versus Commerce, where we explore the messy middle between database and doorstep. I'm Matt Johnson.

Floyd: And I'm Floyd Blaikie. Let's dig in.

Matt: You know what I'm seeing more and more of these days? AI pilots — not just in B2B, but also in retail, and across every department in the businesses that we work with. In my conversations with data, commerce, and supply chain leaders, everyone's got one, maybe two. They've got prototypes running, experiments underway, teams tinkering with generative AI, and building cool proof of concepts.

And honestly, a lot of these ideas are really, really good. They're solving real problems. But here's the thing I keep hearing: "We built this, we proved it works, but we can't actually figure out how to get this into production." That is the gap I'm going to talk about on today's solo episode. Yeah, that's right, it's just me.

Floyd is somehow brave enough to just let me run the show today, on my own. So hang in there. I have some exciting things to talk about when it comes to AI pilots and stage gate governance. Now, the issue that keeps coming up with organizations — or rather, that prevents them from actually scaling their AI investments...

And by the way, I'm talking to a bunch of you right now who are stuck somewhere between a really slick prototype and what I would call an enterprise-grade system. So today I want to talk to you about how to get the most out of your AI pilot. And I'm not talking about picking the right tool or finding smarter people to come in and consult.

I'm talking about process, I'm talking about governance, and I'm talking about the framework that turns experiments into production systems that your department can run on and, most importantly, can rely on. By the end of this episode, my goal is that you'll have a clear roadmap — six gates, to be specific — that will take your idea to production-ready in a way that actually works.

So let me start here. What's actually stopping you from getting an AI pilot to production? Most people think it's a technology problem. But here's the thing: it's not the technology. It's not that you don't have the right people. The real problem I see is governance, or rather the lack of it. A missing roadmap.

It's the fact that many teams are running pilots in complete silos. You've got the data team experimenting with one thing. The marketing team's got their own little pet project going over here. Someone in operations is testing out a new software solution, and nobody's talking to each other. Nobody's got a clear picture of what's actually working and what's a waste of time, and nobody knows how to really move forward.

And look, the thing is, the people building these pilots are smart. They know their domain. They see the friction in their operations better than anyone else, and that's a huge advantage. But if you don't have guardrails around the innovation that's taking place at an incredible pace, you end up with organized chaos, or maybe just chaos.

Experiments that take forever, conflicting priorities, teams duplicating their work, and worst of all, good ideas that just die because nobody knows how to move them forward. So that's the real problem. But I'm not here to talk about problems, I'm here to talk about solutions. And here's the most counterintuitive part of what I'm going to say: governance sounds like restriction, it sounds like bureaucracy, but you need to empower more people to experiment, not fewer.

So what do I mean by that? In my experience — and of course nobody is incredibly experienced at AI, it's all so new — what I've seen is that the most valuable ideas are going to come from the people closest to the problem. The person in operations who sees the exact moment a manual process breaks down. The data manager who knows exactly where your team's wasting hours every week.

The commerce manager who understands the friction in your customer experience. The data person doesn't see the customer experience friction. The marketing person doesn't see what's happening in the supply chain. These people see things that, frankly, I'll be honest, executives miss. And when you give them access to tools like OpenAI or Anthropic, and you teach them how to use these tools effectively, you're unlocking a ton of potential that was already sitting in your organization — they just didn't have the technical capability to make it a reality.

So step one, in our experience — and we've done a lot of this internally here at Pivotree — step one is training. This is where we started. We need to invest in getting our team comfortable with AI and the AI tools available to us. Not everyone needs to be a prompt engineer or, you know, building with Claude Code. Not everybody's a data analyst. But your subject matter experts, the people who understand the business problems best, should absolutely understand what's possible.

They should know the best practices, and they should have permission — and in fact be encouraged — to experiment. This is about creating a culture of innovation, and a lot of businesses aspire to being cultures of innovation, but aspiration and reality are often not the same thing. When you democratize access to these tools, when you give people time to experiment, you're going to get more ideas, better ideas, and ideas that actually move the needle on business outcomes. We always say we want to spend more time working on the business than in the business, and most executives, in my experience, take this burden on themselves.

They try to solve for their teams. But hear me out: what if you let your teams solve for themselves? That's the power of AI when it meets the people on the ground who are actually doing the work. So you can't just run wild. You might hear me say, "Oh, open up AI, let your team loose."

But here's the problem with that: you're going to have 50 people running experiments with AI all over the place, and nobody's coordinating, and you're not going to get efficiency. You're going to get a ton of redundancy. You're going to get abandoned projects that people have burned a lot of hours on, and you're going to burn through tokens like crazy.

And you're going to exceed your budget, and you're going to be wondering, is AI actually generating a return on investment? So once you've created this culture where people can experiment, you have to put some guardrails in place. And I know what you might be thinking — again, governance sounds like restriction.

It sounds like bureaucracy. It sounds slow, and we don't like that. "Governance is going to kill innovation." Wrong. That's not what I'm talking about. What I'm talking about is a stage gate process — a way to effectively move ideas from concept to production in a structured way so that you can drive innovation.

I'm not talking about death by a thousand meetings, but I am talking about a clear roadmap and a process for getting stuff into production. And I'll tell you why I know a little bit about this. Way before the days of AI, and even a lot of digital technology, I was leading R&D at a manufacturing company, and we did this.

This was the stage gates of innovation, and it actually accelerated our time to market. It created clarity — everyone knew what the path looked like. Here's how it worked in practice: ideas would come in from everywhere, from the sales team, from production, from partners, distributors, as well as end customers, and they had a sponsor.

Usually it was someone from the business who saw the problem and believed in the solution. That sponsor would work with our R&D team to build out the basic business case and a working prototype, then we'd present it. This was the first gate. This is where the sponsor — in my case, the head of R&D — would come up and say, "Here, this is a problem. Here's a solution. Here's how we build it."

And at this stage, an idea doesn't have to be perfect, really. It's the proof that the idea actually works, and the business case doesn't have to be super precise either. You don't need to drag in a marketing director at this point and determine your total addressable market, or serviceable market. You don't need to bring in an efficiency analyst. This is really a rough draft. And here's the thing — back then, it required somebody like a head of R&D to be able to interpret this, build that business case, and present it. But today, this subject matter expert, or your sponsor on the ground, can use AI to do a lot of this themselves, so you can actually further streamline this stage gate process.

But once you've got that, you hit this gate, and this gate is a checkpoint — maybe the most important checkpoint. Because before this, it doesn't take a ton of effort. You could spin up an idea, a prototype, a business case in a day. But what happens next can burn a ton of time and resources.

So this requires someone reviewing it: okay, this is worth taking to the next level. It gets a little formal. And what I suggest is you bring in a council — key people from different parts of the organization, so you get a 360-degree view of the business.

What I would do is actually present the prototype and the business case alongside the sponsor. And honestly, it was like Shark Tank. We would come into a boardroom, and in manufacturing, people could actually touch the thing and test it and feel it and see it.

But in digital commerce, you're doing a demo — a solid demo. You have to be able to sell your pilot or your prototype, and you need to be able to present the business case with real metrics. Again, not perfect, but hours saved, costs reduced, revenue increases — whatever is relevant.

This is the second gate. This is where you get budget approval — not just dollars, but also the tokens you're going to need to move forward, and the internal resources and time you're going to need to actually do a real pilot. So the pilot is the next stage in this process, and here's where a little science meets our commerce AI experiments.

A good pilot is a controlled science experiment. You're testing your solution against some sort of control. So if you're working on product data enrichment, as an example — and I see this all the time — maybe you run 10,000 SKUs through your AI solution and through your traditional process.

You compare the results. You grade them. You quantify everything: quality, time, cost, accuracy, whatever matters. I've seen experiments done really well, and I've seen experiments done pretty poorly. But let the data determine the success of the pilot. You're gathering all of this real data and feedback about whether this actually works in the wild.

So the pilot is the third gate. And now here is where you're going to present the results back to that council of executive leaders. The council's going to look at the pilot results: are the numbers what you hypothesized in the beginning? Is the solution working the way you said it would?

Is it actually better than the alternative? Sometimes the alternative is the right answer, and you can save a tremendous amount of friction, headache, and time if you kill a project right now. So if it's a platform or a license you're going to have to buy, at this point you're ready to make a financial investment.

So this is the fourth gate. Gate five is really about production readiness. So you proved your pilot worked, and now it's time to actually put this into production. And this is where a lot of organizations trip up. Honestly, we get here, and they think that production readiness is just, "yeah, it works."

Great. But there's a lot more to it. First of all, is the process clearly documented for the people who are actually going to use it? If someone new joins the team, can they understand, in a few days, how to use the system? And secondly, is it integrated into your existing systems?

What good is an AI tool if it's not connected with the legacy systems and processes you already have in place? Does it play nicely with the other tools and workflows, or is it some standalone thing that people have to use separately and, at the end of the day, doesn't really move the needle in terms of production value?

Third: is it supported and maintained? Who owns the solution? If you're potentially buying a SaaS solution, this is easier to answer. But a lot of teams are building their own tools in-house these days. In a SaaS solution, security, hosting, and support — those boxes are kind of checked.

But when you're building your own stuff, this actually becomes very, very important. Is it supported and maintained? Do you have contingencies internally, and SLAs? Who owns the solution — who owns the fixes when things break? Who handles the upgrades?

Who handles the requests that come in at nine o'clock at night? Those are really important. And here's where I'd say a lot of people are anxious or hesitant about implementing these ideas into production, and that's security. Security is so important because any system worth its salt in your organization is going to carry and utilize sensitive customer data, product data, supply chain data. This stuff needs to be locked down. Is it complying with your data governance policies? You have data governance policies, right? Tell me you do. But these are the things you should be thinking of. This is the fifth gate, and what you're actually asking is: is this thing ready for a production environment?

Now, if every question is answered — really, I think of this fifth gate as a checklist. These questions have to be answered, check, check. But more than the questions being answered, are those issues documented? Is there clear documentation and process that go along with it?

All right, we're at the last gate. Now, if you made it this far, you have proved that you have an idea that meets a true business outcome, that solves real problems, has a defined process, has been proven in a pilot scenario, in an experiment, and has demonstrated the ability to have a return on investment.

It has been documented, processes are clear, all of that is in place. But once you get to gate six, this is about launch strategy and adoption. At this point, you're going to need an adoption plan. How are you going to get people to actually use this? What training do they need? Do you have success metrics?

How do you know this solution is working? How do you measure adoption? How do you know if you're actually solving the business problem you set out to solve? The answer is that you need a feedback loop. Your users need to be able to tell you what's working and what's not. You need to maintain a connection with that AI council who approved the budget and is ultimately responsible for the outcomes, because this isn't a set-it-and-forget-it solution.

This is the sixth gate, and it's all about launch readiness and ongoing improvement. All right, I said there were six gates. There's technically a seventh gate, but it's not really a gate — it's more about ongoing operations and continuous improvement. In this stage, you are monitoring, you are maintaining, you are collecting feedback, you're making improvements, and you keep on innovating because — and I want this truth to drive home — we're living in a world where things are evolving so quickly.

Whatever this solution is, your first solution to a particular problem will not be the best solution, and it will certainly not be the last solution. But if you've got this process in place — this stage gate governance process for AI — you can learn from it, you can iterate, and you can move faster on the next experiment.

Now, some of you might be thinking, "Ooh, that's a lot of gates. Matt, that's quite the mouthful for me to digest." This sounds slow. This sounds overly processed and engineered. But trust me, here's what I've found: it's way faster. It is faster because having 50 experiments running that nobody's managing, that nobody's sponsoring, and then if one of them somehow works, nobody knows what to actually do with it, is that better, or is it better to have a clear process where everyone knows what success looks like and what the path forward is? The gates are not here to block innovation. They're here to accelerate it by creating room for creativity, clarity, and alignment. And here's the last thing: once you've done this two or three times, it does get faster, believe me.

Your council gets better at evaluating ideas, your pilots get more efficient, your team understands what production-ready means, and you start moving good ideas from concept to production in weeks and months instead of years. So in summary, here's what I really want to drive home. The thing that's holding back a lot of organizations from getting real value out of AI isn't the technology.

It's impressive. It can do a lot. It's not that they picked the wrong tool or the wrong platform. It's that they're not thinking big enough. It's that they don't have a clear process for moving from experimentation to production. And it's not hard to build this.

You don't need an expensive AI firm to come in and build this for you. You don't need special software. What you need are four things. Number one: train your team and give them permission to experiment. Number two: set up simple stage gate processes with clear gates and clear criteria for moving forward.

Number three: make sure you've got cross-functional buy-in and executive sponsorship. Number four: treat your pilots like science experiments. Measure everything and let the data drive your decisions. Do those four things and you've got a machine for turning ideas into enterprise-grade, production-ready value multipliers.

Look, if you're sitting in a meeting right now and you're talking about an AI pilot that you're stuck on, if you've got a prototype that works but you're not sure how to scale it, if you've got teams experimenting with AI and you're not sure how to manage all of it, this framework is for you. You've got six gates.

You've got a process. You know what to do. The goal is to remove friction from your operations, to turn the great ideas your team already has into systems that actually work and that your whole company can depend on. And if you want to talk more about this — if you want to discuss AI governance, data, and commerce, how AI tools are impacting those departments, if you've got a specific commerce pilot you're struggling with — I'd love to hear from you.

Reach out to me at pivotree.com and we will talk about it. Until next time — and by the way, I will have some more friends with me next time. Thanks for listening.

Matt: Thanks for tuning in to this episode of Data Versus Commerce. New episodes drop weekly.

Floyd: So if you're responsible for any part of how products get from a database to a doorstep, subscribe now on Apple, Spotify, or wherever you listen.