Keeping 10 to 15 of every 100 internal AI experiments

Episode 22 - Keeping 10 to 15 of every 100 internal AI experiments - Transcript

Matt: Welcome to Data Versus Commerce, where we explore the messy middle between database and doorstep. I’m Matt Johnson.

Floyd: And I'm Floyd Blaikie. Let's dig in.

Floyd: Hey, Matt, it's Q4. We've talked about putting numbers in front of the board this year, and I gotta tell you something. I didn't see myself as a vibe coder before this, but I got so mad at every reporting tool that is on the market that I did end up vibe coding my own. Am I cool now? What do I do next?

Can I put this on my resume?

Matt: I mean, that's a stretch, but... Cool is a stretch.

Yeah, we're so nerdy that saying cool, I don't know if that's gonna happen. But I will say that you've joined the club. This is the new club, and welcome to it.

Floyd: Yes, thank you. And I will say for the record, my D&D group thinks I'm cool, so take that. And speaking of nerds, we've got a couple on the podcast, so I'm sure they love that introduction. But...

Joel: I think you might be nerdier than me with the D&D reference, I'm gonna say.

Aiden: The bar has been raised.

Floyd: Yeah. I've only been playing it for, like, 22, 23 years. 26 years. So, it's casual.

We read some interesting research from Gartner about vibe coding apps, how it's happening in businesses, and how most of them are going to die or get killed, frankly. So we brought our favourite technical folks on the show to talk about it.

Joel and Aiden, do you wanna introduce yourselves real quick?

Joel: Sure, yeah. Joel Farquhar. I'm the chief technology officer at Pivotree, so I lead or try to lead everything technical.

Floyd: Nice.

Aiden: I'm Aiden Parker. I'm a senior product manager here at Pivotree, been here coming up on two years working on our data management suite of products. And yeah, similarly have found myself over the last year being a self-described, it's called senior vibe coder, citizen developer. I don't know what name has stuck yet. Having anything with the word developer in it feels disingenuous, so let's go with builder for now. Product builder.

Floyd: Love it. Vibe coding as a term has been around since, like, not very long. I think it's a 2025 term maybe. Seems like a lot of people are doing it though, and it has picked up attention from important people, industry analysts. Gartner is talking about it, and they are predicting that 60% of all vibe-coded apps that are running in businesses today will either die or be killed by 2028.

I think that's probably good. Joel, what do you think about that number?

Joel: I'll be honest with you, I'm surprised the number is so low. And there's a couple things that I wanna clarify. One, I think vibe coding for whatever reason, especially more recently, has started to have a kind of negative connotation, right? And I think as we start to think about the future, we really are starting to focus much more on AI-assisted application development and how do we shift from vibe coding, which is really just prompting, same way you might have a conversation with AI to generate applications, to how do we systematically build applications with AI?

'Cause that's really where we're focusing a lot more of our attention. But I would argue that the number is probably low. And I don't know if that's bad, and I'm also questioning how far outside of general software development that is. If we think about software as a whole, software has always been disposable, like for a long time.

I joke anecdotally, I was a software developer for the first 15 years of my career and I've been working for maybe 25 years. Not a single piece of software that I ever wrote is still in production, right? So software's always been disposable. It's likely going to get more disposable because it's becoming significantly easier to produce.

But we also have to look at that number, 60% of what? Are we talking 60% of 10 applications? No, we're probably talking 60% of thousands of applications, which means there are still a large number of applications that are going to stick around a lot longer than two years, right? So I think that's the important distinction we have to make as we start to produce bespoke applications much more quickly.

We're gonna build a lot more of them, and yes, a lot of them are gonna be disposable, but we're going to hit a lot that are going to stick around longer.

Floyd: What ones have you seen succeed? You talk to folks who are on the technology side in commerce businesses. Aiden, I'm sure you do as well. What are folks that run e-commerce building right now, and why are they building it?

Joel: Aiden, you wanna jump in there? I can give an answer too, but go ahead.

Aiden: Sure. Well, I've seen it on maybe two ends of the spectrum. So on one, there's just your common workflows that any commerce company is gonna run up against. So for example, writing SEO descriptions or variations of your copy for product pages. Everyone is going to their favourite tool to try and take that off their plate as far as the process workflow.

But on the complete other end of the spectrum, what I've seen is folks who are operating in really niche environments. So for example, I was chatting with a CIO from a company that, they do equipment distribution for film and entertainment industry. So the way he describes it, there's only one major player in that space that was able to get most of the requirements for their needs in order management, delivery, and tracking, but still not get all the way there, and he didn't have a lot of confidence in them going forward.

So they took on the build of those tools themselves to be able to fill that gap, and had no expectation that anyone else could jump in to do that for them.

Joel: The areas that I'm seeing that outlive are maybe not large, broad applications, but smaller solutions that are generating operational efficiencies within the customers that we're working with. So if there's data historically that had to move from one part of an organization to another, and maybe there was a lot of manual process in there, some of our customers are building applications that are making that more efficient and those are tending to stick around.

And that's kind of where, as our customers and as we internally are experimenting, that's kind of where we're seeing a lot of traction. And so mostly stuff right now about operational efficiency is kind of the stuff that seems to be sticking more than I built this really cool dashboard that allows me to analyze something more efficiently.

Those tend to be like, yeah, they're really great in the moment, but once I've gotten what I needed out of that I don't generally need to return to it. I'm not so sure that's a bad thing though. Building small, quick solutions that allow us to do something in a brief moment and then dispose of it.

So it's kind of those two areas that I'm seeing.

Floyd: And Matt, what are you seeing in B2B?

Matt: Well, yeah, I agree with Joel's premise. And I think it's important to distinguish between where we're spending our time. I mean, are we spending too much time creating disposable applications? 'Cause sometimes it's the right thing to do.

Other times, you're totally neglecting your bandwidth on real areas of friction in the business. And what kind of blew my mind was, earlier in the year, we went to this event for distributors called Applied AI for Distributors, and what was interesting is that there was this explosion, like a Cambrian explosion of order automation software, right?

All of these AI tools which are looking inside of the customer's emails, PDFs, right, spreadsheets. The way that industrial buyers are getting quotes and placing orders with distributors, it takes a lot of manual intervention. I got to copy/paste, I got to do a look up, and these tools are doing a lot of that heavy lift for these companies.

But you know what's interesting is that they had a lot of developers at the show not looking to buy, but understanding the way the technology works so that they can go code it themselves. I don't know if you would say it's vibe coded, but these are senior technology people going and saying, "I think we can do something like that ourself."

And it may be, in our situation, it may be better for us to have a custom application because we have a very unique business model. We have a very unique ERP system. It's not easy to integrate with, right? The data lives in these unique places. So rather than going and buying something off the shelf, we're seeing a lot of development happening inside of the areas of the business where most of that manual effort historically has gone.

Aiden: Yeah. And that piece of where are folks at in their maturity journey as far as their understanding of applications of AI? I've definitely seen a shift more recently in, do folks just have a more general understanding of, "Oh, yes, I can apply AI," or, "Should expect AI to work in this part of a workflow. Oh, but this one in is too complex." So being able to right size and have that instinct, yeah, I've definitely seen a shift in the last few months in how folks are coming to the...

Joel: One of the areas that we see though that's the challenge is people are producing, and we've seen it, like domain experts that have really good knowledge can solve problems and build AI solutions, software that solves those problems. But what they struggle with is organizational adoption.

And so why a lot of these applications don't go beyond a single user is most organizations aren't set up to distribute these types of solutions to a broader set of users. And the creator doesn't know how to do that, so it never leaves the creator's desktop, right?

Matt: It's hard enough to adopt an enterprise software that has hundreds of support pages, right? And you think that your company's gonna adopt an app that you wrote where all of the technical information's in your head. I mean, it's... yeah.

I think that's the new challenge that people are running into as well right now.

Floyd: And should they adopt it too? Because I think one of the things that we skip a lot is that vibe coding or using Claude Code or however you're doing it lets you kind of skip over that hands-on-keyboards, the actual coding part, for better or for worse. People get really excited 'cause they can have an idea at breakfast and it could be built by lunch.

But another thing that gets taken out of the equation, other than just the clanging and banging on the keyboard, is the part where we all sit down with other human beings and say, "Is this a good idea? Should we even build it?" You don't have to do that anymore, and I think that's probably bad. And that's why a lot of these apps should be killed and don't end up living on.

So I mean, are we seeing this? People are just building whatever and it's... There's no use for it.

Joel: But I don't see that as a bad thing 'cause I think the alternative was, sometimes you'd spend a lot of time gathering requirements, writing them all down, then having software developers produce something, only to find out four months later that you're in the same boat. Like, we probably shouldn't have done this.

Right? But you spent much more money. So it's a lot faster now to build something, get it in front of potential stakeholders and decide, is this worth investing in further? So I think that's the good side. So like you said, I think you are gonna get a lot of effort spent on things that probably don't have a lot of value, or you have a lot of people potentially in your organization or across the industry producing the same things, right?

Like massive amounts of duplication. I've spent time vibe coding, over the course of a day or two, something that I thought was the world's greatest thing, and then, after like a day, I'm like, "Why did I spend two days on this?" It's really cool, but I don't really need it anymore.

And now I'm questioning why I ever needed it, so. But I do think... I think the positives outweigh the negatives. As a software developer, I just think the old way, it takes so long to produce and decide whether you've created something of value. Now that's been reduced to almost an insignificant amount of time that we can make those "this is garbage" decision much quicker than we maybe were used to be able to in the past.

Aiden: Yeah. The principles of product discovery are still gonna hold up. And I'm thinking of some of our work of just the pure concept of, "Hey, I have this idea," and I can come back to you the next day and say, "Hey, is this what you were thinking? Is this what's in your head?"

And then within another day, we're taking that into an interview and chatting with folks, and the next day it looks different and we've reacted to the feedback, so that loop is... I mean, to me on the product side, that's just... it's worth its weight in gold. But yeah, if you're sitting and developing for three months and haven't shown anyone, that was bad before. It's still not a great thing.

Sponsor: Pivotree is turning the traditional support model on its head with our new Intelligent Managed Services. Instead of reactive support and endless ticket queues, you get continuous monitoring, strategic guidance, and regular proactive health checks, all for a predictable monthly cost. Intelligent Managed Services are now available for Stibo STEP customers.

Start with a free Intelligent Health Check. Visit pivotree.com to learn more.

Matt: I think about our journey internally, right? And all of the experimenting that we've done over the last year. And now we have a much more governed, structured process for bringing apps to market or getting them in the hands of customers or... Or internal users.

The thing that I always go back to in those early days, and I think a lot of folks are working through this challenge now, is, I thought of the quote from Dr. Malcolm from Jurassic Park. He said, "You were so preoccupied on whether or not you could do it, you never stopped to think about if you should."

And I think that's a really important thing because there's limited resources, and there's only so much time, and there's only so many things that you can get people to buy in on. And so, I think the discipline of determining what is a disposable, this is gonna be good for three months, six months, versus this is a very high priority use case.

How is that governed, right? What have you guys learned? I'd be curious. I've seen the advancement, but I'm sure you've learned a ton about governing experiments and making decisions.

Joel: The reality is if you look, we allowed unfettered access to AI in the first half of this year and maybe slightly dangerous from some people's views, right? Just allowing the whole organization to experiment, right? And like you say, we had hundreds of experiments get registered and executed on.

The reality is what came out of 100 of those experiments, I would say 10 to 15 viable or valuable pieces of output. So not a huge percentage, but I could also argue that's the most innovation that Pivotree has created in its history because of the ability to produce so much and decide quickly, like, you know what? Those things are all not worthful, or have worth, but these things here, these have value. Here's where we should invest future dollars.

What then we struggle with this, the second half of the year, is how do we put structure around that? How do we make sure things are built properly, that they can scale, that they're secure, that we can deploy them, that we could manage them, that we can get adoption?

So we've been working heavily on adoption programs internally. How do we turn that one person, Floyd built it on her desktop, into we want the broader organization to leverage this tool. Or better, we want to get this tool in front of customers because we see value in customers leveraging these tools.

So that in the second half of the year, we allowed a lot... First half of the year was experimentation, second half of the year is how do we put structure and how do we build that into a repeatable, executable sort of methodology that we can leverage internally, but then also better, we can take to our customers and teach them how to do it.

Aiden: But Joel, you said registered early, like, experiments registered, and that does speak to, there's right-sizing the governance at each stage in the process. Maybe, yeah, go 30 more seconds on what you meant by registered experiments, 'cause it wasn't that we just threw open Pandora's box and said, "Hey, go..."

Joel: Yeah, we did create a registry internally where folks within our organization, they did have to register it first and say, "Hey, I have this idea, where I think this would be valuable."

And this is... They were asked to describe it. There wasn't a heavy lift of what it would take to register it 'cause we wanted to make that easy.

But we wanted visibility into what everybody was working on. We also built some tools in there to, like, using AI also to identify, were people trying to solve the same problem? 'Cause we did wanna try to avoid duplication within our organization, so we did allow, say, "Hey, I have this idea," and then it says, "Hey, you know what? These other people are maybe doing something similar." But it was a very lightweight registry where us and leadership and other people and the organization as a whole, 'cause it was a wide-open registry, could get visibility into what were the different challenges that people were trying to solve.

And some of those were small. Some of them were very large and ambitious, maybe overly ambitious. And then over time we learned the registry wasn't enough. Like, how do we make sure, especially the stuff we wanna promote into viable solutions, is we layered on sort of a certification process to that registry as to say, has this gone through certain stages, in order to, is it valuable to promote?

And we allowed people, some people didn't work on things for extended periods of time, so we exit it from the registry, right? If people are excited about something one day but then weren't willing to invest in it further, then it's not gonna hang around, so. But it was really a way for us to get visibility, and yeah, I would recommend anybody who's gonna go through massive experimentation to create something similar and to get visibility on what people are trying...

Aiden: Yeah, 'cause I don't know if we ever named it this, but essentially we went through a process of process mining just to find where are the places that had the biggest opportunity for automation and right-sizing the level of AI investment or software investment, which ultimately became a test bed for the things that we're leaning into more heavily now.

A test bed for how do we start leaning into the forward deployed engineering side of things by using the subject matter experts who were already starting to think about and putting pen to paper on how do I get this done with the tools that are available to me?

Joel: And I think you touched on... One thing we did learn through this process, like I've talked about it, right? Pivotree does a broad set of things as an SI that focuses in digital commerce. There's a broad set of platforms we implement. There's a broad set of services, and then within those projects and services, there's a broad set of roles.

So the experimentation could have varied from how do I do better project management to how do I write software better? How do I integrate two internal Pivotree systems better? So the experimentation was all over roles, all over platforms, but it really came from people who had a domain expertise on specifically the problem they were trying to solve, which I don't have, I don't have that skill set to know every problem we have in our organization, but the people closest to those problems do.

And so one of the things, by allowing those domain experts to really think about the problems they're trying to solve, a lot of our problems got solved bottom up, where the people on the ground were solving a lot of our biggest challenges, right?

But the other thing we learned is we... And we've shifted, especially under my guidance, like with the new R and... We had an R&D group that was heavily focused on much more traditional R&D, which is we have an idea, we have product owners and product managers that are gonna try to scope it.

They're gonna write user stories, and then a purely R&D team of developers is going to try to bring it to life, which we know historically is a hard process because often what ultimately gets delivered when it gets back through that process, and it's kind of why Agile was created, is like, "That might have been what I said, but that's not what I wanted," or, "That's not what I meant," right?

Because it's very hard to take somebody's idea, write it down, and then have a third party that really doesn't know anything about the business problem you're trying to solve actually write software to solve it, right?

Matt: Hey, we talk about the game of telephone and product data. There's a game of telephone in software development.

Joel: Right. And now, and I'm a software engineer, taking a lot of the software engineers out of the process or... And now I don't mean totally out of the process because they serve a different role now, but allowing the domain...

Matt: Where they should be.

Joel: Yeah. Allowing the domain expert to really define what it is that they're trying to solve and getting that more instant feedback has been significantly more efficient than what we were trying to do previously. And again, that's why I think it leads to, like, we've innovated more in the last eight months than we did in the history of this...

Aiden: And maybe just to call on the flip side of that as well and some of the success we've seen in bringing our developers and our engineers more into the customer conversation, where it's... To solve that same problem, where we get both sides of the coin now, where the folks who are able to put pen to paper and continue to tackle the most complex problems are able to get closer to the business problem side. So the sort of yin and the yang of the solution experts and the folks that are building the tricky stuff behind the scenes.

Floyd: So if you're a CIO or head of software engineering at an organization where people are right now today vibe coding apps that are touching your commerce stack, is it too late to introduce this kind of governance? Has the cat kind of been let out of the bag? Or can they do something?

Can they take a step today to introduce more governance, more certainty, more process into what's gonna...

Matt: Dinosaurs are already running around the park. They're all loose. They're out of their cages.

Joel: Well, first I would encourage organizations to experiment, right? And again, to move away from vibe coding and start to really learn about AI-assisted engineering because there are starting to become frameworks or organizations that are getting very skilled at building frameworks and solutions that allow for a much more diligent software engineering process using AI.

We might call it agentic software engineering or where we start to think about the software engineering process as a series of agents that are focused on delivering high-quality software, and testing that and building agents that test that software. And so that's where I think organizations need to be focused.

How do you move away from vibe coding and build a process and a structure and a framework for producing high-quality software, right? It's an area that Pivotree's been investing heavily in. Part of what came out of all that experimentation, we're like, all that experimentation was chaos.

How do we put structure around this, and how do we make sure that the output that we get out of the end is a viable, deployable solution that... So we start to think about security. We start to think about observability. We start to think about scalability. We start to think about cost, operating cost, all these types of things that you would think about in a normal software engineering process.

How do you start to think about that more agentically and build tools and structure to do that? And then you have the problem that we're all gonna face and that everybody's facing is, I think I heard a quote from somebody and I'm not sure 100% who to attribute to, so if anybody hears this and it was them that said it, then you can send me a LinkedIn and yell at me.

But the reality is everybody likes to vibe code, but nobody wants to vibe maintain. Which means...

Floyd: Like that.

Joel: What do you do with it? Where are you gonna put it? Who's gonna make sure, if your business is now gonna be dependent on this solution, who's gonna make sure it's up and running 24/7 if that's what's needed?

Who's gonna... Where's the maintenance, as we... If you wanna modify it and deploy new versions, and how are you making sure potentially if it needs to be high availability? These are all things that don't go away just because you vibe coded it, right? So I think organizations are either going to have to invest in, again, building that side of it, like how do you take these solutions and deploy them in reliable ways and manage them, and wrap managed services around them so that they're being managed similar to a traditional software-developed application.

And so, either large organizations will need to figure that out themselves, or they'll need to go to a vendor that's going to help them on that journey.

Floyd: Well, Joel, Aiden, thank you for bringing your expert vibes to the podcast. This has been fascinating, and looking forward to hearing what else you do in the vibe code landscape. Can we call it that? I've coined that. That's mine now TM. Thank you.

Matt: Yeah, it's great having you guys. Thanks for all your hard work and innovation, in finding out where the walls are so the rest of them don't have to hit them.

Aiden: Well, thanks for having us.

Announcer: Thanks for tuning in to this episode of Data vs. Commerce. New episodes drop weekly. So if you're responsible for any part of how products get from a database to a doorstep, subscribe now on Apple, Spotify, or wherever you listen.