AI can connect systems fast, but only if the data is right | Ep. 2

Episode 2 - Integration - Transcript

Welcome to Data Versus Commerce, where we explore the messy middle between database and doorstep. I'm Matt Johnson. And I'm Floyd Blaikie. Let's dig in.

Floyd: Okay, Matt, we're back at it, and it's so weird that we're both wearing the exact same outfit we wore last time. Did we plan this? Do you have other shirts? Because I don't, this is my only one.

Matt: I'm pretty much like Steve Jobs. Make it simple, I don't need to think too hard.

Floyd: Perfect, that comment was not for people listening on Spotify or Apple, this is only for the video viewers, thank you. So just forget I said that if you're not watching us. Okay, so last episode was mainly me complaining about my dishwasher, if we're being frank. If you're a consumer like me, and not a licensed dishwasher technician, trying to find the right part is a difficult process.

You have to look at some really small numbers and do a lot of googling. You think you found the right one, it shows up, and it's actually the wrong one. Very frustrating. The dishwasher is still broken, and to really put the cherry on top, they promised me overnight shipping, I paid extra, and it came three days later.

So what is the problem here? We talked about data last time, but you were a little coy about the fact that the shipping problem was something else entirely. Do you want to talk about what that might be?

Matt: Yeah, let's go. We talked about pointing fingers, I love this, I love pointing fingers. So yes,

Floyd: it's very possible that this is a back-end, invisible, broken-pipe issue, an integration issue essentially. Systems have to talk to each other to make this whole miracle of digital commerce work, right?

Matt: We often talk about product data, and I think that's the most obvious thing: "Oh, that's not describing what I needed, that was false advertising." But what we often don't think about is all the different systems that have to work together to create a frictionless customer experience. So what does that look like?

In your situation, you were sold a bill of goods that this product could ship overnight and be delivered. That's what the e-commerce system told you. But the reality inside the distributor's ERP or OMS was different.

Inside there, they have the ability to identify warehouse location and inventory levels, and often that information never actually connects to the front-end customer experience. So if by default they have a mode that says a product is in stock, that's what they know, so they say, "If it's in stock, it ships overnight." But what they may not know, and what you certainly didn't know, is that the product was actually in stock across the country, in a different warehouse, and that made a huge difference in how quickly you were able to get it.

So location, distribution center location, that's a data point that lives inside a back-end system. The number of products on the shelf, and where that shelf resides, all play a big role in how quickly physical products actually get to where they need to go.

Floyd: Now, we're not talking about one or two systems either, and certainly not simple ones. I've worked with two different ERPs, some other software organizations, and now with an organization that brings all of those together and makes them work. These are complicated, and the data passing from one to another isn't always in the same shape, if that makes sense.

It's not like smooth plumbing where the water is the same shape the whole way through. It comes out as steam and you wanted it as ice, my metaphor is falling apart, but you know what I'm getting at. There's a lot of complexity in these platforms and how they talk to each other.

So that's why I didn't get my dishwasher part overnight, so I could be disappointed faster. Instead I had to wait three days to be disappointed. So, okay, whose fault is that? Who can I yell at?

Matt: Well, again, we could complain about the logistics carrier. We could complain about the warehouse, maybe those guys were on a smoke break for two days, or maybe the warehouse burned down. At the end of the day, it's almost always an issue with systems, multiple systems, because you don't become a national retailer or distributor unless you've built on a lot of complexity.

What that could look like is maybe you've gone through thirty acquisitions in the last three years, and every time you make an acquisition, that comes with new distribution centers and new capabilities. Now you're able to reach your customers faster, with more products and new lines you didn't have before. That's a great growth strategy, and companies have become incredibly successful with that approach. But what we often forget is the significant tech debt that comes along with it.

For example, there's an electrical distributor I worked with in the past who was growing very quickly through acquisition. They were buying up different mom-and-pop or regional distributors and treating them as distribution centers. For that mom-and-pop shop, serving a region like Northwest Florida was their whole business. But to the larger distributor who purchased them, that's just one more distribution center they can use to get products to customers quickly.

What's the problem with this? The problem is that they've now inherited an ERP system and a back-end inventory system with different data than what they use in the rest of the business, or in their master data. So now they have to somehow take all of this disparate data from different systems and get it to talk to one front-end customer experience. It's very difficult.

And who knows, in your situation, you might have had a back-end inventory system that wasn't even plugged into the e-commerce system. So you were told one thing, but if it had been integrated correctly, that information would have been accurate.

Floyd: That seems like a bit of a nightmare, because you think acquiring an organization is such a great success indicator, you're growing, you're expanding your business footprint, and then you inherit all this other stuff that has to plug in. My background is more in marketing, and I remember working with an organization, helping them connect some of their information systems, not necessarily related to their supply chain.

They had grown a lot by acquisition, and, I don't want to give any free promo, but let's be honest, Salesforce doesn't need any help, they had seventeen Salesforce instances. Connecting all of those, and making them all work together because for some reason they absolutely needed to keep them all, was a huge nightmare.

So what does that look like on the ground when all of a sudden you're inheriting all these other systems and they have to work together, or some middle-aged lady in Canada is going to get mad about her dishwasher? You know what I'm saying?

Matt: Exactly, yeah, I do know. It's a mess, let's be honest. It's what keeps a lot of people who wear pocket protectors in business.

Floyd: I forgot mine.

Matt: You have to be. I used to run an e-commerce agency in a past life, and we had nerds and we had ninjas. The nerds were the people who really understood system architecture, the coders and architects who built these complex commerce systems. You'd like to think it would be simple: I have an ERP over here, a PIM here, an e-commerce system here, an OMS, and data just flows simply through it.

It's not like that at all. It's a twisted ball of yarn, is what it is. And the reason is, if I buy a new business, I can't just shut down the software that runs that branch or that distribution center, the business would crawl to a halt.

So you might wonder why they have all these systems, it doesn't seem to make sense, but it's because that particular part of the business depends on it.

Floyd: Right, it's load-bearing.

Matt: Yeah, exactly, you remove it and the whole thing crumbles. So what's the answer?

Floyd: What is the answer, Matt? I need you to tell me. My dishwasher still doesn't work, come on.

Matt: The answer is simple, but it's not easy. The simple answer is that everything has to connect. All the systems have to connect, data needs to flow between systems and get to where it needs to go. But the hard part is that you have to think strategically about your system architecture.

You need to pull it all up on one giant flowchart and figure out where things are broken, where things are being patched together with crusty middleware you've had for ten years. We use the term "brittle integrations," meaning they're likely hard-coded and dependent on someone in IT to maintain them. And then, of course, there are systems that don't even connect.

So it starts with mapping it all out like a true master architect, defining where things need to go and how they'll connect, all the different data points and how the APIs will work together, and then making sure there's a plan to maintain them, because systems do change, and ERPs in particular are famous for this. You make one little change in an ERP, and it has to be custom-built. That custom-built piece that helped one process run smoothly just broke the integration somewhere else.

Floyd: Right, you change one field type and suddenly everything falls apart.

Matt: Yeah, so somebody has to govern it. Somebody needs to architect it, and then continuously monitor and improve it.

Floyd: But wait, who? When you say someone has to map it out, architect it, figure it out, maintain it, are we talking about the ERP people, the OMS people, in-house people with pocket protectors? Whose job is it to make sure it's all working together?

Matt: There has to be somebody in the organization who has the macro view, somebody who understands at the highest level how everything is operating, what systems are connected to what. That person is really in charge of defining the overall structure of the commerce system.

But here's the problem: you have one vendor who set up this platform and built that integration, and over here a third-party contractor who did this one, and these three over here were done in-house by my own team. So you have a bunch of different types of integrations. They're not all done the same way, they're not all maintained the same way, and I have middleware I have to log into and use.

So the problem is, how do I create consistency in my system integrations? I need an integration layer. I need to be able to control my integrations without breaking one piece of the system. This really comes down to having a defined process. It goes back to old-fashioned discipline. Why do people get frustrated in business, in any department? It's usually because things aren't well defined, and there's no clear process or clear outcome. It's organized chaos most of the time, and that's never been truer than in IT and complex commerce systems.

Floyd: I'm sure everyone in IT is nodding along right now, but I think anyone at a complex organization could say the same. So you're layering all this disorganization and unclear ownership on top of more chaos. That seems like a hard hole to crawl out of. How are we fixing this?

Matt: I think one of the most exciting things to come up recently, over the last couple of years, is, and of course everybody knows there's fear around developers and architects losing their jobs to AI, but what AI is allowing us to do is connect systems faster and easier than ever before.

It's really exciting because this is the kind of work, with the amount of data points and complexity involved, that used to take months and months to build. I remember, back around 2018, working on a distributor's commerce system where we had to build two completely different ERP integrations. It took us twelve months to complete and launch the new site, and that's not uncommon, it's not that my team was bad at it, that's just what it took. Now we're seeing AI read APIs and connect data faster than any human developer could.

It's really changing the way we're able to tackle this giant ball of yarn that historically has just been too big and too messy to wrap our minds around. AI integration work is phenomenal not just for making those connections but also for maintaining them, because now we can use agents to tell us proactively when something changes, and prevent something from going down in one of our core systems, our OMS, ERP, or e-commerce system.

Floyd: And that's really exciting, because having a human being in an office maintaining and managing all these integrations is just too much, and it's part of the reason your dishwasher part never made it on time.

Matt: Companies are going to be able to solve this with companies, frankly, like Pivotree, who are leading the way when it comes to developing AI integration frameworks and processes. It's a very exciting time.

Floyd: Let me play devil's advocate for a second, which is, frankly, subverting expectations here, because the girl never plays the devil's advocate, it's always the dude, so you're welcome, just furthering the equality cause here. We talked about how all this data isn't necessarily good, and all of these systems aren't necessarily maintained correctly. Isn't throwing AI at the problem just going to make it worse instead of better?

Matt: It certainly could, because we make an assumption, and I've seen this in the last couple of years. I've seen companies implement AI tools thinking it's going to create a better customer experience because the AI will handle the really hard part of somehow magically translating data and putting it where it needs to go.

It doesn't quite work like that, because those tools, those platforms, those applications and agents, only work as well as the context and data they're given, and they will hallucinate. I've seen order management systems hallucinate. There are a lot of cool things out there, I saw an order management platform that lets a customer or contractor upload a photo, and because the data it was pulling from to draw context was wrong, it misidentified the part. It looked like it was confidently recommending a part and sending it to the shopping cart for checkout, but it was wrong, and that actually creates a worse customer experience.

Floyd: Oh, man.

Matt: It backfires sometimes, Floyd, because the data isn't quite what it needs to be.

Floyd: I feel like we could probably do a whole separate episode digging into the AI stuff, and maybe we should, because honestly, Matt, my dishwasher isn't fixing itself. Can you come over real quick? Do you know anybody?

Matt: No clue. No clue.

Floyd: Okay, we'll put it out to the podcast audience. If anyone wants to help, I'll give you a guest spot, you can be the host, I don't care, I just want a new dishwasher. Anyway, before I give away the entire media property, I guess we should wrap it up.

Thanks for tuning in to this episode of Data Versus Commerce. New episodes drop weekly. So if you're responsible for any part of how products get from a database to a doorstep, subscribe now on Apple, Spotify, or wherever you listen.