Why AI product data enrichment keeps failing the QA test

Episode 13 - Why AI product data enrichment keeps failing the QA test - Transcript

Matt: Welcome to Data Versus Commerce, where we explore the messy middle between database and doorstep. I'm Matt Johnson.

Floyd: And I'm Floyd Blaikie. Let's dig in.

Floyd: Okay, Matt, I don't want to lean too heavily on references to the video portion of this, 'cause I know a lot of people are listening in audio only formats. But for those lucky folks who are watching this on YouTube, you've got a new look, and let me tell you what I think you look like. I think you look like Hopper from Stranger Things, in a good way. Tell me about it. What are we doing here? Did you freshen up?

Matt: Yeah, yeah. Well, couple of things. A few episodes back, when we were talking to Dan, I was talking about how AI was helping me find my new look, and sure enough, that has evolved. A few weeks ago, my wife was curious. She was like, "I wonder what you look like with just a mustache."

So she ran it through ChatGPT, came back with something. She was like, "You know what? You should go for it." And I was like, "Well, I can't get rid of the whole beard, but yeah, let's try it out." And it is freaking hot in Central Florida right now, so it kind of works out.

Floyd: Okay, and does your wife like it as much as she did in the mock-up? Has your marriage been enriched by AI, is what I'm asking.

Matt: I love that. Yes, it has, as a matter of fact. So yeah, we're good. You gotta keep... In marriage, guys, I know you didn't come here for marriage tips, but keep it spicy. Spice things up, change things up every once in a while. That's the key to a good relationship.

Floyd: I love it. Well, if you did come to this podcast for marriage advice, I can probably tell you what's wrong is you don't know where to go for advice. But if you want advice on enriching things with AI, this might be the right place. We track a lot of industry trends across manufacturing, distribution, retail, and all I'm seeing is we can enrich your product data with AI.

And sometimes AI gets it wrong. Obviously not with your facial hair choices, it nailed that one. But I have seen AI be confidently incorrect in a lot of cases. So I get a little worried about this, especially knowing that your ability to sell things, to get things sold, is highly dependent on that data being correct.

So what are people buying when they buy AI product enrichment, and is it a good idea, Matt?

Matt: Well, they're buying something that is super valuable because this is one of the most painful tasks in commerce. And especially in B2B commerce, where sometimes you're dealing with hundreds, even thousands, of supplier feeds. You're dealing with hundreds of thousands, million plus, part numbers.

This is one of the most manually intensive jobs, and it's really why there's such a huge gap between the big players, the MSCs, the Amazon B2Bs of the world, and most mid-market distributors who are desperately trying to catch up. So it's smart, I get it. It makes a lot of sense why you would turn to AI and automation to take some of the manual pain out of the process.

So yeah, the claim, the promise, to do something that historically has been very painful, is real, and I totally get that. And for a lot of folks that are listening, probably don't know, but I actually didn't start off as a podcaster. I started off doing this work. Even in college, I was scraping — and when I say scraping, I mean copy and pasting — supplier data out of sales sheets into spreadsheets and ultimately into a PIM.

So I know firsthand what's involved, and we've made a lot of advancements. There's a lot of technology out there that helps streamline the process. But at the end of the day, we're seeing that customers are still struggling with AI enrichment, or enrichment in general.

Floyd: So make that real for me for a second. Tell me about back in the ye olden days when we had to copy and paste things out of spreadsheets. What did that look like for you? How long did it take, and how long would it take now? What would it look like with one of these tools?

Matt: Yeah. So what we had to do first of all was, you kind of had to be experts in the product categories that you were building. You had to understand the different suppliers that you were collecting that raw data from, to be able to essentially normalize it into a structure that could then be published, either on the e-com channel or, back when I started, in your print catalog.

But what that means is you had to understand the different attributes — and when we say attributes, all we mean is the sales information that describes the product. You had to understand those, and they varied category to category, supplier to supplier. And so, first of all, if you didn't understand the products, this was really hard to do.

You couldn't just bring somebody in and say, "This is the process, get at it." They had to do a lot of research and homework in order to confidently map supplier data to the format that your customers needed in order to make purchasing decisions.

Floyd: So now, do you have to still know anything? Can you get an AI tool and not know anything and it'll help you do the thing? Or do you still need some sort of human information floating around in your brain to make this work?

Matt: Well, that really gets at the heart of what I would say if I was speaking to a distributor or even a retailer about this very topic. The misconception I think we have — in every area where AI is gaining traction, and this is certainly one of those areas, I see new AI enrichment tools pop up all the time.

One of the things a customer recently told me on a phone call was, "Enrichment is now table stakes." "Enrichment is a commodity." He said, "PIMs do it, offshore teams do it, AI enrichment tools do it, even the birds and bees do it." It made me laugh, but it made me also think, man, you're right, everybody is claiming to do AI enrichment.

But there's always a shadow side to the AI hype. And most of what we're seeing is frustration around why some of these tools — whether you're using it inside your PIM or subscribing to a very specific enrichment platform — why they are disappointed, and why a lot of these experiments simply don't pan out.

I have a story about one of these scenarios. I've actually had several conversations over the last few months with different distributors who have run AI enrichment experiments, and one of the things that keeps popping up is the lack of confidence in the output.

So what does that mean? That means when I import my file and say, "Go enrich this product data," it will do it. And when you see how fast it's able to pull data — skipping all of that copy/paste work that I used to do — it gets really exciting, and you're like, wow, this is amazing.

Then you open up the file and start QA-ing it, and you realize, okay, this isn't quite right. It filled in a lot of gaps, but I am starting not to trust this. And as you go through your review, you start realizing, oh boy, there's a lot of information here, and that means I have a lot of review ahead of me, and a lot of corrections to make.

So what we heard was that the net gain, in terms of total cost of ownership for these solutions, was a wash. At the end of the day, they've spent as much time doing the QA and correcting the source data as they did before, when they were doing it one supplier at a time. And that has become a bit of a theme.

So recently we heard from a rather large national distributor about one of these experiments. They ran an AI enrichment tool versus Claude, and I love this example because it's so relevant to many, many distributors and data teams right now.

They wanted to know what works best. The AI tool was specific about being an industry AI enrichment tool — in other words, "we do this industry" — and it was industrial supply. Claude isn't marketed as a product data enrichment tool, but the results came back and Claude killed this AI enrichment tool, and it wasn't even close.

Floyd: So you're telling me that the same technology that tells you what to do with your facial hair is better at enriching SKUs than a purpose-built tool? What does that even mean for the industry? That's nuts.

Matt: It really is crazy. But when we dug in and learned more about why that was, it came back, ironically enough, to what made product data teams successful 20 years ago. It's context, Floyd. It's all about context. It's context, it's governance, it's process, and no AI tool, at least today, is able to do that for you.

The way it worked well for this company using Claude was that they had category managers who understood the products and understood the suppliers, the source of the data. And they were able to build very thorough, very strict contextual documentation to feed Claude. They had a very clear set of attributes and potential attribute values.

So Claude couldn't hallucinate values or attributes. They prioritized those attributes — "This is absolutely required, tier one attributes. These are nice to have, and these are even nicer to have. But this product data is not complete until these attributes are filled out, and here are the ways you can fill them out.

And if you deviate from that, your task fails.

Floyd: Oh, wow. Okay.

Matt: And so, yeah, because what Claude is really good at doing is following rules. And this is probably one of the insights that people in the industry can take from this: governance, the very thing that made catalog data work years and years ago, is the very thing that's going to make these AI tools work.

So what does that mean? It means I don't create attributes that aren't approved. I don't fill attributes in ways that are not consistent with my style guide or my normalization standards. When you have that kind of structure, and you've built out a workflow for Claude that complements your real human experts, now you've got something that can really work.

So that was the takeaway: context, rules, governance with Claude seem to work better than AI tools who claim to know the industry, who claim to know process and attribute fill.

Floyd: I think that really illustrates two different ways that people think about AI, and one of them I think is wrong: is AI a tool that you have to use like any other tool, or is it a magic wand? Honestly, I have this argument the most with my seven-year-old, because she'll ask me something like, "When does the ice cream store open?"

I'll say, "I know that it opens at 4:00. It opens at 4:00. We can't go till 4:00." And she'll say, "No, you should ask Gemini. You should ask Gemini." And I think she honestly thinks that A, it knows everything, and B, it can also magically make the ice cream store open at noon. It's just this mentality where people think it knows things, but it doesn't.

AI doesn't know anything. It's not a person. It's not a brain. It is a tool. So it sounds like what you're saying is that you have to put the tool in the right context. You have to use it the right way. You can't just buy something that has four manufacturers slapped on the title and expect it to do the right thing.

Matt: Yeah, exactly right.

Floyd: I'm just having this ice cream argument way too much, I think, and that's really—

Matt: —sounds pretty great right about now. But yes, you are correct. Claude cannot make the product data automatically clean, consistent, complete, ready for the digital channels. But what it can do is fit inside of a very well-structured product data process.

And if that structure's in place, if the rules are in place, if you know your stuff, Claude can really speed things up. And here's the thing — I don't want to throw a bunch of shade on these AI enrichment tools, because they require the same thing. Even in our work, and we do this sort of thing all the time as well, this is one of the differentiating factors between the way that we do product data and the way that some of the tools, or the teams doing it themselves, are handling it.

We always start with that context. Even way before AI tools were a thing for product data enrichment, we always started with the context — the SDS work, which we call Strategic Data Services, that defines what that schema is, what those rules are, what's acceptable, so that you can get a file back that you can trust.

And then, of course, there should be gates and governance about what gets published, but that's the starting point — that foundational work. And even really successful distributors, hundreds of millions and billions of sales, struggle with this, because this is an area where there's a lot of internal conflict and friction.

People can't agree on how to standardize a product category or a part type. And if you can't even agree on that, then what luck do you have in terms of feeding AI the context that it really needs?

Floyd: So what do you want me to take away from this? If I'm in charge, if I've got 400,000 SKUs, they're my babies, I'm worried about them, they keep me up at night — are you saying don't bother with any of these enrichment tools, put your best and most senior data person alone in a room with Claude and let them hash it out?

Or is there something else I need to be thinking about here? Is that going to work for me or what?

Matt: It might not work, and the reason why is, first of all, that person will go insane. They're going to be out of here in three months. But the reason why is because it really starts with strategic conversations at the senior level. And this is what I see as well, in addition to some of the tactical things around enrichment that we covered — it's even more important for you to get aligned with your director of sales and your head of marketing and e-commerce to understand where the data lives downstream, because you're going to reverse engineer the context and your governance practice around the downstream channels.

That could be your e-commerce platform, but it could also be your CRM. It could also be a marketplace channel. And those channels really determine how you're going to create that structure and that documentation. So it starts with strategy, and then it's really important that you lock yourself in a room with the stakeholders who are responsible for both your upstream channel partnerships as well as those downstream channels, and determine what does great look like for us.

Not what MSC does, not what Amazon does, but for our customers and our sales team — how do we create the definitions that are going to drive our product team and the tools they're using?

Floyd: Sounds like it's context dependent. You like that? Was that good?

Matt: 100%, 100%. And I can always count on you, Floyd, for great puns. But—

Floyd: —thank you.

Matt: —yes, it is. And this is really where, if I'm honest, it all starts with a conversation about strategy. So if a client comes to us, we're not going to just throw out a quote on what it would cost to enrich your product data.

We're going to ask questions to understand where you are with the strategy, your downstream channels, your documentation, that context. That's the hard work that these AI tools can't do. And it's the work that most teams haven't done, and it's what we happen to specialize in.

Floyd: Yeah, bottom line, there's still a ton of work that has to be done before you can pull in a tool that might seem like a magic wand. And that's how you know you're going to be successful — putting those guardrails in place up front. Okay, I'm going to go ask Claude if I should grow my bangs out or not, 'cause seemingly, in addition to beating the specialized tools in AI product data enrichment, it makes great hair decisions.

So I look forward to talking to you about what it has decided. Thanks so much, Matt.

Matt: Let's go. Yeah, mix it up. Can't wait.

Matt: Thanks for tuning in to this episode of Data Versus Commerce. New episodes drop weekly.

Floyd: So if you're responsible for any part of how products get from a database to a doorstep, subscribe now on Apple, Spotify, or wherever you listen.