The robots.txt setting that hides your catalog from ChatGPT | EP 4.

Episode 4 - The robots.txt setting that hides your catalog from ChatGPT - Transcript

Matt: Welcome to Data Versus Commerce, where we explore the messy middle between database and doorstep. I’m Matt Johnson.

Floyd: And I’m Floyd Blaikie. Let’s dig in.

Matt: Guys, welcome back. Dan, I’m so excited to get into a conversation with you today about what’s happening in the world of retail and agentic commerce. Before we started, I thought it might be fun to tell you a little anecdote about a real experience I had recently around agentic shopping. So, sitting with my kids one night, I said, “Hey guys, we launched this new podcast.” I was looking at myself on the camera and thinking, “I look so basic — just another bald guy with a beard. What can I do to amplify my style?” And they said, “Well, why don’t you ask ChatGPT?” So that’s what I did. I took a picture of myself, uploaded it to ChatGPT, and said, “Hey, I need to revamp my style — I need a totally different look. What are some things that would work for me?” It gave me back some really good suggestions. One of them was get glasses, but then I thought, I’ll look just like Dan — so I can’t do that. But it gave me good suggestions, and then I said, “Okay, I like this — but where should I get it? I don’t want to spend a fortune, I’m not going to Nordstrom’s, but I don’t want some Chinese knockoff either. Tell me where to get it.” It sent back a list of links to different storefronts, and it got me thinking: how does it know which storefronts to send me to? Why were some visible and not others? I’m pretty sure there weren’t any ads involved — if there were, I’ve got a complaint, but maybe there are. You’re here to fill us in, Dan. What’s up with that? How does this whole thing work?

Dan: Fascinating question. In my role as our industry lead for retail, I’ve spent a lot of time over the last few months trying to figure that out — it’s the question I get from our retail customers. They’re trying to figure out, in the first place, why they’re even getting traffic from ChatGPT, Perplexity, or Gemini, because in many cases, up until a couple of months ago, they weren’t doing anything in particular to drive traffic out of the AI platforms. In other cases, they want to know how to get recommended. It’s a really interesting rabbit hole to lose a lot of your life down, trying to figure out how these large language models determine why they’re sending you to your local Running Room store as opposed to somewhere else. I guess it unwraps in layers. If you use ChatGPT a lot and it’s relatively present on your device, it does keep track of where you go, where you shop, and your preferences. So it’s possible some of those results are based on somewhere you visited recently, in terms of getting a sense of personal preference.

Floyd: It recommended glasses.

Dan: Beyond that, it looks at a number of different components to decide who to recommend, and why. Much like a human shopper, it’s trying to solve a mission — what are you looking for, how quickly can you get it, what’s the best price — and it’s trying to optimize that model. Especially in a general search like the one you provided — “Hey, I want to change my look around, give me some ideas, now give me some stores that represent those ideas” — versus a more specific mission like, “I’m looking for a specific black shirt, check these five stores.” That’s a different scenario. But the first thing it’ll look at is the core product data. Let’s say we’re looking for that magnificent shirt you’re wearing today, Matt — find me a black shirt in this size. Those two dimensions are the first thing it looks for: who has this shirt, in this color, in my size? Then it moves on to price and availability, and trust signals like return policies and certifications about product quality — where it’s made, where it’s manufactured, whether it’s sustainable — different things that drive consumer decision-making. The last thing it looks at — and this became even more prominent in the last couple of months — is third-party references to validate that this store is a place it should recommend. So if it’s someone like Nordstrom, for example, it doesn’t necessarily know Nordstrom’s brand positioning, and it doesn’t care about any of that, because it’s a machine. But it will look for reference sites — say, Vogue, GQ, or Reddit — that say, “Yes, Nordstrom is a high-quality seller, that’s a place people go for good quality, higher-priced items,” and it’ll recommend it because there’s one near you, the inventory is available, and it’s at a price point you’ve indicated you’re willing to pay.

Floyd: I think there’s a really interesting disconnect between what you tried to do with ChatGPT, Matt, and what you were saying, Dan, about how it looks for information. When Matt bought his beautiful black shirt, he probably bought it the old-fashioned way. He could have typed in, “I need a black button-down shirt, here are the sizes and specs,” but he didn’t — he did a vibes-based search. He went onto ChatGPT and said, “How do I make myself look cool?” He roasted you a little bit in the process, Dan — I don’t know if you should stand for this “bald guy with a beard” hate, but you can sort that out offline.

Dan: Takes one to know one.

Floyd: That’s very much a vibes sort of thing — it’s very subjective. I think a lot of retailers have invested a lot in this, because Matt’s probably going to be swayed by beautiful lifestyle photography, Matt’s going to be swayed by a brand story, because he’s trying to become something. It’s inherently an emotional search — sorry, Matt, but it is. So how does that work in agentic commerce search? Do AI agents care about any of that?

Dan: They don’t care so much about it directly — they care about it in the context of looking for a third-party signal indicating that this is a brand or store to recommend. They don’t care that the copy on a retailer’s website creates an amazing emotional association with the shirt and is going to make you the coolest, most amazing person ever — that, it’s not going to care about. But it will look for the way we think of information, and certain adjectives we’d associate with a product that are kind of meaningless to a machine — like “soft,” or “the drape,” things like this. Those become attributes that need to be added into the product information in a way the language model can interpret, so that when the search is something like, “I’m looking for a winter jacket that’s really puffy and warm,” it can figure that out based on a down fill rate and a temperature rating. There’s an interpretation that needs to happen. The other thing — when all this AI stuff came out and we started talking about it, you saw this all over LinkedIn: “data is the key thing, if you don’t have product data, your company will never be found.” That’s only partly true. What we’ve found running our agentic readiness assessment for our retail clients and other companies we work with is: first, make sure your security settings aren’t blocking all robots — robots.txt. There are many sites set up for fraud, denial-of-service, and other cybersecurity reasons that stop Perplexity, ChatGPT, and Gemini from even coming to the site. So the first step is, “I need to let these things in” — while keeping out the malicious actors, because otherwise you won’t even be considered, you’ll just be skipped. The next layer is the product information — the basic unit of measure, UPC codes, GTIN. Looking for a pair of size 8.5 Saucony Gel-Kayano running shoes — yes, this is in fact this product. Then it moves on to information stored elsewhere that matters to completing the mission — inventory availability, shipping costs, taxes, other elements like that. But the emotional elements still remain important, because as a seller, you can train these models to look for you. There are ads coming out too, Matt, to the joke you made earlier — ads are starting to come out, I think on ChatGPT or Gemini for sure. They’ll be at Google eventually, but ChatGPT has started it, where you can influence the models to some degree with ads. But even beyond that, getting the recommendation does require content marketing, because the models look at that content marketing, and you can train the models to look for certain keywords so they’ll recommend you. It takes time, and there are tools out there that help you monitor how you’re doing. The other big difference is: most people, in an example like Matt’s, when going shopping on ChatGPT or doing a search, have a wide-open context and intent, as opposed to an SEO-type search, which is usually very specific to a kind of product, if not a brand or a store. These are intent-oriented — “I’m going to a beach wedding in July, what should I bring?” That’s wide open, and trying to understand whether you’re going to show up in these prompts, and which prompts result in the models recommending your store or brand over someone else’s, has become a big part of what marketing and merchandising teams are starting to figure out.

Matt: That makes so much sense.

Dan: Yeah. And while the percentage growth in traffic is high, what we’ve seen talking to our retail customers is the numbers aren’t necessarily high yet — they’re not selling millions through these channels yet, but it’s growing fast, so they end up with yet another channel to support. At first it was your store and analog advertising, which we all know and love. Then it became e-commerce — how do I influence the search engines to find me and my site, and what do I do on my site? Then we had social channels — how do I sell through and promote myself, either through influencers or ads, on TikTok, YouTube, Instagram? And now there’s another channel, with its own behavior and its own quirks, and things that need to be put in place to influence it. The marketing race just continues and expands, and then it becomes a question of where to spend your dollars and how to get the most bang for your buck. The good thing is that, at least as it relates to on-site information — product data, availability, price — that also helps SEO. So it’s not like this investment only benefits one channel; it helps all of it. There’s benefit to be gained across the effort.

Floyd: Can that be measured yet? My background is marketing, so I can go into a tool and see whether I’m number three or number four for a high-intent search term, but I don’t think there’s really a way for retailers to do that with LLM search yet. How can they know it’s working, so they can double down on that channel?

Dan: There are emerging platforms out there that are starting to measure your recommendation rate — how much you’re getting recommended versus others, where that traffic is coming from, and the prompts that are resulting in your brand or store coming up. Some platforms, like Shopify, have recently released embedded analytics that show your traffic coming from the different LLMs, in addition to all the other channels, as part of that referral traffic. The key missing piece in the journey from an LLM to your store is the scenario where I don’t click through. So trying to figure out the attribution when somebody does a search on, say, ChatGPT, gets a bunch of examples, and chooses to go to ABC store — if I exit ChatGPT and go straight to ABC store, you don’t necessarily know where that came from. In those scenarios, companies are starting to try to figure it out by looking at the shopping journey from their site, trying to gauge it, because people coming from ChatGPT to the store have very high intent — they move down the funnel to convert much faster, and we’re seeing that. Referrals from the LLMs convert at a much higher rate than they do through organic or paid search. But it’s still a bit hard to attribute, and there’s still a lot of effort going into figuring out how you’re ranking, and how to continuously train those models so you don’t fall behind — like we did with SEO, continuously making sure all the keywords that could be out there are there. Now it’s which prompts, and in which scenarios, am I showing up. You can get pretty specific with some of these tools, down to introducing personas and segments to see if your target segment is actually being influenced by the effort you’re putting out. But it is certainly early days to figure out this race.

Matt: Dan, this all makes so much sense. One of the things I was thinking about, hearing you talk about conversion through LLMs, was the fact that as consumers, we’re changing our buying behavior — we’re changing the way we think. For so many years we’ve been overrun by so many options that, at least from my own personal experience, having ChatGPT tell me exactly the five shirts and two pairs of jeans I need to buy is an unbelievable time-saver, instead of spending hours potentially shopping or going to different stores in person. What does this trend toward AI tell us about the consumer, and how do retailers need to think about where to spend their time — particularly around content marketing, since that’s where this context is coming from? How does it know that I like rugged industrial outfits if there isn’t some rugged industrial blog or Reddit thread out there?

Dan: That’s exactly right. I think it’s somewhat category-dependent. Of all the different things we buy as consumers, some lend themselves to this AI agent really easily, because they’re very defined and repetitive. I think of grocery as a prime candidate — my list is already in my grocery app. If it takes me five minutes to place my order every week and have it delivered, that’s already a lot. The extra step of saying, “Order the same stuff, and add a watermelon this week because I’m having a picnic,” all through voice command, is a major time saver for categories that are very well defined like that. There are many consumer categories like that — hardware stores and other things that are less emotionally driven and more specification-driven. Those become very important when it comes to the data piece, and figuring out, as a retailer in those categories, how to make sure you’re the one that gets chosen. In fashion, apparel, home furnishings — things that are more emotionally oriented and have a lot of style and taste to them — it becomes a bit different. But in many ways it comes back to the age-old question of brand. If you’re a brand-oriented company, you’re going to continue to need to invest in brand, brand awareness, and brand positioning, so the consumer actually puts you into the search criteria. You want to be a favorite that’s there and recommended — and if not, it becomes, much like it is today, how do I get into that consideration set, what do I need to know about those models to get there? In some ways, depending on the category, sellers who’ve been selling on Amazon for years have a bit of a leg up, because they’re playing the same game in the LLM models that they’re playing on Amazon — leaving ads aside for the moment. It’s all about how do I get to the top of that list, the content, the product, the trustability, the visuals, and the rest of that in the LLM context — you still need to understand what gets you to the top of that recommendation list. Those who don’t compete in that channel or in marketplaces have that dual challenge of brand awareness and positioning across different sources, as well as their own content on their own website — because I could be recommended, I could have the product, but if I don’t have shipping policies, I’m going to rank lower in the recommendation list than someone with very clear shipping policies that are advantageous to the consumer.

Floyd: The marketer in me loves to hear that brand isn’t dead, loves to hear that content is still important — and I think we’ve barely scratched the surface, especially with the level of your expertise, Dan, and how you can optimize for a kind of agentic search. But let’s meet a lot of retailers where they’re at, because this is so new. I think step one is just making sure you show up at all. So if you’re an e-commerce leader, if you’re in charge of merchandise, if you’re in charge of digital — what do you check tomorrow to make sure you’re showing up in the first place?

Dan: You check the basics of product information, making sure it’s complete in terms of the attributes the models will be looking for to confirm that what the shopper is looking for is in fact at your store. You check the other information that rounds out that decision — price, inventory availability, whether pickup in store or delivery matters, whether store inventory can be surfaced as well. As a second step, start looking at the places you’re going to be cross-referenced, and make sure you’re showing up. Some of it is also geography-based — most of these models are trained in the United States, so there’s much more content about brands in the United States if you operate there. If you don’t operate there, you’re at a bit of a disadvantage in some ways — something to consider for global operators selling into North America. But really it comes down to the basics of product data first, and then the content on your site, and other information we don’t necessarily think about as marketers, that determines that decision-making online — price, availability, and other policies that influence my decision to buy from you.

Matt: Man, this has been so great, Dan. I think we really need you to come back and tell us more about what this looks like in terms of converting — it’s one thing to get found, and then we can talk about the other end of it, in terms of what a good customer experience looks like, what we need on the page to make a decision. But this was so helpful. Thank you.

Dan: It was my pleasure. Looking forward to coming back. Thanks for having me.

Floyd: See ya.

Matt: Thanks for tuning in to this episode of Data Versus Commerce. New episodes drop weekly. So if you’re responsible for any part of how products get from a database to a doorstep, subscribe now on Apple, Spotify, or wherever you listen.