Ad Quality And User Experience: Interview With Google’s Nick Fox And Diane Tang
When we’re talking about the search user experience, we’re always balancing two sources of information, organic and sponsored listings (at least, for now), with the overall objective of delivering the user the most relevant results based on their query. At this time, nobody has done that more successfully, from the user’s perspective, than Google. And the quality of the ads is a key component of that. So I wanted to talk to Google’s Nick Fox, who was joined by Diane Tang from the Ad Quality team. The topic: How Google’s Ad Quality score improves the user experience.
I’ve stated in this column before that Google is obsessively focused on the user experience, which, of course, is music to my ears. In fact, I believe it’s the primary reason that Google has captured the huge market share they have. This user focus permeates every conversation I have with a Googler. Nick quickly assured by that this is not just by chance.
Nick: I think it comes from the top and it comes from the roots. If we were doing a proposal to Larry and Sergey and Eric where we’re saying, “Hey, let’s show a bunch of low quality ads”, the first question they’re going to ask is “Is this the right thing for the user?” And if the answer is no, we get kicked out of the room and that’s the end of conversation. So you get that from the top and it permeates all the way through. You hear it when you speak to Marissa and Matt and us. It permeates the conversations we have here as well. It’s not just external when we talk about the user; it’s what the conversation is internally as well. It just exudes through the company because it’s just part of what we think.
I wouldn’t say that there isn’t a focus on the advertiser too, it’s just that our belief is that the way you get that balance is by focusing on the user, and as long as the user’s happy, the user’s clicking on the ad, and as long as the user’s clicking on the ad, the advertiser’s getting leads and everything works. If you focus on the advertiser’s in the short term, maybe the advertisers will be happy in the short term, but in the long term that doesn’t work. That used to be a hard message to get across. It used to be the case that advertiser’s didn’t really get that. And one of the most rewarding things for me is that the advertisers now see that, they get that. Some of the stuff we do in the world of ad quality is frustrating to advertisers because in some cases we’re preventing their ads from running in cases where they’d like it to run. We’ve seen that the advertiser community is actually more receptive to that recently because they understand why we’re doing it and they understand that in the long term, they’re benefiting from it as well. I think that you are seeing that there is a difference in approach between us and our competitors. That we believe the ecosystem thrives if you focus on the users first.
So, how does this focus on user experience translate into the day-to-day quality scoring of ads? Four weeks ago in the same column I called the Canadian sponsored ads the “slums of search.” In every user test we’ve ever done, the relevancy and quality of sponsored ads are a critical component in the success of that user experience. In less mature markets, such as Canada and, to a greater extent, China, we’ve seen less interaction with sponsored ads, especially in the critical top sponsored position. It’s because in these emerging markets, the quality of information in the sponsored ads just can’t match the quality of information in the top organic listings. Major brands and major vendors haven’t moved into the sponsored space to any great extent so you don’t see the brands you’re expecting to see reflected in sponsored listings. I asked Nick about this.
Nick: I think a lot of the things you’ve picked up on are very accurate. In terms of the focus on top ad quality..in general, the focus on quality..I think what you picked up on in your various reports as well as the study in Canada are pretty accurate and pretty much what drives what we are working on here. The big concern that I would have, the main motivation for why I think ad quality is important is, as a company, we need to make sure users continue to trust our ads. If users don’t trust our ads, they will stop looking at the ads, and once they stop looking at the ads they’ll stop clicking on the ads and all is lost. So what we need to make sure we are doing in long run is that the users believe that the ads will provide them what they are looking for and they will continue looking at the ads as valuable real estate and to continue to trust that. So that is what we are going for.
Another distinguishing factor of Google’s approach to showing sponsored ads has been a less aggressive presentation of ads on the page. This was most notable in our last eye tracking study, where we compared Google against Yahoo and Microsoft. In our study, which involved very commercial terms, we only saw top sponsored ads appear in a little over half the sessions on Google. Contrast this against Yahoo, where top sponsored ads appeared in well over 80% of the sessions and Microsoft where we saw top ads in over 70% of the sessions. Additionally, Google usually showed two or, in rare cases, three ads as opposed to three or four from Yahoo. When you multiply both of these factors together, you’re 2 1/2 times more likely to see a sponsored ad on top of a Yahoo search page then you will on top of a Google search page. (By the way, since the study Yahoo has dialed back a little on the presentation of top sponsored ads) Nick commented on the reasoning behind this more stringent threshold.
Nick: I think as we look at the competitor’s landscape as well, we see a lot of what you see. We certainly have historically, and continue to do so, much more of a focus on the quality of the ads. Making sure we’re not doing things where we trade off the user experience against revenue. We all have the ability to show more ads or worse ads, but we take a very stringent approach, as you’ve noticed, to making sure we only show the best ads that we believe the user will actually get something out of. If the user’s not going to get something out of the ad, we don’t show the ad. Otherwise the user is going to be less likely to consider ads in the future.
But what I was particularly interested in was what was the threshold that dictates whether an ad will sit on the right rail or be promoted up into the Golden triangle real estate as a top sponsored ad. When an ad makes that jump, you can expect the corresponding 3 or 4X jump in click throughs.
Nick: It’s based on two things. One is the primary element is the quality of the ad. The highest quality ads get shown on the top. The lower quality ads get shown on the right hand side. We block off the top ads from the top of the auction, if you really believe those are truly excellent ads.
Diane (Tang): It’s worth pointing out that we never break auction order...
Nick: One of the things that’s sacred here is making sure that the advertiser’s have the incentive. In an auction, you want to make sure that the folks who win the auction are the ones who actually did win the auction. You can’t give the prize away to the person who didn’t win the auction. The primary element in that function is the quality of the ad. Another element of function is what the advertiser’s going to pay for that ad. Which, in some ways, is also a measure of quality. We’ve seen that in most cases, where the advertiser’s willing to pay more, it’s more of a commercial topic. The query itself is more commercial, therefore users are more likely to be interested in ads. So we typically see that queries that have high revenue ads, ads that are likely to generate a lot of revenue for Google, are also the queries where the ads are also most relevant to the user, so the user is more likely to be happy as well. So it’s those two factors that go into it. But it is a very high threshold. I don’t want to get into specific numbers, but the fraction of queries that actually show these promoted ads is very small.
After having sat on a few ad quality panels at various shows, a question that typically arises is trying to balance click through performance against conversion performance. Panelists often mentioned that ads that get a high click through (whether through intentional or unintentional seeding of copy with factors that tend to boost the likelihood of the user to click) do great from a quality score perspective in that they get promoted up the ranks but performed miserably in terms of conversion.
One example that was mentioned in the panel was for a fantasy football league. The fantasy football league was being promoted by ESPN and the advertiser experimented with putting fantasy football as the key element in the title versus putting ESPN as the main element in the title. The version that promoted the ESPN brand got a substantially higher click through rate but the conversion rate dropped through the floor. In understanding how users interact with listings, this isn’t surprising. ESPN is a compelling brand and will get a number of curious clickers clicking through to see what it was about without actually reading the ad content. Whereas the fantasy football clicks would be clicking through specifically to find out more about ESPN’s offering.
We’ve actually seen the inclusion of well-known brands in sponsored ads sometimes works at cross purposes with the actual intent of the advertiser. A compelling brand will often draw “comparative” click just to check with the landing page is about and then the user will click back to the results page and move further down into the subsequent listings, whether they’re sponsored or organic. You’re capturing the click through but you’re not converting on it. It’s just one more factor in the delicate balance that goes through the user’s mind when they’re interacting with the search results page. Nick Fox and Diane Chang believe that with Google’s approach; ultimately the market will correct itself.
Nick: I think there are two things. One is, in general, an ad’s that’s being honest, and gets a high click rate from being honest, is essentially a very relevant ad and therefore gets a high click through rate. We’ll typically see that that ad also has a high conversion rate. In cases where the advertiser’s not being dishonest, the high click through rate is generally correlated with a high conversion rate. And it’s simply because that ad is more relevant, it’s more relevant in terms of getting the user to click on that ad in the first place, it’s also more relevant in delivering what that user is looking for once they actually got to the landing page. So you see a good correlation there.
There are cases where advertisers can do things where they’re misleading in their ad text and create an incentive for a user to click on their ad and then not be able to deliver, so the advertiser could say “great deals on iPods” and then they sell iPod cases or something. In that case, the high click through rate is unlikely to be correlated with a high conversion rate because the users are going to be disappointed when they actually end up on the page. The good thing for us is that the conversion rate typically gets reflected in the amount that the advertiser’s actually willing to pay, so that’s one of the reasons why the advertiser’s bid is a relatively decent metric of the quality, for example in this iPod cases case, because that conversion rates likely to be low, the advertiser’s not likely to bid as much for that. The click just isn’t worth as much to them, therefore they’ll bid less and end up getting a lower rank as a result of that. So, in many cases, this doesn’t end up being a problem because that just sort of falls out of the ranking formula. It’s a little bit convoluted.
Gord: Just to restate it to make sure I’ve got it here. You’re saying that if somebody is being dishonest, ultimately the return they’re getting on that will dictate that they have to drop their bid amount, so it will correct itself. If they’re not getting the returns on the back end, they’re not going to pay the same on the front end and ultimately it will just find it will just find its proper place.
Nick: What an advertiser should probably be thinking most about is mostly ROI per click…it’s actually ROI per impression. From the ad that’s likely to generate the most value for the user, and therefore the most value to Google as well as the most value to the advertiser, all aligned in a very nice way, is the ad that’s the most likely to generate the most ROI per impression. And because of our ranking formula, those are the ads that are most likely to show up at the top of the auction. And the ones that aren’t fall out. So the advertiser should care click through rate, but they shouldn’t care about click through rate exclusively to the extent that that results in a low conversion rate and a low ROI per click for them.
One of the recurring themes that I’m asking everyone who will listen about it is what happens to ads when we start seeing images on the search results page? In a few recent quotes, Marissa Mayer has definitely left the door open to that possibility in the future. Of course, I asked Nick and Diane the same question and how it would impact the quality scoring of ads.
Nick: We need to see. I don’t think we know yet. Ultimately it would be our team deciding whether to do that or not, so fortunately we don’t have to worry too much about hooking up the quality score because we would design a quality score that would make sense for it. The team that focuses on what we call Ad UI, that’s the team that’s looking at …it’s sub group within that, that’s the team that essentially thinks about what should the ads actually look like.
Diane: And what information can we present that’s most useful to the user?
Nick: So in some cases, that information may be an image, in some cases that information may be a video. We need to make sure in doing this that we’re not just showing video ads, because video happens to be catchy. We want to make sure that we’re showing video ads because the video is what actually contains the content that’s actually useful for the user. With Universal Search we found that video search results, for example, can contain that information, so it’s likely that their paid results set could be the same as well. Again, just as in text ads, we’d need to make sure that whatever we do there is user driven rather than anything else and that the users are actually happy with it. There would be a lot of user experimentation that would happen before anything was launched along those lines.
So, for me, the natural follow-up would be to turn the conversation to my favorite topic, personalization. And I’m nothing if not predictable. Nick took the opportunity to take a not-so-subtle swing at competition that’s pursuing personalization or ad serving based on demographic profiles, i.e. Microsoft.
Nick: Yes. So we have been looking at some… I’m not sure if the right word is personalization or some sort of user based or task based…what the right word is..changes to how we think about ads. We have made changes to try to get a sense of what the user’s trying to do right now. Whether they’re, for example, in a commercial mind set and alter how we do ads somewhat based on that type of an understanding of the user’s current task. We’ve done much less with trying to..we’ve done nothing really…with trying to build profiles of the user and trying to understand who the user is and whether the user is a man or woman or a 45 year old or a 25 year old. We haven’t seen that that’s particularly useful for us. You don’t want to personalize users into a corner, you don’t want to create a profile of them that’s not actually reflective of whom they are. We don’t want to freak the user out. If you have a qualified user you could risk alienating that user. So we’ve been very hesitant to move in that direction and in general, we think that there’s a lot more we can that doesn’t require profiles down that path.
Diane: You can think of personalization in a couple of different ways, right? It can manifest itself in regards to the results you actually show. It can also be more about how many ads or even the presentation of those ads with regards to actual information. Those sorts of things. There are many possible directions that can be more fruitful than, like Nick points out, profiling.
Nick: For example, one of the things that you could theoretically do is, as you know, we changed the background color of our top ads from blue to yellow, because we found that yellow works better in general. You might find that for certain users, green is better, you might find that for certain users, blue is actually better. Those types of things, where you’re able to change things based on what users are responding to, is more appealing to us than these broad user classification types of things. It seems somewhat sketchy.
Finally, I wrapped up by asking Nick a question every search engine marketer wants to know: what’s in the quality scoring black box? There have been increasing degrees of transparency around this as Nick has shed light in previous interviews on some of the factors in both the minimum bid threshold algorithm and the actual ranking algorithm. He’s also gone on for on record as saying that ideally he would like to see some of these signals from the minimum bid algorithm find their way into the relatively more straightforward ranking algorithm. I asked Nick where Google was at with this.
Nick: There are probably two things. One is that when setting the minimum bid, we have much less information available to us. We don’t know what the specific query is that the user issued. We don’t know what time of day it is. We know very little about the context of what the user is actually trying to do. We don’t know what property that user’s on. There’s a whole lot that we don’t know. What we need to do when we set a minimum bid is much coarser. We just need to be able to say, what do we think this keyword is, what do we think the quality of the ad is, does the keyword meet the objective of the landing page and make a judgment based on that. But we don’t have the ability to be more nuanced in terms of actually taking into account the context of how the ad is likely to actually show up. There’s always going to be a difference in terms of what we can actually use when we set the minimum bid versus what we use at auction time to set the position.
The other piece of it though is there are certain pieces that only affect the minimum bid. Let me give you an example. Landing page quality normally impacts the minimum bid but it doesn’t impact your ranking. The reason for that is mostly from the standpoint of our decision to launch the product and what we thought was the most expedient way to improve the landing page quality of our ads rather than what we think will be the long term design of the system. So I’d expect things like that, where signals like landing page quality should impact not only the minimum CPC but also rank which ads show at the top of the page and things like that as well. That’s where you’ll see more convergence. But there’s always going to be more context that we can get at query time to use for the auction than we can for minimum CPC.
Like most things on the results page (and yes, I am still working on the columns looking at what the results page will look like in 2010. It’s taking longer to get all the interviews than I anticipated. Trust me, it will be worth the wait. And we may have some surprises coming up for some panels in San Jose!) the ads will be in a constant state of flux over the next few years. Nick, Diane and I did touch on a few other topics in the course of the interview, so if you want the full transcript, it’s posted on outofmygord.com.
Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.