Yes, Google Says It’s Having Problems Listing iTunes Preview Pages

Yes, it is harder to find pages to iTunes apps in Google. But no, the company says, it’s not part of some nefarious plot. Rather, Google’s having technical problems gathering iTunes Preview pages, an issue it’s working to solve.

Google has confirmed the issue, saying:

We’ve been having some issues fetching pages from the iTunes web servers, and as a result some people may have had problems finding iTunes apps in search easily. We’re working with the team there to ensure search users can find what they’re looking for.

In particular, the head of Google’s web spam team Matt Cutts, said on Twitter that this was due to Apple sending 403 “forbidden” errors, when Google tried to access some pages. From his tweet:

@ourielohayon I assume you saw appleinsider.com/articles/13/04� ? The iTunes web servers were sending lots of 403s when fetching from certain IPs.

� Matt Cutts (@mattcutts) April 3, 2013

He further added that “it wasn’t anything on our side.”

If you want to understand more about what’s been going on, and what people were wondering, you can read my original story below, when I was trying to figure out what was happening from afar.

Where’s iTunes?

The story starts with a Facebook post today by Ouriel Ohayon, cofounder of Appsfire, who seems concerned that the iTunes page for his own WhatsApp messaging app doesn’t appear at the top of Google’s results and points to the same as not happening for Twitter as a problem. From his post:

Something really weird starts to happen on Google. It has become impossible to find iPhone and iPad apps looking for them in the search engine with a normal query. Eg “twitter iPhone app” which used to bring instantly the iTunes link first. What happened ? Google playing games? See that screenshot taken from Google own iPhone app.

His screenshot shows the homepage for WhatsApp coming first, followed by his own download page and then, a link to where the app can be found in Google Play. That’s very similar to what you see for Web search results (all results shown in this article are from me being logged out and in “Incognito” mode in Chrome, to minimize the impact of personalization):

Developer’s Own Site Coming First Is An Issue?

My immediate thought is that this is a pretty odd complaint from a developer, being upset that Google is listing their own site above iTunes or anyone else. If you go to the WhatsApp homepage, listed first in the results, you can easily get to a page listing where it’s available for download on a variety of platforms, including iOS. That download page itself is listed second. A page from WhatsApp for BlackBerry users comes third. The first three arrows show all these.

Of course, Google does list some pages for the app within actual app stores. Google Play gets fourth billing above, followed further down by listing for the app in the BlackBerry, Nokia and Windows Phone app stores.

A Duplicate Content Problem?

Where’s iTunes? We’re trying to figure that out for ourselves. It might be that Google has done something dastardly. But we suspect the problem might be on Apple’s end. Consider this:

That’s a search on Google specifically to find pages that are about the WhatsApp App. You can see that WhatsApp Messenger is listed twice. The arrows point out how the top listing is for the page in the US version of Apple’s iTunes store; the second listing, with the “LC” in the URL, seems to be for the St. Lucia version.

This suggests that Apple might have a duplicate content issue happening, perhaps one that’s become recent, for some reason. With duplicate content, you have two or more pages that are virtually identical to each other. That can confuse search engines and sometimes have the effect of “splitting the vote” when it comes to ranking, so that neither page wins.

But Bing Gets It Right

Of course, this isn’t happening on Bing. There, iTunes does well (while Google Play, BlackBerry and even the Windows Phone app store are nowhere to be seen on the first page):

Bing also manages to do this despite dealing with the same duplicate content issue as Google:

Impossible To Find Apps? No.

Now let’s step back and look at another example, drawing from Ohayon’s original statement:

It has become impossible to find iPhone and iPad apps looking for them in the search engine with a normal query. Eg “twitter iPhone app” which used to bring instantly the iTunes link first. What happened ?

To clarify, it’s not impossible. It’s actually pretty easy. It’s just harder to find the apps directly on iTunes versus the developer’s own site. Using his own Twitter example:

It’s exceedingly easy to find the Twitter for iPhone app. It’s listed right at the top of the page, on Twitter itself. Yes, you have to click from there to do the actual download from iTunes, but it’s hard to say that it’s not relevant to show the official page, from the official developer, first. As for iTunes, it appears listed third — after two pages from Twitter. (FYI, on Bing, iTunes comes first, followed by Twitter’s page).

Both Google & Bing Have Issues

Another example comes from The Next Web, which ran several queries to judge the�visibility�of iTunes. When I do that search, I get the Facebook official page at the top (similar to what happens with WhatApp and Twitter). iTunes does have a page that appears, but oddly for the Facebook Pages Manager. That’s at least better than Bing, where iTunes doesn’t appear at all:

Weird, and it makes it seems that perhaps this isn’t just a Google thing.

Google Favoring Itself? Not Necessarily

Then again, The Next Web points out that for searches on “google maps iphone” or “youtube iphone,” Google manages to list iTunes first. Gotcha!

Well, no. Google doesn’t really have a page for Google Maps for iPhone, not like how Twitter or Facebook have pages for their own apps. Really, it doesn’t. Nor one for YouTube, either. What it has is one giant page where all of these are listed, each with a link leading over to iTunes (in contrast, there is a YouTube app�page for Android).

Skipping over to TechCrunch, there’s a flat-out declaration in the opening paragraph that “Google search is making it more difficult to surface iOS applications.” This isn’t correct on two fronts.

First, it’s pretty clear Google is listing apps but directly from the developers, rather than iTunes — and that doesn’t necessarily make them harder to find.

Consider Snapchat, an example that TechCrunch uses as supposedly being missing in action for a search on “snapchat iphone.”�It’s not. It’s listed right at the top of the page — the official site.

But agreed, it is odd that an iTunes link isn’t showing (which does show at Bing). But that leads to the unsupported statement that Google is making iTunes listings harder to find on purpose. We don’t know that. Maybe it is, and on purpose. Maybe it is as a consequence of some change that’s impacting many results. Maybe there’s a chance something on Apple’s end is having an impact.

I’ve got messages out to Google and Apple, to see if they can shed any light on the situation. Until then, personally, I’d reign in the paranoia. While there’s no love lost between the two companies, this is far more likely some technical issue than Google attempting to unwisely go thermonuclear on iTunes.

Oh, and developers, this is a good lesson why you never, ever, should assume that your page on anyone’s app store is a substitute for having your own actual page about your apps. Have your own page, and you’re always directly in control of at least one entry point for those searching on Google or Bing.

Postscript (4pm PT): Google has sent me this statement:

We’ve been having some issues fetching pages from the iTunes web servers, and as a result some people may have had problems finding iTunes apps in search easily. We’re working with the team there to ensure search users can find what they’re looking for.

Postscript 2 (11:50pm PT): This story was further updated above to add in the tweet from Matt Cutts.