The World Series Spidering ProblemPosted by willcritchlow
Since we were
kindly introduced as 'global associates' here on the blog by Scott, we have been helping work through a bit of a glut of Q&A. As many of you know, being able to ask consulting-style questions to SEO experts is one of the great benefits of PRO membership (and I don't even get paid for saying that!). Seeing the process from the inside, and seeing some of the private questions that have been solved, I know it is often really helpful.
But now it's time to make an appearance writing here on the blog.
I have unsurprisingly been getting assigned quite a lot of the internationalisation issues (or should it be internationalization? That joke never gets old). There has been one question that has been coming up again and again so I thought I'd put my mind to answering it publicly.
The issue I want to talk about is geo-delivery i.e. delivering different content to different visitors depending on their geographic location. When you don't know more information about the visitor (from sign-up information, cookie, etc.), the only way of doing this is through determining their location from their IP address. Whenever you start talking about selectively delivering content based on IP address, the topic of cloaking inevitably comes up.
Google have just this week written up their definitive guide to
when IP delivery is cloaking. The key take-away message from this post is:
Googlebot should see the same content a typical user from the same IP address would see.
Now hold on a second.
You know things are going down a dubious route when you find yourself writing 'many of my closest friends are American' to head off criticisms of xenophobia. But they are. I love you guys. But the analogy I want to use to explain the problem with this definition of cloaking is very similar to the way US sporting events are billed as world championships. Didn't America win the World Series again last year?
Now, I can assure you that there is a world outside the US. Over in the UK live some 60 million people whose first language is English. Quite a few people search (particularly for business stuff) across Northern Europe in English. And of course, and there are our friends down under and in SA as well. This, coupled with the ubiquity of the .com domain name extension across the English-speaking world (if only .us was used for all US sites) means that search engines can have a hard time working out which results to show to who.
We see a lot of cases where UK-based companies are ranking in google.com and not in google.co.uk because they have tripped some filter that sees them as not 'UK enough'. Similarly, we get a lot of questions from US-based companies looking to expand into the UK and wanting to know how they should go about it.
I want to target a specific countryStarting with the basics of international targeting, in this case, it is important to let the search engines know where your business is based in as many ways as possible. These might include:
- cc tld for your domain (e.g. .co.uk)
- local hosting
- physical local address in plain text on every page of your site
- Google Webmaster Central geo-targeting setting
- links from local websites
- etc.
If you are starting from scratch, getting these all lined up will give you the best possible chance of ranking in the local country you are targeting.
However, this isn't the end of the story - if you have an established .com domain, you might want to leverage your domain weight to target the new territory rather than starting from scratch.
The main point of this post, therefore (and the reason for the 'World Series' jibes above) was to point out the pitfalls of one particular approach that I see suggested surprisingly often by businesses considering this problem.
How to avoid the 'World Series' ranking issueWhat I see suggested is:
Why don't we create multiple versions of our site and determine where the user is in the world before either delivering the appropriate content or redirecting them to the appropriate place in our site (or even a sub-domain hosted in the target country)
The problem with this is that the spiders' view of the world is surprisingly like like that of the baseball commissioner - they are all US, all the time.
Because the major search engines spider from the US, their IP addresses will be US in your lookup and will therefore be delivered the US content. This problem is exacerbated if you are going even further and geo-delivering different language content as only your English language content will be spidered unless you cloak for the search engine bots.
This kind of IP-delivery is therefore a bad idea - you should make sure that you do not blindly geo-deliver content based on IP address as you will ignore many of your markets in the search engines' eyes.
So what is the right way of doing things?The best practice remains one of two approaches depending on the size and scale of your operations in the new countries and how powerful / established your .com domain is.
If you have strong local teams and (relatively speaking) less power in your main domain, then launching independent local websites geo-targeted as described above (hosted locally etc.) is a smart move in the long-run.
If, on the other hand, you only have centralised marketing and PR and / or a strong main domain, the best practice is to create localised versions of your content on either sub-domains (uk. au. etc.) or in sub-folders (/uk/ /au/ etc.).
Either the sub-domains or sub-folders approach allows you to set your geo-targeting in Google Webmaster Central and under either method you have to be equally careful of duplicate content across regions. In the sub-domain example, you can host the sub-domain locally if you choose, while in the sub-folder case, more of the power of the domain filters down.
It's a shame the right answer isn't more clear-cut. Hopefully we will see one way of doing things become significantly better than the others over the coming months, in which case hopefully you'll hear it here first.
In the meantime, happy language issues...
Do you like this post? Yes No
read more: 
