Allowing search engines to spider 'hidden' content - best practice?

Hi Ashley,

This is something we've raised with several clients, as one of them had 300,000+ pages hidden behind a login screen, all potentially very valuable data in not only increasing the volume of long-tail phrases the site would be found for but also in providing more content on important core topics that then in turn would help the home page rank higher (as I have frequently found that if you have 500 pages on a topic on your site, rather than any single one of them ranking very high, it just pushes up the rankings for the home page).

The most obvious but difficult solution to the problem would be to be able to provide Googlebot a username and password to 'unlock' that content itself, allowing it to index content in your secure area, but then when a search engine user clicked on the link in the Google results they would be prompted to login to read the full article. However, this approach would require an association with the Google team that not many sites or companies do or would ever possess.

Another option that still avoids the potentially risky 'redirection based on bots or users' would be to skim the first 50 words from each piece of content that is currently secure, and publish it on your front end with appropriate Meta data to encourage high rankings.

Search engines regard the first 50 words on a page as the most important anyway, so you would satisfy them in terms of providing enough content to get the SEO benefit, while only providing a teaser to visiting people, who would still have to register and login to read the rest of it.

If you do go down the route of redirecting bots or users and return different content it is definitely a gray area. Google themselves have been doing it for years with Adwords, i.e. allowing you to only display ads to people from different towns, cities, and countries, but then again, that doesn't mean it's a license for everyone to do it.

My thoughts on redirection based on bot or browser has always been that if you have a good reason for doing it, you will be fine. You will never get automatically banned or penalised by an engine, only potentially flagged for a manual review of suspicious activity, and assuming that reviewer has any sense, they will realise you are not trying to get an unfair advantage or fool them, you are actually just trying to improve the quality of their search database by sharing your content with search engine users.

However, saying that, I'd go with the first 50 words on a page option as you get the benefits of the extra traffic from search engines, there's no redirection involved so no risk there, and you still get the value of the memberships / logins from people who want to read the full article.

I hope this helps.

Regards,

Paul

Head of Search
Barracuda Digital