Allowing search engines to spider 'hidden' content - best practice?

Quote: So you could have a mechanism of restricting access at the chapter/topic level – first 50 words maybe, but make all the detailed content sub-topic available, and so exploit the tail

On 08:57:28 22 August 2007 DaveChaffey wrote:

 

I've come late to this party - holidays! and can't add much from an SEO POV, but I think there are some other alternatives to consider.

What’s not really covered is the impact on conversion rate. My gut feel is that of the two models – #1 partial content preview (the 50 words one) and #2 time-limited preview (the Webmasterworld one) #1 would work best for conversion. It is also not subject to tech savvy people getting around the security on #2.

On the other hand, I must say I have always liked the Webmasterworld model since all their content is indexed so shows up frequently in long tail searches – and option #2 will therefore be much better for awareness / reach.

So, balancing higher conversion against reach I think option #2 will work best.

Which ever option you chose, but especially #2 it obviously needs to be flexible enough to revert if the engines introduce a new approach / rule on cloaking or a tag for subscription content.

The other aspect not really mentioned and this is probably option #3 - is that for all your reports you have a 3 or 4 layer hierarchy, essentially chapter:topic:sub-topic. So you could have a mechanism of restricting access at the chapter/topic level – first 50 words maybe, but make all the detailed content sub-topic available, and so exploit the tail, but the full picture – horizontal navigation - isn’t available.

HTH Dave Chaffey
www.davechaffey.com

On 16:02:04 7 August 2007 Ashley wrote:

 

I should know this but I'm intrigued to hear from any SEO experts out there what the latest thinking / best practice is for allowing search engines to index restricted content e.g. content that sits behind a pay-access log in, or other barrier?

We're looking at converting all our current file content (e.g. Word files, PDFs etc.), which are mostly paid-access only research and guides, into XHTML so that we can display them as HTML or allow users to convert them (e.g. to PDF) on the fly. This will also make it easier to syndicate our content, present it on other devices, "reskin" the presentation layer and so on.

But it also would allow us to make all the contents of a report (e.g. a 200 page Word file) available for indexing by a search engine. In theory this is good because there is a lot of great, niche, content in these documents which would be great for long tail SEO and attracting high-converting traffic.

But, of course, we wouldn't want the user to actually get access to the full content itself without first paying. Nor would we want all this pay-access content existing in Google's cache.

I guess, in theory, we could allow the Googlebot and chosen other spiders to index this content as HTML but not allow real humans or other agents to do so. I believe we might be able to use the robots.txt protocol to prevent caching too?

But in the case of the above we are showing Google something that we are not showing our users - and isn't this cloaking?

However, Google's Book search seems to work in just this way so Google don't appear to be averse to indexing intellectual property / content in this way but without revealing it all?

Any thoughts / pointers / experiences welcome...

Thanks

Ashley Friedlein
CEO
E-consultancy.com