Do you run a website with a premium member’s only area? or simply an area that requires users to log-in first? Previously, the Google Bot that would attempt to crawl these pages to determine what ads to run would not be able to crawl the pages properly because it would be denied access.

Google has added a new feature to enable their crawler to access login-protected pages. It involves logging into Google SiteMaps and Google Webmaster Tools to make the appropriate changes. For detailed instructions, login to your Google AdSense account (AdSense Setup > Site Authentication), or visit the help page.

Of course you will also need to ensure that you are not blocking access to the crawler in your robots.txt file. The name of the crawler used for AdSense is called: Mediapartners-Google

To allow this Googlebot full access to your website, add the following lines to your robots.txt:

User-agent: Mediapartners-Google*
Disallow:

Did you like this post? If so share with others: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Netscape
  • Reddit
  • del.icio.us
  • StumbleUpon
  • Digg


Other posts you may be interested in:

  • Agloco is Officially Dead
  • Google SERPs Updated With My New Permalinks
  • No Penalty For VW From Google
  • Lower Your Ad’s CPC and Still Rank Higher!
  • Beyond.ca Experiences The Digg Effect