AdSense crawler errors: Check your robots.txt file for improved ad targeting and relevancy

This is the first post in our two-part AdSense Crawler Errors series.

There are many ways that publishers can go about optimizing their site for AdSense; opting-in to text/image ads, upgrading to our preferred ad formats, and increasing ad coverage across a site are just a few of the more well-known ones. But did you know that there’s another straightforward optimization tip that many publishers often overlook?

A bit of context
Your site’s robots.txt file essentially acts as a gatekeeper that determines which web crawlers, web robots, and search engines have access to your site and which do not. Those that are granted permission can do things like view your pages and index your site. Those that don’t have permission are not able to view or index specific sections of your site, depending on what you’ve specified.

AdSense ads are displayed through the use of an AdSense web crawler. That crawler scans your page’s content and determines which ads to display, according to specific keywords. If our AdSense crawler is being blocked by your robots.txt file, we’re going to have a difficult time displaying relevant ads on your site. As a result, your users may see less relevant ads, which can lead to a lower CTR.

How you can help yourself
View the contents of your robots.txt file by going to [yourdomain.com]/robots.txt.  (If you have a subdomain, it likely has a robots.txt file as well, located at [sub.yourdomain.com]/robots.txt.) Be sure that the file is configured to allow our AdSense ad crawler to view your site. You can do that by simply adding the following two lines to the very top of the file:

User-agent: Mediapartners-Google
Disallow:

This will ensure that our AdSense ad crawler can access your site and will help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. Please note that making this change will not impact your Google search rankings. Adding these two lines to your robots.txt file will only help to deliver better, more relevant ads to pages with AdSense code already on them. Pages that don’t have AdSense ad code will not be affected.

If you have URLs with any errors, you can see what they are by logging into your AdSense account and clicking on ‘Account Settings’ from the home page. From there, click on ‘View errors’ under ‘Access and Authorization.’

Stay tuned for the second post in our AdSense Crawler Errors series, where we’ll cover crawler login issues and how you can solve them.

Set up a crawler login for increased ad relevancy on pages behind a login

This is the second post in our two-part AdSense Crawler Errors series.

Recently we discussed ways to improve ad targeting and relevancy by making sure that the AdSense crawler can view all pages that have AdSense code on them. Sometimes though, it can become problematic when a publisher has a members only site that requires the user to log in to view the site’s content. In cases like this, the AdSense ads that are located behind the login aren’t as relevant and targeted as they could be.

If you’re a publisher with content behind a login page, be sure to set up an AdSense crawler login so that your ads are as well-targeted as possible. Doing so will give the AdSense crawler the ability to scan content on these pages, resulting in more relevant, targeted ads that can potentially increase your CTRs and revenue. Please note that adding a crawler login will only help to deliver better, more relevant ads to pages with AdSense code already on them, and will not have any effect on which pages of your site Google Search will index.

To grant the AdSense crawler access, log in to your AdSense account and click on ‘Account settings.’ From there, scroll down to the ‘Access and authorization’ section and click on ‘edit logins’ next to crawler logins.

Then click on ‘Add a Login.’ You’ll be prompted to fill in pertinent information that the AdSense crawler will use to log in to your site and crawl your content.

After that, you’ll need to log in to Webmaster Tools to verify your website, if you haven’t already done so. Once verified, the crawler will be able to log in successfully and scan your content to deliver more targeted ads.

For additional information, take a look at our help center article regarding displaying ads on login-protected pages.