I’m still waiting for Matt Cutts to confirm if this is a real bug or not, but I can’t find any other causes..
I think it has to do with the latest BigDaddy update and cache sharing. Here’s how that works:
When adsense visits your site, it caches it. When the search spider needs to visit your site, it says “oh, adsense was just there.. I’ll use that.” This is a great feature, as it saves both Google and webmasters bandwith.
Note, that using adsense does NOT get you indexed faster, it just stops the bot from making an unnecessary visit. But, I do think this is causing me to get indexed slower. Here’s what happened:
In April I bought a domain and put up a “coming soon” type message on it. I then created a subdirectory and started developing the site there. Security by obscurity right?
Anyway, I made a test.html page in the root directory a few days before going live. This test.html page linked to www.mydomain.com, and had adsense included on it.
I noticed after a few visits to my test page (121 actually), that ads were starting to get relevant. I verified this because I saw Googlebot had visited twice (Yahoobot has visited once too, as my alternate ads are set to yahoo)
I thought nothing of it, and switched the site live.
It’s important to realize that at this time, there were a total of 0 in-links to the site (in google, msn, yahoo, ask, etc), and the site had not been previously owned, nor submitted to any search engine.
But it was listed on Google’s cache. Not the test page, just the main index page, and it’s “coming soon” message.
How did that happen? Is it possible that the adsense spider searched other pages it found (to determine relevance), then cached them too? So when I submitted the domain to Google the next day, it grabbed that old adsense cache even though it technically didn’t exist anymore?
Ordinarily this wouldn’t be a problem, but another BigDaddy improvement was to not revisit spam sites as often. I can see how Google wouldn’t want to revisit a 4 word website, and so it’s been almost a month since the last googlebot visit (despite some fresh in links and a manual submission)
It looks like putting adsense on my page here actually hurt me.
The good news is, it’s an easy fix for Google. Don’t let the adsense bot make the first cache of a webpage. Or, if you do.. don’t let the search engine spider use the adsense cache if it’s the first time it found the URL from a different site, or from manual submission.