The URL in our sitemap is in the format:

http://www.domain.com/index.html?whatever=value

The problem with Googlebot is that even though that is the URL we put in the sitemap, it doesn’t use that URL to make the request – it contracts it down to:

http://www.domain.com/?whatever=value

So our server sees this ‘incorrect’ URL, issues a 301 with the ‘correct’ URL (that has the index.html bit in it), but then Googlebot doesn’t follow that URL faithfully and again tries to request the URL without index.html in the path. So our server again issues a 301 redirect, with the correct URL and here we go off on our infinite loop. So no wonder we get the error message:

URLs not followed.... [sitemap] contained too many redirects.

John Mueller, Webmaster Trends Analyst at Google Zürich replies:

In this case it actually is something that we’re doing — we strip “/index.html” from URLs because that’s generally irrelevant and only makes the URL longer and look more complicated to the user. We do this when processing the URLs in your Sitemap file so if you *need* to have “/index.html” in the URLs, they generally won’t work like that. At the moment, there is no solution for using these URLs in Sitemap files if you need to have “/index.html” in them. I would generally recommend dropping the “/index.html” part, but I realize that this is sometimes not easily done.

That said, we will still crawl the website normally, so if those URLs are reachable through a normal web crawl, we’ll still find and index them normally.

Useful advice there for anyone putting together a sitemap and wondering why Google was throwing an error on URLs requiring a default filename. I assume this would also apply to the other “default” page names such as index.html index.htm index.cgi index.pl index.php index.xhtml, index.asp and perhaps default.html etc.

Technorati Tags: Google, Google Webmaster Groups, JohnMu, Sitemaps

So as part of my contribution to the Linux community, I’ve been trying to help in the Wine project. This software attempts to allow you to run Windows programs in a Linux environment, and is completely free. The list of applications they have working is impressive and the work of the Wine community has definitely enabled many people to cross to Linux.

One of the first tools I have managed to get working there, is GSiteCrawler. This program will scan your site and automatically create and upload an XML sitemap for Google or Yahoo! Once uploaded, their location can be added to the respective webmaster panels.

How to Install GSiteCrawler for Ubuntu

If you already have Wine installed then jump to point 4.

1. Add the Wine respository to your system:
# sudo wget http://wine.budgetdedicated.com/apt/sources.list.d/hardy.list -O /etc/apt/sources.list.d/winehq.list

2. Update your system package information:
# sudo apt-get update

3. You can now install Wine by clicking this link. Alternatively, you can install by going to Applications->Add/Remove and searching for Wine.

4. Install the following at the terminal:
# wget kegel.com/wine/winetricks && sh winetricks jet40

5. Download the full installation copy of GSiteCrawler from here and save to your desktop.

6. Right click the program install file and select “Open with Wine Windows Program loader”.

You’re done. Of course, any of my Windows readers can also download GSiteCrawler and install this handy application. I’m building a list of Ubuntu-friendly SEO/SEM tools as a reference for online marketers who have crossed over or are thinking about the switch. Feel free to list any must-have Windows tools in the comments and I’ll add them to my list for testing.

Technorati Tags: Google, GsiteCrawler, JohnMu, Sitemaps, Ubuntu, Wine