[Search for "wp_list_pages" - one of the functions of WordPress]

[Search for "limitations of google analytics" - spotted by TalentGurus]

[Search for "html anchors" brings up w3.org]

In all cases Google is picking up the internal in-page navigation (HTML anchors) and pulling it into the SERPs. All site examples here are large, popular sites so this feature, as with site search box, maybe reserved for such cases. However this is definitely something to experiment with. In-page navigation is a bonus for usability so this could be a win-win tactic.

Also worth noting that Google has recently been experimenting with Breadcrumb navigation in their SERPs too.

Update: This new feature is explained over at the Google WebmasterBlog by Raj Krishnan (Snippets Team).

There are a few things you can do to increase the chances that [these links] might appear on your pages. First, ensure that long, multi-topic pages on your site are well-structured and broken into distinct logical sections. Second, ensure that each section has an associated anchor with a descriptive name (i.e., not just “Section 2.1″), and that your page includes a “table of contents” which links to the individual anchors.

Technorati Tags: Google, SEO, SERPs, Snippets

The URL in our sitemap is in the format:

http://www.domain.com/index.html?whatever=value

The problem with Googlebot is that even though that is the URL we put in the sitemap, it doesn’t use that URL to make the request – it contracts it down to:

http://www.domain.com/?whatever=value

So our server sees this ‘incorrect’ URL, issues a 301 with the ‘correct’ URL (that has the index.html bit in it), but then Googlebot doesn’t follow that URL faithfully and again tries to request the URL without index.html in the path. So our server again issues a 301 redirect, with the correct URL and here we go off on our infinite loop. So no wonder we get the error message:

URLs not followed.... [sitemap] contained too many redirects.

John Mueller, Webmaster Trends Analyst at Google Zürich replies:

In this case it actually is something that we’re doing — we strip “/index.html” from URLs because that’s generally irrelevant and only makes the URL longer and look more complicated to the user. We do this when processing the URLs in your Sitemap file so if you *need* to have “/index.html” in the URLs, they generally won’t work like that. At the moment, there is no solution for using these URLs in Sitemap files if you need to have “/index.html” in them. I would generally recommend dropping the “/index.html” part, but I realize that this is sometimes not easily done.

That said, we will still crawl the website normally, so if those URLs are reachable through a normal web crawl, we’ll still find and index them normally.

Useful advice there for anyone putting together a sitemap and wondering why Google was throwing an error on URLs requiring a default filename. I assume this would also apply to the other “default” page names such as index.html index.htm index.cgi index.pl index.php index.xhtml, index.asp and perhaps default.html etc.

Technorati Tags: Google, Google Webmaster Groups, JohnMu, Sitemaps

What is the current listing status for www.davidnaylor.co.uk/?

This site is not currently listed as suspicious.

What happened when Google visited this site?

Of the 29 pages we tested on the site over the past 90 days, 0 page(s) resulted in malicious software being downloaded and installed without user consent. The last time Google visited this site was on 09/18/2008, and suspicious content was never found on this site within the past 90 days.

Has this site acted as an intermediary resulting in further distribution of malware?

Over the past 90 days, www.davidnaylor.co.uk/ did not appear to function as an intermediary for the infection of any sites.

Has this site hosted malware?

No, this site has not hosted malicious software over the past 90 days.

OK, so we assume Google did not see Dave’s site as compromised. This would explain why Dave did not get any warning messages from Webmaster Central. Both that system and SERPGuard work on the Google Safe Browsing API and without a positive listing of DaveNaylor.co.uk – neither would have reason to send out an alert.

According to Dave Naylor’s latest blog post, his site appears to have suffered a text-book hack.

While I was in the meeting Becky texted me to say they had found something Patrick at Blogstorm ( I’m not linking out just in case I pass bad karma) and Josh from JaeWeb, had spotted an issue. It was spot on, the server had been comprised and the site was cloaking links to google of antidepressant drugs and we had a fake adsense code injected into the blog.

Patrick Altoft discovered a useful way of detecting compromises in your blog, by setting up Google Alerts on key terms. I imagine one of these searches revealed the cloaked pages that Dave had unknowingly served to Google.

The real advantage of Patrick’s technique, is that it might be able to catch this kind of compromise before the site makes it onto Google’s blacklists. I’ve always assumed that SERP penalties would only be applied after the site is listed by Google as being compromised.

So the real question here is why would Google apply a penalty without first listing DaveNaylor.co.uk? Is this a case of the left hand not knowing what the right is doing? ie. are malware/phishing penalties applied irrespective of the Safe Browsing list? Or does Google not consider this incident malware/phishing related?

There are still a lot of unanswered questions for me with this incident. I look forward to Dave revealing more about the hack and any feedback Google can give on why he was penalised but not listed? If the Safe Browsing project is to have any use to webmasters, we should know it accurately reflects Google’s opinion of the site.

Technorati Tags: Dave Naylor, Google, Hacking, malware, phishing, Safe Browsing API, Security

I don’t know if this is official Google policy but it seems an excellent idea to me and a novel way to reach out to webmasters and SEO/SEMs who follow John’s updates. If nothing else, it’s a notable use of Twitter as a company brand building tool and an interesting experiment in 140 character Q&A sessions.

Questions & Answers

I’ll be updating this post as more questions and answers appear. John’s offer is open to everyone so grab the opportunity and head over to Twitter

Pierre Far: What’s the value of a nofollowed link in the eyes of Google’s algorithm? (status link)

JohnMu: Nofollow is for “ignorable” links. Recent ex: site with many URLs blocked with meta robots, nofollows help us to crawl better. (status link)

NickWilsdon: What is an acceptable number of IBLs to get from the “wrong” geographic area or language? i.e. UK/English site + Chinese links? (status link)

JohnMu: An old site of mine got many Chinese links – no problem. Within reason, they’re just links. Perhaps put a translation up? (status link)

PatrickAltoft: Will you be removing these urls, people are concerned their feeds are being indexed (I know robots.txt blocks) http://bit.ly/1b6Dyu (status link)

JohnMu: I’ll pass that on, thanks for bringing it up! (status link)

AndyBeard: Can an old, out of date XML sitemap prevent a site from being fully indexed? (status link)

JohnMu: If you know that the XML Sitemap is outdated, then fix it :). We use the Sitemap for hints, but we still crawl normally. (status link)

NickWilsdon: Beyond the user experience issue, can having broken links and 404’s negatively impact a site’s overall ranking in Google? (status link)

JohnMu: Having 404s should not affect ranking, but it can affect crawling (wasted crawls trying to reach 404s instead of good content) (status link)

mvandemar: Not too long ago Matt announced that a bunch of penalties had expired in Google… do all penalties expire? (status link)

JohnMu: For some penalties it makes sense to expire (eg when the things really have no impact anymore). Fixing issues is faster though:) (status link)

Not that many questions this week but John seems happy to do this again next Friday. Make sure you’re following his account here before then.

John’s welcome to clarify in the comments but I’d assume he’s looking for more practical questions on sites and indexing issues. I’m not sure how fair it is to throw Google policy questions at him, especially in a rapid-fire environment like Twitter. This is a useful opportunity to question a Google employee who answers literally hundreds of webmaster questions each week.

I look forward to seeing how this develops.

Technorati Tags: Google, Google Webmaster Help, John Mueller, JohnMu, Twitter

We’ve been working on the Jaiku service over the weekend after
finding an issue with one of our servers on Friday. As part of the
solution, we’re moving Jaiku to a Google data center.

This is something that we’d planned to do anyway, as part of our
future transition to Google App Engine. Now that we’ve moved, we’ll
need to ask you to review and accept a new terms of service and
privacy policy.

As a special thank you for your patience, we’d like to throw a
little nest-warming party and open unlimited invitations for Jaiku.

Twitter must be feeling rather nervous right now. The Google App Engine promises to solve the infrastructure promises to solve the problems of reliability that have plagued Twitter over recent months. As Google themselves say, “Google App Engine makes it easy to build scalable applications that grow from one user to millions of users without infrastructure headaches”. This will be a very public demonstration of that statement and the validity of the platform.

There have also been suggestions that Python, the language used in the Google Engine will be a better choice for this kind of application. Most agree it is is faster than Ruby on Rails (RoR). However RoR fans point the finger at the framework behind the project, a key advantage Google holds over Twitter.

It’s likely there will be a migration to Jaiku, even just to check out the relaunch so ensure you have your branding/username secured. If you need an invite, then leave a comment below. As @seocopyandstrat just tweeted to me:

I wonder if we’ll all migrate to Jaiku next time the Fail Whale appears, and then stay there?

Technorati Tags: Google, Jaiku, Twitter

So as part of my contribution to the Linux community, I’ve been trying to help in the Wine project. This software attempts to allow you to run Windows programs in a Linux environment, and is completely free. The list of applications they have working is impressive and the work of the Wine community has definitely enabled many people to cross to Linux.

One of the first tools I have managed to get working there, is GSiteCrawler. This program will scan your site and automatically create and upload an XML sitemap for Google or Yahoo! Once uploaded, their location can be added to the respective webmaster panels.

How to Install GSiteCrawler for Ubuntu

If you already have Wine installed then jump to point 4.

1. Add the Wine respository to your system:
# sudo wget http://wine.budgetdedicated.com/apt/sources.list.d/hardy.list -O /etc/apt/sources.list.d/winehq.list

2. Update your system package information:
# sudo apt-get update

3. You can now install Wine by clicking this link. Alternatively, you can install by going to Applications->Add/Remove and searching for Wine.

4. Install the following at the terminal:
# wget kegel.com/wine/winetricks && sh winetricks jet40

5. Download the full installation copy of GSiteCrawler from here and save to your desktop.

6. Right click the program install file and select “Open with Wine Windows Program loader”.

You’re done. Of course, any of my Windows readers can also download GSiteCrawler and install this handy application. I’m building a list of Ubuntu-friendly SEO/SEM tools as a reference for online marketers who have crossed over or are thinking about the switch. Feel free to list any must-have Windows tools in the comments and I’ll add them to my list for testing.

Technorati Tags: Google, GsiteCrawler, JohnMu, Sitemaps, Ubuntu, Wine

1. Non Disclosure Contract (NDA)

An NDA is an essential tool for a search marketer, or anyone in a consultancy role. This contract creates a confidential relationship between two parties and prevents either side discussing sensitive information to third parties. This obviously comes in handy when quoting on a project but I have also used an NDA with industry peers when asking for advice. You can read more on NDAs here.

2. Presentation Backgrounds

Giving talks and presentations are an essential part of marketing yourself or your company in the search world. It may not be SMX, SES or PubCon but sooner or later your going to find yourself knee-deep in PowerPoint. Larger client contracts often require you to make a presentation to their team.

3. Company Newsletters

These templates can give you or your client the push they need to start email marketing to their customer base. Not only is the R.O.I. higher when marketing to existing clients but as Garrett French writes email marketing can form part of your linking strategy.

4. Basic Google Adwords Proposal

Google’s released two templates to help you with their Adwords system. This one is a basic proposal document for a new Adwords client. You can show trends to clients, sample keywords/creative and give estimated costs for the campaign.

5. Adwords Creative Generator

This template will help you create adverts for your Adwords campaigns. I’m not well versed in PPC but it looks like the aim of this sheet is to auto-generate advert headlines based on your Ad Groups. It will mix custom terms with Ad Group keywords and highlight adverts that exceed the character limit. Once you have finished here, you paste into Adwords Editor or a bulksheet.

6. Travel Expenses and Itinerary Templates

Going to conferences is an essential part of the search industry. It’s a great way to get a beer pick up clients. These templates can help you keep track of your travel details and expenses. Having them on Google Docs is perfect when you’re on the move.

7. Video Christmas Template

It’s depressing to think about Christmas already but for many of us involved in e-commerce the planning starts soon. Assemble your mailing list and send out this card at the start of the season, with a reminder of the month’s best deals.

The template wording needs tweaking but it comes in 5 colours and the embedded video is an interesting angle. Alternatively use it for your own marketing clients or just to give your designer some inspiration.

Technorati Tags: Adwords Templates, Google, Google Adwords, Google Docs, NDA, Powerpoint Templates, templates

What are Google’s Safe Browsing Blacklists?

If Google detects malware or phishing attacks coming from your website they will put the URLs onto their blacklists and take steps to warn their users. They will place a message next to your results listing, an interstitial warning page when the user clicks the result and a pop-up alert if the Google Toolbar is installed. You can view examples of these warnings here..

This is a useful initiative to stop innocent users infecting their computers but being listed by Google in this way can have a serious effect on an online business. Users will be worried about the warnings shown to them and rankings/traffic can drop noticeably. The quicker you deal with this situation the better so early alerts are essential.

Introducing SERPGuard

SERPGuard is a Ruby on Rails project using the new Safe Browsing API that Google has helpfully made available. We built this application as part of our in-house client suite but decided to release it publicly.

Why would I use this tool?

Google has made their API available for exactly this kind of real-time reporting. Although they are already sending emails out from Webmaster Central you may want an additional checking mechanism. When it comes to something as serious as lost rankings, clicks or sales; you can never have enough alerts.

This tool will also be useful for webmasters who did not want to link up their entire portfolio within Google Webmaster Central. Our tool provides an independent and relatively anonymous way of checking all your sites in one place.

SERPGuard is RSS-powered. You will have a unique RSS feed which will update should any of your sites become infected. This gives you more choices in how to be alerted. One suggestion would be to create a new account at Twitter and feed the SERPGuard RSS stream to that account via Twitterfeed. Just follow that new account to receive instant updates by RSS or SMS to your mobile.

Technorati Tags: API, Google, Google Toolbar, malware, phishing, safe browsing

In summary, Lyndon Antcliff posted a satirical story on a client’s site, which ended up being quoted on several mainstream news channels. Critics have claimed that Lyndon manipulated these sources by not labeling the material as fake, in order to gain links for his client. In his defense, none of these news channels or reporters seem to have tried to check the legitimacy of the story.

This is hardly the first time news channels have carried fake news. For example, FoxNews, the main channel to carry Lyndon’s story was reprimanded in 2004 for placing fabricated quotes attributed to John Kerry on their website.

In this thread at Sphinn, Matt Cutts has now suggested that such behavior comes under the jurisdiction of Google.

My quick take is that Google’s webmaster guidelines allow for cases such as this:

“Google may respond negatively to other misleading practices not listed here (e.g. tricking users by registering misspellings of well-known websites). It’s not safe to assume that just because a specific deceptive technique isn’t included on this page, Google approves of it.”

There’s not much more deceptive or misleading than a fake story without any disclosure that the story is hoax.

Does this mean penalties for misleading information on websites? This certainly opens the doors to a whole host of issues, from fake reviews to less than honest product marketing. Also, can we expect to see the FoxNews website suffering any penalties in the near future?

Technorati Tags: Fox News, Google, Matt Cutts

As you can see, there are several references to articles on this blog, NickWilsdon.com. The data from the two sites seems to be merged. I’ve tried in other browsers and refreshed cookies but the problem remains. Anyone else got this issue?

Technorati Tags: Analytics, Google, Google Analytics