Ajax and SEO
Until a few hours ago, Knowcal used Ajax to load the directory data in the background after the page was loaded. The main benefit of this was that page response times were very low and the page could be navigated while the background data 'thread' was doing its thing. As I had set it up to batch-load results in packets of 20, up to a maximum number of 10 batches, this seemed a viable option to stop the user having to around for all potential 200 results (over 10 web service calls) to load.
However, with the launch of the site, the side effects of this strategy (which were always niggling me in my subconcious) became clear.
Firstly, Google has flatly refused to index the site, even after setting up a substantial sitemap. I know it has only been a week since launch so I may be looking too deeply into this, but I have an increasingly nagging feeling it may also have something to so with the pages being devoid of any content - all this is 'injected' by the browser through the ajax functions.
Secondly, this problem is further compounded by each page being virtually identical (give or take a few headings and titles), which goes against the webmaster standards.
Thirdly, as all the links were inserted through DHTML, again using ajax, there is no page-to-page navigation providing nothing for other search engines who do not understand the sitemap files to crawl.
This morning I took the decision to change the design of the site to be a hybrid between traditional server-side and 'web2.0' ajax content rendering. The category and geography taxonomy links are now rendered directly in lovely HTML, along with the first page of results. The second (and subsequent) results pages are then injected by ajax in the background as before.
Although this means an additional trip to the web service it will hopefully provide more meaningful information to the search engines.
The new code is now live, but whether Google immediately starts indexing the site remains to be seen!