Google Maps Mania
My attention was drawn today that knowcal was mentioned in Google Maps Mania.
Many thanks to whoever made the suggestion to those in charge.
My attention was drawn today that knowcal was mentioned in Google Maps Mania.
Many thanks to whoever made the suggestion to those in charge.
My Google site index has dropped from over 40,000 to just 800 in the space of a week. The traffic to my site has dropped to a trickle and the revenue is down substantially. That is the bad news.
The good news is that Google is finally indexing the new links (as shown by the date header on the Google results) so it looks like the index should be going up again (hopefully).
I was getting worried that I had been banned for some reason - maybe the Ajax background loading of pages, but hopefully I have just been 'sandboxed' whilst the index sorted itself out.
No more messing with the site now - I am definately going to sit tight and see what happens.
This shows the real risk of relying on one major partner for your business. In my case, with Google currently bringing in the majority of the traffic, I am in trouble if they drop me from their index. I am also using Google Adsense and Google Maps, so a ban on these would also cause me great problems.
This is a high risk strategy for any business and as Google becomes even more powerful it does worry me that they may start to use that power to create the sort of Internet they want, rather than what the 'poeple' want - I am, in many senses, a competitor to them as they have their own local search product. How long before they take exception to this and kick off all their competitors?
Whilst traffic (from Google) has gone quiet, I have taken the opportunity to restructure the URL's as some of them were causing problems.
For example, ampersands have now been replaced with the word 'and', spaces (which appear as %20) have been replaced with a dash, and dashes (in place names such as Stoke-on-Trent) have been replaced with an underscore.
All these changes should (!) make them more search engine friendly and easier to read. Importantly, they are backwards compatible with the old URL style so current pages in the search engines will still work.
Now I will have to sit tight for a few weeks (or months) whilst the search engines catch up and build a new index from the fresh links. I expect my current page index count to drop and then slowly rise again, hopefully bringing back the large quantities of traffic I had a few days ago.
One can hope!
Oh dear, I knew things were going too well!
The number of pages Google has indexed has been steadily gorwing, from 400 a few weeks ago to 20,000 last week to (as of yesterday) 48,200. My visitor numbers had also been steadily growing in response to this and, with adsense now optimised on the site to the best of my current ability, my click-through-rate and revenue had been increasing rather nicely.
...and then Bang!
Looking at the stats for the last 3 days, my visitor numbers have been dropping for the first time since launch - currently down around 65% from the peak. This seems very odd, but I notice many others have experienced the same phenomenon.
Having racked my brain, the possible reasons for this slump could be:
1) At the end of last week I redesigned the front page to include more category links as these seemed to be the ones that Google ranks most highly, and those correlated closely with the search terms bringing people to the site.
2) I also disabled my sitemaps as they appeared to be interferring with my stats (although that is probably nonsense).
3) The site was down for a few hours on Monday whilst I was out (Mr Sod and his damn law at work again) which did not go unnoticed by the google sitemap reports (which carry on working, even with no sitemaps listed!).
To get things going again (hopefully) I have reinclude the sitemaps and made provision to reduce the chances of the site going down again (the problem appeared to be a scarcity of disk space and java memory).
The main lesson to come out of this is to be patient. Changes to the site can take weeks to be finally reflected in Google's index so tweaking the site in future needs to be done in small stages, measuring the impact contnuously until things settle down before making the next change.
I will now sit tight to see what happens in the next few weeks.
This morning I have added some new functionality to the results, namely an image of the website next to the address (if the website is registered).
These images come from Alexa, although only websites with enough traffic have an image available.
As both google and other search engines (Yahoo, AOL and others) are now actively indexing the site the user numbers are growing steadily and the Adsense clickthrough rate looking much more healthy. With only 0.1% of the site currently indexed the possibility of 1000 times the traffic (and adsense revenue) is both exciting and daunting.
Users are also staying longer and performing more searches on the site with increasing numbers staying between 2 and 5 minutes and a quarter of them return visits. I don't want to make too much of these statistics yet, but I think it shows the SEO strategy is working and that the users like the novely of the service - it is certainly more compelling than the competitors' sites but I would say that!
The logs are also telling me that the results are coming in at the number one spot on Google quite often, with the vast majority within the top ten. However, I understand the bias in these numbers - if the results are way down the list then the chances of a clickthrough are slim so there will be nothing for the log to measure!
I have also added some google sitemaps to the site and further simplified the URL structure which makes it clearer for us humans and simpler for the search engines to follow (now only 2 path levels rather than 3 and no annoying paranthesis). It will be interesting to see if this makes a big difference and hopefully the sitemap reports will give me more indepth positioning information.
All-in-all good progress this week.
The next step is to increase my page rank through gaining more links - the hardest part of SEO.
It has been two weeks since I updated the site structure and the results are now starting to pay off.
The google bot has made over 65,000 page crawls and over 400 pages are now indexed (and growing daily). Bearning in mind there are nearly 3 million potential pages in the site this is still incredibly small, but is a promising start and already I am getting hits on some truely obscure search terms - the long tail in action!
What does confuse me is the way Google shows the search terms in its results. For example, a search for Galleries in Baskingstoke shows up as the first result and Google highlights all of 'Galleries', 'in' and 'Basingstoke' on the page title but fails to highlight either 'Galleries' or 'Basingstoke' in the URL line. Looking at results further down both of these terms appear in bold for other sites.
Can anyone shed any light on why Google fails to think these terms are not important in my URL? I'm stumped!
Until a few hours ago, Knowcal used Ajax to load the directory data in the background after the page was loaded. The main benefit of this was that page response times were very low and the page could be navigated while the background data 'thread' was doing its thing. As I had set it up to batch-load results in packets of 20, up to a maximum number of 10 batches, this seemed a viable option to stop the user having to around for all potential 200 results (over 10 web service calls) to load.
However, with the launch of the site, the side effects of this strategy (which were always niggling me in my subconcious) became clear.
Firstly, Google has flatly refused to index the site, even after setting up a substantial sitemap. I know it has only been a week since launch so I may be looking too deeply into this, but I have an increasingly nagging feeling it may also have something to so with the pages being devoid of any content - all this is 'injected' by the browser through the ajax functions.
Secondly, this problem is further compounded by each page being virtually identical (give or take a few headings and titles), which goes against the webmaster standards.
Thirdly, as all the links were inserted through DHTML, again using ajax, there is no page-to-page navigation providing nothing for other search engines who do not understand the sitemap files to crawl.
This morning I took the decision to change the design of the site to be a hybrid between traditional server-side and 'web2.0' ajax content rendering. The category and geography taxonomy links are now rendered directly in lovely HTML, along with the first page of results. The second (and subsequent) results pages are then injected by ajax in the background as before.
Although this means an additional trip to the web service it will hopefully provide more meaningful information to the search engines.
The new code is now live, but whether Google immediately starts indexing the site remains to be seen!
Public Launch
This is the new blog for Knowcal - where Google Maps meets UK Business directory.
I notice from the logs that many people have already found the site and I hope are finding it useful, although I should stress to non-UK users that searches on towns in Italy, Belgium or France will not return any results (you know who you are!).
Feel free to play around, and any bugs, corrections or suggestions will be gratefully received.