Why the January 2012 update is coming to you late next weekend

Dear UKC followers and subscribers,

Normally, we aim to release our update within a week after Police.uk release theirs and often much sooner. So we owe you an apology and an explanation on why it’s later than usual.

First of all, Police.uk aim to release on the 25th of the month but this time, it was released on the morning of the 1st March. There was no explanation or less still, an apology on their behalf. In our view, this near-zero level of interraction with 3rd party developers has a lot to answer for. We have a number of outstanding queries too – some months old that require attention – just see our Forum page.

Secondly – and we’ve checked this a lot because we can hardly believe it – Police.uk have actually changed the underlying historical source data files located here without telling anyone – all the way back to December 2010 !

We know they’ve done this because we keep published historical records of all our data for everyone to see. This means that Police.uk has acted in direct contravention to the Home Office Crime Data Guidance which states;

 http://data.gov.uk/sites/default/files/Crime_Data_Guidance20101207.pdf  “When errors are discovered, or files are changed for other reasons, rather than ‘silently’ changing the file it is recommended to publish a revised and differently named version (together with the original files), e.g. 02_2011_v2, 02_2011_v1.

Unlike the Home Office funded website, at UKCrimeStats, we comply with this guidance – if you’ll forgive the management-speak, it seems like “best practice”. See here http://www.ukcrimestats.com/Disclaimer/ and scroll approx. 80% down the page to the data entry for February 2011 for Cheshire Constabulary and few others, all of which we spotted and asked to be fixed – all of which took far too long. 
We haven’t done a full audit but we can confirm that the following data has been changed on www.police.uk/data ;

1) The Eastings and Northings of the street csv files – i.e. the geolocational data of several hundred thousand “snap points” (the on or near location where said crimes/ASB incidents are mapped to  – have all been changed. Eastings and Northings are meaured in metres – prior to a few months ago – specifically prior to the December 2011 update, these were shown as for example Easting 40892 and Northing 91364 – this corresponds to Fir Vale Road in Bournemouth. Last month, all the crimes registered to the snap point in that street were changed to 40892.0008 and 91363.99942. So in this case, they have been moved less than a millimetre in one direction.

What this means is that we now have 2 separate snap points for the same street http://www.ukcrimestats.com/Streets/50.721836105634/-1.874109374612/ and http://www.ukcrimestats.com/Streets/50.721827128272/-1.874123564855/ . 
Obviously, that’s not right – so we’ve having to fix this and run scripts that literally take days of computing power to join these streets as well as everything else. We are still awaiting an explanation from the authorities.
2) Neighbourhood crime csv files have also been changed on police.uk/data – quite substantially for Kent and Devon and Cornwall – we suspect there are others too. For these two examples, they have increased significantly the number of recorded crime from;

Police Force Month NHOOD CSVs: Crimes and ASB (with and without location) NHOOD CHECK – 27/2/2012 Nhood check difference
Devon and Cornwall Constabulary April-11 13,286 13,455 169
Devon and Cornwall Constabulary May-11 12,286 12,902 616
Devon and Cornwall Constabulary June-11 12,154 12,484 330
Devon and Cornwall Constabulary July-11 14,342 14,630 288
Devon and Cornwall Constabulary August-11 14,076 14,468 392
Devon and Cornwall Constabulary September-11 11,964 12,322 358
Devon and Cornwall Constabulary October-11 12,541 12,916 375
Devon and Cornwall Constabulary November-11 11,168 11,878 710
Kent Police September-11 13,763 14008 245
Kent Police October-11 14,346 14808 462
Kent Police November-11 12,460 12658 198

On neighbourhoods too, we are still awaiting an explanation from the authorities. It takes a lot of time to work out what has gone wrong, figure out  a solution and then process it through. We are now on the processing stage.

As a final word, we think this is an open and shut case of why it is so important to open up the crime data to third parties and developers. The more eyes look at it the better. Monopoly oversight and control of crime data is dangerous and prone to errors that will either not be discovered or silently changed from the inside – as appears to have happened here.  For open data to succeed, it has to be really ok for governments to say “Sorry, we’ve made a mistake” and publish the correction. After some upset, this openness builds public confidence in the crime data because new pressure is created to prevent a re-occurence. Quite the worst thing of all is to try and change it on the quiet and hope no one will notice.

Next week I have meeting with some officials at the Home Office to discuss points like these and the week after with the Crime and Justice Transparency Sector Panel of the Ministry of Justice.

It was a great achievement of the government to get 43 Police Forces to start releasing data simultaneously once a month. It is the largest publicly available crime dataset in the world by far. With open data like crime, we think the role of government should be to continuously improve it’s quality as well as it’s quantity as long as that does not undermine judicial process or anonymity. However, you can’t just release the data and walk away and say job done without engaging with those who use it.

One thought on “Why the January 2012 update is coming to you late next weekend

Leave a Reply