Why the January 2012 update is coming to you late next weekend

Dear UKC followers and subscribers,

Normally, we aim to release our update within a week after Police.uk release theirs and often much sooner. So we owe you an apology and an explanation on why it’s later than usual.

First of all, Police.uk aim to release on the 25th of the month but this time, it was released on the morning of the 1st March. There was no explanation or less still, an apology on their behalf. In our view, this near-zero level of interraction with 3rd party developers has a lot to answer for. We have a number of outstanding queries too – some months old that require attention – just see our Forum page.

Secondly – and we’ve checked this a lot because we can hardly believe it – Police.uk have actually changed the underlying historical source data files located here without telling anyone – all the way back to December 2010 !

We know they’ve done this because we keep published historical records of all our data for everyone to see. This means that Police.uk has acted in direct contravention to the Home Office Crime Data Guidance which states;

 http://data.gov.uk/sites/default/files/Crime_Data_Guidance20101207.pdf  “When errors are discovered, or files are changed for other reasons, rather than ‘silently’ changing the file it is recommended to publish a revised and differently named version (together with the original files), e.g. 02_2011_v2, 02_2011_v1.”

On neighbourhoods too, we are still awaiting an explanation from the authorities. It takes a lot of time to work out what has gone wrong, figure out  a solution and then process it through. We are now on the processing stage.

As a final word, we think this is an open and shut case of why it is so important to open up the crime data to third parties and developers. The more eyes look at it the better. Monopoly oversight and control of crime data is dangerous and prone to errors that will either not be discovered or silently changed from the inside – as appears to have happened here.  For open data to succeed, it has to be really ok for governments to say “Sorry, we’ve made a mistake” and publish the correction. After some upset, this openness builds public confidence in the crime data because new pressure is created to prevent a re-occurence. Quite the worst thing of all is to try and change it on the quiet and hope no one will notice.

Next week I have meeting with some officials at the Home Office to discuss points like these and the week after with the Crime and Justice Transparency Sector Panel of the Ministry of Justice.

It was a great achievement of the government to get 43 Police Forces to start releasing data simultaneously once a month. It is the largest publicly available crime dataset in the world by far. With open data like crime, we think the role of government should be to continuously improve it’s quality as well as it’s quantity as long as that does not undermine judicial process or anonymity. However, you can’t just release the data and walk away and say job done without engaging with those who use it.