Big and open data and the funding base of the third sector

Much has been made of the transparency in the public sector that will be stimulated by the release of data on public procurement. At the same time social scientists are enthused by the potential of “big data” to improve our understanding of social processes. What can we glean from the data that’s available so far? My interest in this derives from work on the funding base of the third sector, but also from research into the changing balance between public and private provision of public services.

There is more on this work at but briefly, based on a presentation today at the International Society for Third Sector research (which you should be able to find at the above website), I would raise some questions about the utility of the data we have while also arguing that it can help us answer important questions if we have data of appropriate quality. One criticism of “big data” in social science is the “big data, small questions” argument – in other words, the really difficult questions that societies face won’t be answered simply by large-scale data manipulation. However few would argue, in relation to the public sector, that the balance between commercial, voluntary and statutory provision is not an important question.

So far, though, the potential has yet to be realised. Procurement data don’t necessarily have the volume, velocity and variety of consumer databases, though they are pretty large: our databases on local authority procurement include c. 26 million transactions. This poses challenges especially when public agencies don’t comply with codes of good practice: they are supposed to disclose company and charity ID / registration numbers, but rarely do, which means that analysts have to develop sophisticated name-matching methods. There are other glitches and inconsistencies, such as varying practice in terms of redactions of information.

More manageable datasets do, however, suggest there is great potential in the use of open data. Clinical commissioning group data for the NHS, for example, are smaller-scale because they only have to disclose transactions above a £25000 threshold. Of 300 000 + transactions we can identify around 10 000 unique suppliers. We can see from this some interesting emerging patterns. Charities receive relatively limited shares of funding – perhaps under half a percent; social enterprises get rather more, in the range of 2%, but this is mainly organisations that have been “spun out” of the NHS as part of the mutualisation agenda. Commercial contracts for clinical care are larger, at c. 2.5%. Some of these could be underestimates, because at the moment we cannot take account of subcontracting relationships. But the data point in interesting directions. For example, David Cameron talked of creating the “largest social enterprise sector in the world” by moving much NHS care into non-profit companies such as Community Interest Companies. Thus far this has had limited effect. There is some suggestion from the data we have that private contractors are picking up more contracts, but that really requires long-term data. The data also raise questions about what the normative balance might be. While advocates of the voluntary sector might claim that there ought to be a significant expansion in their role, there is no way of determining what the appropriate proportion might be. One can see though that this information does provide an important baseline against which we might monitor significant changes in the NHS – and it also raises questions about how well public agencies are complying with the codes of practice which are supposed to govern their release of data.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s