Data culture or data for culture? What survey metrics in decision-making can and can’t do for the arts

There has been recent media coverage for a test in which Arts Council England (ACE) experimented a set of standardised metrics in the assessment of art produced by its funding recipients.

The test is the third one in a period of 3 years, where the first two focused on Manchester-based organisations while the most recent was carried out on a national level.

The system was initially discussed within the Department of Arts and Culture of Western Australia under the supervision of researcher and entrepreneur John Knell, now involved in ACE and the organisation Culture Counts, which is a research group that deals with the development of this open-source framework of metrics.

At the same time, Culture Counts is also a registered company offering the online platform that can manage the data of the framework, allowing the organisation to keep control of it and share findings with both ACE and other organisations.

In practice, the framework consists in a standardised collection of survey-based data, coming from three separate entities involved in the cultural products of the assessed: the people who participated in the production (self assessment), a number of experts in the field (peer-assessment), and the general audience of that production (public-assessment).

According to the framework, most metrics apply to all three these groups, except two of them about which the general public is not asked to give an opinion. They can be found on the ACE’s website here.

The framework, the way it would be used by ACE, and the general idea of a quantified assessment of art quality have all been subjects of critique.

We think some of the complaints about the entire project are properly argued whereas others stem from ideological positions, most of them defending an independence of the arts from an institutionalisation that supposedly comes from a quantified-assessment of them.

Concerning CC framework itself, the problems pointed out by some of the organisations that participated in the last pilot are hard to disagree with.

The widely reported one is about the idea that the same quality metrics should be the same for the entire field of the arts.

Intuitively, it is hard not to see how concepts of quality differs between an experimental music show, a circus night, an exhibition of paintings and opera theatre for example.

A standardised approach across all the sector cannot be received but with scepticism by the sector.

And even if we were to go beyond the intuition, the data from the last test that was made available by ACE doesn’t help in feeling at ease with the proposed standardisation.

From the way this data is structured, it is impossible to understand whether there is a specificity in the responses according to cultural sector.

For example, it is impossible to understand whether sculpture peer reviewers are more likely to give positive feedback than music peer reviewers.

Or whether cinema audiences are less picky than ballet audiences.

Or if self assessment for an art curator differs from that of a literature curator.

Assuming this cross sector examination data is available to ACE, we think it is important to use it to understand the different specificities existing within the very broad field of culture, so in the future different frameworks for every one of its subsystems might be implemented.

Such differentiation and indeed sophistication of the general approach would also help avoiding a strange competition within the sector, a scenario where a specific art form could come across as “better” in the data simply because the general response to it best suits the standardised metrics.

Identifying sector specificity may be a complex and ambitious task, some work that nonetheless seems necessary for the framework to provide relevant information.

We also believe that a sophistication of the system would also help clarifying some of the quality metrics for their users, working to respond to a lack of clarity of the terms expressed in the metrics, a problem that was pointed out by some of the participating organisations in the test (for more information, the reader can refer to the independent analysis of the test commissioned by ACE to consultancy firm Nordicity here.

Another relevant issue concerns how data from the framework would be used by ACE in their decision-making process, both in the assessment of already funded recipients and in the evaluation of future possible ones.

The topic is partially investigated in a paper commissioned by ACE called Counting What Counts, written at the beginning of 2013 by media expert Anthony Lilley and art policy professor Paul Moore.

The paper praises a so called data-driven decision-making process in sectors such as the arts, referencing to the important work on biases and heuristics of Daniel Kahnemann and how the hidden rules in general human decision-making can be easily exposed by statistical data.

As crucial as the synergy between human and machine in the process of selecting from a set of options is, it is important that its workings are well communicated from the beginning.

The point here is to understand how this hybrid decision-making would take place, and unfortunately ACE has been quite vague (or still unprepared) to extensively define the process in the communication campaign for its quality metrics framework.

At the moment, all the stress is on the importance of obtaining larger quantities of data but very little is said about its integration in the existing decision-making process.

For example, would figures on peer reviewer satisfaction count more or less than those on the audience satisfaction when the assessor needs to make an assessment about a certain production/organisation?

A possible approach to research how this human/data relation could work is a straightforward comparison between the proposed metrics or KPI (key performance indicators) and the form that is already in use by art assessors at ACE (link) in their duties.

Unfortunately the institution didn’t respond to our request to share this document.

The last argument considers an expectancy of independence of the arts from institutionalisation through quantified approaches to quality.

In this case, institutionalisation can’t be naively referring to a generic institutionalisation since any organisation automatically (and intentionally) looses a deal on independence when it becomes a funding recipient.

The point is to argue whether a quantified approach would rather make the relationship between the funded and funder more transparent, fairer and more fruitful for both audience satisfaction and the development of an art discourse.

In our opinion, the paper by Lilley and Moore rightly claims that, if a system of metrics that assesses art quality or even social capital is not in place, the only approach to quantification would be based on data that is already collected: economical figures and size of audiences.

In this case, the risk would be to simply confirm a very current “model that tends to trap many cultural organisations in a survivalist, financial mindset” (Lilley/Moore).

To conclude, the work of ACE and CC in the last year has been one of the most interesting experiments in Europe in developing alternative decision-making strategies when it comes to public funding bodies.

We have here attempted to show that there is room for improvement in the proposed framework and any possible changes should come from a conversation about the topic that is freed from both ideological dismissal and a too positivist and superficial view on a concept of data-driven (or even machine-based) decision-making.

November 25, 2020