In need for a drastic change: on the "evidence-based" debates in cultural economics and cultural policy research
Andrej Srakar Institute for Economic Research Ljubljana; Faculty of Economics, Ljubljana firstname.lastname@example.org
Abstract Cultural policy research is overloaded with speaking about »evidence-based cultural policy«. But, has cultural policy research gone far enough in the production of policy-relevant knowledge? Indeed, is it asking the appropriate research questions at all? Is evidence-based cultural policy research not merely an example of bullshit, i.e. currently prevailing rhetoric with no meaning at all, »a signifier without the signified«? Our study surveys the examples of evidence-based cultural policy research, following two streams: economic impact of culture, and composite indicators. We show that in both streams, cultural policy research satisfies itself with rhetorical figures and descriptive analysis, unable to answer the most basic research questions. At this point, therefore, evidence-based cultural policy is only a rhetoric concept, by policymakers and researchers. We provide a broad set of research questions to be addressed in future, examples of methods and datasets, and good practices from other sectors of public policy. Keywords: cultural economics, cultural policy, bullshit, evidence-based, economic effects, statistical methodology
JEL classifications: Z11, Z18, H41, R11, C43
Introduction: on the concept of evidence-based policy research
In her often-quoted article, Belfiore (2009) introduced the concept of bullshit in evidence-based cultural policy by referring to (mis)using statistical calculations with the aim of cultural policy persuasion and demagogy. She pointed to this problem as »statisticulation« (referring to the usage by Darrell Huff, see Huff, 1954), presenting several misusages of statistics, related to cultural policy practice in United Kingdom. Although her article received good response in cultural policy research, we claim that it might have caused misusage for presenting statistical research as »the root of evil« in cultural policy research. Furthermore, we briefly demonstrate that the debates in cultural policy research, particularly emphasizing the concept of evidence-based cultural policy, are misplaced. Most of these debates rely on the contrast between, on the one hand, strongly theoretically based arguments and, on the other, very basic and “rudimentary” methods to solve the highly complex problems of cultural statistics and, consequently, try to avoid the issues it should be preoccupied with: improvements of statistical methodology and a consequent more proper, concise and content-rich answer to the problems under consideration. We use the concept of bullshit as a grounding point. Most of the discussion on this concept started with the work of American philosopher Harry G. Frankfurt, namely the essay »On bullshit« in 1986 (later reprinted in a book form by Princeton University Press, see Frankfurt, 2005) where Frankfurt presents a theory of bullshit that defines the concept and analyzes its applications in the context of communication. As such, bullshit “can be neither true nor false; hence, the bullshitter is someone whose principal aim is to impress the listener and the reader with words that communicate an impression that something is being or has been done, words that are neither true nor false, and so obscure the facts of the matter being discussed” (Frankfurt, 2005: 30-34). In our article, we demonstrate that the present discourse on evidence-based cultural policy research is a clear example of bullshit: it aims to impress the reader / listener that something is being done / researched (or even better to say: thought / reflected / critiqued) with a large disparity over what was promised / supposed to be done and what was actually done. We demonstrate this by using two topics which we consider as excellent examples of bullshitting related to evidence-based cultural policy research: economic impact studies and the construction of composite indicators in culture. While previous studies (Belfiore and Bennett, 2007; Belfiore, 2009; 2010) already identified the first (economic impact) as an example for bullshit in cultural policy practice, we provide a step ahead from their elaboration: we see a potential of the debate on economic impact of culture which has been misused, but not (or not only) by the practitioners and policymakers but rather by the researchers themselves. It is necessary to firstly concisely define the term “evidence-based policymaking”. As stated by Sutcliffe and Court (2005: 1), “The idea of using evidence to inform policy is not new. As far back as ancient Greece, Aristotle put forward the notion that different kinds of knowledge should inform rulemaking. This would ideally involve a combination of scientific knowledge, pragmatic knowledge and value-led knowledge”. As stated by Davies, evidence-based policymaking is an approach that “helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation” (Davies, 2004: 3). Such discourse has become popular among a range of policy communities, those within government departments, research organizations and think-tanks (Sutcliffe and Court, 2005). Shaxson (2005) argues that we need evidence in policymaking, in terms of policy, to understand the policy environment and how it’s changing; and appraise the likely effects of policy changes so we can choose between different policy options and subsequently assess their impacts. In terms of strategy, we need it in order to demonstrate the links between strategic direction, intended outcomes and policy objectives; and determine what we need to do to meet our strategic goals or intermediate objectives. In terms of outreach, we need it to influence others so that they help us achieve our policy goals and take them through to delivery; and to communicate the quality (breadth and depth) of our evidence base to meet the open government agenda (Shaxson, 2005:106-107). When speaking about the research foundations for evidence-based policymaking, Sutcliffe and Court state that “evidence-based policy should be based on research-based evidence” (Sutcliffe and Court, 2005: 3). On the other hand, they adopt a very general and widely accepted definition of research as any systematic effort to increase the stock of knowledge (see also OECD, 1981). Thus, to their opinion, such research can include “all kinds of evidence as long as they have been collected through a systematic process” (Sutcliffe and Court, 2005: 3). Simply said, almost anything that could count as research could be the foundation of evidence-based policymaking. Our methodological approach will be descriptive and based on the presentation of two case studies to support our main claim. Firstly, we will present the extant evidence in the literature on the economic impact of culture, the critiques of the existing methods, some of the related discussions in the field of cultural policy and solutions in cultural economics that are emerging in recent years. Secondly, we will present also the emerging field of composite indicators in culture (related to the field of cultural statistics in more general terms) and similar problems (with a similar structure of the presentation of our arguments) emerging there. The remainder of the article is organised as follows. Section 2 presents the debate on economic impact of culture (related also to the work of John Myerscough, the main topic of the special issue), its problems, the proposed solution and plethora of possibilities of future work. Section 3 develops similarly the debate on cultural indicators, very lively presently in both cultural policy and cultural economic research. Finally, section 4 concludes by pointing to the other fields of cultural policy research which also succumb to the problem of bullshitting and a reflection on the future usage of statistical methods in the research of cultural policy phenomena and practice.
Evidence-based research on the economic impact of culture
The debate on the economic impact of culture, at least in cultural economics, started in the 1970’s, with the American »monetary experiment« (Barsky and Kilian, 2000) when the US started to use extremely restrictive monetary policy to solve the problems of stagflation. To this reason, many areas of public economy have come under closer public scrutiny and were faced with a significantly changed financial environment. To adjust, they started using economic arguments to justify their support (in culture, this debate is nicely summarized in the work of Radich, 1993). The debate spurned so-called economic impact studies, starting with two influential studies: the 1977 report, Economic Impacts of Arts and Cultural Institutions: A Model for Assessment and a Case Study in Baltimore (Cwi and Lyall, 1977); and the 1983 study, The Arts as an Industry: Their Economic Importance to the New York-New Jersey Metropolitan Region (Port Authority of New York and New Jersey – Cultural Assistance Center, 1983). Such kind of studies pretended to calculate the “net economic impact” of a certain cultural event, mainly to show its large benefit for the economy and community in general. In an influential essay, Frey labelled the proponents of such studies as “arts people” which “focus more on the economic effects of the arts than economists do. Or conversely: arts economists concentrate more on the artistic aspects than arts people do” (Frey, 2005: 2). In Europe, such “economic” arguments have been glorified and used in favour of public support of the arts in the study published in 1988 by John Myerscough and entitled The Economic Importance of the Arts in Britain (1988). As stated by Belfiore (2003: 1), it “was a highly controversial publication, strongly criticized particularly by cultural economists, yet, it opened the way to an increasing number of similar studies claiming to be able to prove and measure the importance of the arts sector to the local and national economy”. Myerscough demonstrated, through the use of a multiplier-based analysis, that direct spending on the arts led to spending in other sectors which in turn enhanced wealth and job creation on the city and country level. This study had a far-reaching impact on the cultural sector and strengthened its argument for the economic impact of the arts as a powerful justification for continued public funding. Studies on economic impact are of course clear examples of research based evidence, as defined by Sutcliffe and Court. But it is also necessary to claim that such kind of evidence is misplaced. Although it uses some rudimental statistical methods, it clearly uses them a) wrongly – the numbers, calculated by such methodology are clearly overblown and not supported by the ex-post evidence (Seaman, 1987; Seaman, 2006); b) purposefully – the numbers are clearly calculated with the purpose of showing large impacts to justify the economic value of the event and convince the funders that it is economically profitable to invest in (Frey, 2005). This has commonly led researchers to conclude on the inappropriateness of such approach and even proposing different types of agendas, focused on defying instrumental rationality and pronouncing critical approach (see e.g. Belfiore, 2010). In cultural economics, such critical approach is probably best described by the research agenda on the usage of contingent valuation method to study the individual preferences and “total economic value” (see Peterson and Sorg, 1987) of the event, encompassing both use and non-use values and including sometimes also cultural values, being broadly defined as values of culture outside of the economic / monetary valuation (the debate on cultural values is today very widespread, see e.g. Klamer, 1996; Hutter and Throsby, 2008; Oakley et al., 2006; Hesmondhalgh et al. 2014; Oakley and O'Brien, 2015). Such debate and usage of methodology originates from environmental economics and was transferred to cultural economics in the 1980’s by the study of Throsby and Withers (1986). Today, the debate on the economic impact and value of cultural events in academic circles has been predominated by the usage of contingent valuation methodology. Some researchers also use other methods, like life satisfaction approach (Steiner, Frey and Hotz, 2015), new internet possibilities, such as Google Trends and Google News (Plaza et al., 2015) or even referenda (Frey, 2000). As a consequence, many researchers in academic cultural economic and cultural policy research believe that we should completely forget the economic impact studies due to their numerous flaws and problems. Although this speaks in favour of using statistical methods (which are significantly more complex in contingent valuation studies than in “classical” economic impact studies), we should ask ourselves: is this the appropriate path of research and does it answer to the research questions under hand? If the purpose of the analysis is to measure economic impact of the arts, the contingent valuation method and other above mentioned methods clearly do not answer the main questions: a) do the art events have significant economic effects, as measured by e.g. new income, employment spaces, additional tourism and taxes raised; b) how large precisely are such effects; c) on what characteristics do they depend upon. We can certainly agree that “value” of the arts is multidimensional and cannot be completely encapsulated in either use or monetary amount. Nevertheless, the responses currently provided do not answer the original questions, pretending they are impossible or largely unimportant. At present, there is a new way that appears promising to solve the present conundrums of economic impact research which, as a paraphrase of Frey, is stuck in the futile dichotomy between the “arts people”, using economic impact studies (providing wrong numbers), and the “arts economists”, using contingent valuation and similar approaches (answering the wrong questions – and being prone to numerous own methodological problems, exemplified and warned against by e.g. Diamond and Hausman, 1994). This methodological path is called ex-post econometric verification, and has been used in culture very seldom (Skinner, 2006; Srakar and Vecco, 2016; Srakar, Slabe-Erker and Vecco, 2016). It originates in sport economics, starting with work of Baade and Dye (1988). The method uses existing statistical data after the event takes place (ex-post) and econometrical methodology to discern a “blip” (Gergaud and Ginsburgh, 2013), caused by the event, in the data. There are many advantages of the method which clearly answer all of the above challenges (and in an easy manner): a) it is done after the event; b) it uses a methodology, which suffers from no additional problems, characterising both economic impact studies and contingent valuation (overblown results, hypothetical bias, micro vs. macro focus); c) it employs statistical data, measured under commonly accepted methodology; d) its results can be compared across events, regions, countries; e) it is not expensive or methodologically over-complex. Although there exist several issues also with this methodology, such as whether it is possible to really discern the “blip” from the data and in which cases is this even impossible due to e.g. small event in a large city, inadequacy of data, many other competing events and happenings at the same time, etc., it is justified to say that if any method is able to answer to the above pointed research questions in best manner, it is probably this method. The possibilities provided are extensive: the method can be used to study almost any cultural event under question and even to relatively easily compare them (a problem of “benefit transfer” that contingent valuation is hardly able to answer, see Whitehead, Morgan and Huth, 2015). At present, the methodologies for studying the ex-post economic impact of a cultural (or sport) event can be broadly classified into two types: time series methodology, which can be applied to small events and very few variables with adequate data, as demonstrated by Skinner (2006); and panel data methods, which can be applied when the data allow richer possibilities and comparison among different individual units. Therefore, the method allows to be used in numerous different settings and it remains to be developed and explored in its possibilities in future research. It also demonstrates a key finding for our article. The impotence and flaws of previously used economic impact studies do not mean that the question of economic impact cannot be studied methodologically and even using traditional statistical and econometrical framework. In cultural economics, the debate has so far been caught between two contested and futile options, both having significant methodological and ideological problems of their own. In cultural policy research, the prevailing misplaced economic impact studies of the “arts people” have led the researchers to conclude on the inappropriateness and bullshit character of such studies and to the need of the program of research on arts impacts that would “not be confined to the demands of an instrumental rationality”. In both research fields, the main research questions to our opinion remain largely unanswered despite the amount of articles and studies done in past decades. To our opinion, the focus of research should be significantly changed in future and drawn back to the original economic questions as stated above. Yet, it should use a different methodological approach and agenda, and, furthermore, should not be “purposefully” oriented. Interestingly, as demonstrated by the evidence in existing sport economic ex-post econometric studies, almost never they find an economic impact as predicted by the ex-ante studies, even more, such effect can be significantly smaller and sometimes even negative (Seaman and Price Elton, 2016; Srakar and Vecco, 2016). The findings which are, therefore, more realistic, appear to distract the usages by the “arts people” in future and promise an interesting and fertile research agenda for the future.
Evidence-based research on composite indicators in culture
Another contested topic of evidence-based research in cultural policy is the construction and usage of composite indicators in culture. According to the OECD glossary, “a composite indicator is formed when individual indicators are compiled into a single index on the basis of an underlying model of the multi-dimensional concept that is being measured” (OECD, 2007). In the presence of an ever wider need for measurement of composite and multi-dimensional concepts, the need for a developed methodology for constructing composite indicators has come to the forefront of attention in many fields of research. This has been summarized in influential studies of OECD (Nardo et al., 2008), which provides a detailed description and elaboration on the main required steps in building any composite indicator, and Bandura (2008) who provides an inventory of over 400 country-level indices, with topics spanning from economic progress to educational quality. In culture and the arts, the haze of making cultural indices is also in a significant rise. Endeavours such as the US National Arts Index, Arts Index Netherlands, recently published Indicator Framework on Culture and Democracy, several efforts to construct a European Cultural Index (see e.g. Inkei, 2013), British NCA Arts Index, ARC Creative City Index, Creative Community Index, Florida’s Creative Cities Index, Euro-Creativity Index, Cultural Life Index, Creative Vitality Index, Intercultural Cities Index, and research and overview articles such as Srakar, Verbič and Čopič (2015), Kregzdaite et al. (2016) and Rodríguez Ramos et al. (2016) show the intense efforts into construction of an appropriate composite indicator to measure the condition of culture. Yet, as pointed out by Srakar, Verbič and Čopič (2015), even the most basic methodological principles for constructing composite indicators, such as appropriate considerations of weighting, multivariate analysis and sensitivity analysis, are for the most part absent from all of the above mentioned indices. The need for improved cultural statistics has been exemplified in studies and reports such as Bína et al. (2012) and there are many problems of cultural statistics, not least being the comparability of data across countries due to different definitions of culture. Furthermore, Eurostat as the main European statistical institution does not provide any regular / yearly data on cultural indicators, so most of the studies have to rely on sporadic Cultural Statistics Pocketbooks, provided by the same institution (at present there have been three editions, published in 2007, 2011 and 2016). This justifies special consideration provided to statistical indicators in culture. Yet, this provides also reasons for wonder why so far no institutional effort on developing a comparative composite indicator of condition of culture (i.e. cultural index) that would follow more closely the statistical guidelines of OECD (Nardo et al., 2008) has been provided. Is this merely the lack of statistical knowledge among researchers in cultural policy – but, if this is so, this surely provides reasons for serious concern. As we note in conclusion to this article, at present almost no topic in cultural policy research has been provided a solid and complex statistical framework of research, not least to mention that it would be adequately researched in statistical and/or econometric terms. One short example we will use is National Arts Index as developed by the organization Americans for the Arts. The index, composed of 83 indicators, comprises all sectors: non-profit organisations, for-profit businesses, individual artists, as well as amateur levels of activity. On a broad level, the indicators are grouped into four dimensions: (1) financing, (2) capacities, (3) participation, and (4) competitiveness. Each dimension adds up to a respective index. Interestingly, the selection of dimensions does not follow any previous statistically developed analysis and is merely guided by intuition (as is characteristic of almost any existing institutionally provided index in culture). Also, no particular consideration is provided to weighting methods, such as factor analysis, principal components, structural equation modelling, etc. which is clearly contrary to suggestions of Nardo et al. (2008). Despite its problematic statistical structure, National Arts Index is used in numerous policy publications and is even used as a reference by e.g. the Arts Index Netherlands (see Boelhouwer et al., 2013), which is another example of weak statistical structure – composed by mere basic summation over intuitively composed set of dimensions without any used weighting scheme. Not much difference could be found for e.g. NCA Arts Index as the main cultural index for United Kingdom. The most recent composite indicator attempt is the Indicator Framework of Culture and Democracy, described in research reports of e.g. Council of Europe (2016) and published in beta version in October 2016. The indicator framework was intended “to launch a medium-term working process that should include work on indicators of the impact of cultural activities on democracy as well as the economic efficiency of financing culture in order to improve the effectiveness of cultural policies” (Council of Europe Conference of Ministers of Culture, 2013). It consists of 8 dimensions – 4 for respectively each culture and democracy; in total it includes 177 variables, transformed using basic z-score normalization, for 37 Council of Europe member states. The data are compiled for one cross-sectional period, although gathered many time for different years, due to inaccessibility of data. This indicator framework is very ambitious in its attempt to provide a tool to “be used by governments to adjust cultural policy in order to spend money where it is most needed, make access to culture easier where required, assist marginal and excluded groups where necessary and let the private sector and civil society take responsibility where needed and possible” (Council of Europe, 2016). Furthermore, it claims to be able to analyse the causal relationships between culture and democracy. Causal inference is an important part of contemporary statistical and econometric analysis, receiving an extensive coverage with some of the best known works by Angrist and Pischke (2009) and Morgan and Winship (2014). It shows that when data are considered in an inconsistent statistical manner, it can soon lead to problematic and wrong conclusions about causality. Recently, problems of overly simplified statistical evidence when analysing the effects of institutional (e.g. political, such as democracy) characteristics of a country on e.g. economic growth have been exemplified by Pozuelo, Slipowitz and Vuletin (2016). It is, therefore, reasonable to question whether the attempts such as this indicator framework do not succumb to the problem that Diamond and Hausman nicely labelled as “is some number better than no number” (1994). Definitely, it should be taken with careful consideration and could easily lead to rush and oversimplified / wrong “statistical” conclusions. Again, the shortly presented debate has pointed to our main claim: the problem of using statistical indicators and / or methods to study culture does not lie in the usage of statistical methodology per se, but rather in its inadequate usage in present day research in cultural policy. It is not accidental that despite numerous existing attempts to construct cultural indexes by cultural organizations and institutions, there are to date to our knowledge no published scientific articles with statistical methodology on this topic. On the other hand, the possibilities for (statistical) research on this topic are rich. A clear one is better exploration of the characteristics of cultural statistics, specific for this domain. How to take into account the problems of different definitions of culture – would any particular statistical methodology be able to provide a more appropriate and timely answer to this question. Furthermore, what means could be used to take into account the missing data problems in existing cultural statistics? What is the relation between cultural indicators and indicators of sustainable development – economic, social, environmental – and could those relationships be used to better take into account problems with existing cultural indicators? Could perhaps the methods of multivariate analysis (e.g. structural equation models, correspondence analysis, tree modelling, modern methods in clustering, etc.), taking into account the latent / unobserved nature of many cultural phenomena be used to study cultural statistics in a more appropriate manner? Those are just some of the many research questions and possibilities that would not only enrich the research agenda in cultural policy but also improve the knowledge in statistical methods and econometrics in general.
Discussion and conclusion
In conclusion, let’s firstly resume the debate and our arguments. In the introductory section we defined the concept of evidence-based policy research and pointed to some of the problems when applied to the existing research practice in the field of cultural policy. We presented the existing evidence in two large areas of cultural policy research: economic impact of culture and composite indicators in culture. We pointed to large problems of existing studies which cannot be attributed simply to misuse in practice or problems of statistical methods, but mainly to inadequate research work and lack of usage of appropriate statistical methodology. To our opinion, we could attribute this in large part also to professional affiliation of the researchers themselves: in particular in the field of cultural policy research, more sophisticated statistical analyses are extremely hard to find and even to convince they are needed. To our opinion, this is contributing to both a) over-pronouncement of theory over statistical work; and/or b) misuse and false arguments, using poorly done and “purposefully” oriented statistical work, as demonstrated on the cases of two chosen fields. This does not mean the theory should be abandoned for the purpose of empirical and statistical work, quite the contrary: the more solid and profound evidence that is missing at present would, on the one hand, need to be supported by even stronger theory, interpreting it and putting it to the (changed) context, while, on the other, surely leading to significant theoretical developments in future, which are at present, paradoxically as this may sound, almost impossible – with missing evidence and missing answers (mentioned previously and in below paragraphs), leaving the field with mainly theoretical speculations of "what should be there". A related question is, surely, the implications: for both cultural policy research and cultural economics. Related to the first topic, economic impact of culture, the over-exaggeration of this debate and its consequences for the field of culture in general (presented in Section 2) has definitely contributed to the severe and strange situation, faced at present – a strange mix of "bullshitting" on the side of practitioners and avoidance of answering more serious questions on the side of the researchers (both of cultural policy and cultural economic provenience). If such situation will persist, it will surely have an additional and strong adverse effect on the perception of both fields in scientific, policy and more general and wide public circles. The problematic situations, when the proclaimed economic effects are simply "not there" (as described shortly in Section 2), cannot but contribute to marginalization of both cultural policy research as well as cultural economics. The main intention and novelty of the article, indeed, is to point to this: although there is a plethora of already existing critiques, they seem to be misfounded and simply leading nowhere. Misplaced economic impact studies are still done, even quite frequently, and it is just to say that until the methodological development will not be able to catch up with the real problems, laying unanswered in the field, such studies might even prosper, develop and overthrow all the efforts and critiques by the researchers – because simply there is demand for such research. And if such demand is left unmet and taken unseriously (as, unfortunately, is the situation at present), this will surely and gradually lead to even more bitter consequences as were in the past decades since the first such studies have been done. Not much different is the situation with composite indicators in culture (and, indeed, many or even most of the fields of empirical cultural policy research – as described in more detail below). Here, the field is much "younger", still only emerging, but, indeed, already with full of "bullshitting", as defined in our article: with a lot of poor statistics, with a potential of leading also to a lot of problematic and wrong conclusions. The only solution that this article is able to provide is clear, but demands dire changes in the present situation, in particular for cultural policy research: significantly more effort into more demanding and sophisticated (but focused on problems, not methods per se) statistical and econometric work. Only with this will the field be able to catch up with the development of other scientific fields at present, and, indeed, be able to provide more concrete and developed answers to many research questions, left open. In the article, we did not satisfy ourselves with the description of present condition and presented possibilities of corrections and pathways for future research, which seem many. As noted, on our opinion, however controversial this may sound, to date almost no topic in the empirical and statistical research on cultural policy has been adequately covered. Even more, to date we could hardly find any existing study providing any more solid and complex / sophisticated statistical evidence on those topics. Numerous topics, beside economic impact and composite indicators, come to mind: public financing of culture – to date there exist almost no econometric cross-country analyses on the determinants, dynamics and characteristics of public budgets in culture and its relationship to other macroeconomic and policy variables. Some studies (e.g. Čopič et al., 2013) point to lack of knowledge of the relationship between central and local public budgets for culture, yet the data on both are clearly available in relatively long time series, provided by Eurostat’s COFOG database. Furthermore, the relationship between public financing and employment in culture has remained un-modeled and under-researched – one would clearly expect a causal relationship, with public financing positively affecting the employment, but no evidence has been provided so far to our best knowledge. Extremely large and unstudied topic is the effects of the implementation of different policy measures on the outcomes of cultural policy. Many other fields of policy analysis use counterfactual methods (e.g. Angrist and Pischke, 2009; Morgan and Winship, 2014) deriving from a large econometric field, called program evaluation methods. Yet, to our knowledge, usages of this methodology to provide “evidence” on the effects of cultural policy measures can hardly, if at all, be found in the literature. The decisions of expert commissions have also remained a largely unexplored topic, although receiving some literature in past years (e.g. Meskell et al., 2015). Large macro-models, like microsimulation models and different types of general or partial equilibrium modelling have also remained largely a void in the field, although used in many other policy areas (e.g. education, social policy, health care, labor market). Studying efficiency of public institutions has gained momentum in past decade, following works like Cuccia, Guccio and Rizzo (2013) and Zieba (2011). Still, several methodological issues remain open here as well, like comparison of different estimators, studying both technical and allocative efficiency, and, furthermore, finding a method which would be able to capture not just basic quantitative aspects of the efficiency of organizations, but also some of the more qualitative aspects and, indeed, the complexity of the problem of efficiency. Also, international trade with cultural goods is still very much under-reseached with only handful of existing studies (e.g. Marvasti and Canterbery, 1992; Disdier et al., 2009; Qu and Han, 2011). The list is not conclusive – one could list many more topics in the “evidence-based” research on cultural policy which are at present completely blank. The final point of the article, therefore, seems clear, but brutal: research that could be the foundation of evidence-based policymaking, as defined in the start of this article following Sutcliffe and Court (2005), is extremely undeveloped. Most of the existing debates are theoretical and critical with insufficient focus on the development of appropriate statistical methodology to study the phenomena and practice of cultural policy in an adequate manner. Until something changes, evidence-based cultural policy research is an example of bullshit. It is the task of future work in cultural policy research (and cultural economics) to change this in a significant and drastic manner.
References Angrist, J. D., and Pischke, J.-S. 2009. Mostly Harmless Econometrics: An Empiricists Companion. Princeton: Princeton University Press. Baade R. A. and Dye, R. F. 1988. Sports stadiums and area development: a critical view. Economic Development Quarterly, 2(3), pp. 265-275. Bandura, R. 2008. A Survey of Composite Indices Measuring Country Performance: 2008 Update. New York: United Nations Development Programme. Barsky, R. and Kilian, L. 2000. A Monetary Explanation Of The Great Stagflation Of The 1970s. CEPR Discussion Papers 2389, C.E.P.R. Discussion Papers. Belfiore, E. 2003. Economic impact – Inconclusive evidence. Arts Professional, No. 43. Belfiore, E. 2009. On bullshit in cultural policy practice and research: notes from the British case. International Journal of Cultural Policy, Vol.15, No.3, pp. 343-359. Belfiore, E. 2010. Beyond the “Toolkit Approach”: arts impact evaluation research and the realities of cultural policy‐making. Journal for Cultural Research, 14(2), pp. 121-142. Belfiore, E. and Bennett, O. 2007. Rethinking the Social Impacts of the Arts’, International Journal of Cultural Policy, vol. 13, no. 2, pp. 135–151. Bína, V., Chantepie, P., Deroin, V., Frank, G., Kommel, K., Kotýnek, J. and Robin, P. 2012. ESSnet‐Culture Final Report. Luxembourg: European Commission, Eurostat (ESTAT). Boelhouwer, J., van den Broek, A., van Eyck, K., Smithuijsen, C., Vinken, H. and van Woersem, L. 2013. Arts Index Netherlands. Key information about the cultural sector 2005-2011. Journal Boekman #97: The State of Culture 2013. Council of Europe. 2016. Indicator Framework on Culture and Democracy. Policy Maker’s Guidebook. Strasbourg: Council of Europe. Council of Europe Conference of Ministers of Culture. 2013. Final Statement 10th Council of Europe Conference of Ministers of Culture. MinConfCult (2013) 7 – 16 April 2013. Strasbourg: Council of Europe. Cuccia, T., Guccio, C. and Rizzo, I. 2013. Does Unesco inscription affect the performance of tourism destinations? A regional perspective. ACEI Working Paper Series AWP-04-2013, Association for Cultural Economics International. Cwi, D. and Lyall, K. 1977. A model to assess the local economic impact of arts institutions: the Baltimore case study. Baltimore: Center for Metropolitan Planning and Research the Johns Hopkins Univ. Čopič, V., Inkei, P., Kangas, A. and Srakar, A. 2013. Trends in Public Funding for Culture in the EU. EENC Report, August 2013. Davies, P. 2004. Is evidence-based government possible? Jerry Lee Lecture, presented at the 4th Annual Campbell Collaboration Colloquium, Washington DC. Diamond, P. A. and Hausman, J. A. 1994. Contingent Valuation: Is Some Number Better than No Number? Journal of Economic Perspectives, 8 (1994), pp. 45-64. Disdier, A.-C., Tai, S. H . T., Fontagné, L. and Mayer, T. (2009). Bilateral trade of cultural goods. Review of World Economics, 145(4), pp. 575-595. Eurostat. 2007. Cultural Statistics, 2007 Edition (Pocketbook Series). Brussels: European Commission. Eurostat. 2011. Cultural Statistics, 2011 Edition (Pocketbook Series). Brussels: European Commission. Eurostat. 2016. Cultural Statistics, 2016 Edition (Pocketbook Series). Brussels: European Commission. Frankfurt, H. G. 2005. On bullshit. Princeton/Oxford: Princeton University Press. Frey, B. S. 2000. Arts and Economics: Analysis and Policy. Berlin: Springer Verlag. Frey, B. S. 2005. What Values Should Count in the Arts? The Tension between Economic Effects and Cultural Value. CREMA Working Paper Series, Center for Research in Economics, Management and the Arts (CREMA) Gergaud, O. and Ginsburgh, V. 2013. Measuring the economic effects of cultural events with special emphasis on music festivals. ULB Institutional Repository, ULB – Universite Libre de Bruxelles, Belgium. Hesmondhalgh D., Nisbett M., Oakley K. and Lee, D.J. 2014. Were New Labour’s Cultural Policies Neo-Liberal? International Journal of Cultural Policy, pp. 1-18. Huff, D. 1954. How to lie with statistics. London: Penguin. Hutter, M. and Throsby, D. 2008. Beyond price: value in culture, economics and the arts. Cambridge: Cambridge University Press. Inkei, P. 2013. Considerations about a European Cultural Index. Amsterdam: European Cultural Foundation. Klamer, A. 1996. Value of Culture. On the Relationship between Economics and Arts. Amsterdam: Amsterdam University Press. Kregzdaite, R., Cerneviciute, J., Strazdas, R. and Jancoras, Z. 2016. Problems of Evaluation Impact of CCI: Constructing Indexes. 19th International Conference of the Association for Cultural Economics, International, June 21-24, 2016. Marvasti, A. and Canterbery, R. 1992. Intra-Industry Trade in Culture: The United States and Canada. International Trade and Finance Association Proceeding, pp. 159-174. Meskell, L., Liuzza, C., Bertacchini, E. and Saccone, D. 2015. Multilateralism and UNESCO World Heritage: decision-making, States Parties and political processes. International Journal of Heritage Studies, Vol. 21, Iss. 5, pp. 423-440. Morgan, S. L. and Winship, C. 2014. Counterfactuals and Causal Inference: Methods and Principles for Social Research (Analytical Methods for Social Research). 2nd ed. New York: Cambridge University Press. Myerscough, J. 1988. The Economic Importance of the Arts in Britain. London: Policy Studies Institute. Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A. and Giovannini, E. 2008. Handbook on Constructing Composite Indicators: Methodology and User Guide. Paris: OECD. Oakley, K., Naylor, R. and Lee, D. 2006. Giving them what they want: the construction of the public in ‘public value’. CRESC Media Change & Social Change Conference, 6-8 September 2006, Oxford University. Oakley, K. and O'Brien, D. 2015. Cultural Value and Inequality: A Critical Literature Review. Report. Arts and Humanities Research Council. OECD. 1981. The Measurement of Scientific and Technical Activities: Proposed Standard Practice for Surveys of Research and Experimental Development (Frascati Manual). Paris: OECD. OECD. 2007. Glossary of Statistical Terms. Composite indicator. http://stats.oecd.org. Accessed 09 December 2016. Peterson, G. and Sorg, C. F. 1987. Toward the measurement of total economic value. November 1987. Fort Collins, CO. Prepared for the USDA Forest Service. General Technical Report RM-148. Plaza, B., González-Casimiro, P., Moral-Zuazo, P. and Waldron, C. 2015. Culture-led city brands as economic engines: theory and empirics. The Annals of Regional Science, Springer, Western Regional Science Association, Vol. 54(1), pp. 179-196. Port Authority of New York and New Jersey, Cultural Assistance Center. 1983. The Arts as an Industry: Their Economic Importance to the New York-New Jersey Metropolitan Region. New York, NY: Port Authority of New York and New Jersey. Ruiz Pozuelo, J., Slipowitz, A. and Vuletin, G. 2016. Democracy does not cause growth: the importance of endogeneity arguments. IDB Working Paper Series, 694, Inter-American Development Bank. Qu, R.-X. and Han, L.-L. 2011. The Factors on Trade of Chinese Cultural Goods: an empirical analysis based on panel data. Chinese – Hungarian International Conference – Economics of Crisis, Education and Labour, July 2011, Budapest, HG. Radich, A. J. 1993. Twenty Years of Economic Impact Studies of the Arts: A Review. Washington DC: National Endowment for the Arts. Rodríguez Ramos, A., Navarro, J. L. M., Palma Martos, M. L. and Martínez Camacho, M. I. 2016. Indexes of creativity: A measurement proposal for Spain and its Autonomous Communities. 19th International Conference of the Association for Cultural Economics, International, June 21-24, 2016. Seaman, B. A. 1987. Arts Impact Studies: A Fashionable Excess. In Radich, A. J. (ed.). The Economic Impact of the Arts: A Sourcebook (pp. 43-76). Denver, Colorado: National Conference of State Legislatures. Seaman, B. A. 2006. The Relationship Among Regional Economic Impact Models: Contingent Valuation Versus Economic Impact in the Case of Cultural Assets. Working Paper, Atlanta: Georgia State University. Seaman, B. A. and Price Elton, J. J. 2016. How Reliable are Ex Post Econometric Studies of Regional Economic Impacts. 19th International Conference of the Association for Cultural Economics, International, June 21-24, 2016; Powerpoint presentation. Skinner, S. J. 2006. Estimating the real growth effects of blockbuster art exhibits: a time series approach. Journal of Cultural Economics, 30, pp. 109-125. Srakar, A. and Vecco, M. 2016. Ex-ante vs. ex-post: Comparison of the effects of the European Capital of Culture Maribor 2012 on tourism and employment. Journal of Cultural Economics, forthcoming (in press). Srakar, A., Slabe-Erker, R. and Vecco, M. 2016. Economic effects of Venice Carnivals: An ex-post econometric verification approach. In Cuffy, V. (2016, ed.). Carnival, Culture and Tourism. Oxfordshire: CABI Publishing, forthcoming. Srakar, A., Verbič, M. and Čopič, V. 2015. European "cultural models" in statistical perspective : a high-dimensionally adjusted cultural index for the EU countries, 2005-2009. ACEI Working Paper Series, Association for Cultural Economics International. Steiner, L., Frey, B. S. and Hotz, S. 2015. European Capitals of Culture and Life Satisfaction. Urban Studies, 52.2, pp. 374-394. Sutcliffe, S. and Court, J. 2005. Evidence-Based Policymaking: What is it? How does it work? What relevance for developing countries? Overseas Development Institute, November 2005. Shaxson, L. 2005. Is your evidence robust enough? Questions for policy makers and practitioners. Evidence and Policy: A Journal of Research, Debate and Practice, Vol. 1, No. 1, pp. 101-111. Throsby, D. and Withers, G. 1986. Strategic bias and demand for public goods: theory and an application to the arts. Journal of Public Economics, 31, pp. 307-327. Whitehead, J. C., Ashton Morgan, O. and Huth, W. L. 2015. Benefit Transfers with the Contingent Valuation Method. In Johnston, R.J., Rolfe, J., Rosenberger, R., & Brouwer, R. (Eds.) Benefit Transfer of Environmental and Resource Values. A Guide for Researchers and Practitioners. Series: The Economics of Non-Market Goods and Resources, Vol. 14. Berlin: Springer Verlag, pp. 119-140. Zieba, M. 2011. An Analysis of Technical Efficiency and Efficiency Factors for Austrian and Swiss Non-Profit Theatres. Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 147(II), pp. 233–274.