.jpg)
printAssessment of Scientific Productivity and Social Utility of Scientific Studies:
The Lessons from the U.S. Record
Istomin I.A.,
Cand. Sci. (Polit. Sci.), Associate Professor, Acting Head of the Department of Applied International Political Analysis, MGIMO University, i.istomin@inno.mgimo.ru
elibrary_id: 333124 | ORCID: 0000-0002-8334-6343 | RESEARCHER_ID: A-8494-2017
DOI: 10.17976/jpps/2018.06.09
Rubric: Orbis terrarum
For citation:
Istomin I.A. Assessment of Scientific Productivity and Social Utility of Scientific Studies: The Lessons from the U.S. Record. – Polis. Political Studies. 2018. No. 6. P. 127-141. (In Russ.). https://doi.org/10.17976/jpps/2018.06.09
Abstract
The last decades witnessed the rise of innovative development, based on scientific inquiry, as a priority of national policies in numerous countries (including Russia). Growing attention to the research activities is accompanied by the search for better methods to assess scientific productivity of individuals, institutions and entire professional communities. The current article focuses on the bottlenecks in this process as well as possible ways for their overcoming building upon recent record of the United States. This country represents a particular significant case for study as it maintains global leadership in both fundamental and applied research. The article highlights the necessity to differentiate between two levels of assessment: the one dealing with immediate scientific outcomes and the other reflecting broader social consequences of research. Both state and society are more interested in the latter; however, they are dependent on the whole innovation system of the state, rather than on scholars alone. The study of scientific productivity relies on bibliometric methods and expert evaluation. Despite the surge of interest towards quantitative assessments, their uncritical application could lead to the significant distortions of the real impact of individual scholars in professional community. They could play useful supplementary role, but they do not provide solutions for making certain managerial decisions. Aggregated data is not specifically useful for forecasting productivity of individual scholars. In the assessment of broader social contribution of scientific research in the U.S., there is a growing appreciation of multidimensional analysis, relying on Big Data methods. Currently, it is underdeveloped; however, the demand from American federal authorities stimulates rapid development of evaluation instruments for better targeting of policy.
Keywords
expert assessment; tenure; science policy; bibliometrics; h-index; impact factor score; STAR metrics.
References
Balatskii E.V. Arrhythmia Syndrome in the Higher Education. – Voprosy Ekonomiki. 2014. No. 4 (24). P. 111-120. (In Russ.)
Feigel’man M.V., Tsirlina G.A. Bibliometricheskij azart kak sledstvie otsutstviya nauchnoj elspertizy [Bibliometric Enthusiasm as a Consequence of the Lack of Scientific Expertise]. – Upravlenie bol’shimi sistemami. Spetsial’nij vypusk 44: “Naukometriua i ekspertiza v upravlenii naukoj” [Management of Large Systems. Special Edition 44: Sciencemetrics and Expertise in Management of Science]. Moscow: Institut problem upravleniya im. V.A. Trapeznikova RAN. 2013. P. 332-345. (In Russ.)
Furthering America’s Research Enterprise. Ed. by R.F. Celeste, A. Griswold, M.L. Straf. National Academy of Sciences. Committee on Assessing the Value of Research in Advancing National Goals, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. 2014. 216 p.
Gorokhov V.G. Problema izmerimosti produktivnosti otdel’nykh uchenykh i tselykh institutov [The Problem of Measurement of Productivity of Individual Scholars and Entire Institutions]. – Upravlenie bol’shimi sistemami. Spetsial’nij vypusk 44: “Naukometriua i ekspertiza v upravlenii naukoj” [Management of Large Systems. Special Edition 44: Sciencemetrics and Expertise in Management of Science]. Moscow: Institut problem upravleniya im. V.A. Trapeznikova RAN. 2013. P. 190-209. (In Russ.)
Gorraiz J., Gumpenberger Chr. A Flexible Bibliometric Approach for the Assessment of Professorial Appointments. – Scientometrics. 2015. Vol. 105. No. 3. P. 1699-1719. https://doi.org/10.1007/s11192-015-1703-6
Grigorieva E.I., Zaripova Z.R., Kokarev K.P. How Good Are the Journals in Which You Publish Your Articles? – Polis. Political Studies. 2015. No. 3. P. 147-159 (In Russ.) https://doi.org/10.17976/jpps/2015.03.10
Holden G., Rosenberg G., Baker K., Onghena P. An Assessment of the Predictive Validity of Impact Factor Scores: Implications for Academic Employment Decisions in Social Work. – Research on Social Work Practice. 2006. Vol. 16. No. 6. P. 613-624. https://doi.org/10.1177/1049731506292570
Holden G., Rosenberg G., Barker K. Bibliometrics: A Potential Decision Making Aid in Hiring, Reappointment, Tenure and Promotion Decisions. – Social Work in Health Care. 2005. Vol. 41. No. 3-4. P. 67-92. https://doi.org/10.1300/J010v41n03_03
Istomin I.A. U.S. Research Universities: Benefits and Tradeoffs of Education and Science under One Roof. – USA and Canada: Economy, Politics and Culture. 2015. No. 12. P. 70-84. (In Russ.)
Istomin I.A., Baykov A.A. Russian and International Publication Practices a Comparative Study of IR Scholarly Journals. – International Trends. 2015. No. 2. P. 114-140. (In Russ.) https://doi.org/10.17994/IT.2015.13.2.41.9
Ivanova E.A. The Use of Metrics of Scholars’ Publishing in the Practice of Science Management (Review of the Issues under Discussion). – Sociology of Science and Technology. 2011. Vol. 2. No. 4. P. 61-72. (In Russ.)
Keohane R. After Hegemony. Cooperation and Discord in World Political Economy. Princeton: Princeton University Press. 1984. 290 p.
Keohane R.O., Nye J.S. Power and Interdependence: World Politics in Transition. Boston: Little, Brown. 1977. 273 p.
Klochikhin E. Collaborative Innovation Beyond Science: Exploring New Data and New Methods with Computer Science. 2015. URL: https://ecpr.eu/Filestore/PaperProposal/813b30e1-640a-43a8-ab68-fd27fc3b73d4.pdf (accessed 01.10.2018).
Kuhn T. The Structure of Scientific Revolutions. (Russ. ed.: Kuhn T. Struktura nauchnykh revolyutsij. Moscow: AST. 2003. 605 p.)
Lane J., Bertuzzi St. Measuring the Results of Science Investments. – Science. 2011. Vol. 331. URL: http://cssip.org/docs/meeting/measuring_results_of_sci_investments,_science_feb_2011.pdf (accessed 01.10.2018).
Latour B. Science in Action: How to Follow Scientists and Engineers Through Society. (Russ. ed.: Latour B. Nauka v dejstvii: sleduya za uchenymi i inzhenerami vnutri obschestva. St. Petersburg: EUSP Press. 2013. 414 p.)
Lee C.J, Sugimoto C.R., Cronin B. Bias in Peer Review. – Journal of American Society for Information and Technology. 2013. Vol. 64. No. 1. P. 2-17. https://doi.org/10.1002/asi.22784
Ludlow Chr.L., Kent R.D. Building a Research Career. San Diego: Plural Publishers. 2011. 212 p.
Mirkin B.G. O ponyatii nauchnogo vklada i ego izmeritelyakh [About the Notion of Scientific Contribution and Its Measurement]. – Upravlenie bol’shimi sistemami. Spetsial’nij vypusk 44: “Naukometriua i ekspertiza v upravlenii naukoj” [Management of Large Systems. Special Edition 44: Sciencemetrics and Expertise in Management of Science]. Moscow: Institut problem upravleniya im. V.A. Trapeznikova RAN. 2013. P. 292-307. (In Russ.)
Mole B. NSF Cancels Political-Science Grant Cycle. – Nature. 2013. https://doi.org/10.1126/science.caredit.a1200035
Monastersky R. The Number That’s Devouring Science. – The Chronicle of Higher Education. 2005. Vol. 52. No. 8. P. A12-A17.
Olson S., Merrill S. Measuring the Impacts of Federal Investments in Research: A Workshop Summary. Washington D.C.: National Academy of Sciences. 2011. 206 p.
Orlov A.I. Dva tipa metodologicheskikh oshibok pri upravlenii nauchnoj deyatel’nostiyu [Two Types of Methodological Errors in Management of Scientific Activity]. – Upravlenie bol’shimi sistemami. Spetsial’nij vypusk 44: “Naukometriua i ekspertiza v upravlenii naukoj” [Management of Large Systems. Special Edition 44: Sciencemetrics and Expertise in Management of Science]. Moscow: Institut problem upravleniya im. V.A. Trapeznikova RAN. 2013. P. 32-54. (In Russ.)
Paulus F.M., Rademacher L., Schafer T.A.J., Muller-Pinzler L., Krach S. Journal Impact Factor Shapes Scientists’ Reward Signal in the Prospect of Publication. – PLoS ONE. 2015. Vol. 10(11): e0142537. https://doi.org/10.1371/journal.pone.0142537
Romanova V.V., Matskevich A.V. Pokazateli byudzhetnykh raskhodov v sfere obrazovaniya [Indexes of Budget Expenditures in the Field of Education]. Moscow: HSE Publishing House. 2012. 58 p. (In Russ.)
Shulman S. Undermining Science: Suppression and Distortion in the Bush Administration. Berkeley: University of California Press. 2006. 202 p.
Tretyakova O.V. Indeksy nauchnogo tsitirovaniya: vozmozhnosti i perspektivy v otsenke rezul’tatov nauchnoi deyatel’nosti [Indexes of Scientific Citation: Opportunities and Prospects of Assessment of the Results of Scientific Activity]. Vologda: ISERT RAN. 2014. 52 p. (In Russ.)
Varshavskii A. Problems of Science and Its Productivity. – Voprosy Ekonomiki. 2011. No. 1. P. 151-157. (In Russ.)
Waltz K.N. Man, the State, and War: A Theoretical Analysis. New York: Columbia University. 1959. 263 p.
Waltz K. Theory of International Politics. New York: McGraw-Hill. 1979. 250 p.
Weinberg B., Owen-Smith J., Rosen R., Schwarz L., McFadden Allen B., Weiss R., Lane J. Science Funding and Short-Term Economic Activity. – Science. 2014. Vol. 344. No. 6179. P. 41-43. https://doi.org/10.1126/science.1250055
Yurevich M.A. Methodological Problems of Assessment of Scholar’s Effectiveness. – Science. Innovations. Education. 2014. No. 16. P. 28-41. (In Russ.)
Zolas N., Goldschlag N., Jarmin R., Stephan P., Owen-Smith J., Rosen R.F., McFadden Allen B., Weinberg B.A., Lane J.I. Wrapping It up in a Person: Examining Employment and Earnings Outcomes for Ph.D. Recipients. – Science. 2015. Vol. 350. No. 6266. P. 1367-1371. https://doi.org/10.1126/science.aac5949
Content No. 6, 2018
See also:
Grigorieva E.I., Zaripova Z.R., Kokarev K.P.,
How Good Are the Journals in Which You Publish Your Articles?. – Polis. Political Studies. 2015. No3
Savinov L.V.,
Russian political science and its scientometrical characteristics. – Polis. Political Studies. 2012. No3
Romanova M.D.,
Influence of Cultural Context on Formation of Science Policy (French Experience). – Polis. Political Studies. 2015. No5
Talagayeva D.A.,
Norway: the state science policy. – Polis. Political Studies. 2014. No1
Information,
Introducing the «MGIMO-University Bulletin». – Polis. Political Studies. 2009. No6
.jpg)
.jpg)