IDEAS home Printed from https://ideas.repec.org/a/gam/jpubli/v11y2023i4p49-d1279756.html
   My bibliography  Save this article

Migrating 120,000 Legacy Publications from Several Systems into a Current Research Information System Using Advanced Data Wrangling Techniques

Author

Listed:
  • Yrjö Lappalainen

    (Library and Learning Commons, Zayed University, Dubai P.O. Box 19282, United Arab Emirates)

  • Matti Lassila

    (Tampere University Library, Tampere University, 33014 Tampere, Finland)

  • Tanja Heikkilä

    (Finnish Geospatial Research Institute (FGI), National Land Survey of Finland (NLS), 02150 Espoo, Finland)

  • Jani Nieminen

    (Tampere University Library, Tampere University, 33014 Tampere, Finland)

  • Tapani Lehtilä

    (Tampere University Library, Tampere University, 33014 Tampere, Finland)

Abstract

This article describes a complex CRIS (current research information system) implementation project involving the migration of around 120,000 legacy publication records from three different systems. The project, undertaken by Tampere University, encountered several challenges in data diversity, data quality, and resource allocation. To handle the extensive and heterogenous dataset, innovative approaches such as machine learning techniques and various data wrangling tools were used to process data, correct errors, and merge information from different sources. Despite significant delays and unforeseen obstacles, the project was ultimately successful in achieving its goals. The project served as a valuable learning experience, highlighting the importance of data quality and standardized practices, and the need for dedicated resources in handling complex data migration projects in research organizations. This study stands out for its comprehensive documentation of the data wrangling and migration process, which has been less explored in the context of CRIS literature.

Suggested Citation

  • Yrjö Lappalainen & Matti Lassila & Tanja Heikkilä & Jani Nieminen & Tapani Lehtilä, 2023. "Migrating 120,000 Legacy Publications from Several Systems into a Current Research Information System Using Advanced Data Wrangling Techniques," Publications, MDPI, vol. 11(4), pages 1-16, November.
    • Handle: RePEc:gam:jpubli:v:11:y:2023:i:4:p:49-:d:1279756
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2304-6775/11/4/49/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2304-6775/11/4/49/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Helen Shen, 2014. "Interactive notebooks: Sharing the code," Nature, Nature, vol. 515(7525), pages 151-152, November.
    2. Kuhn, Max, 2008. "Building Predictive Models in R Using the caret Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 28(i05).
    3. Otmane Azeroual & Joachim Schöpfel, 2019. "Quality Issues of CRIS Data: An Exploratory Investigation with Universities from Twelve Countries," Publications, MDPI, vol. 7(1), pages 1-18, February.
    4. Helen Shen, 2014. "Interactive notebooks: Sharing the code," Nature, Nature, vol. 515(7525), pages 152-152, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ian L Morgan & Omar A Saleh, 2021. "Tweezepy: A Python package for calibrating forces in single-molecule video-tracking experiments," PLOS ONE, Public Library of Science, vol. 16(12), pages 1-18, December.
    2. Prabal Das & D. A. Sachindra & Kironmala Chanda, 2022. "Machine Learning-Based Rainfall Forecasting with Multiple Non-Linear Feature Selection Algorithms," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(15), pages 6043-6071, December.
    3. Jie Zhao & Ji Chen & Damien Beillouin & Hans Lambers & Yadong Yang & Pete Smith & Zhaohai Zeng & Jørgen E. Olesen & Huadong Zang, 2022. "Global systematic review with meta-analysis reveals yield advantage of legume-based rotations and its drivers," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    4. Piaopiao Chen & Agnès H. Michel & Jianzhi Zhang, 2022. "Transposon insertional mutagenesis of diverse yeast strains suggests coordinated gene essentiality polymorphisms," Nature Communications, Nature, vol. 13(1), pages 1-15, December.
    5. Paulo Infante & Gonçalo Jacinto & Anabela Afonso & Leonor Rego & Pedro Nogueira & Marcelo Silva & Vitor Nogueira & José Saias & Paulo Quaresma & Daniel Santos & Patrícia Góis & Paulo Rebelo Manuel, 2023. "Factors That Influence the Type of Road Traffic Accidents: A Case Study in a District of Portugal," Sustainability, MDPI, vol. 15(3), pages 1-16, January.
    6. Ephrem Habyarimana & Faheem S Baloch, 2021. "Machine learning models based on remote and proximal sensing as potential methods for in-season biomass yields prediction in commercial sorghum fields," PLOS ONE, Public Library of Science, vol. 16(3), pages 1-23, March.
    7. Banks, Jonathan & Rabbani, Arif & Nadkarni, Kabir & Renaud, Evan, 2020. "Estimating parasitic loads related to brine production from a hot sedimentary aquifer geothermal project: A case study from the Clarke Lake gas field, British Columbia," Renewable Energy, Elsevier, vol. 153(C), pages 539-552.
    8. Crespo, Cristian, 2020. "Two become one: improving the targeting of conditional cash transfers with a predictive model of school dropout," LSE Research Online Documents on Economics 123139, London School of Economics and Political Science, LSE Library.
    9. Alexander Wettstein & Gabriel Jenni & Ida Schneider & Fabienne Kühne & Martin grosse Holtforth & Roberto La Marca, 2023. "Predictors of Psychological Strain and Allostatic Load in Teachers: Examining the Long-Term Effects of Biopsychosocial Risk and Protective Factors Using a LASSO Regression Approach," IJERPH, MDPI, vol. 20(10), pages 1-20, May.
    10. Tang, Kayu & Parsons, David J. & Jude, Simon, 2019. "Comparison of automatic and guided learning for Bayesian networks to analyse pipe failures in the water distribution system," Reliability Engineering and System Safety, Elsevier, vol. 186(C), pages 24-36.
    11. Daifeng Xiang & Gangsheng Wang & Jing Tian & Wanyu Li, 2023. "Global patterns and edaphic-climatic controls of soil carbon decomposition kinetics predicted from incubation experiments," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    12. Joel Podgorski & Oliver Kracht & Luis Araguas-Araguas & Stefan Terzer-Wassmuth & Jodie Miller & Ralf Straub & Rolf Kipfer & Michael Berg, 2024. "Groundwater vulnerability to pollution in Africa’s Sahel region," Nature Sustainability, Nature, vol. 7(5), pages 558-567, May.
    13. Bellotti, Anthony & Brigo, Damiano & Gambetti, Paolo & Vrins, Frédéric, 2021. "Forecasting recovery rates on non-performing loans with machine learning," International Journal of Forecasting, Elsevier, vol. 37(1), pages 428-444.
    14. Tranos, Emmanouil & Incera, Andre Carrascal & Willis, George, 2022. "Using the web to predict regional trade flows: data extraction, modelling, and validation," OSF Preprints 9bu5z, Center for Open Science.
    15. Štefan Lyócsa & Petra Vašaničová & Branka Hadji Misheva & Marko Dávid Vateha, 2022. "Default or profit scoring credit systems? Evidence from European and US peer-to-peer lending markets," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-21, December.
    16. Arjan S. Gosal & Janine A. McMahon & Katharine M. Bowgen & Catherine H. Hoppe & Guy Ziv, 2021. "Identifying and Mapping Groups of Protected Area Visitors by Environmental Awareness," Land, MDPI, vol. 10(6), pages 1-14, May.
    17. Joachim Schöpfel & Stéphane Chaudiron & Bernard Jacquemin & Eric Kergosien & Hélène Prost & Florence Thiault, 2023. "The Transformation of the Green Road to Open Access," Publications, MDPI, vol. 11(2), pages 1-12, May.
    18. Marcos Rodrigues & Fermín Alcasena & Pere Gelabert & Cristina Vega‐García, 2020. "Geospatial Modeling of Containment Probability for Escaped Wildfires in a Mediterranean Region," Risk Analysis, John Wiley & Sons, vol. 40(9), pages 1762-1779, September.
    19. Natalia Pardo-Lorente & Anestis Gkanogiannis & Luca Cozzuto & Antoni Gañez Zapater & Lorena Espinar & Ritobrata Ghose & Jacqueline Severino & Laura García-López & Rabia Gül Aydin & Laura Martin & Mari, 2024. "Nuclear localization of MTHFD2 is required for correct mitosis progression," Nature Communications, Nature, vol. 15(1), pages 1-23, December.
    20. Giovanny Pillajo-Quijia & Blanca Arenas-Ramírez & Camino González-Fernández & Francisco Aparicio-Izquierdo, 2020. "Influential Factors on Injury Severity for Drivers of Light Trucks and Vans with Machine Learning Methods," Sustainability, MDPI, vol. 12(4), pages 1-28, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jpubli:v:11:y:2023:i:4:p:49-:d:1279756. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.