Evaluating a collaborative IT based research and development project

Khan, Zaheer; Ludlow, David; Caceres, Santiago

Abstract

In common with all projects, evaluating an Information Technology (IT) based research and development project is necessary in order to discover whether or not the outcomes of the project are successful. However, evaluating large-scale collaborative projects is especially difficult as: (i) stakeholders from different countries are involved who, almost inevitably, have diverse technological and/or application domain backgrounds and objectives; (ii) multiple and sometimes conflicting application specific and user-defined requirements exist; and (iii) multiple and often conflicting technological research and development objectives are apparent. In this paper, we share our experiences based on the large-scale integrated research project - The HUMBOLDT project - with project duration of 54 months, involving contributions from 27 partner organisations, plus 4 sub-contractors from 14 different European countries. In the HUMBOLDT project, a specific evaluation methodology was defined and utilised for the user evaluation of the project outcomes. The user evaluation performed on the HUMBOLDT Framework and its associated nine application scenarios from various application domains, resulted in not only an evaluation of the integrated project, but also revealed the benefits and disadvantages of the evaluation methodology. This paper presents the evaluation methodology, discusses in detail the process of applying it to the HUMBOLDT project and provides an in-depth analysis of the results, which can be usefully applied to other collaborative research projects in a variety of domains. © 2013 Elsevier Ltd.

• Journal Article Type: Article
• Publication Date: Oct 1, 2013
• Publicly Available Date: Jun 7, 2019
• Journal: Evaluation and Program Planning
• Print ISSN: 0149-7189
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 40
• Pages: 27-41
• DOI: https://doi.org/10.1016/j.evalprogplan.2013.04.004
• Keywords: integrated project research and development, collaborative research project evaluation, participatory evaluation, methodology, questionnaire and user-based evaluation, user and technical evaluation
• Public URL: https://uwe-repository.worktribe.com/output/927242
• Publisher URL: http://dx.doi.org/10.1016/j.evalprogplan.2013.04.004
• Related Public URLs: http://www.sciencedirect.com/science/article/pii/S0149718913000347