Critical BIM qualification criteria for construction pre-qualification and selection

ABSTRACT Despite emerging research on Building Information Modelling (BIM) capability, there remain no studies that specifically look at criteria for the pre-qualification and selection of construction supply chain (CSC) firms for BIM projects. As a consequence, there is insufficient theoretical insight and empirical justification for critical BIM qualification criteria for the CSC context. In order to bridge this knowledge gap, a study was conducted to ascertain the most critical BIM capability attributes, as well as relevance to the pre-qualification and selection of suitable CSC firms for BIM-enabled projects. The relative importance of these criteria was determined based on their perceived influence on overall BIM delivery success on recent BIM-enabled projects. A sequential exploratory mixed methodological research design was adopted involving semi-structured interviews (n = 8) and a Delphi survey (n = 25) of experts, as well as a survey of CSC firms (n = 64) on BIM-enabled projects. The findings reinforce the criticality of past BIM experience in delivery success. Furthermore, it revealed that while technological infrastructure-related capacities (such as software and hardware availability) are perceived as very important qualification criteria, their actual contribution to delivery success is not as high in comparison to criteria related to previous BIM experience. This study provides insights on prioritising BIM capability criteria based on evidence on their contribution to delivery success in practice rather than their perceived importance as capability metrics.


Introduction
Building Information Modelling (BIM) is recognised as a critical collaborative information technology that could assist the Construction Supply Chain (CSC) in achieving integrated practice (Vrijhoef, 2011;BIS 2013a;. This is expected to be achieved through centralised digital exchange of data to eliminate current information flow inefficiencies that contribute to poor performance (Arayici et al., 2012). BIM is, therefore, increasingly becoming a prerequisite for construction projects, yet wider uptake of BIM across project supply chain remains a challenge due to varying levels of capability or even willingness to use it among other reasons (Gu & London, 2010;Succar et al., 2012;Papadonikolaki et al., 2016). There is a significant risk of failure if the CSC selected for projects lacks the ability to operate within a BIM environment as well as the capacity to adopt the processes and related technologies. Ample evidence demonstrates the need for metrics in evaluating organisations' ability to deliver BIM as well as attain BIM success amidst a lack of standardised and accepted approaches for establishing capability (Succar et al., 2012;Haron, 2013).
Despite the surge in interest in BIM capability evaluation, there remains a lack of studies specifically tailored for CSC qualification. Furthermore, there is a lack of insight and empirical justification for the relationship between often pre-emptive qualification criteria and BIM delivery success especially in the CSC context. Despite the proliferation of frameworks and toolsets for evaluating the BIM performance of firms, there remains a lack of a specifically tailored approach to predicting a firm's propensity to succeed in the delivery of BIM during the pre-qualification and selection stage. In order to address this, there is a need for the identification of BIM capability attributes relevant for the qualification process (prequalification and selection) (CIC, 2013). Furthermore, there is a need for a deep understanding of the relative importance of such criteria in relation to their contribution to BIM delivery success .
In order to address the identified gaps and limitations this study aimed to establish the following:  Develop an understanding of BIM capability attributes, their uses as qualification criteria for the CSC, as well as their role in successful delivery of BIM in the supply chain context;  Identify and categorise BIM qualification criteria in order to develop a hierarchy of assessment criteria for CSC pre-qualification or selection purposes; and  Identify the most critical criteria and prioritise them based on their relative contribution to the successful delivery of BIM.

BIM Capability and Supplier Selection
A supplier in the CSC context represents any firm that contributes to the effective delivery of a project, or activities of a client or main contractor. Suppliers could, therefore, be consultants, sub-contractors or other relevant service providers for the building process (Briscoe & Dainty, 2005). The selection process allows the determination of competent firms to be part of the delivery of a project or to be part of the team of a main contractor or client for long term supply chain management (SCM) purposes. This process remains one of the most critical milestones in the construction project life cycle (Holt et al., 1994). There are two main activities associated with supplier selection. The first may be a pre-qualification phase to develop a shortlist of qualified suppliers eligible to prepare tender or proposals for a project, work package or a series of projects (Plebankiewicz, 2012). The second is the actual selection, where suppliers are assessed in order to determine or predict the most suitable candidate to be selected as part of a CSC or a specific project (Holt et al., 1994). The selection of suitable suppliers to be engaged on a construction project has significant effect on the success and attainment of objectives (Jaselskis & Russell, 1991). In order to minimise the risk of selecting incapable suppliers, the process for evaluating alternative candidates must be methodical, thorough and complete (Holt et al., 1994;Plebankiewicz, 2012). Both clients and main contractors are faced with the challenge of assessing and prioritising potential candidates as a result of the need for consideration of their capability in many relevant areas. An emerging core competence area for successful completion of projects is the ability to deliver through BIM (Van Berlo et al., 2012).
There are several emerging guidance documents, protocols and standards requiring principal suppliers (e.g. main contractors and consultants in contract with the client) to demonstrate that the rest of their CSC can deliver through BIM (PAS1192:2, 2013; Al- Ahbabi & Alshawi, 2015). The UK Government's standard pre-qualification questionnaires now include a section specifically dedicated for BIM qualification (PAS 91, 2013). There is great emphasis on the CSC's BIM capability; hence, requiring principal suppliers to submit a CSC BIM capability summary or evaluation for each project as part of the qualification process (PAS1192:2, 2013). Despite the need for demonstration of BIM capability, there remains lack of standardised approaches for qualifying the CSC based on their BIM capabilities. Indeed, there is a lack of tools and benchmarks for assessing firm's BIM performance, thus, resulting in false claims of firms' ability to deliver BIM services or products on projects (Succar, 2010;Van Berlo et al., 2012). Consequently, main contractors and clients are, therefore, faced with the challenge of assessing the competence; proficiency and willingness to deliver BIM for firms they choose to deliver services during projects amidst a proliferation of mostly theoretical propositions on assessing BIM capability and maturity (Succar, 2010;Van Berlo et al., 2012;Haron, 2013). Despite the proliferation of capability assessment frameworks and tools, to date, there are no frameworks specifically tailored for qualification of the CSC for projects.

BIM Capability Assessment Tools and Frameworks
The dearth in literature and lack of specifically tailored BIM assessment frameworks for qualification of CSC on projects leaves a significant research gap that needs to be filled. The qualification process is, however, unique and requires a more holistic, but concise approach, as well as a precise prediction of the likelihood of success (Doloi, 2009;PAS 91, 2013). Existing capability frameworks are, however, limited in this regard. Several limitations exist in relation to the use of existing BIM capability assessment frameworks and toolsets.
Some of the existing frameworks and tools include the Capability Maturity Model (CMM) by the National Institute of Building Science (NIBS, 2007;; the Indiana University BIM proficiency matrix (IU, 2009); the BIM capability framework by Succar (2010); the BIM owners' competency framework by Giel & Issa (2015); the BIM Quickscan (van Berlo et al., 2012); and the BIM readiness frameworks by Gu & London (2010) and Haron (2013). The major limitation of these frameworks is the fact that they have all been designed to measure internal organisational maturity for the purposes of BIM implementation rather than qualification for projects. Other frameworks such as the BIM scorecard by Kam et al. (2014) was defined to measure project BIM implementation and performance rather than the assessment of an organisations qualification.
Despite some relevance to the qualification process, the Computer Integrated Construction (CIC) matrix (CIC, 2013) has a number of limitations. The CIC (2013) framework is generally an implementation advice and guidance document. Despite the provision of guidance for assessing Request for Qualifications (RFQ) and Request for Proposals (RFP), it is unclear how all the elements in the documents can be synergised for a comprehensive qualification of potential CSC's for BIM enabled projects. Furthermore, priority weightings for the RFQ and RFP assessment criteria have not been provided, thus, it is unclear, the relative importance assigned to these criteria.
In relation to the criteria relied on by these tools, most of the existing tools and frameworks focus on hard measures pertaining to the physical resources and processes required to deliver BIM models as opposed to other competency and organisational factors (Sebastian & van Berlo, 2010;Chen et al., 2016). Sackey (2014) described BIM capability discourse as hard technologically deterministic to the neglect of the socio-technical nature of its use in practice. Indeed, most existing frameworks heavily focus on assessing BIM as a product or technical process to the neglect of many people-related competencies and attributes (NIBS, 2007;IU, 2009;Du et al., 2014). The soft human behavioural or organisational factors that influence the competence to deliver BIM have not been adequately considered by most frameworks despite evidence of the role of these factors in BIM delivery success (Sebastian & van Berlo, 2010;Haron, 2013). There are, however, a few frameworks that have considered all these dimensions of capability (see Succar, 2010;van Berlo et al., 2012;CIC, 2013;Giel & Issa, 2014;Kam et al., 2014). None of these were, however, developed specifically for the purposes of selection or for the UK CSC context. The complementary application of different frameworks is also challenging since the type of criteria considered and the nature of criteria weighting renders them generally incompatible (Sebastian & van Berlo, 2010).
In addition to limitations related to criteria used, there are also methodological challenges such as lack of empirical validation of most existing frameworks. Firstly, there is a relative lack of reliance on robust computational methods for prioritising criteria used in existing BIM capability frameworks (Mahamadu et al., 2015). Generally, the relative importance of criteria in these frameworks has either been arbitrary allocated or based on their importance as capability metrics (Succar, 2010;CIC, 2013) rather than their contribution to delivery success. Secondly, scientific underpinning for validating most existing frameworks and tools is unclear (van Berlo et al., 2012;Kam et al., 2013b). This is attributed to the fact that a good number of these frameworks were not developed through academic research and empirical enquiries (Giel & Issa, 2014). Furthermore, some of these tools were developed as part of BIM implementation guidance rather than for academic purposes, thus, lack the necessary academic rigour in the determination of criteria weight or importance (see Succar, 2010;CIC, 2013;PAS1192:2, 2013PAS91, 2013).
BIM capability criteria used in existing tools and frameworks Succar et al's. (2012) BIM competency hierarchy provides a broad and generic description of the necessary criteria used in BIM capability evaluations namely technology, process and policy. The technology category of criteria describes specific abilities related to physical artefacts including software, hardware and data or networks. The process category is used to describe resources, activities, workflows, products, services, leadership and management related capacity for delivering BIM. Finally, Succar et al. (2012) describes a policy category which encompassing contracts, benchmarks and guidance for attainment of BIM implementation objectives. Dib et al. (2012) identified the following as the critical areas for the attainment of BIM maturity: planning and management of process and technology; team structure; hardware; process definition; and information management. The Pennsylvania State University BIM guide (CIC, 2013) evaluates organisations maturity in the following key areas: strategy, BIM uses, process, information, infrastructure and personnel. The CIC (2012) BIM implementation document includes a guide for evaluating RFP's and RFQ's with proposed criteria relevant to pre-qualification and selection process such as price, experience and proposed deliverables. However, importance of the maturity criteria relative to the proposed RFP and RFQ criteria is not specified in this document. Thus, there remains a need for further exploration of criteria that are most critical for evaluations as well as their relative importance in relation to their contribution to delivery success.

BIM Capability Criteria and Delivery Success
Thus, these emerging standards, frameworks and tools provide basis for identifying appropriate BIM qualification criteria for selecting CSC on BIM-enabled projects (Succar, 2010;van Berlo et al., 2012;NIBS, 2012;CIC, 2013;Kam et al., 2013a, b;Succar et al., 2013;Du et al., 2014;Giel & Issa, 2014). However, none of these initiatives provide the necessary links between BIM utilization capacity of an organisation and delivery success particularly. While some studies have explored the role of BIM maturity in project performance generally , there remains no studies specifically looking at BIM delivery success rather than overall project success especially in the CSC context.  surveyed Dutch organisations to identify the influence of CIC (2013) maturity elements on project performance. Surprisingly, few statistically reliable associations were found between BIM maturity and project success indicators (time and cost) with inconclusive findings on the effect on delivery of project quality. , thus, caution against over optimism in expectations of BIM's influence on project performance. Despite the relevance of this study, the performance factors that were investigated related to project success rather than success in the delivery of BIM itself. It is unclear which capability criteria are most important at this phase considering the need for concise yet detailed evaluations which need to provide accurate predictions of likelihood of successful delivery. Despite the provision of insight about the role of BIM process predictability and control on BIM business performance, the relevance of the proposed maturity factors, as well as their relative importance in the qualification and CSC context, is unclear The main weaknesses of existing frameworks for BIM capability assessment that render them challenging to use in the pre-qualification and selection of CSC is presented below:  Lack of academic validation and theoretical underpinning;  Criteria prioritisation not based on empirically established relationships between capability attributes and BIM delivery success;  Framework or definitions are not intuitive to practitioners, thus, remain obsolete to the pre-qualification and selection context;  Lack of comprehensive consideration of product, process, people notwithstanding hard and soft behavioural and attitudinal aspects in measurement of capability;  Commercial interest and involvement in tool development;  Challenges in complementarily use of the different frameworks and tools due to variations in the type of criteria focus and weighting allocated to each criteria; and  Most importantly, there is no existing study that has investigated BIM capability attributes in relation to their specific use as qualification criteria, as well as their influence on BIM delivery success in practice.

Methodology
To address the aims of the study, a sequential exploratory mixed method research strategy is adopted based on a pragmatic philosophical stance. The first phase explores BIM expert views on appropriate CSC BIM qualification criteria based on their experience as part of the tender evaluation process. This is achieved through semi-structured interviews with eight construction BIM experts with managerial roles in leading construction organisations. Participants included BIM managers, digital engineers and commercial managers and quantity surveyor with an average of 16 years' industry experience. All interviewees were professionals in charge of the development and implementation of BIM within a major construction project or their organisations in the UK. Interview protocol sought information about interviewee's awareness and use of BIM capability frameworks and tools, as well as opinions about appropriate BIM qualification criteria for CSC pre-qualification and selection. This was used to generate a wide range of possible qualification criteria for the CSC, as well as a comparison of proposed criteria in exiting BIM capability frameworks and toolsets. Thus these semi-structured interviews were used to generate a list of BIM capability attributes that are relevant to the pre-qualification and selection of CSC to be part of projects. Interviewing is embedded in an interpretive philosophical stance, the qualitative interviewing in the research offered fluidity and enabled effective contextualisation of issues (Knight & Ruddock, 2008). The interviews also provided an opportunity to ignore priori ideas about BIM capability in order to draw on the knowledge and experience of experts about appropriate BIM capability criteria that need to be used specifically for the CSC selection process. Interviews lasted 30-40 minutes and transcribed verbatim for subsequent analysis. These formed part of the exploratory phase of the research and were terminated by the eighth interview as a result of saturation (Guest et al., 2006).
The interview phase was followed by Delphi survey of 30 construction practitioners with BIM experience resulting in the return of 25 valid final Delphi responses about the most critical capability criteria for the qualification of CSC. This was determined through statistical determination of consensus by a computation of interrater agreement (rwg) among Delphi participants. The Delphi technique was developed by Dalkey & Helmer (1963) as a method for achieving convergence of opinion among groups of people. The Delphi technique is, therefore, regarded as a group decision making process where controlled communication is used to collate expert opinion about a subject in iterative rounds of data collection, discussions and feedback (Sourani & Sohail, 2014). Despite relatively lower use of Delphi within construction management studies, it has gained popularity recently (Hallowell & Gambatese, 2010). Previous use of the Delphi technique included the determination of contractor selection criteria (Hatush & Skitmore, 1997), as well as BIM competence criteria prioritisation for owner organisations (Giel & Issa, 2014). The Delphi technique adopted here allowed a quantitative approach to determining the most relevant out of the 45 proposed criteria (Table 1) for the qualification of CSC for projects.
The hybrid epistemological status of Delphi, makes it both positivist and interpretivist in nature (Powell, 2003), therefore, it suits the broader pragmatic philosophical stance of this study where multiple methods were used to achieve the overall aim. Typically, contractor evaluation in itself is undertaken by a few experienced people, thus, the use of an expert data collection technique, such as Delphi, is a natural choice in the determination of qualification (Hatush & Skitmore, 1997) or competence related criteria (Giel & Issa, 2014;. Subsequently, a survey of practitioners on (n = 64) BIM enabled projects was used to establish the influence of the critical BIM qualification criteria on overall BIM delivery success. This was achieved through requesting senior project participant's independent evaluation of the BIM delivery performance of CSC firms on their respective projects as well as their perception on the influence of capability attributes on their overall performance. Multivariate statistical analysis techniques were engaged to identify qualification criteria influence on success with the aid of SPSS (version 19) software.

Data Analysis and Results
The approach to data analysis and results from the various stages of the study is presented in this section.

BIM Capability Criteria for Pre-Qualification and Selection
Based on the phenomenological principles, thematic analysis was adopted to establish a hierarchy of BIM qualification criteria for CSC. Thematic analysis offers flexibility for unearthing themes that deepen the understanding of topics that have not been adequately explored in a particular context (Thomas & Harden, 2008). A scarcity of studies specifically for the pre-qualification and selection process, meant thematic analysis was deemed appropriate contextualisation of criteria in the CSC and qualification context. Based on the coded responses, interviewee's opinions on BIM qualification criteria were further categorised into distinctive but related concepts, leading to the development of a three tier hierarchy of BIM qualification criteria as presented in Table 1.
The first tier consisted of four categories of BIM qualification assessment, eleven distinctive qualification criteria areas. The categories were (

Capacity and resources
Administrative and Strategic Capacity

Identification of the Most Critical BIM Qualification Criteria
Out of the 60 practitioners contacted for the Delphi study, 35 responses were received. The responses were scrutinized from the initial 35 responses resulting in the elimination of five responses due to missing data, thus resulting in an overall response rate of 50%. After the first round descriptive statistics (mean) was computed and a summary provided to participants as part of the Delphi procedure requirements. This was sent with another set of questionnaires to the 30 participants resulting in 25 second round Delphi returns representing 83% retention rate.
Since the Delphi technique focuses on eliciting expert opinions, the criteria relied on must ensure the selection of the most knowledgeable and experienced people in the subject area. This may also consist of individuals who are primary stakeholders or have considerable interests in the subject (Hsu & Standford, 2007). Delphi participants were recruitment through the extension of invitations participant organisations in UK government BIM promotion events as well as professional and academic conferences in UK. Majority of respondents were Architects (26.7%) followed by Quantity Surveyors (20%) then Project and Construction Mangers (16.7%). Majority of respondents held at least a Masters Degree (46.7%) with a substantial proportion (23.3%) holding a Doctorate degree. This is summarised in Table 2. In addition to industry practitioners, there were also several academic contributors to the Delphi study. This included BIM researchers (10%), as well as Professors (3%) or Senior Lecturers (10%) in construction collaborative technologies. Delphi participants in this study were adequately experienced both within the construction industry as well as in the delivery of BIM and other Virtual Digital Construction (VDC) technologies. Respondents had an average of 16 years industry experience with some participants having as much as 40 years construction industry experience. Most respondents had an average of 7 years in BIM or VDC and an average of 10 years involvement in tender, pre-qualification or selection activities.

Analysis of Delphi Study Data
Since Delphi is conducted in several rounds there is a need for statistical determination of stability between rounds in order to terminate the process when there is no significant shift in participant opinion between Delphi rounds. Spearman's coefficient normally denoted by rho was adopted to assess stability between Delphi rounds. Spearman's rho is a non-parametric test for statistical dependence between two variables (Jamieson, 2004). It compares the medians of these variables, thus, making it a preferred option for correlation analysis for ordinal data (Field, 2005). Hence, considering the ordinal nature of the data gathered from the questionnaires, Spearman's rho was deemed approriate for this study. The correlations tests between Delphi rounds resulted in statistically significant correlations, indicative of no significant shift in participant opinion about the criticality of criteria. Competence related criteria recorded rho = 0.857 (p < 0.01); Capacity and Resources related criteria recorded a rho = 0.964 (p < 0.01); while Culture and Attitude related criteria recorded a rho = 0.816 (p < 0.01).
After the second round of Delphi data administration, there was therefore no significant shift in participant's opinions, thus, the Delphi study was terminated.
The Delphi participants were asked to provide a rating on each of the 45 sub-criteria (Table 1) constituting the third level of the proposed BIM qualification criteria hierarchy. The questionnaire requested participants to state the extent of their agreement with the use of the proposed criteria in determining suitable CSC candidates on BIM enabled projects. The rwg was used in computing consensus among participants in relation to their agreement that a criterion is relevant and critical to the qualification of CSC for BIM enabled projects. The rwg provides a statistically significant measure of the consistency of agreement among raters (Mandrekar, 2011). It was considered as the most appropriate to ensure confidence in interpreting the results from a review of previous construction management studies (Manu, 2012;Bashir, 2013). It was concluded that the adoption of a robust computational approach to determination of inter-rater agreement will provide more confidence in the measurement of consensus. Despite a conventional acceptance rwg values equal to or greater than 0.7 as adequate indicator of consensus, it is acknowledged that sample size and number of variables affect reliability of scores (Harvey & Hollander, 2004). According to Cohen et al. (2001), the rwg index values of 0.7 may, therefore, be inadequate for showing agreement in some circumstances. A method for the determination of a minimum acceptable rwg is proposed and incorporated within the 'R' software package. In this study, 10,000 simulation runs were relied on to ascertain minimum acceptable rwg values (Bliese, 2000). A minimum threshold agreement was found to be rwg = 0.75, specifically, for the data collected from this Delphi study. From this analysis, only criteria with rwg ≥ 0.75 were considered as haven reached consensus. The mean scores of all criteria achieving rwg ≥ 0.75 was then examined further.
Based on the analysis of the rwg values and mean ratings, all criteria that recorded acceptable (rwg ≥ 0.750), as well as a mean scores equivalent or above 'agree' were retained. This was based on the five-point rating scale used in the Delphi survey (1 = Strongly Disagree, 2= Disagree, 3 = Neither Agree or Disagree, 4 = Agree, 5 = Strongly Agree). From the principles of mathematical approximation, BIM qualification criteria with mean values ≥3.5 were accepted as critical provided there was consensus among participants (i.e. rwg ≥ 0.750). This was to ensure that the rules of Delphi were met, as well as confidence that the mean rating was statistically acceptable to most participants in the Delphi survey. From this analysis 28 qualification criteria were identified as critical, representing 62% of the initially proposed criteria from the interviews. This is summarised in Table 3.   (Table 3). This category however also retained the most number of critical criteria with BIM Standards (Mean = 4.400 and rwg = 0.792) and Network Infrastructure Availability (Mean = 4.280 and rwg = 0.863) emerged as most critical after the analysis of consensus.
Three culture and attitude related criteria were also eliminated namely: reputation in relation to Relationship with Principle Supplier (Satisfaction) (Mean = 4.120 and rwg = 0.653), Youthfulness of Staff (Mean = 2.360 and rwg = 0.547); and Number of Graduates in Firm (Mean = 2.480 and rwg = 0.703). The criteria considered as the most critical to assessing culture or positive attitude towards technology was the assessment of evidence of IT Support to Core Business and Processes within CSC Firm (Mean = 4.120 and rwg = 0.861). There was general consensus that cost / price of BIM service is critical to the BIM qualification process of CSC for projects (Mean = 3.920 and rwg = 0.753).

Qualification Criteria Influence on BIM Delivery Success
The general survey was used to ascertain the perceived influence of the 28 critical BIM qualification criteria (from the Delphi study) on BIM delivery success. The response rate for the survey was 13.3% (n = 64) after three-month administration across identified UK BIM enabled projects as well as within online professional BIM networks. The response rate and number of responses is deemed adequate in view of the response statistics similar to other construction management related studies (Ankrah, 2007).
From an analysis of the background of respondents, the majority were BIM Managers or Technicians (31.4%) followed by Project, Construction Managers (15.6%) and Quantity Surveyors (15.6%). The majority of respondents (46.7%) had between 11-15 years industry experience. Many respondents (35%) also had between 4-6 years' experience working with BIM or other relevant digital construction technologies. Bachelor's degree as their highest educational qualification with a substantial number of respondents holding higher degrees, such as Masters (29.7%) or a Doctorate (7.8%). This is indicative of substantially well experienced and knowledgeable group of respondents whose opinions are most likely to be valid and relevant to the research.
Most of the CSC organisations assessed as part of the survey belong mainly to the top or middle tier the project supply chain. The majority were Design Consultants, with Architects representing 34.4%, while Engineering Consultants represented 25% of the organisations assessed. Among the least assessed organisations were Material Suppliers (4.7%) and Sub-Contractors (6.3%). In relation to the background of projects on which these firms were assessed, 19.3% were notably large scale with estimated project values above £50 million. Significant number was (80.7%), however, had less than £50 million in value but with more than half above £25 million. Most of the projects (40.3%) had an intermediate level of CSC integration with a substantial (35.5%) number of the project CSC's considered as fragmented. Most of the projects had some middle tier CSC involvement in the project BIM process with only 1.6% reporting lower tier participation. A large proportion (90.3%) of the projects surveyed were buildings, with only 9.7% being civil engineering projects.

Analysis of Survey Data
Mean ratings of the main qualification criteria themes was computed based on an index of consisting of the mean contribution of each of its constituent criteria. In addition to the mean contribution statistics, one-way analysis of variance (ANOVA) was used to determine whether there are any significant differences between the means of three or more independent groups within a data set (Field, 2005). ANOVA was used to analyse whether certain CSC firm characteristics could be statistically differentiated in relation to the attainment of BIM delivery success on projects. The ANOVA tests the null hypothesis expressed in Equation 1.
The results of the survey are summarised and compared with the results from the Delphi study in Table 4.
From a review of the data, it is evident that there is high similarity between main criteria in relation to their importance as qualification criteria and their importance to delivery success.
Thus the existence of a capability to delivery BIM is highly related to the ability to delivery successfully. However, a detailed look at individual criteria reveals varying levels of importance. In order to ascertain the statistical significance of these differences, a test of agreement (spearman's coefficient) was conducted between survey and Delphi ratings. The RII of criteria from the general survey was based on the contribution of the criteria to BIM delivery success. The Relative Importance Index (RII) was used to convert frequencies on each ordinal data point into an aggregated index of scores to aid ranking of criteria. This ranking aided both prioritisation and subsequent correlation analysis (Babatunde et al., 2010).
The RII of criteria in the Delphi survey was based on practitioner's views about the criticality of the criteria as a qualification metric. A comparison was, therefore, made between the RII rankings from the two surveys to identify statistically significant differences ( Table 4). This is used to measure whether or not there is a relationship between the levels of importance of attributed to a criterion as a qualification metric and its influence on delivery success in practice. The test of agreement between RII's resulted in non-significant degrees of associations: competence related criteria [rho = -0.018, p > 0.05]; capacity and resources criteria [rho = -0.047, p > 0.05]; culture, attitude and cost related criteria [rho = 0.058, p > 0.05] and overall for all criteria [rho = 0.039, p > 0.05].
From these results, it is evident that despite the similarity in criteria importance rating, there was no statistically significant (p > 0.05) correlation between criteria contribution to BIM capability and its contribution to delivery success. Overall Competence criteria were rated as most important both in survey (Mean = 3.449 [1]) and Delphi study (Mean = 4.173 [1]). Experience was rated as the single most important qualification criteria as well as contributor to BIM delivery success. However, Organisation's Experience (Mean = 4.260 [1]) was rated as very important in the assessment of a CSC's ability to deliver BIM, while Staff Experience (Mean= 3.883 [1]) was the single most important influencer on BIM delivery success. This is indicative of the fact respondent's rated competence related criteria as important indicator of qualification as well as highest contributors to overall delivery success. Similarly, Capacity and Resources criteria was also rated as second most important in the survey (Mean = 3.244 [2]) and Delphi study respectively (Mean = 4.062 [2]). In relation to this category Technical (Physical) Resources (Mean = 4.213 [2]) was regarded as the second most important qualification indicator while Specific BIM Modelling Capacity (Mean = 3.426 [2]) was second most important contributor to success. Capacity and Resources criteria was however the most influential category of criteria based on overall mean weighted contribution of all its subcriteria. Cost (Mean = 3.188 [3]) was rated as the third most important contributor to success, Culture and Attitude (Mean = 3.987 [3]) was regarded as the third most important indicator of BIM qualification (i.e. an ability to deliver BIM).
From a review of these findings, it is evident that there is high similarity between main criteria in relation to their importance as qualification criteria and their importance to delivery success. However, a detailed look at individual criteria reveals varying levels of importance. Thus, while some criteria are generally regarded as important predictors of success during the qualification process, perceptions about their contribution to delivery success in reality are lower.
The one-way ANOVA between-groups was used to analyse the effect of CSC firm's type on the attainment of delivery success, as well as the level of influence of BIM qualification criteria thereof. From this analysis firm type had an effect on the perceived level of influence of Organisational Structure (F = 2.186; p < 0.05, n = 64) on overall delivery success ( Table 4). This is indicative of the fact that no statistically significant (p < 0.05, n = 64) effect was noticed as a result of differences in the types of firms surveyed. Thus the priority weightings of criteria importance are applicable to many CSC firm types. It is, however, worth noting that the firms assessed in the survey were mostly first and second tier suppliers with high design responsibility (i.e. Architect, and Engineering design consultants).

Discussion
Findings were reviewed and interpreted in light of existing knowledge as well as criteria proposed for BIM capability assessment within existing frameworks. Suggested criteria were largely similar to criteria relied on in five of the most relevant BIM capability frameworks ( Table 1). The findings further highlight the uniqueness of BIM qualification from generic capability concepts in a number of ways. The BIM qualification process supports the contention that, both generic and contextual indicators of ability must be considered. Thus, a significant number of the proposed BIM qualification criteria were directly related to the specific context within which candidates are to be assessed (i.e. specific project or client requirements). One critical contextual criteria proposed during the interviews was the cost/tender price for BIM service. Most capability frameworks do not fully consider cost, except for the CIC BIM planning guide (CIC, 2013), which similarly acknowledges the importance of cost as a qualification criterion. The other selection specific criteria included 'Suitability-BEP's for Project'; 'Innovativeness-BEP's for Project'; 'BIM Vendor Involvement and Support'; 'Suitability of Privacy and Security Plans'; and previous 'Experience on Similar Projects'. Other proposed criteria from the interviews that have generally not been considered in existing capability frameworks include 'Youthfulness of Staff' and 'Number of Graduates in Firm'. Furthermore, some criteria were more specific to the CSC context including 'Relationship with Principle Supplier (Satisfaction)' and 'Reputation -Performance on Past BIM projects'.
Existing BIM capability frameworks have mainly considered the generic indicators of BIM maturity or competence, particularly process, information and technological infrastructure capability (Chen et al., 2014;. These additional criteria have not been adequately considered in BIM capability assessment in existing frameworks, where the focus has often been on BIM implementation and internal organisational process maturity (Succar, 2010;van Berlo et al., 2012;NIBS, 2012;Kam et al., 2013;Haron, 2013;Giel & Issa, 2014). As a result of their focus on internal process maturity, most existing frameworks place less emphasis on historical indicators of capability such as experience. The qualification hierarchy derived from the interviews, however, places significant emphasis on the proposal of seven different historical measures of capability relating to individual staff and organisations experience.  (5) 3.244 (2) 3.933 (9) 4.062 (2 (CIC, 2013;Haron, 2013). It is worth researching whether or not this may however be more important in relation to principal suppliers rather than their supply chains who were the focus of this study.
Based on the findings, an alternative hierarchal structure of criteria is proposed with cognisance to its relevance to BIM qualification of CSC candidates for projects. Qualification is often aimed at measuring anecdotes of likely success, thus, it requires measurement of various attributes that have a more holistic view of capability. The proposed criteria in this study meet this requirement since they cut across the different concepts of BIM capability attributes as theorised in previous studies (Giel & Issa, 2013), including process, people, product driven, technology or information driven criteria (Succar, 2010;Giel & Issa, 2013;Chen et al., 2016).  (Succar, 2010;NIBS, 2012). However, the findings highlight the importance of historical and evidential demonstration of competence through knowledge and skills in BIM delivery within organisations. The emergence of Organisations BIM Experience as one of the most critical BIM qualification criteria aligns with the general view of contractor and consultant selection theories, where past experience is often regarded as the single most important qualification criterion (Hatush & Skitmore, 1997;Doloi, 2009). Many existing capability frameworks relate to internal implementation and benchmarking, thus often focusing on process maturity or technological infrastructure availability to the detriment of historical indicators of capability (Chen et al., 2014). However, in the pre-qualification and selection context, it has emerged that a demonstration of prior experience with BIM is mostly critical to qualification.
BIM Qualifications category of criteria relate to the possession of externally validated evidence of capabilities and competencies (academic and professional). This includes certificates, licenses or degrees for individual staff or an organisation as a whole. While these have been acknowledged in the BIM capability literature (Succar et al., 2013), this study highlights its particular importance in a pre-qualification and selection scenario. Since qualification often happens within limited timescales (Holt et al., 1994;Arslan et al., 2008), the thoroughness of capability assessment can sometimes be impaired. Thus, from the findings, the possession of evidence from recognised third party institutions about an individual's or firm's ability to deliver BIM is particularly important to the qualification process.
These findings are consistent with both BIM capability theories which have alluded to the importance of historical indicators of competence, (Succar et al., 2013) hard technology centric BIM maturity theories (NIBS, 2012;Sackey 2014). The role of Managerial Staff BIM Experience is also highlighted in this study. Despite the recognition of management buy-in as the most important criterion in BIM competence assessment (Giel & Issa, 2014), the focus on management has never been looked at from the perspective of the management's BIM experience. Giel & Issah's (2014) study, however, pertains to owner organisation's BIM competence, thus, significantly different to the principal suppliers' or CSC context.

Conclusion
The findings generally aligned with existing knowledge about the importance of process, technology and information management attributes as the primary indicators of capability. It was also revealed that there is a good degree of relationship between the possession of a BIM capability and likelihood of delivery success in general. However, a more detailed analysis of criteria revealed varying levels of importance in relation to each individual criterion's importance as a capability metric and, on the other hand, its contribution to various dimensions of BIM delivery success, respectively. For instance, technological and infrastructure requirements are perceived as important qualification criteria, however, they were found to contribute less to delivery success in comparison to criteria such as experience and specific BIM modelling process maturity and expertise. Thus, the findings support a notion that BIM capability assessments for implementation or a basic determination of the suitability of candidate requires available technological infrastructural capacities. However, when it comes to leveraging some of the tangible and intangible benefits of BIM, information related process maturity and availability of knowledge and skills among individual personnel is paramount.
The relationship between BIM capability and delivery success is multi-dimensional rather than unilateral, as theorised in many existing BIM capability assessment studies. While this study revealed that most of the BIM capability attributes are reliable predictors of success, thus, important for BIM qualification process, their contribution to success in practice may be to a lesser extent. Thus, the prioritisation of criteria during pre-qualification or selection must be based on their relative contribution to success in practice. Prioritisation of criteria in existing frameworks is, however, based only on the relative importance as capability metrics rather than their relative contribution to success in practice based on empirical evidence as investigated in this study. Furthermore, capability or maturity only denotes the basic abilities to perform BIM related tasks efficiently rather than the actual attainment of the objectives expected from BIM deployment. Thus, this study provides insights into the role of prioritising BIM capability criteria based on evidence on their contribution to delivery success in practice. There must be a recognition of the role of BIM performance and success measurement as part of BIM execution planning in practice. This should facilitate future assessment of the specific contributions of various BIM capability elements within an organisations and its impact on the various dimensions of BIM delivery success. The need for prioritising BIM capability criteria based on their contribution to delivery success must be recognised in standards such as Publicly Available Specifications (PAS) of the UK in order to enhance the feedback of post project evaluation into the pre-qualification and selection process.