Augmented and virtual reality have the potential to provide a step-change in productivity in the construction sector; however, the level of adoption is very low. This paper presents a systematic study of the factors that limit and drive adoption in the construction sector-specific context. A mixed research method was employed, combining qualitative and quantitative data collection and analysis. Eight focus groups with 54 experts and an online questionnaire were conducted. Forty-two limiting and driving factors were identified and ranked. Principal Component Analysis was conducted to group the identified factors into a smaller number of factors based on correlations. Four types of limiting factors and four types of driving factors were identified. The main limitation of adoption is that AR and VR technologies are regarded as expensive and immature technologies that are not suitable for engineering and construction. The main drivers are that AR and VR enable improvements in project delivery and provision of new and better services. This study provides valuable insights to stakeholders to devise actions that mitigate the limiting factors, and that boosts the driving factors. This is one of the first systematic studies that presents a detailed analysis of the factors that limit and drive adoption of AR and VR in the construction industry. The main contribution to knowledge of this study is that it grouped and characterized a myriad of limiting and driving factors into easily understandable categories; so that, the limiting factors can be effectively mitigated, and the driving factors potentiated. Also, a roadmap with specific short term and medium-term actions for improving adoption has been outlined.