A systematic review of study results reported for the evaluation of robotic rollators from the perspective of users

Purpose: To evaluate the effectiveness and perception of robotic rollators (RRs) from the perspective of users. Methods: Studies identified in a previous systematic review published 2016 on the methodology of studies evaluating RRs by the user perspective were re-screened for eligibility based on the following inclusion criteria: evaluation of the human-robot interaction from the user perspective, use of standardized outcome measurements, and quantitative presentation of study results. Results: Seventeen studies were eligible for inclusion. Due to the clinical and methodological heterogeneity across studies, a narrative synthesis of study results was conducted. We found conflicting results concerning the effectiveness of the robotic functionalities of the RRs. Only a few studies reported superior user performance or reduced physical demands with the RRs compared to unassisted conditions or conventional assistive mobility devices; however, without providing statistical evidence. The user perception of the RRs was found to be generally positive. Conclusions: There is still no sufficient evidence on the effectiveness of RRs from the user perspective. More well-designed, high-quality studies with adequate study populations, larger sample sizes, appropriate assessment strategies with outcomes specifically tailored to the robotic functionalities, and statistical analyses of results are required to evaluate RRs at a higher level of evidence.


Introduction
The maintenance of mobility is fundamental for the quality of life, wellbeing, and autonomous life of older people [1,2], and being physically active is associated with numerous positive health outcomes in this population [3][4][5]. Impaired mobility is, however, common among the elderly [6,7] and has been shown to be a risk factor for subsequent disability, loss of independence, and mortality [2,8,9].
To enhance mobility, extend independent living and, ultimately, to improve the quality of life of affected people, assistive mobility devices (AMDs) such as walkers, which are used more than any other AMD except the cane [10], have been developed with early focus on physical support [11]. However, as mobility in the elderly may not only be restricted by motor but also by sensorial and/or cognitive impairments [12], conventional AMDs (i.e. canes, crutches, walkers, rollators) may not be sufficient to cover the needs of persons suffering from such additional geriatric deficits.
Recent advances in robotics have made it possible to develop a new class of more intelligent walkers by integrating robotic technology, electronics and mechanics [13]. According to the user's needs, these so-called 'smart walkers', 'robotic walkers', or 'robotic rollators' (RRs) are not restricted to their primary focus, i.e. physical support, but are capable of providing mobility assistance in different functional domains [14,15]. Overall, RRs have evolved to provide physical support, sensorial and cognitive assistance, and/or health monitoring [16].
An important part in the development process of RRs represents the verification of the technical capability of the devices and their functionalities. However, in addition to such technical testing, an evaluation that considers the user perspective in terms of the user's performance, physical demands and satisfaction with the RRs is also essential to enable and optimize a user-focused development, to prove the usability and effectiveness, and to document the potential added value of the innovative, robotic functionalities for the intended user group [33]. In general, to ensure that assistive technology devices meet the needs, requirements and preferences of users and to become successful on the market, the product development and such evaluation processes have to be closely aligned and guided by continuous end-user input at all stages [31,34,35].
The evaluation of RRs from the user perspective seemed to be associated with significant methodological challenges [31,36]. In our recent systematic review on the methodology of studies evaluating RRs by the user perspective, the identified studies showed large heterogeneity in study population, design of studies/test scenarios, and assessment methods. No generic methodology to evaluate RRs from the user perspective could be identified [19]. We also found major methodological shortcomings related to insufficient sample sizes, lack of appropriate standardized and validated assessment methods, and lack of statistical analyses of study results.
The evidence of the effectiveness and positive user perception of the RRs might have been substantially influenced by these study limitations and different methodological approaches.
However, as we did not report the results of the studies identified in our previous review, we were so far not able to address this topic. To our knowledge, also no other systematic review has been published on the results of studies evaluating RRs by the user perspective. Therefore, the purpose of this article is to summarize and review study results reported for the evaluation of RRs from the user perspective.

Methods
This review involved studies identified in our previous systematic review on the methodology of studies evaluating RRs by the user perspective [33]. The literature search, inclusion criteria, and study selection process of the previous systematic review have been described there in detail, so only relevant information for the analysis of study results are reported here. The systematic literature search in the electronic databases PubMed and IEEE Xplore, reference lists of relevant publications, and key author's own databases was performed there until December 31, 2014. The studies identified by this search were re-screened and assessed for eligibility in the current review based on the following inclusion criteria: (1) evaluation of the human-robot interaction (HRI) from the user perspective; (2) use of a standardized outcome measurement, and (3) quantitative presentation of study results. The selection process was performed by two independent reviewers (C.W. and P.U.). Disagreement was resolved by consensus or third-party adjudication (K.H.). After inclusion, relevant data were extracted by 1 researcher (C.W.) and confirmed by another researcher (P.U.).

Results
After removing duplicates, screening titles and abstracts, and assessing the full-text articles, our previous systematic review covered 28 studies [33]. Of these, 11 studies were excluded after re-screening for eligibility in the current review as four did not present quantitative data on study results, four did not use standardized outcome measurements, two did not provide sufficient information on the outcome measurement used, and one did not evaluate the HRI by the user perspective (see Figure 1).
[ Figure 1 near here] The remaining 17 studies 1 were reviewed and review results were extracted in table format, containing information on the names of RRs, study sample, robotic functionality to be tested, design of studies/test scenarios, assessment methods, and study results (see Table 1).
The methodology of identified studies was described and discussed in detail in our previous systematic review [33]. In this article, we extracted only information on the study methodology relevant for an adequate presentation, understanding, and discussion of the study results.

Design of studies and test scenarios
Seventeen articles described comparative studies or test scenarios in which RRs were compared with conventional AMDs or unassisted walking/STS transfers ('inter-device comparison') [17,18,23,28,37,[40][41][42], or in which different assistance levels (e.g. activated vs. nonactivated navigation assistance) [22,23,25,27,28,38], development stages [18,37] or userinterface designs [26] of the same RR were compared to each other ('intra-device comparison'). Three articles reported on observations and provided only descriptive data without any reference or comparative values for classification of study results [16,25,28]. Two articles described interventional studies that evaluated the effects of an RR-assisted ambulation training compared to traditional ambulation training on parallel bars [39] or of the repeated use of a RR over six consecutive days [17]. One article described a test scenario in pre-post-test design in which the subjective user perception of the overall RR functionality was assessed before and after a series of trials [23].

Physical support
The ability of the RR in supporting users' gait and motor-functional performance was assessed by clinically well-established walking and functional mobility tests ( [41,42], gait analysis methods [17,42], self-designed walking paths [37], a subjective expert rating of abnormal gait patterns (festinating gait, freezing of gait) [17], or a single dichotomous question on the ease of walking with a RR [40]. The most frequently used outcome of these assessment methods was gait speed or RR velocity [37,41,42].
The STS functionality of the RR was evaluated by a self-designed user questionnaire on the ease and confidence of standing up with the RR [18].
The physical demands when using the RRs was evaluated by measuring the exertion of force applied to steer the RR [28,37], the oxygen consumption and metabolic cost of transport (COT, metabolic cost per unit of mass and distance travelled) [37], the torso kinematics and/or the muscle activity in lower limbs [22,41] during time-based performance tasks (navigation trail, 10MWT) or during walking with standardized gait speed.
To investigate the potential of the RR as rehabilitation training device, the subjects' gait and motor-functional performance and ability in activities of daily living (ADLs) were assessed by the 6-Min Walk Test (6MWT), 10MWT, Performance Oriented Mobility Assessment (POMA), and the Barthel ADL Index [39].

Cognitive assistance
Robotic functionalities that aimed to assist navigation and localization were evaluated on selfdesigned navigation trails [25][26][27][28]. Outcomes related to subjects' navigation performance covered simple quantifiable outcomes (e.g. task completion time, target achievement [28]) and more detailed, technique-based outcomes (e.g. deviation from optimal path [25,27], walking distance [28]) which were specifically tailored to the functionality to be tested and most frequently derived from the data flow created by the robot-integrated sensing technologies (e.g. laser range finder). One study used a dichotomous subjective question to assess subjects' preference of two user different user-interface designs of the RR's navigation assistance system [26].

Sensorial assistance
Obstacle avoidance and guidance functionalities of the RRs were evaluated on self-designed obstacle courses/walking paths [23,38] or during navigation trials [25,28]. The subjects' sensorial performance with the RRs was assessed by simple quantifiable outcomes such as task completion time or number of collisions [23,28,38], or by more technique-based, tailored outcomes such as the distance to obstacles [25,28] or the deviation from a path marked on the floor [38].

Overall functionality
Assessment methods that addressed the overall functionality of the RRs covered self-designed structured questionnaires with different items and different multistage rating scales to evaluate the subjective user experience with the RR [16,23,25,28,37,40]. The most frequently used questionnaire item addressed the manoeuvrability of the RRs [16,25,37,40].
In the following, we present the study results related to the different assistance functionalities to be evaluated in the identified studies.

Physical support
Out of the studies that compared robot-assisted walking and walking with conventional AMDs or without support of an AMD [17,[40][41][42], two reported superior gait performance with the RR, as indicated by a smaller number of abnormal gaits and lower gait variability (i.e. SD of gait speed) [17] or more positive responses on the ease of walking in robot-assisted walking [40]. The other two studies reported an inferior gait and motor-functional performance with the RR in clinically established walking or functional mobility tests, documented by an increased TUG completion time, increased step time and double limb support time dur-ing the TUG, and/or a slower gait speed (4MWT, 10MWT) [41,42]. In one of these studies, subjects achieved a higher gait speed (10MWT) with the RR when compared to walking in parallel bars [41].
One study reported the highest questionnaire scores for the use of the most recent development stage of the robotic STS assistance system, indicating that subjects perceived the STS transfer with this new development stage as being easier and associated with less fear of falling than with the previous development stage or without any assistance [18].
The study comparing subjects' gait performance with two different HRI systems reported no significant differences in the mean and SD of the RR speed between the newly developed and the traditional, state-of-the-art HRI system and that subjects were able to achieve a similar good speed control to the targeted speed with both HRI systems [37].
In two studies, walking with motorized RRs was reported to be more physically demanding than with conventional walkers, documented by an increased VO2 and significant greater COT [37], or substantially higher forces applied to control the RR [28]. In contrast, another study presented a lower muscle activity in lower limbs and trunk acceleration during robot-assisted gait when compared to walking with conventional AMDs [41]. One of these studies also compared the forces required to steer the RR when using two different HRI systems (traditional vs. newly developed system) and showed that these forces were significantly higher with the most recent version [37]. In another study assessing physiological demands in ambulation with different levels of RR's BWS system, muscle activity in lower limbs seemed to decrease with increasing BWS [22].
In the RCT study, robot-assisted ambulation training resulted in significant improved gait speed (10MWT) and motor-functional (POMA) and ADL performance (Barthel ADL Index), compared to the conventional ambulation training on parallel bars [39].
The interventional study performing gait analyses on six consecutive days reported the same positive level of subjects' gait performance over the entire 'intervention' period in terms of low gait variability and a small number of abnormal gait patterns in robot-assisted gait [17].

Cognitive assistance
In specifically tailored outcomes of the navigation trails, three studies reported superior user performance with the activated navigation assistance of the RRs in terms of smaller deviations from an optimal path [25,27] or a reduced walking distance [28] when compared to that with a conventional AMD or the same RR with non-activated navigation assistance. In less specific outcomes, however, one of these studies reported an inferior user performance in robot-assisted navigation, documented by a longer walking time and a slower maximum speed [28].
In all studies comparing different assistance level of the navigation assistance (e.g. shared user-robot vs. robot motion control), subjects achieved the highest user performance (smallest path deviations [25,27], shortest walking distance [28]) when the RRs provided maximum navigation assistance by the full robot motion control modes in which the subjects had no control over the motion direction of the RR but followed the RR rigidly along the robotplanned path.
When having the choice (dichotomous question) between two different user-interface designs for the navigation assistance system of a RR, most subjects (75%) seemed to prefer a map-based design when compared to a text-and-arrow based design (25%), as reported in one study [26].

Sensorial assistance
On obstacles courses, walking paths or during navigation trails, subjects tended to show a superior sensorial performance with the RRs with activated obstacle avoidance and guidance assistance when compared to that with a RR with non-activated sensorial assistance or a conventional walker, or without any AMD. Three out of four studies reported larger distances to the obstacles [25,28], a reduced number of collisions [28,38], or smaller deviations from a path marked on the floor [38] when using the RR with activated sensorial assistance. In one study, which performed a statistical data analysis, descriptive data indicated also fewer collisions but a longer walking time with the sensorial assistance of the RR; however, these trends could not be confirmed as statistically significant [23].
Out of the studies that compared different assistance levels of the RRs, one out of three reported a superior sensorial performance documented by larger distances to obstacles when maximum assistance was provided by the full robot motion control mode [28]. In the other studies, no apparent [25] or significant [23] differences in outcomes such as the distance to obstacles, number of collisions, or task completion time were observed.

Overall functionality
Independent of the different items included in the self-designed questionnaires (e.g. manoeuvrability, safety, comfort), a high number of positive responses [40] and positive average or median scores in the upper half [16,23,25] or even in the upper quartile [37] of the scales were achieved, suggesting, for instance, that the RRs were easy to manoeuvre or subjects felt safe and comfortable using the RR [16,23,25,37,40].
The study comparing subjects' user experience with two different development stages of the RR's HRI system reported positive average scores in the upper quartile of the rating scales for both the traditional and the newly developed HRI system, with no significant differences in any questionnaire item (e.g. comfort, overall experience, speed control) [37].
In the only study that assessed subjects' perception of the RR before and after the use of the RR, favourable average scores in the upper half of the rating scale were observed at preand post-test assessment with the tendency of more positive scores after participating in the study; however, the statistical analysis showed no significant differences between pre-and post-testing [23].

Discussion
The purpose of this systematic review was to summarize the results of studies evaluating RRs from the perspective of users. Included studies showed large clinical and methodological heterogeneity (sample characteristics, study design, assessment methods, outcomes), and findings of studies were mainly based on the authors' subjective appraisal without statistical data analysis or reference values for comparison. Such evaluations are of very limited value at a low level of evidence and rather comparable to mere use case descriptions. The overall evaluation of the effectiveness and user perception of the RRs is therefore severely hampered. Although hard to compare, a limited number of studies reported a superior user performance in specific outcomes when using the robotic functionalities compared to unassisted conditions or the use of conventional AMDs; however, these studies were performed with small sample sizes and without providing statistical evidence. The users' physical demands seemed not to be reduced with the RRs when compared to that with a conventional AMD. The overall functionality of the RRs evaluated by subjective user questionnaires was generally rated as positive by the users.

Physical support
Clinically established functional or walking tests such as the 4MWT, 10MWT, or TUG show various methodological qualities; however, they do not prevent a misuse of an inappropriate study outcome. When using a motorized RR with limited maximum speed and comparing it to a conventional walker or walking without any AMD, it is almost mandatory that the subjects achieved an inferior gait speed or task completion time with the RR, as reported in two studies [41,42]. Choosing such inappropriate and unidimensional outcomes underestimate or even completely miss the potential benefits of a RR to support users' gait and motor-functional performance. Augmenting established clinical performance-based measures (e.g. 4MWT, TUG) with technical assessment measures, such as done in one study by a video-based gait analysis [42], allows for a multidimensional analysis of subjects' gait by further temporalspatial gait parameters such as stride length, step time, or double limb support time. However, as such parameters are highly associated with gait speed and subjects' gait speed was limited in this study by RR's maximum speed, it is not very surprising that the subjects achieved superior performance also in these outcomes with the conventional walkers by which they were able to walk much faster. In contrast, studies evaluating subjects' RR-assisted gait and motorfunctional performance by less time-/speed-dependent outcomes but more qualitative performance outcomes (e.g. number of abnormal gaits, gait variability) or by more user-based outcomes (e.g. subjective perception on ease of walking/standing up) reported superior user performance and satisfaction with the RR when compared to with a conventional walker or without support of an AMD. These findings suggest that RRs may well have the potential to provide an added value for subjects' gait and motor-functional performance; however, the documentation of this seems to depend substantially on the choice of an appropriate outcome.
The development of AAL systems should involve a multi-stage iterative process, including iterative refinement of robotic prototypes/functionalities and their regularly evaluation during development process ('iterative design-development-testing procedure' [43]). As reported in one study, the most recent development stage of the STS assistance system was more positively perceived and rated by the subject than the previous one [18]. In the sense of an iterative development process, such findings indicate that the re-design and optimization of this robotic functionality seems to have been successful in this study. In contrast, in another study that developed and evaluated a new, alternative technical approach for the HRI system , such re-design seems to have been less effective, as indicated by the significant higher physical demands and similar gait performance reported for the subjects when using the more recent approach compared to the traditional, state-of-the-art HRI system [37].
RRs are augmented with a lot of technical hardware components substantially increasing their weight and inertia. The motion control of such heavy-weight, high-tech devices using HRI forces is still a challenging problem in the development of RR [25]. Since the forces required to control them and users' physical demands were reported to be higher compared to low-weight, conventional AMDs [28,37] and further improvements of traditional HRI systems appear to be difficult to achieve [37], there seems to be still no generic and optimal solution for the HRI making the handling of RRs comparable to that of a conventional AMD. In one study, the substantially higher user-applied forces may, however, also be caused by subjects' attempt to exceed robot's limited maximum speed [28]. When choosing a maximum RR speed without having in mind subjects' maximum gait speed, it is not surprising that subjects intuitively push hard to further accelerate the RR. These findings may indicate not only methodological flaws in the design of this study but also less optimized technical solutions in the design of the RR.
The reduced trunk accelerations and EMG signals in lower extremities in robot-assisted gait compared to walking with conventional walkers might be a direct consequence of subjects' lower gait speed with the RR [41]. Since gait speed may be closely related to torso kin-ematics and muscle activity in lower extremities, these findings seem to be almost inevitable and may indicate shortcomings in the design of a study. To ensure comparability of outcomes such as muscle activity, it is mandatory to standardize subjects' gait speed when using different types of AMDs, such as done in [37].
When using a RR for gait rehabilitation purpose, it is crucial to have the possibility to specifically tailor the amount of robotic assistance according to the user's individual gait performance. Since the muscle activity of lower extremities decreased with increasing assistance level of the BWS system evaluated in one study [22], this robotic functionality seems to be high adaptable allowing a user-specific adjustment of RR's assistance levels in rehabilitation process.
Based on clinically established assessment methods (i.e. 6MWT, 10MWT, POMA, Barthel ADL Index) and adequate statistical analyses, results of the RCT study [39] indicate that RRs may not only be used as an intelligent AMD to support users directly in functional tasks of daily living (e.g. walking, STS transfer, navigation), but also for training purposes in rehabilitation practice.
In the other interventional study [17], the similar positive gait parameters without obvious changes over the 'intervention' period may suggest that either subjects did not require much time to get used to the RR and the RR allowed already initially a very satisfactory gait performance or that the repeated use for only a six times in the restricted intervention period may not be sufficient to achieve further improvements in outcomes.

Cognitive and sensorial assistance
Studies evaluating RRs that provided navigation assistance or obstacle avoidance showed promising but not conclusive results. In outcomes less specifically tailored to the robotic functionalities (e.g. walking time, walking speed), conventional, low-tech AMDs seem to allow a superior user performance when compared to RRs [23,28]. In more specifically tailored outcomes (e.g. walking distance, path deviation, distance to obstacles), however, users seem to achieve a superior performance rather by using a RR that actively provide robotic assistance [25,27,28,38]. These findings suggest that such specific outcomes, which can often be captured by the sensing technologies already integrated on the RRs to realize the high-tech assistance, may be much more appropriate to demonstrate the added value of robotic functionalities than rather unspecific outcomes.
Full robot motion control modes of the RRs provide maximum assistance in navigation, guidance, or obstacle avoidance and may allow highest user performances [25,27,28]; however, as the RR just tracks its self-generated path (around obstacles) without considering users' input in such modes, subjects may complain about having too little control about the motion of the RR [25]. From a clinical and user perspective, the motion control of a RR should rather be based on a sophisticated HRI which sufficiently bears in mind the user's input, provides adequate assistance only when needed, and gives the user a feeling of being in control of the RR at all time.

Overall functionality
In general, results of questionnaire-based surveys on the user-perceived overall functionality of the RRs suggest that subjects had positive experiences with the RR. The comparability and a more precise classification of study results is, however, severely limited due to the large variety of questionnaires, items and rating scales used to evaluate the subjective user experience. One of the most remarkable finding here may be that the manoeuvrability of the RRs was rated by the subjects as quite high [16,25,37]. As a lot of hardware components are required to realize intelligent robotic functionalities, it seems almost inevitable that RRs are heavier and probably also bulkier than conventional walkers. The high manoeuvrability re-ported for the RRs, however, highlights that there are already engineering approaches available that successfully address this issue in a user-satisfying manner.
In the study evaluating the user perception before and after the use of the RR [23], the positive results already obtained at pre-test without significant changes after the actual use of the RR indicated that subjects seemed to have initially no negative prejudices against the RR.
Referring to descriptive data, the authors of this study also stated that the RR was slightly more positively rated after having used it for a few times (post-test); however, they could not confirm this trend as statistically significant. Since the user satisfaction of an AMD was reported to be related to the number of times it was used [44], giving the subjects the opportunity to use the RR more frequently or over a longer period of time may have further increased the positive impact on the user perception.

Conclusions
Overall, this systematic review has revealed that the evaluation of RRs from the user perspective is still understudied. So far, very limited data on the evidence for the effectiveness of RRs in improving users' mobility and functional performance or in reducing their physical demands as well as for the positive user perception of RRs are available. Only tentative conclusions can be drawn from the identified studies, which show large heterogeneity and mostly lack sufficient methodological quality. Intelligent functionalities of the RRs may have the potential to be beneficial for users, and RRs seemed to be generally perceived as positive; however, more well-designed, high-quality studies with adequate study populations, larger sample sizes, appropriate assessment strategies with outcomes specifically tailored to the robotic functionalities, and a statistical analysis of results are required to evaluate RRs from the user perspective at a higher level of evidence.

Disclosure of interest
The authors report no conflicts of interest.   Obstacle course: walking time, number of collisions/reorientations Self-designed questionnaire: appearance, ease of use, usefulness, embarrassment (1 = best score; 5 = worst score) Walking time: AMD < own/without AMD < Guido: n.s. differences Number of collisions: Guido < own/without AMD < conventional AMD: n.s. differences Number of reorientations: AMD < own/without AMD < Guido: n.s. differences Appearance: n/a Ease of use, usefulness, embarrassment: post-test < pre-test score: n.s. differences Mean/SD of RR velocity: n.s. differences between motion controllers; all subjects achieved a very good speed control to the targeted speed of 1 m/s with both motion controllers Pushing force: recent > previous motion control * COT: conventional rollator > without AMD *, previous/recent motion control > without AMD or conventional rollator *, n.s. differences between both motion controllers VO2: RR > conventional rollator > no assistive device # Comfort, intuition, speed control, exertion, overall experience: n.s. differences between both motion controllers; similar positive user experience for both motion controllers (for all items: score ≥4)