Enhanced teleoperation performance using hybrid control and virtual fixture

ABSTRACT To develop secure, natural and effective teleoperation, the perception of the slave plays a key role for the interaction of a human operator with the environment. By sensing slave information, the human operator can choose the correct operation in a process during the human–robot interaction. This paper develops an integrated scheme based on a hybrid control and virtual fixture approach for the telerobot. The human operator can sense the slave interaction condition and adjust the master device via the surface electromyographic signal. This hybrid control method integrates the proportional-derivative control and the variable stiffness control, and involves the muscle activation at the same time. It is proposed to quantitatively analyse the human operator's control demand to enhance the control performance of the teleoperation system. In addition, due to unskilful operation and muscle physiological tremor of the human operator, a virtual fixture method is developed to ensure accuracy of operation and to reduce the operation pressure on the human operator. Experimental results demonstrated the effectiveness of the proposed method for the teleoperated robot.


Introduction
Over the last 30 years, the research has focused on the replacement of humans by robots in unknown or dangerous environments.Robots have been widely used in several areas, such as industrial applications, for deep-sea exploration, and for medical service (Bolopion et al., 2013;Chan et al., 2014;Perera et al. , 2014;Zhang J. et al. , 2017).Since then, the telerobot developed into a research hot-spot in the field of robotics.
A typical teleoperated system includes four parts, i.e., the human operator and the master device, the communication channel, a slave device, and its surrounding environment (Li et al. , 2015;Lu et al. , 2017;Yang C. and Wang, X. et al. , 2017).To guarantee the stability and transparency of the teleoperated system, numerous related studies have been developed (Jiang et al., 2016;Yang et al. , 2016).Previously (Li et al. , 2014;Shi et al. , 2002), the authors assumed that master and slave device were linear models and consequently proposed adaptive control technology to control a teleoperated system with uncertainties of different cases.Yang et al. (Yang C. and Wang, X. et al. , 2017) proposed radial basis function neural networks (RBFNNs) with wave variable method to deal with influences of system delay and uncertainties.In (Zhai D. and Xia Y. , 2017), Zhai et al. developed a finite time method to deal with problems such as model uncertainties, actuator, and time-varying delay for a nonlinear teleoperated system.Li et al. (Li et al. , 2016) combined neural network-based control with the parameter adaptive method to handle both kinematic and dynamic uncertainties.Javier et al. developed a haptic assistance method to enhance the tracking performance and human machine interaction of a teleoperation system (Corredor et al., 2017).In (Farooq et al., 2016), the authors presented a state convergence-based control method with a Takagi-Sugeno (TS) fuzzy model for nonlinear teleoperation system.Havoutis et al. (Havoutis et al., 2017) developed an integrated method involving optimal control and online learning to accomplish a manipulation task for underwater remotely operated vehicles during supervisory teleoperation.Additionally, Daniel et al. proposed a user-controlled variable impedance method with implicit haptic feedback for unstructured environments (Walker et al. , 2010).Panagiotis et al. used an EMG signal with a switching regime model as control interface for real-time operation (Artemiadis et al., 2011).In (Ajoudani et al., 2011;Ajoudani A. et al., 2011), the authors introduced a tele-impedance method based on a surface electromyography (sEMG) signal with good manipulation performance for human-robot interaction (HRI).
It is difficult to provide sufficient real time perception for a teleoperation system.Moreover, due to the unskilful operation and muscle physiological tremor of the human operator, the natural performance cannot guarantee secure operation (Li et al. , 2017;Liu et al. , 2014Liu et al. , , 2015;;Zhao S. et al. , 2017).Thus, it is important to enhance the interaction capability of the teleoperation system.Virtual fixture is an alternative method to improve teleoperation performance.Virtual fixture was first proposed to extract relevant information between the human operator and the remote environment for HRI (Rosenberg, L. B , 1993).In (Fehlberg et al., 2014), a virtual fixture control strategy was presented to improve the manipulation performance of the active handrest.Brian et al. developed a derivation of virtual fixtures based on the motion of the instrument in real time for system control (Becker et al., 2013).In (Hong et al., 2016), the authors proposed a forbidden region virtual fixture with robust fuzzy logic controller to improve the human manipulation performance during laparoscopic surgery.A virtual fixture method based on the position error was presented to add an augmentation force on the master device to improve the task quality (Maddahi et al. , 2015).In (Selvaggio et al. , 2016), Selvaggio et al. proposed an online virtual fixture and task switching mechanism that utilizes a stereo camera system to provide position information, thus improving teleoperation performance.In (Quintero et al. , 2017), a flexible virtual fixture method with force-vision-based scheme was developed to reduce cognitive load and improve the task performance.
This paper proposes a combined scheme of hybrid control and virtual fixture to improve teleoperation performance.In the proposed hybrid method, muscle activation is introduced to indicate the variable stiffness for the slave in the process of HRI.Combination of the proposed method with the variable stiffness and proportionalderivative (PD) control method can provide a natural and secure interaction for the teleoperation system.Moreover, a virtual fixture method is presented to alleviate the detrimental influences of unskilful operation by the human operator and reduce operation pressure on the human operator.Finally, experimental results have demonstrated the performance of the proposed method.
The remainder of this paper is structured as follows.In Section 2, the proposed control strategy is developed to improve the system's HRI capability and to enhance the manipulation performance.Experimental results are presented in Section 3 to verify the effectiveness of the presented hybrid control and virtual fixture approach.Section 4 presents the conclusion and future directions.

System description
The teleoperated robot is a novel implement that provides an interaction mode between the human operator and the telerobot, thus enhancing human perception and motion and integrating the human intelligence with the advantages of the robot under long distance constraints.
The block diagram shown in Fig. 1 explains the teleoperation system.The teleoperated robotic system utilizes a master-slave structure and the slave device follows the master motion, operated by the human operator.In this paper, a novel algorithm is developed that involves muscle activation and stiffness control of the human operator with virtual fixture to obtain satisfying performance.A schematic diagram and overview of the proposed method with virtual fixture and hybrid control are presented in Fig. 2 and Algorithm 1 .The proposed method is composed of a master side module, slave side module, haptic interface module, and sEMG signal processing module.The master side involves virtual fixture based on position error.PD control and variable stiffness method enables the slave side to provide a force feedback to the haptic device through the haptic interface.The control stiffness with regard to muscle activation can be changed by varying the generated force and haptic force reflection.

Teleoperation system
The teleoperation system employs a master-slave frame to accomplish the required manipulation task.In general, both master and slave can be extended to a multifreedom robotic system.The dynamics of master and slave are presented as where H s (q s , qs ) = C s (q s , qs ) qs + G s (q s ) ( where i = m, s indicate the master device and the slave device, respectively.M i (q i ) represents the inertia matrix for the master and the slave.C i (q i , qi ) represents the Coriolis and Centrifugal force matrix.G i represents the gravitational force matrix.q i represents the joint variables.τ i represents the control inputs.J T i represents the transpose of the Jacobian matrix.H i (q i , qi ) represent the nonlinear coupling terms for the centripetal force, Coriolis force and gravity.f m (t) and f s (t) represent the disturbances (Coulomb friction and time-delayed jamming) of master device and slave device, respectively.F h (t) represents the human operator applied force to the robot.F e (t) represents the interaction force between the slave and the environment.

Control method
The control scheme is shown in Fig. 3.A PD controller and a hybrid method of position controller and stiffness controller are proposed for the master device and the slave device, respectively.As shown, the overall control scheme consists of a PD control module and a variable stiffness control module in Cartesian space.

PD control of the master
A PD controller is employed for the master and can be defined as where K m and D m are positive parameters of the PD controller for the master.x me = x m − x s is the deviation between desired trajectory x m and the actual trajectory x s .
2.3.2.Hybrid control of the slave 2.3.2.1.PD Control.As shown in Fig. 3, the PD controller can be represented as where K s and D s are positive parameters of the PD controller for the slave device.x e = x sd − x s indicates the deviation between desired trajectory x sd and actual trajectory x s .
2.3.2.2.Variable stiffness control.The human operator can adjust the muscle activation according to the external force applied to the slave device.The trend of muscle activation change is based on the deviation from the force that is exerted by the human operator and the feedback force of the slave manipulator1 .
In this study, the collected sEMG signal u emg can be obtained as where u raw are the raw sEMG signals and i = 1, 2, ..., N are the sEMG signal detection channels.
Using a moving average filter yields where W f represents the size of the moving window.Based on Eqs. ( 11) and ( 12), the relationship between the raw sEMG signals u raw and the muscle activation a(k) can be presented as (Yang C. et al. , 2017;Yang C. and Luo, J. et al. , 2017).
where α represents the muscle activation.u(k) represents the processed sEMG signal.−3 < β < 0 is a parameter involved in the sEMG signal.Through sEMG signal processing, a linear function that describes stiffness can be represented as where K α max represents the maximum of K α , and K α min represents the minimum of K α , i.e.K α min ≤ K α ≤ K α max .α k min and α k max are the upper and lower bound of the muscle activation, respectively 2 .The variable stiffness parameter K α describes the generation of the optimal stiffness from human operator for slave manipulation.The values of K α max and K α min are devised to rely on experimental experience generated in advance.
According to Eqs. ( 11)-( 14), the variable stiffness control can be defined as where K α > 0 is the variable stiffness that indicates strength of muscle activation.
In general, the muscle activation of the human operator varies with the manipulation and the external environment during teleoperation.By sensing the information of the remote environment, the human operator can initiate a correct operation/demand 2 Obtaining the muscle activation is processed by series of treatment steps.The processing procedure includes the following sections: Rectification⇒Squaring⇒Moving average⇒Low pass filter⇒Envelope.The parameters of the muscle activation α k min and α k max are determined by a previously conducted pilot experiment.
in HRI.Considering the linear relationship between muscle activation and sEMG signals, the proposed variable stiffness control F α is used to describe the human control operation/demand during teleoperation.By changing the stiffness value K α , the human operator can initiate the correct control of the slave device.Moreover, the control intention of the human operator during teleoperation can be quantitatively analysed via the proposed stiffness control method.
2.3.2.3.Hybrid control.During the process of HRI, the control force F r involves the muscle activation F a and the generated force F pd , as shown in Fig. 3.This can be obtained by Eqs. ( 10), ( 15) and ( 14) as follows: where F r represents the force integrated variable stiffness with PD control based on the Cartesian space.In the variable stiffness control method, variable parameter K α is introduced that reflects the muscle activation necessary to acquire of optimal force in the process of HRI.The proposed controller synthesises the virtue of both feedback and the human operator's factor.This schedule achieved a good realization of the incorporation between human intention and the dynamics of the robots.The control law of the slave device can be represented as The control law (17) achieves the hybrid control with regard to position and stiffness in Cartesian space.

Virtual fixture
When the slave follows the master, the position of end effector of the slave can be defined as where x s , y s , z s represent the positions in the XY Z coordinate system.The desired trajectory of the slave is The joint variables (q s 1 , q s 2 , • • • , q s n ) of the slave can be obtained by using inverse kinematics.
The position and desired position of the master are presented as where the P m and P md represent the actual and desired position of the master, respectively.Thus, the position error of the slave end effector as For the master device, where P e m is the position error of the master.The generated force is proportional to the position error of the haptic control, which is presented as where K vf represents the matrix of the virtual fixture which indicates the guiding ability of the virtual fixture.
As shown in Fig. virtual-fixture, when P e m = 0, F vf is either a positive/negative force and leads the master moves as expected.

Convergence of tracking error
The generalized tracking error of the slave can be defined as e vse = K se e s + ės (24) where e s = q s − q sd , K se = (K s + K α )D −1 s .The control input of the slave can defined as Based on ( 25), ( 7) can be represented as where q sv = qsd − K se e s .Combining ( 25) and ( 26), According to (Yang C. and Wang, X. et al. , 2017), the uncertain dynamics of the slave device with the input z s can be represented as In this paper, the dynamics of the slave device are assumed to be available for trajectory tracking.Thus, ( 27) can be rewritten as: This results in: Therefore, V is negative definite.When t −→ ∞, e vse ∈ L 2 ∩ L ∞ , and ėvse ∈ L ∞ , e vse can be asymptotically converged to 0 (Zhou Q. et al. , 1993).

Stability analysis
Proof.Consider a Lyapunov function as below where e m = q m − q md .According to Eq. ( 33), we have It can be assumed that the human operator and the external environment are passive (Yang C. and Wang, X. et al. , 2017).Then, we obtain where V mh (t) and V se (t) are bounded.
According to Eqs. ( 33) and (37), we obtain which can guarantee the boundedness of V 1 .
The proof is completed.

Experimental Setup
To demonstrate the performance of the proposed integrated algorithm, an experimental platform was built as shown in Fig. 5.

Experimental evaluation
To evaluate the effectiveness of the proposed approach, the root mean squared error (RMSE) was used which is defined as follows: where y(i) represents the master trajectory and ŷ(i) represents the slave trajectory.Table 1 shows that the tracking errors of the hybrid control are 0.0108, 0.0081, and 0.0160 in X, Y, and Z directions, respectively.However, the tracking errors of  the PD control are 0.0176, 0.0093, and 0.0173 in X, Y, and Z directions, respectively.The values of RMSE are smaller in the integrated control mode compared to the PD control mode.Consequently, the tracking performance of the proposed hybrid control method is superior to that of the PD control method.

Virtual fixture experiment
Based on the tracking experiment, a typical trajectory of the slave device end-effector in the Cartesian space is presented to verify the performance of the proposed method using virtual fixture.In this experiment, the trajectory error had to be smaller com-pared to that of the tracking experiment.
Figs. 14-18 show the trajectory performance in the operation space.The solid red lines indicate the tracking performance of the master device.The dashed black lines are the trajectories of the slave device.As shown in Figs. ( 14)-( 15), the slave can precisely follow the master all the time.As shown in Figs. ( 16)-( 17), the slave can also almost track the master by using virtual fixture in comparison to Figs. ( 14)-( 15).The generated force by using virtual fixture is shown in  16), the completion time of the task in case of virtual fixture is below that for the case of without virtual fixture.The experimental results observation as shown in Table 2, indicate that the RMSE of the tracking trajectory between the cases with and without virtual fixture are 0.0011 and 0.0025, respectively.It should be noted that the RMSE in the virtual fixture experiment is smaller than the RMSE in the tracking experiment.The time spend on the trajectory task without virtual fixture is 12.33 s, while the completion time is 10.72 s in case of virtual fixture.Figs.19-20 show the trajectories of the variable stiffness gain K α and the human force.The curves of K α and human force vary with the position trajectory.The values of K α and human force are increased under both conditions of with and without virtual fixture.It can be concluded that the variable stiffness gain K α is positively correlated to the human force.

Conclusion
This paper proposes a novel scheme that combines hybrid control with virtual fixture and achieves a good manipulation performance of the telerobot.The hybrid control method integrated with both PD control and variable stiffness control and the proposed method could provide both natural and secure interaction for the human operator by adjusting their hand muscle activation.Based on the hybrid control structure, a virtual fixture method was presented to improve the manipulation performance of the human operator.The experimental results verified the performance of the proposed method.In future, more human physiological informations will be introduced, i.e. electroencephalogram (EEG) and electro-oculogram (EOG) to enhance both the perception and interaction experience of the human operator for the teleoperation.

Figure 1 .
Figure 1.Block diagram of the proposed system.

Figure 2 .
Figure 2. Schematic diagram of the proposed method with virtual fixture and hybrid control.

Figure 3 .
Figure 3. Schematic diagram of the proposed control approach.

Figure 4 .
Figure 4. Diagram of the virtual fixture.

M
s ėvse + C s e vse + D s e vse = 0 (29) and M s ėvse = −(C s + D s )e vse (Ṁs e vse − e T vse C s ėvse − e T vse D s ėvse = −e T vse D s ėvse < 0 (33) Hardware equipment.The experimental hardware equipment consists of the Touch X, the simulated Baxter robot, and the MYO armband.• Software environment.The software environment includes the MATLAB software, Visual Studio 2013 (VS 2013), and the Windows 7 operation system.• Experimental Parameters.For the experiments, a PD controller is used to control the master.An integrated controller is used to control the slave.The experimental parameters of the master are set as follows: PD controller parameters: K m = 50 and D m = 30; virtual fixture related parameter K vf = 20.The following experimental parameters of the slave are set: PD controller parameters of K s = 50 and D s = 30, the parameter of the muscle activation β = −0.6891.

Figs. 6
Figs. 6-9 show the tracking performance when the PD control mode in X/Y/Z directions is used.The solid red lines indicate the tracking performance of the master device.The dashed black lines are the trajectories of the slave device.As shown in Figs.6-8, the slave can not follow the master in 0-5 s, but it can preferably follow the master after 5 s in the X/Z directions except for the Y direction.The result of tracking error in PD control is shown in Fig. 9.

Figure 9 .
Figure 9. Tracking error with PD control.

.
The results of tracking experiment in hybrid control mode are shown in Figs.10-13.The solid red lines indicate the tracking performance of the master device.The dashed black lines show the trajectories of the slave device.In Figs.10-12, the trajectories of the slave do not match the tracking the master in 0-3 s due to different initial positions in the task space.However, the slave almost exactly tracked the trajectories of the master during the last 3-7 s.The tracking error of the slave is depicted in Fig.13.Compared to the PD control, the error of the tracking trajectory is smaller in case of the hybrid control mode, and the hybrid control mode achieves better performance in trajectory tracking experiments.

Figure 13 .
Figure 13.Tracking error with hybrid control.

Figure 18 .
Figure 18.Generated force by using virtual fixture.

Figure 19 .
Figure 19.Performance of variable stiffness gain and human force without virtual fixture.

Figure 20 .
Figure 20.Performance of variable stiffness gain and human force by using virtual fixture.

Table 1 .
Comparisons of tracking performance between PD control and hybrid control: root mean square error (RMSE).

Table 2 .
Performance comparisons between with and without virtual fixture control: root mean square error and completion time of the task.