Upper limb motion intent recognition using tactile sensing

Focusing on upper limb rehabilitation of weak stroke patients, this pilot study explores how motion intent can be detected using force sensitive resistors (FSR). This is part of a bigger project which will see the actuation and control of an intent-driven exoskeleton. The limited time stroke survivors have with their therapists means that they can not often get enough training. During active-assisted training, therapists guide the paralysed limb through a movement only after detecting visual or haptic cues of the motion intent from the patient. Aiming to replicate therapist practices of recognising patients' intention to move, a pilot study of a tactile system is performed. The system will perform consistently even with patients who have low muscle strength and control ability. Currently available devices for detecting muscle activity do not offer the robustness and performance necessary; Electromyography (EMG) sensors, a well-established method, is affected by factors like skin moisture and BCI (Brain Computer Interface) has a slow response time. The proposed tactile sensing system is a simple yet robust solution both from a sensing as well as a usability point of view. Pilot experiments have been performed with a healthy subject emulating low muscle activation conditions. An overall accuracy of 80.45% is achieved when detecting forearm and arm muscle contractions and hence motion intent.


Upper Limb Motion Intent Recognition Using Tactile Sensing
In Stroke Patients Stefanou T. 1 Turton A. 2 Lenz A. 3 and Dogramadzi S. 4   Abstract-Focusing on upper limb rehabilitation of weak stroke patients, this pilot study explores how motion intent can be detected using force sensitive resistors (FSR).This is part of a bigger project which will see the actuation and control of an intent-driven exoskeleton.The limited time stroke survivors have with their therapists means that they can not often get enough training.During active-assisted training, therapists guide the paralysed limb through a movement only after detecting visual or haptic cues of the motion intent from the patient.Aiming to replicate therapist practices of recognising patients' intention to move, a pilot study of a tactile system is performed.The system will perform consistently even with patients who have low muscle strength and control ability.Currently available devices for detecting muscle activity do not offer the robustness and performance necessary; Electromyography (EMG) sensors, a wellestablished method, is affected by factors like skin moisture and BCI (Brain Computer Interface) has a slow response time.The proposed tactile sensing system is a simple yet robust solution both from a sensing as well as a usability point of view.Pilot experiments have been performed with a healthy subject emulating low muscle activation conditions.An overall accuracy of 80.45% is achieved when detecting forearm and arm muscle contractions and hence motion intent.

I. INTRODUCTION A. Background
Rehabilitation helps stroke survivors to learn new ways of movement with the potential to help them regain use of the affected limbs.Recovery is influenced by quantity of training and the specific tasks practised [22].Physicians' guidelines suggest a combination of repetitive task training (RTT), including constraint-induced movement (CIMT), depending on the individual's ability [29].During active-assisted exercises the therapists wait for the initiation of the movement by the patient; the cues are either visual or haptic.They then support and guide the limb through the completion of the exercise.The visual feedback, of the movement of the limb that occurs following muscle contraction, and proprioception of the patient improves the rebuilding of neural pathways, replacing the damaged ones [1].Discussions with therapists in the field indicated that where there is a lack of mobility they feel the soft tissue in the proximity of the actuation Given the actual ratio of occupational therapists to patients in the NHS (1.1:10), compared to the ideal (3.3:10) [20], the therapy time for each patient is about three times lower than that required.Robotic devices have great potential in assisting therapists with rehabilitation [19] and have already been implemented in the medical sector [29].The aim of the system proposed is to mimic the recognition of movement intention in this therapist-patient interaction.Recognising intention, by using the same cues as therapists do, a safe input for an upper limb rehabilitation exoskeleton can be used to guide the limb through a motion.
Within the last two decades, multiple upper-limb rehabilitative devices/exoskeletons have been developed.They can support all upper limb degrees of freedom of the shoulder, elbow, forearm and wrist through the full range of motion [24] [2].Commercially available rehabilitation devices are being used in conjunction with traditional therapy [32].Intention-driven ones [27] usually lack robustness [9] as they are unable to detect intent under different conditions.
A variety of sensors have been used in movement intent recognition systems; these include sEMG, joint torque sensors and EEG (Electroencephalography) sensors, recording brain activity for BCI (Brain Computer Interface) systems.EMG signals sense motor-neuron train spikes.They have been widely implemented in assisted living, rehabilitation and prosthetic systems; nonetheless, EMG controlled systems have still not reached acceptable performance consistency [9].There are limitation on the conditions under which they are being used [7].For example, it has been shown that adipose tissue (fat) can affect amplitude and create crosstalk between signal recordings [17], while skin moisture also affects signal acquisition [28].
Kiguchi et al [14] have used EMG signals to detect the intent of motion in the upper limbs.Together with Gopura they developed EMG based control methods for a 7DOF upper-limb exoskeleton [10] [15].EMG sensing here was complemented by torque sensors on the exoskeleton, creating a more robust system.The limitations arise from the fact that the system relies on the user's ability to produce detectable joint torques.
Steering away from EMG triggered controllers, developments by Barsotti et al [3] included a Motor Imagery (MI) based BCI to control the BRAVO (Brain computer interfaces for Robotic enhanced Action in Visuo-motOr tasks) exoskeleton [4].Tests with stroke patients demonstrated a classification accuracy of 82.5% during reaching-grasping exercises.The limitations in this approach are seen in the response time of the system; it takes about (3.45 ± 1.60)s to initiate a movement.This could cause issues if the patient needs to urgently terminate the movement as even greater delays would occur taking into account the actuation stopping time.
In prosthetics, force Myography (FMG), also referred to as tactile imaging, has shown promising results when used for intent recognition.The idea that the volumetric and shape changes that take place within the muscle can be monitored on the skin surface and used as an indication of motion intent was first captured by Moromugi et al [21].They implemented push buttons, with load sensors, indented in the skin to capture "muscle stiffness" for the purpose of actuating a prosthetic hand.Wininger et al [33] performed one of the first studies implementing FSR sensing to predict grip force in hand prostheses.Using a grip dynamometer, Wininger mapped the readings measured during gripping and the pressure exerted by the forearm on the FMG cuff.Testing the concept on healthy young adults, they concluded that FMG is a useful alternative to EMG.Furthermore, a high resolution tactile sensor system developed by Schürmann was used in a proof of concept study carried out by Castellini et al to create tactile images of the anterior forearm [5].This was followed by a feasibility study that indicated tactile sensing offers more stability than sEMG [26].The same sensory technology was embedded and tested in a tactile sensor bracelet [16].Lastly, a feasibility study performed by Cho indicated that FMG resulted in classification accuracies of over 70% in all grip classifications attempted [6].In this paper we investigate how tactile sensing can be used to detect motion intent in stroke patients with upper limb paralysis in order to actuate an exoskeleton that will help them perform their rehabilitative exercises.The challenges lie in the stroke patient's muscular strength that needs to be detected by the system, which could be as low as 5% [8] of their nominal strength.In this paper we describe the methodology and set up for the three sets of experiments performed using this tactile motion intent recognition system.The experimental section that follows presents the validation of the sensory system and the testing and evaluation of the concept using hand grasping and elbow flexion.

II. METHODOLOGY
The FMG approach has been adopted in this study aiming to find a robust solution that offers a fast, consis-tent and accurate motion intent recognition system for stroke patients.The aim is to mimic the way therapists use their sense of touch to detect the intention of motion.A series of experiments have been designed to emulate the low muscle activation, as would be the case with weak stroke patients.Three different experimental stages were completed A) Proof-of-concept experiment to validate the sensitivity of the sensors B) Testing of the sensory system to detect motion intent during griping C) Testing of the sensory system to detect motion intent during griping Indications as to whether a particular movement is being attempted is acquired by monitoring the activity in the identified muscle areas.The sensors are placed on an arm brace designed to tightly fit on the arm.Muscle contraction or relaxation will alter the shape of the proximal tissue area, causing contact force changes between the arm or forearm and the brace.
Gripping-induced muscle contractions was the first concept we tested.The muscles providing the main forces during gripping are the three extrinsic ones [18] located in the forearm; flexor digitorum superficialis (FDS), flexor digitorum profundus (FDP) and flexor pollicis longus (FPL), Fig. 1.
Fig. 1: The flexor digitorum profundus (FDP), flexor digitorum superficialis (FDS) and flexor pollicis longus muscles (FPL) -in all three diagrams the volar compartment of the forearm is shown [34].Indicated are also the approximate sensor posiions when in contact with the forearm.
These are, therefore, the muscles targeted.The sensors will be located in the upper central half on the anterior and posterior parts of the forearm for the gripping motion; in proximity to the muscle belly, the thicker part of the muscle, as seen in Fig. 1, where all muscle fibres come together.

A. Stage1
For the initial stage of experiments we determined that the sensor sensitivity is good enough to detect contact force changes in the forearm during low grip strength motions; as would be expected in a weak stroke patient.To perform these experiments, a measurement device was used to measure the grip forces exerted.
1) Grip Holder: An analog grip strength meter, a Saehan hydraulic hand dynamometer [25], was utilised to monitor the grip strength used by the subject.Such devices are mainly used to evaluate grip strength after hand surgery or during the rehabilitation program.Fig. 2: Sehan hydraulic hand dynamometer used to measure grip strength [25].
The minimum strength requirement the sensors needed to detect was evaluated.Stroke survivors are at their weakest right after the stroke incident That is when their affected limb has on average 18% of the unaffected side's nominal grip strength [30]; this can be as low as 5-10%.According to a study performed in UK [8], the peak optimum median grip is 51kg and 31kg for male and female respectively.Thus, 5-10% of the latter calculates to 1.55-3.10kg(15.21-30.41N).Seeing as the resolution of the device is only 2kg (19.62N), the indicator was kept just below that.Therefore, the goal was to detect the contact force changes that take place between the forearm and the sensors, during gripping motions with what is determined to be as less than 6.45% of the average nominal strength of females.
2) Sensing: The next step was to choose the sensors.Piezoresistive tactile sensors were preferred over capacitive sensors as the latter are very susceptible to noise [31].The Interlink Electronics flexible FSRs were chosen as they provide the largest active surface area for the cheapest price [11].
The FSRs have a conductor substrate with a printed interdigitated circuit pattern and another one coated with carbon-based ink; when a force is applied, the conductive substrate deforms and contact is made with the printed circuit lines varying the resistance and proportionally the measure voltage.FSRs require a simple interface and their size (thickness of 0.46mm and 18.3mm diameter) [12] and allow for an easy integration.These sensors have a minimum force detection of 1N and a maximum repeatability error of ±2%.The sensor sensitivity has an exponential behaviour, Fig. 3. Experiments determined its sensitivity by recording of the readings as 100g weights (adding up to 1kg) were added on the sensor's active area.The sensors were centrally placed on the anterior and posteriorforearm,in proximity of the gripping muscles, Fig. 1.Two sensors were used; sensor1 was placed at the top, dorsal part of the forearm (palm facing upwards) and sensor2 bottom of the inner lining of the support (volar forearm), Fig. 4.
3) Forearm Support: Finally a forearm brace was built.To ensure a good contact of the forearm with the sensors a an adjustable interface that allows a tight fit was created.Avoiding obstruction of movement and minimising the weight and cost were also important factors.
Although a cheap adjustable strap would have offered good comfort levels and a tight fit, the sensors would not function as well if mounted on a flexible surface [12].The forearm brace was designed to have adjustable height and separately adjustable width, to accommodate different arm sizes.
4) Integration: The sensors were fixed on the brace and their circuit outputs were fed into the Arduino analog inputs which provide 10bits of resolution.The data broadcast on the serial port was recorded using a MAT-LAB script.Their acquisition frequency was 104Hz.The frequency typically used in body movement monitoring or human movement classification implementations [13] is 100Hz.

B. Stage2
Having shown that the contact force changes between the arm and the brace can be detected when low grip forces, up to 6.45% of the nominal, were used, the grip meter was replaced with a microswitch [23].This provided a ground truth on the gripping state of the hand.The switch closes when the two gripper levers come together during gripping.A torsion spring integrated in the system pushes the gripper open.Hence its ON state indicates contraction of the muscles, and its OFF state indicates relaxation.It was attempted to keep the grip force used as low as possible, just about exceeding the resistive torque of the spring, 2.10 * 10 −5 Nm, with stiffness 0.6582Nmm/rad and the switch maximum operating force, 0.25N.For the elbow flexion, monitoring of the   biceps/triceps muscles will be required; in which case, the brace will be worn on the arm.Interface of both sensors and the microswitch was done using using the Arduino UNO board and the data transmitted through a serial connection to MATLAB.
Interface of both the sensors and the microswitch was done using the Arduino UNO board and the data was transmitted through a serial connection to MATLAB.

C. Stage3
Finally, the concept was tested on biceps/triceps activity detection.The gripper with the switch was attached on the underside of a table with the arm supported just underneath.As the elbow flexes and the forearm is raised the switch closes, Fig. 7.The two gripper handles were constrained at about 10mm apart, just enough to keep the switch open.To close the switch an average weight person would have to surpass the 0.113Nm torque needed to raise their 1.149kg forearm, as well as the 0.25N required to close the switch.
The sensors were placed in the proximity of the biceps (sensor1) and triceps muscles (sensor2).The forearm was resting on an arm-rest at about a 90 • angle, just below the bottom of the gripper.The wrist joint was kept fixed at all times to ensure that the biceps was the one working to close the switch and not the flexor carpi radialis/ulnaris (wrist flexing muscles);

III. EXPERIMENTS A. Proof of Concept
In this first stage of experiments, tests were performed with the forearm muscles relaxed or contracted (using a grip strength of approximately 2kg, see II-A), at certain pre-determined in time.The forearm, which was placed inside the brace, rested on the table with the palm facing upwards, supination position.A total of 10 initial experiments were run, that lasted 20-25s each, where 2-3 gripping motions were performed.There were slight variations in the contact points of the sensors (brace) with the forearm across experiments as the arm-brace was removed at the end of each one.The arm and shoulder were kept relaxed.
The forearm had good contact with sensor2 since its weight was resting on it.As it can be observed in Fig. 8, there are distinct voltage variations when gripping is performed.Visual inspection of the complete signal indicates certain emerging patterns.As the hand grips the dynamometer, the flexor digitorum contracts while the extensor digitorum (volar forearm compartment) relaxes completely; this causes a decline in the contact forces between the volar part of the forearm and sensor2.The first step change of the recorded voltage indicates that a gripping motion is taking place and one in the opposite direction signals the relaxation of the muscles.The forearm placement, its shape and weight and exerted force, affect the contact forces between the sensors and the forearm.With the chosen forearm orientation, contact with sensor1 was inconsistent; hence the focus was on sensor1 readings.The results in this first set of experiments indicated that it may be possible to detect small muscle contractions and hence recognise motion intent.They also suggest that the sensitivity of the sensor is good enough to detect the contact force changes that take place between the forearm and the brace when a grip strength of 2kg(19.6N) is used; across all stage1 experiments the contact force changes gave rise to a detectable potential difference changes.As observed, a grip strength of 19.6N (2kg) in this experiment produced a potential difference change on the FSR sensors from 2.565V to 1.285N was recorded, which corresponds to contact force change of 5.30N, Fig. 3.

B. Intent Recognition: Forearm
With the conclusion of these first experiments, it was clear that the sensor system was sensitive enough to detect contact force changes between the arm and its brace when grip strength was limited to 19.6N.During the second stage experiments the switch state was also recorded alongside the sensors.This allowed us to perform state prediction and determine the accuracy of the results.In total over twenty 2-3min experiments were performed with an arbitrary number of gripping movements (>2).
No discernible contact force changes were detected on sensor1, Fig. 10.This was expected, as its contact with the forearm was weak; therefore, the sensor2 readings were the ones analysed, Fig 9 .As it can be observed in Fig. 10, where the force derivatives are presented, large gradient changes hint towards a grip state change.Negative gradients indicate muscle contraction of the gripping muscles, which causes the volar part of the forearm and the sensor to loose contact.Following that, increasing contact forces hint towards return to the initial state.Drifting can be observed, Fig. 9, which can be attributed to the increased blood flow in the forearm as well as the slight shifting of contact points due to the solid nature of the brace.
An algorithm was developed to classify the state of the forearm, Algorithm.1, whether it is 'at rest'(muscle relaxation) or under tension (muscle contraction).The binary classification algorithm developed determines the state based on the emerging signal features, such as large gradient step changes and their direction with respect to the baseline as well as previous tendencies.The accuracy is determined by comparing the algorithm output to the microswitch state recorded.The algorithm focuses on the patterns that emerge with every new, incoming data set;  aiming for a state change detection within 1s.The initial state is always assumed to be 'at rest'and a baseline value is thus determined.Calibration was performed during the preprocessing of the data; this included the use of a moving average filter to reduce noise which also introduced a time delay of 0.12s.The algorithm's classification accuracy, averaged over all Stage2 experiments, was found to be 77%, with a standard deviation of 11%.This was calculated by comparing the state of the switch that represents the actual state with the predicted state output.Drifting observed in the FSR sensor readings did not affect its performance.It was important to avoid false positives, false prediction of rest-to-tension state change.As long as it is not a consistent occurrence, non detection is not as problematic.The gripper had to move quasi-statically which increased the delay between muscle contraction and movement reaction detection.This was evident in certain cases where the detection of tension happened prior to the switch indication.This can be seen in Fig. 11a where the first tension period is detected 0.10s beOct2016 fore there is any indication from the switch state.The maximum delay that is estimated to be 0.455s.This could be improved by using a continuous gripping force measurement device; this would enable detection of the movement instantaneously.
When the algorithm successfully detected the state change, the maximum delay found was 0.6s.At the absence of key patterns1 from the waveform the algorithm will not make a decision until it is certain of the change.
In about a fifth of the data points the forearm intention to move was wrongly classified.This was due to the delays in detecting muscle relaxation and early detections of tension when the switch state change delays with respect to the motion onset.One cause of the false positives though that could cause issues would be the complete failure to detect the arm has gone back to rest; such as in Fig. 11b, at t=23.24s.

C. Intent Recognition: Arm
Stage3 involved testing the motion intent detection system performance during elbow flexion.As mentioned earlier, II-C, to experiment with elbow movement the arm brace was fitted on the arm.Similar contact force patterns emerged, Fig. 12, during elbow flexion, as previously in the forearm when gripping was performed.A total of 10 experiments were performed with an arbitrary number of elbow extensions (>2) per minute as each one lasted between 2 and 5 minutes.
As the biceps contract to raise the forearm, muscle flexion causes an increase in the contact forces.The arm brace was fitted tightly around the arm which caused the contact forces monitoring the biceps and the triceps vary in a similar manner.The sensor in proximity to the biceps recorded changes of slightly higher magnitudes than the triceps, by an average of 20%.
The classification algorithm parameters were tuned accordingly for use on the forearm, producing an average accuracy of 83.9%.As Fig. 13 illustrates, the adapted algorithm was successful at correct state prediction with higher confidence levels than gripping.During this set of experiments, there was a slight movement of the brace, and sometimes even a small rotation, as the biceps contracted.This affected the accuracy of the data acquisition as the point of contact of the sensor with the arm was shifting, influencing classification.Nonetheless, the state classification accuracy was improved compared to gripping which is a more complex movement involving a greater number of smaller muscles.

IV. DISCUSSION
In this pilot study on the use of tactile sensing for motion intent recognition in stroke patients the results were promising for the development of a reliable, quick detection system, as part of an actuated exoskeleton device.Evidence suggests that using two FSR sensors we can detect low activations of muscles, when using no more than 6.45% of average nominal strength input.Detection occurs within 1s in both the upper and forearm.
The accuracies of this pilot study system when classifying the state of the arm/hand averaged at 80%.

A. Limitations
There were some limitations and shortcomings in the experiments completed.Firstly, there was a visible delay between the initiation of the movement and the change in the microswitch state in some cases; this created an uncertainty in the results as it is not possible to know the exact time a state change took place.Hence, instead of a binary gripper, we intend to introduce a continuous gripping force measurement device.Furthermore, this system has not yet been tested on stroke survivors.Nonetheless, an attempt was made at setting some force limitations during experimentation based on the minimal strength data for stroke patients.Whether these could be sensed as well with a thicker forearm and weaker muscles is yet to be seen.Additionally, the slight movement of the arm brace, during the Stage3 arm experiments, affects the trustworthiness of the results.While keeping the sensors attached on solid surfaces they will be held together on the arm using a tight fitted arm band.With a hard-coded feature algorithm there are limitations when it comes to generalisation.Alternatively, a supervised learning, classification algorithm will be trained on the features using the experimental data.

V. CONCLUSIONS AND FUTURE WORK
This paper demonstrates a new approach to motion intent recognition in stroke patients.The results in this pilot study were promising despite aforementioned limitations, which will be taken into account for the improvement of the system.The addition of more sensors will be c onsidered, as well as the integration with EMG, as a new lighter, easy-to-wear brace is built.Future work will include the actuation of an upper limb brace and its use to support the elbow flexion/extension motion using a tactile sensing driven controller.

Fig. 9 :
Fig. 9: Plot of sensor2 forces and microswitch data indicating the state of the arm.

Fig. 10 :
Fig. 10: Force derivatives of sensor1 and sensor2.The former barely records any changes.

Fig. 11 :
Fig.11: Examples of predicted state, during gripping, when the algorithm is run against sensor readings along with the actual state.

Fig. 12 :
Fig. 12: Sensor1 and sensor2 recorded contact forces in the arm during elbow flexion.
During most experiments the algorithm Algorithm overview; some of the indicative parameters that were adjusted for each muscle group are ChangeThres, ReturnThres LowerLimit and UpperLimit, and BaselnThres was able to correctly detect the state changes within 1s.