REALISTIC AND INTERACTIVE HIGH-RESOLUTION 4D ENVIRONMENTS FOR REAL-TIME SURGEON AND PATIENT INTERACTION

ABSTRACT

There are no financial and/or personal relationships that might bias the authors' work and the authors have no conflicts of interests in relation to this manuscript.No potential conflicts exist.The paper describes original research that has not been reported elsewhere, has not been simultaneously submitted elsewhere and has not been accepted for publication anywhere else.The manuscript has also been approved by each of the co-authors; and we have permission to publish.The research received ethical approval from the NHS South-West REC and the Ethics Committee at the University of the West of England, Bristol.

INTRODUCTION
In today's society the requirements for prompt medical consultation are rapidly expanding; and an increasingly serious shortage of consultants is arising in many medical areas.The dispersion of the population and its increasingly aged demographics are also important factors.
Consequently the need for efficient consultations and effective use of the available medical resources is acute.The increased demand being placed upon healthcare systems world-wide means that they are being forced to change the way they support patients.State-of-the-art medical technologies need personnel with advanced qualifications, whose availabilities are limited.As a result, specialists with the latest advanced capabilities are rare and inaccessible in general.A potential solution for this problem is to support healthcare systems through additional approaches such as remote consultancies and treatment follow-up.By enabling effective telemedicine, with remote consultations and provision of accurate clinical data for computerassisted diagnoses, it is possible to optimise the experts' time use as well as the quality and effectiveness of healthcare and medical services.
Solutions and technologies oriented to remote medical implementations have been slow in progressing due to factors such as non-standard and variable image quality, which limits the utility of telemedicine.The long-term aim is to progress from the present-day 2D audio-visual telepresence technologies (e.g.videoconferencing) to a 3D tele-immersive experience; i.e. virtual reality of sufficient realism to be useful clinically.Here, high-resolution interactive visualisations are the key to providing a realistic experience where useful clinical data can be gathered and where users are virtually co-present, allowing them to interact with each other and the data more intuitively.This can be achieved through accurate recovery and remote visualisation of the position and morphology of the head and body of both the patient and the consultant, as well as high resolution (spatial and/or temporal) recovery and analysis of clinical data.The latter would provide information of particular interest/importance for the patient concerned (e.g.analysis of 3D and 2D skin textures for identification of suspicious potentially cancerous lesions).
It is suggested here that dramatic improvements in the quality of remote visualisations will facilitate a step-change in the capabilities and application of telemedicine, thereby enabling widespread implementation of remote consultancy.The expected benefits thus attainable include facilitating:

•
Remote consultancy technologies to assist with early diagnosis, treatment planning and follow-up processes.(In the case of some diseases, such as skin cancer, early diagnosis is critical for successful treatments and favourable long-term prognoses.) • More time-efficient consultations -providing more availability for earlier and more focused patient consultations.
• Increased utilisation of consultant surgeons' time by avoiding travel; using the system in their clinic will enable a greater number of patients to be examined in a given time.
• Data capture systems will be relatively low-cost and operators will not require a high degree of clinical training, thereby facilitating widespread use for screening and detection of commonly occurring conditions (e.g.respiratory diseases) in all locations, including remote areas and developing countries.
• Enabling quantitative post-surgery physiotherapy and analysis, with faster and more frequent patient remote care, will assist with improving clinical outcomes and identification of preferred interventions.For example, in the case of scoliosis an improved outcome might be one where the respiratory function is improved and/or the Cobb angle is reduced.

•
The proposed approach will assist with training of surgeons for pre-operative preparation.
• Computer-assisted technologies and 4D video-imaging exchange and collaborative work will assist with clinical knowledge exchange, thereby helping to establish best practice.
The authors' aim has been to help realise these benefits by undertaking advanced machine vision research for enabling production of a system for real-time remote medical consultancy and diagnosis.By employing realistic interactive high-resolution 4D environments for real-time surgeon and patient interaction, with morphological anatomical analyses as well as skin morphology and 2D/3D texture analysis, the system is expected to assist clinicians by providing an advanced computer-aided detection and diagnosis functionality.This requires much more than simple telephone, video-consultancy, or conventional image and medical data transfer.Rather, techniques that are identified here as enabling the next generation of remote consultation include: • High resolution machine vision technology (spatial and temporal).
• Recovery of 4D (dynamic real-time 3D) as well as 3D and 2D patient data.
• Incorporation of modelling and simulation for analyses of patient data.
• Remote interactive on-line consultation as well as off-line patient appraisal.Regarding psychological conditions, VR offers the advantage of enabling the therapist to control the quality, intensity and the frequency of the exposure therapy in the treatment of anxiety disorders, or even to stop it if the patient cannot tolerate it.In 2013 work was reported on use of VR and mobile phones in the treatment of Generalized Anxiety Disorders (GAD) (4).Although the patient group was small at 25 and the authors say there is a need for further analysis, they concluded that VR is suitable for use in the treatment of GAD, and that mobile devices offer patients advantages in provision of the VR experience.Other psychology-related work that has been reported includes fear reduction in the case of phobias (5), and body image disturbances for patients with eating disorders (6).Application of VR to PTSD is particularly well reported, with numerous researchers working in the area (7)(8)(9).A 2011 paper reported on a small, randomized, controlled trial of VR exposure therapy versus conventional treatment, for PTSD for Active Duty military personnel (10).Although the trial was relatively uncontrolled, the authors concluded that VR is a safe and effective treatment for combat-related PTSD.

LITERATURE REVIEW
Regarding research into virtual environments for simulating surgery, a number of systems have been developed, but they are reported to have only been applied to a limited range of surgical procedures.For example, in 2015 Wu et al. described a real-time environment for surgical simulation (11) but this system has so far only been used for percutaneous coronary intervention and the benefits do not seem to have been quantified.Other work has included a study of the impact of virtual reality (VR) surgical simulation on improvement of psychomotor skills relevant to the performance of laparoscopic cholecystectomy.Here it was reported that surgeons who received VR simulator training showed significantly greater improvement in performance than those in the control group (12).Similar developments include a software platform for manipulation of 4D cardiac images (13), where real-time visualisation (including a virtual scalpel) can be used for planning and guidance of surgery.The main difference between these systems and the approach proposed here is that we address 3D skin surface/morphology analysis (on various scales) rather than modelling of specific organs such as the heart.Also, our main aim is to employ virtual reality with innovative machine vision techniques, to enable remote real-time surgeon and patient interaction, rather than just guiding surgeons in specific operative procedures.
To summarise, although the potential wide-ranging benefits that could result from application of realistic virtual reality (VR) technology to clinical applications such as surgery has been appreciated for more than ten years (14), the application of VR technology to remote clinical consultation has so far remained limited.Reasons for this may have included a lack of appreciation of the capabilities of advanced imaging technologies; and a lack of standard techniques for implementing the consultations.For example, in primary healthcare the visualisations would need to be performed in a controlled and documented fashion and the results collated, standardised and analysed in conjunction with other patient data.So far it has also not been clear that the imaging technologies available could provide information of sufficient resolution, accuracy and repeatability to be of use clinically.This situation has now potentially changed by the common availability of relatively low-cost technologies in the form of high-resolution/quality machine vision cameras, with associated vision analysis techniques and computing power.As well as providing a virtual consultation experience, this technology can be incorporated into innovative devices for generating new types of medical data that will provide enhanced clinical functionality.Before describing how we implement this in practice, it is worth reviewing the capabilities of current telehealth systems.
Existing telehealth systems often employ conventional video conferencing for consultation between patients and clinicians, in addition to transmission of vital signs data from conventional medical devices.For example, the proposed vital signs include: blood pressure (systolic and diastolic), heart rate, oxygen saturation, body temperature, spirometry (lung volumes) and blood glucose level (15).Although such an approach has the benefits of being low-cost and able to communicate some useful data, its utility is severely limited due to, for example, the limitations imposed through use of relatively low resolution visualisations and the single fixed camera/noninteractive nature of conventional video conferencing technology.Clinicians report that the usefulness of tele-health consultation systems tends to increase when factors such as the number of cameras, resolution, 3D capabilities and potential inter-activity of the systems increase.In 2005 Welch et al. reported an attempt to obtain desired camera views and depth perception by employing a small array of cameras to synthesize a spatially continuous range of dynamic threedimensional (3D) views of remote environments and events.The approach was evaluated in simulations of life-critical trauma situations but was reported to be of insufficient resolution for high quality reconstruction with an insufficient range of views to enable reconstruction over a large viewing volume (16).
Despite the above developments, research and system development in the area of 4D (moving 3D) environments for real-time surgeon and patient interaction has so far been relatively limited.
In primary healthcare, for example, advanced vision based computer assisted technologies have been under-utilised and have not yielded their potential benefits.Systems that have been implemented that are intended to provide a facility for remote consultation are generally conventional telemedicine web sites.Consultants can access remotely to view patient images captured using conventional digital cameras, or patient information such as data from X-rays, ventilators, infusion pumps and heart monitors.Teleradiology, where clinicians can remotely review data such as CT and magnetic resonance imaging (MRI) scans, is also well established.It has been estimated that in 2011 up to 20% of the total amount of imaging procedures were reported remotely (17).However, this pronounced increase in teleradiology has not been matched by systems incorporating the realistic interactive 3D vision capabilities that would enable users to benefit more fully from remote collaborations or consultations.Teleradiology applications often employ collaboration tools that include conventional 2D teleconferencing, and basic image navigation/editing.In such a system one user requests the object and applies changes on it, while all other users cannot make changes to it until the object is 'released' (18).Therefore the relatively basic nature of the data involved and the sequential rather than concurrent approach to analysis has limited the potential of this approach for remote clinical interaction.If it were possible to progress remote consultations away from simple reviews of patient literature towards real-time interactive visualisations, it would be expected that the range of applications and utility of possible recommendations would increase dramatically.
Many of the VR medical systems outlined above have necessitated construction of complex, elaborate and/or highly dynamic virtual environments.For example, effective treatment of PTSD may require construction of virtual battlefields, armoured vehicles, weapons and even aircraft.
However, in order to facilitate remote medical consultations, much of this 'computer gaming' type technology is not needed.It is true that a reasonably realistic virtual consultation room might make the VR and hence the overall experience more familiar for the patient; but even this is not absolutely necessary.However, for a virtual consultation to be effective, the mutually realistic appearance of the patient and doctor is important.There is a crucial necessity for the patient's 3D and 2D skin textures to be high-resolution and realistic; and for the face/head/body/limb morphologies to be free of distortions.It is also important for the patient/doctor visual and audio interactions to occur in real time without any delays.All of these considerations are important for generating an effective immersive experience that will give the patient confidence in the validity of the consultation and the doctor useful and reliable information on the patient's appearance.To ensure that the consultations have genuine clinical value, it is also necessary to employ medical devices to generate compatible data that can assist doctors with their diagnoses.Since the consultations will be remote, these devices will actually constitute 'virtual instruments'.In other words, although they will provide useful clinical data, they will not be physically the same as currently existing medical devices.

Implementing realistic high-resolution virtual interactions
The innovative vision systems that have recently been developed in CMV at UWE provide the high spatial and temporal resolution that is needed to implement realistic interactive visualisations.The real-time combination of geometric stereo (GS), which provides good accuracy, and photometric stereo (PS), which enables high-resolution surface topographic texture recovery, provides a powerful means of recovering realistic and precise 2D and particularly 3D data from patients' skin.As well as producing a realistic virtual presence of the patient that the clinician can interact with in a convincing and useful way (i.e. as close as possible to physically being with the patient), the intention is to use our integrated 3D and 2D vision techniques to enable interaction with the skin data on various scales, through use of virtual analysis and design tools.Such a system would offer potential to assist a clinician, such as a skin cancer surgeon, in early identification of a suspicious lesion along with virtual modelling of the expected skin appearance following lesion removal and skin reconstruction.Early identification and treatment will assist with obtaining positive prognoses.Modelling of desired outcomes will help with the planning of operations, particularly when involving virtual interactions with other relevant clinicians, as well as improving the patient's understanding of expected results.

New Virtual Instruments for Assisting with Remote Consultation
The usefulness of 3D technology on the macroscopic scale, for increasing the realism of a remote consultation where the clinician is in discussion with the patient, is generally understood.
However it is also necessary that this 3D realism be maintained during other parts of the consultation, which may be on other scales.Examples of data capture on microscopic and macroscopic scales are given by the two case studies below.In the case of medical examination of the skin for identification of potentially cancerous lesions, accurate and repeatable texture detail is required at a microscopic level.For the nature of the examination to be similar to that attainable in person, it is essential that 3D data should be provided and that the skin visualisations should be realistic and interactive; for example allowing lighting, pose and range to be readily altered.This is illustrated by the first case study below: the 'Skin Analyser', which shows how an advanced machine vision instrument can be employed virtually to generate new types of clinically useful data.

Case Study 1: The Skin Analyser
The findings of the Literature Review, as well as the factors discussed below, led to the development of the UWE Skin Analyser (SA), which is a vision based device designed for accurate and repeatable recovery of 3D and 2D textures from human skin.
The conventional approach to tele-dermatology has been to simply take photographs of a lesion with a digital camera and to email them to the specialist.However, such images often contain highlights associated with specular reflections, particularly for oily and smooth skin.Raised skin lesions can also result in shadows that are dependent on the lighting direction.Even relatively minor topographic details can modulate the brightness and apparent colour of the skin surface.
When a clinician examines a lesion in a clinical environment, either with the naked eye or with a simple magnifying lens, these aberrations can be mitigated by both dynamically and interactively varying the lighting and viewing directions.When examining photographs however, the option to make such intuitive adjustments is no longer available, and the presence of such artefacts tends to severely limit the diagnostic usefulness of stored images.The lesion may exhibit pigmentation at various places around its margin, but it is not always easy to distinguish how much of the apparent variation in colour is due to the shading caused by local curvature of the lesion surface or actual pigmentation.Therefore, the current practice of simply using a normal digital camera where images are taken at various distances from the lesion and under various lighting conditions, leads to a lack of repeatability that limits the measurement potential.In contrast to this, the location of the camera and the nature of the lighting in the SA are controlled and repeatable.Figure 1 shows the SA, where it can be seen to employ one central colour camera, surrounded by six LED lights.This, in combination with employment of the PS technique, enables us to separate 3D (surface shape and texture) from 2D (true colour data).By employing built-in redundancy from the 6 lights used in the SA, images of a given lesion can be taken at different times, which are repeatable, artefact free, and suitable for facilitating automation of ABCD skin lesion feature calculation and metrology tasks for lesion change detection.Therefore the SA is not just a camera for taking snapshots of interesting skin features.Rather, it is a calibrated instrument, able to recover 2D and 3D textures with sufficient accuracy for them to be used in realistic 3D remote interactive visualisations.This is achieved by employing the PS technique to combine the six images, for generation of the high resolution (sub-pixel) topographic bump map (3D texture) and albedo (2D colour) data sets (19).A rendered bump map and albedo for an example lesion are shown in Figure 2.
Rendered Bump map (3D data) Albedo (2D data) Figure 2.An example of bump map and albedo recovery from a lesion using the Skin Analyser.
These data sets are transmitted to the specialist who can view and analyse them on their remote computer; Figure 3 illustrates this concept.The remote user can then choose to display/analyse the information in several ways, depending on their personal requirements.Also, importantly, the instrument provides repeatable data for the accurate measurement of lesion features in both 2D and 3D (as shown in the case of the wound area measurement in Figure 4); and unlike many vision based surface recovery systems, data can be recovered at very high resolution.In fact, the resolution of the SA is limited only by the choice of camera sensor, with the current device being able to resolve features as small as 20 microns.Since the device is unique in reliably recovering 3D data from pigmented lesions in vivo, it enabled the authors to find relationships between 3D lesion textures and the presence of cancer (20).The highresolution and quality of the images have also enabled experiments to be conducted on providing an immersive experience from the PS data.We have, for example, employed data from the SA in virtual 3D visualisations, where active (synchronised spectacles) and passive (two projected images) systems have been used to provide the impression of depth in the skin visualisation.
Clinical staff employing this system have reported an impression of being immersed in the scene, which is taken as the first step towards the immersive experience of a remote clinician achieving an in-person consultation.The interactive control of synthetic viewpoints and lighting conditions in real time adds to the more immersive and intuitive viewer experience, closely replicating the natural interaction with the patient's skin.For diagnosing skin tumours, it is also helpful to be able to visualise true skin pigmentation variation in isolation, free from 3D topographic influences and surface highlights.This is also an important characteristic of the existing technique known as epiluminescence microscopy; and since the SA shares this benefit and provides additional data, it is considered to be complimentary to conventional dermoscopy.To summarise, the use of the SA provides an opportunity for introducing objective metrics and remote realistic and interactive high-resolution consultations into lesion analysis and by extension to other skin conditions, including wounds / pressure ulcers (21) and burns; thereby potentially enabling a step-change in the utility of tele-dermatology.The development of such methods for assisting with early diagnosis may be considered urgent, since The World Health Organization estimates that cancer incidence rates will almost double by 2030.

Case study 2: NORM
The second case study illustrates application of novel machine vision techniques at a more macroscopic scale; specifically for enabling non-contact measurement of tidal breathing for NHS patients.Whereas skin analysis involves examination of microscopic textures, here machine vision needs to capture and track motion details of the chest and abdomen walls, recording changes in the 3D shape and dimensional variation of the body dynamically during breathing.
Although breathing measurement represents a clinically useful technique for detecting and monitoring many medical conditions, currently employed methods (e.g.spirometry) are often not adapted to measurement of tidal breathing.They are also usually contact based, which may adversely affect the results and often requires a level of patient interaction; meaning that they are not suited to patients such as the very young or critically ill.This leads to delays in diagnoses and/or non-optimal treatments that are undesirable for the patients concerned and also a waste of money.Since conditions such as, for example, wheeze are quite common, and respiratory specialists have many demands upon their time, remote consultancy may be beneficial in this case.Rather than travelling a long distance to a specialised clinic, a patient could access this system in a primary healthcare setting, or even in their own home, with data transmission to a remote clinician.This would be expected to assist with early identification of the condition, enabling optimal dispensation of drugs or other therapy for appropriate and tailored treatment.
These considerations led to the development in CMV of an optical system that employs a technique designated as 'Novel Non-invasive Assessment of Respiratory Function' (NORM) (22).NORM also employs PS for 3D surface recovery (23), but on a larger scale, and here the recovery is continuous and in real time (i.e.4D or moving 3D).In-vivo experiments using the NORM system have demonstrated proof of principle of measurement of volume-time signals for tidal breathing of volunteers, as shown in Figure 5.  Further, the absence of any environmental structuring means that data can be acquired from patients with an unconstrained pose and position.This presents a facility for diagnosis of conditions such as Obstructive Sleep Apnoea (OSA), thereby potentially assisting with obviation of lengthy, laborious and expensive conventional polysomnography.Also, the high spatial and temporal resolution that has been found to be achievable with the NORM approach means that it offers a capability to monitor or diagnose a number of respiratory related disorders.Examples include neurological disorders, muscle motion and orthopaedic conditions.The fact that NORM can recover the 3D shape of the whole of the torso gives it the ability to detect asymmetries and localised regional movement, providing further diagnostic functionality of benefit in, for example, assessing scoliosis.The approach may also enable detection of early signs of respiratory dysfunction in high risk patients, or localised diagnoses of particular lung conditions and identification of regions of dysfunction.Data from NORM may also be useful for analysis of cardiac disorders and the monitoring of any conditions that lead to changes in torso morphology.
An example of the latter is provided by scoliosis, where NORM can address diagnostics and possible appraisal of surgical interventions (24).Scoliosis, in the more severe forms, is often associated with impaired pulmonary function, which in turn directly impacts on life quality.
Neurosurgeons, orthopaedic surgeons and respiratory clinicians have expressed interest in quick and easy non-contact measurement of respiratory function before and after corrective spinal surgery.The ability to detect and quantify outcomes has motivated the focus of NORM on scoliosis, where capture of the morphology of the back of a patient in real time can provide objective measurements of the Cobb Angle, before and after surgery (25).Vision based screening or diagnostics offers benefits over the conventional approach of employing x-rays, which include avoidance of the use of ionizing radiation, and potential for eliminating the error in measured Cobb Angle that can result from parallax effects when employing orthogonal x-rays to image a twisted spine.
The ability of NORM to work for patients while in bed and its relatively low-cost, means that it can be engineered into a discrete device mounted on the ceiling above the bed.This offers potential for it to be very widely employed in healthcare, or even potentially in a domestic setting.A particularly interesting possibility is the employment of this type of technology in primary healthcare.Here the system could be mounted above a couch at a health-centre, while a doctor in a separate room (which could be in a different city or even country), could remotely monitor breathing and other data, while interactively visualising the torso through use of suitable high-definition screens.NORM's potential for providing both tidal breathing and torso morphology data with high temporal and spatial resolution, as well as the unobtrusive and flexible nature of the technology, means that it offers potential to generate new types of clinically useful data as well as providing the doctor with the impression of being in the same room as the patient.

Overview of technologies for remote consultations
The systems described above provide illustrations of how techniques such as PS, which allows high resolution 3D data capture at low-cost, can be employed for capturing skin texture and shape.This can then be employed for medical diagnoses; and for facilitating remote realistic interactive and immersive consultations involving patients and clinicians.PS is a technique that has undergone much development in CMV and which has been found to offer a number of benefits, including: capability to recover surface 3D textures at very high resolution, capture of 3D and 2D surface texture and morphology information at low cost and with high resilience to changes in background lighting.The use of multiple lights also provides a degree of redundancy that enables removal of 2D image artefacts such as highlights and shadows (19) -this capability is expected to be particularly useful for implementing automated analysis.The authors have also begun combining PS with GS, in order to access the complimentary benefits of the two techniques.GS employs triangulation for accurate and large scale recovery of the 3D surface shape information from the skin of the face or body.Although GS cannot to recover 3D texture data from the skin surface with as high a resolution as PS, it is able to recover the global surface morphology more accurately.This is illustrated in Figure 7, where 3D facial recovery is shown.
The data on the left were captured in CMV by employing a GS system that triangulates from patterns of projected light, giving high positional accuracy of up to 150 microns, but low 3D texture resolution.The opposite is true for the data on the right, which were captured using the UWE 'Photoface' PS system (26).Experiments are being conducted in CMV using hybrid GS and PS equipment that employs invisible infra-red illumination in order to maintain unobtrusive operation.Employment of both GS and PS enables capture of 3D information from the face and body that contains low-distortion morphologies as well as high-resolution 3D textures, as shown in Figure 7. Combining such 3D data with high-resolution 2D data (pigment) provides an opportunity to realise very realistic virtual patient reconstructions.
Geometric stereo (GS) recovery Photometric stereo (PS) recovery The remote consultation system can be envisaged as being desk-top based, and consisting of the following equipment at two separate locations: high resolution cameras, sets of lights (of selected frequency) with the image capture for implementing PS and GS, and highresolution screens (with optional immersive 3D display capability), which the user will use to visualise and interact with the remote person.Thus PS will be implemented continuously in both directions, allowing the remote person to be reconstructed in real-time 3D on the display.
Furthermore, the user will be able to interact with the 3D reconstruction enabling them to, for example, view the remote face/body from various angles.It is also important to note that, since a change in the user's head position will be detected by the PS equipment mounted on their desk, this change in angle with which the user may wish to view the remote person can be specified by the user moving their head rather than using a mouse.In this way the system will allow the user and remote person to change views by moving their heads and bodies in a natural way at will, and to see automatically updated views of the other person.This capability is expected to significantly increase the realism of the interaction between the two people.By mapping the very high resolution 2D and 3D PS captured textures onto the surfaces recovered using GS (by triangulating with projected patterns of infra-red (IR) light and IR cameras), and reconstructing in real-time, the expectation is that a highly immersive effect will be produced.This will provide the good accuracy as well as the high temporal and spatial resolution that is needed to generate realistic dynamic visualisations.
2. Post-surgery analysis.A remote 4D system will enable faster and more frequent patient care.
Here it may be possible for the patient to attend a local GP practice and use the 4D system there to allow the remote surgeon to analyse their physical response to the surgery.Since this would be quicker and cheaper than having the patient travel to the surgeon, post-surgery analysis could be performed more frequently, which again would be expected to result in a number of benefits.
The technology will also be useful for assisting rehabilitation therapies, since an interactive, adaptive approach will allow for more personalised treatment to be given thus accelerating rehabilitation of patients.
3. Collaborative reviews.The existence of the 3D face/head reconstruction, with a potential facility for modifying the captured data with suitable computer-based tools (e.g. a virtual scalpel), is expected to enable 3D interactive planning of surgical procedures involving remotely located surgeons.If the skin surface data were combined with internal tissue and bone structure information obtained from, for example, a CT scanner, then it may even be possible to simulate the whole operation on a specific patient, thereby enabling identification of potential problems and solutions before any real surgery has occurred.The incorporation of suitable haptic devices would provide further functionality in simulation of the tactile behaviour of virtual surgical instruments such as a scalpel, thereby providing further realism.Here, in addition to planning of operations, the long-term aim would be for the remotely located surgeons to undertake the actual surgery, by employing 4D PS for an enhanced immersive interactive visualisation of the patient, with haptics for their instrument control.Some remote surgery is currently possible, for example with the Da Vinci system (28), but this is essentially a tele-operator system with a passive stereo vision provision, without haptic feedback or a capability to measure or analyse 3D or 2D tissue shapes or textures.
4. Platform for analysis and clinical appraisal of 3D and 2D patient data for computer-aided detection and diagnoses.It is envisaged that specialised machine vision devices will be employed to recover 3D and 2D skin surface data, as appropriate for analysis of a range of common but serious medical conditions.A number of conditions have already been addressed by research in CMV that has confirmed the utility of high resolution 3D machine vision for generating clinically useful data.Example conditions, and their associated medical vision devices are as follows: skin cancer (the SA) (20), respiratory illnesses and scoliosis (the NORM system) (22), plagiocephaly (PLAGIO -here both GS and PS were employed) (29), and 3D expression analysis (the UWE 4D PS System) (27).The latter enables real-time analysis of 3D changes in facial expression (e.g. during smiling) for appraising the results of plastic surgery.It can also be used for correlating changes in 3D facial expression to visual stimuli (e.g.videos of comedies or cartoon animations), as a method for diagnosing conditions such as clinical depression -here some data capture has been completed by CMV.In all these cases the devices have been developed for research purposes, and used to capture vision data in a novel way in order to provide additional indicators that may be used for diagnoses and clinical appraisals.Further development of these devices is needed to make them ward-ready and they need to be employed in clinical trials in primary healthcare and specialised clinics, as appropriate, before they can generate data that will be used for diagnostic purposes and be employed in clinical practice as part of the 4D medical machine vision system proposed here.

Figure 1 .
Figure 1.Skin Analyser (SA) with cover removed to show six LED lights surrounding central

Figure 3 .
Figure 3. Flow chart describing the concept of the SA tele-dermatology system.At REMOTE

Figure 4 .
Figure 4. Interactive remote SA visualisation of a skin lesion (left) and diabetic leg ulcer (right)

Figure 5 .
Figure 5. NORM data showing a good correlation between breathing volume measurement from

Figure 6 .
Figure 6.A real-time NORM image, where infra-red illumination is employed.

Figure 7 .
Figure 7. Showing 3D facial recovery with accurate morphology from GS and high-resolution

Figure 1 .
Figure 1.Skin Analyser (SA) with cover removed to show six LED lights surrounding central

Figure 2 .
Figure 2.An example of bump map and albedo recovery from a lesion using the Skin Analyser.

Figure 3 .
Figure 3. Flow chart describing the concept of the SA tele-dermatology system.At REMOTE

Figure 4 .
Figure 4. Interactive remote SA visualisation of a skin lesion (left) and diabetic leg ulcer (right)

Figure 5 .
Figure 5. NORM data showing a good correlation between breathing volume measurement from

Figure 6 .
Figure 6.A real-time NORM image, where infra-red illumination is employed.

Figure 7 .
Figure 7. Showing 3D facial recovery with accurate morphology from GS and high-resolution

Prior art for remote consultation or diagnosis
(2)dy by Shema et al. of a five week treadmill training program where VR was used to enhance gait(2).Although their trial only included 60 patients, they concluded that: 'treadmill training with VR appears to be an effective and practical tool that can be applied in an outpatient physical therapy clinic'.In 2013 Kurillo and Bajcsy provided a review of previous research on collaborative virtual reality, and described their work, which has concentrated on use of stereo vision and the Kinect device for implementation of tele-immersion.However, the implementations described appear to be mainly orientated towards