Yiru
Zhou†
ab,
Yao
Tang†
ab and
You
Yu
*ab
aSchool of Biomedical Engineering, ShanghaiTech University, Shanghai, China. E-mail: yuyou@shanghaitech.edu.cn
bState Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, China
First published on 22nd December 2023
Medical robotic sensing is a developing field that combines mechanical technology with medical engineering. Medical robots can play important roles in healthcare monitoring, prosthetic assistance, early diagnosis, and surgical operations. However, the increasing demand for medical robots comes with the need for greater sensing abilities, accurate control, and low-cost manufacture. Accordingly, this review highlights the recent progress in sensing technologies for medical robotic applications, including physical sensing, chemical sensing, and biological sensing. Subsequently, printing technology and human–machine interfaces are described. Our aim is to highlight the potential of medical sensing robotics and inspire further innovation.
However, integrated sensors on e-skin are associated with several challenges given that the sensors have to work properly during stretching, bending, and deforming together with the robotics operation. In the last few decades, soft robotics has emerged, which requires sensors with stretchability and softness.7,8
In contrast with sensors on a hard platform, sensors on the soft e-skin suffer from nonlinearities and non-repeatable results due to traditional fabrication processes. Thus, to address these issues, specific methods have been introduced for the fabrication of sensors including 3D printing, inkjet printing, laser ablation and textile weaving.
On the other hand, e-skin with multimodal sensors is crucial to achieve the goal of enhancing the robotics awareness of circumstances, such as pressure,9–11 temperature,12 pH,13 warfare agents, and biohazards. With the help of e-skin, robotics can perform more complex tasks in various fields, such as medical surgery, prosthetic equipment, and even pandemic research.14–16 COVID-19 was spread globally, and thus the detection of this virus without the risk of exposure to medical staff became an urgent topic. In this case, the use of a medical robot in place of medical staff will not only improve the efficiency and reduce the work pressure but also greatly reduce the risk of further spread. Similarly, there are often potential threats in agriculture and industrial production,17,18 such as the detection of residues of organophosphorus fertilizers and microorganisms in the environment.19,20
This review aims to summarize the recent advances in integrating a variety of sensors that can be mounted on robots and soft devices. These advances address the critical issues related to robotic sensing and human–machine interaction (HMI) in smart robots. In the following sections, we describe in detail the sensing mechanism, materials, and fabrication technologies for robotic e-skin. Firstly, we introduce the physical sensors on robotic e-skin, which are the most common and diverse, where efforts are being devoted to improving the sensitivity and accuracy of sensors by changing their shape and structure. Subsequently, we review the chemical sensors and biological sensors for e-skin, describing the specific detection of hazardous chemicals and biomolecules in the environment. Besides, we also present the HMI through e-skin with a variety of algorithms such as machine learning and deep learning, and advanced fabrication technology for soft e-skin with multimodal sensors (Scheme 1).
Pressure sensors are one of the most common types of physical sensors. There are various ways to achieve pressure information through signal conversion. In this case, resistive pressure sensors are widely used, which obtain the variation in pressure by measuring the resistance changes given that force affinity forms a larger connection, resulting in a change in resistance. This type of sensor has a very fast signal response and stable performance. However, flexible resistive pressure sensors are affected by their stretchable and bendable characteristics. Thus, to enhance their performance, various materials have been applied in their production, such as graphene,21 molecular perovskite (TMFM),22 MXenes,23–25 and polyvinyl alcohol (PVA) fibers.26 For instance, a resistive pressure sensor was fabricated using a polyimide (PI) substrate with spiky-structured polyurethane (PU) combined with MXene nanosheets, which were spray-deposited.23 The complex materials and specially designed structure (Fig. 1B) exhibited high sensitivity (Fig. 1C) and self-healing capabilities. Considering that sensors are prone to being damaged during continuous operation, in this case, the sensitive layer could autonomously “cut and heal” after 18 h, significantly prolonging its lifespan. Thus, self-healing is a significant function for the persistence of sensors.27 Due to its high sensitivity, the robotic arm with this resistive pressure sensor could grip delicate and fragile objects such as balloons and tofu (Fig. 1A). In addition to resistive pressure detection using irregular contact surfaces formed on the surface of the material, pressure sensors have been specially designed to improve the sensitivity of pressure detection with regular shapes.28 Besides, we can also use the change in resistance value as a piezoresistive sensor and a temperature sensor simultaneously.
Another common way to sense pressure is to calculate the change by capacitance.29–35 Generally, a common capacitive pressure sensor consists of three layers, i.e., a conductive layer on top and bottom sides and an insulating elastomer in the middle layer as a sandwich structure. The application of force results in a change in capacitance, which may be caused by two reasons, i.e., a change in the state of the electrodes or the distance between the top and bottom sides. Many different materials have been optimized to realize higher sensitivity and larger detection range. For instance, a type of highly responsive capacitive pressure sensor utilized a hybrid ionic nanofibrous membrane as a sensing material, which was positioned between micro-structured PDMS electrodes coated with gold (Au/M-PDMS).29 This sensor could have both high pressure range and high pressure measurement accuracy (Fig. 1D and E), respectively. Also, the accuracy and durability of the measurements were verified (Fig. 1F). By assembling multiple pressure sensors into an array, it was possible to capture surface pressure changes in real time and estimate the pressure distribution at both the edge and center of the object. Moreover, there is a special structure in biological skin called a spinous process.30,31,36,37 A new type of capacitive sensor incorporated this spine structure.31 Because different forces exerted on a 3D map create different pressure changes (Fig. 1H), this structure had higher sensitivity, in addition to better ability to distinguish among normal, shear and tilt forces (Fig. 1I). When added to a robotic arm, this sensor could be used to interact with, grasp and place light and easily deformable objects such as ping pong balls and raspberries (Fig. 1G). Using graphene, another high-performance capacitive pressure sensor with high sensitivity (3.19 kPa−1), fast response (30 ms), ultralow detection limit (1 mg), tunable-sensitivity, high flexibility, and high stability was obtained.38 In addition, higher-performance capacitive pressure sensors can be obtained by changing the material between capacitors.39
In addition to pressure sensing, the detection of friction is also an attractive pathway for robotic sensing. Therefore, numerous flexible friction electric nanogenerators (TENG) have emerged.40–42 Novel e-skin with TENG can distinguish materials from objects with indistinguishable smooth surfaces more sensitively than human skin. Also, piezoresistive pressure sensors (PPS) and friction electric nanogenerators (TENG) have been prepared.43 A piezoresistive pressure sensor (Fig. 2A) employed a stabilized PEDOT:PSS thin-film resistor, creating additional conductive paths, which disassembled the total resistance to be composed of four aspects (Fig. 2B). Due to its ingenious structure and circuit design, it could sensitively capture pressure changes, and also change the sensitivity of the sensor by changing the resistance in the circuit to meet different usage scenarios. The TENG part was composed of a doped electrically conductive PEDOT:PSS layer with high specific surface area as the charged layer, and the ordinary PDMS was treated as the anti-friction layer. It could precisely recognize different bending angles (Fig. 2C). This composite sensor was verified to be able to recognize materials at different positions simultaneously in a 3 × 4 array.
Based on electromagnetic sensing technology, a pressure sensor with high sensitivity and low detection limit was proposed. It consisted of an independent membrane of polymer magnets and a magnetic sensor integrated in an air gap structure (Fig. 2D).44 When subjected to external force, the freestanding membrane deformed, thereby causing a change in the magnetic field. The inductive magnetic sensor detected the change in the magnetic field and formed an LC oscillating circuit with a capacitor (Fig. 2E). Subsequently, it was possible to calculate the magnitude of the applied external force by detecting the magnitude change (Fig. 2F). It utilized a giant magnetoimpedance (GMI) material, i.e., Co-based amorphous wire (CoAW), which has various advantages including high sensitivity of 500% Oe. The sensor was flexible in size and shape, making it easy to integrate into prosthetic limbs or smart robot surfaces of different shapes and sizes. Furthermore, the sensor exhibited a sensitivity of 4.4 kPa−1 (equal to 120 N−1) and detection limit of 0.3 Pa (equal to 10 μN) in the range of 0–1 kPa. Experiments showed that this sensor could accurately sense the mass of a drop of water and even the movement of a small worm with a mass of 0.8 mg. In addition, it could reflect pressure changes through electromagnetism, realizing the direct transformation of force stimuli into digital frequency signals, where the frequency increased with an increase in the external force. This is consistent with the normal reflection of external forces in living organisms, indicating the potential of this sensor to be applied in smart prostheses.
When light is transmitted in an elastic optical waveguide, some energy is radiated into the environment, and the more it becomes shaped, the more light energy is lost. Thus, an inherent drawback of optical waveguides is that they incur losses during conduction, but taking advantage of this can also allow for the detection of pressure changes through the loss of light.45,46 Specifically, a sensing waveguide is a columnar elastomer with a high reflectivity material on the inside and a low reflectivity material on the outside. Also, light-emitting diodes (LEDs) and photodetectors (photodiodes) are located at each end of the waveguide. One type of waveguide has a transparent polyurethane rubber interior material and a highly absorbent silicone composite material (ELASTOSIL 4601A/B) wrapped around its exterior (Fig. 2H).45 This sensor acquired pressure changes with sensitivity and low signal-to-noise ratio and could be scanned to obtain information about the surface characteristics of an object by moving it in a certain direction on the surface at a suitable pressure (ΔP = 100 kPa) (Fig. 2I). Thus, the robotic arm integrated with this optical waveguide pressure sensor (Fig. 2G) could reconstruct the shape of a mouse, and also sense and determine the ripeness of a tomato while ensuring that it was not damaged if it was soft. The pressure signal obtained through the optical waveguide was not only of the pressure magnitude, but also retained information about the direction of the force. The isolation between different sensors is fantastic. When three waveguides were placed in one finger, the top, middle and bottom, respectively, their functions were not same. The top and middle waveguides had an immediate response to the inflation of the finger, which is unique to show the proprioception of the robotic itself. Specifically, optical waveguides are anisotropic in force sensing. By cross-stacking the waveguides, normal and shear force data can be obtained more accurately.47 A robotic arm integrated with this type of sensor can perform the action of grasping a key and unlocking a door with relative excellence.
In the case of soft robotics, self-sensing is a complex and important function given that the state of the robotics is unknown when it is controlled by a valve. Accordingly, many methods have been developed such as the use of a waveguide45 and piezoresistance48 to determine the curvature of the chamber. One of the latest methods is the use of a soft material-based self-sensing tensile valve (STV) that is capable of self-sensing and proportional control of soft pneumatic actuators from a single, constant supply pressure.49 By changing the inlet and outlet channel, the soft robotics can be controlled freely, and by measuring the fluid pressure in the inlet channel and outlet channel simultaneously, the state of the chamber can be calculated (Fig. 2K). Also, by changing the STV setup conditions post fabrication, the chamber pressure curve can be further programmed. For instance, by changing the constant supply pressure, Ps, to different levels, the maximum chamber pressure, Pch, of the STV is adjusted accordingly at ε = εmax (Fig. 2L). Furthermore, by inversing the inlet and outlet of the STV, the airflow direction passing through the channels is reversed.
Human skin has 4 different mechanical sensors that sense static and dynamic mechanical stimuli. The static mechanical stimulus sensors acutely capture low-frequency stimuli, whereas the fast-adapting mechanical sensors can capture dynamic pressure or vibration. The former is mainly realized by pressure sensors, while the latter is realized by friction sensors. Friction sensors help in recognizing the surface texture of an object and detecting sliding characteristics (Fig. 3A). An easily fabricated friction sensor that can be bonded to cotton textiles was prepared using Teflon.46 Teflon has a stronger electronic attraction and excellent mechanical strength compared to other common friction electrical materials such as PDMS and PET. This sensor was essentially a single-electrode TENG that is well suited for tactile sensing (Fig. 3B). When the sensor comes into contact with a fringed object, Teflon, with its strong electron affinity, adopts a more negative charge, whereas the surface of the object adopts a positive charge. When the object is separated from the sensor, the potential difference between the two friction layers will gradually increase, and transient electrons will flow from the Cu electrode to the ground, generating an output voltage for the external load. When the sensor and the object are in contact again, electrons will flow from the ground back to the Cu electrode, generating the reverse signal. This sensor can be used to accurately classify materials through artificial neural network (ANN) training (Fig. 3C). Compared to the classification by machine learning methods,43 that by neural networks has higher accuracy.
Synaptic transistors are another way to mimic electronic skin. Obviously, this type of sensor is realized by mimicking the synapses found in living organisms, which is a major breakthrough compared to simply recording pressure through transistors alone.50 When a current pulse is applied to a synaptic transistor, current transfer occurs in it, and subsequently the current pulse is passed on in steps.51 Recently, large-area, array-distributed synaptic transistor sensors were achieved by printing ZnO NWs with highly uniform (synaptic) transistors (Fig. 3D and E).52 A major problem with the application of conventional transistors in flexible smart robots is that the bending of these devices results in significant changes in their performance. However, after testing, this ZnO NW-based transistor sensor showed no significant performance changes during device bending, demonstrating its stability during mechanical deformation (Fig. 3F). This sensor directly simulates the synapses in biological skin, and subsequently the data obtained from the synapses can be directly used for neural network simulations to obtain certain conclusions.
To further improve the sensor performance and reduce the crosstalk between different sensors when combining them in multiple layers, multimodal designs that combine several different sensors have been developed, in addition to physical sensors that are designed separately and independently. However, the integration of multimodal sensors generally requires complex fabrication given that different sensing mechanisms are not very perfectly compatible. A sensor was cleverly designed by utilizing the thermoresistive and thermoelectric effects to combine pressure and temperature sensors.53 This design enabled the simultaneous independent detection of pressure and temperature (Fig. 3G), where the bimodal sensing of pressure and temperature stimuli plays an important role in artificial electronics and human health monitoring. This sensor consisted of a combination of two different temperature-sensitive sensors and a porous elastomer employing a constant-temperature-difference (CTD) feedback circuit to collect temperature and pressure data independently (Fig. 3H). The two temperature-sensitive resistors were made of platinum (Pt), where the smaller temperature-sensitive resistor was made by a larger temperature-sensitive resistor surrounded by two resistors and the thermal conductivity was detected based on the size of the pressure changes in the PDMS elastomer. The peripheral temperature-sensitive resistor itself had a kilo-ohm resistance, its own Joule heat was negligible, and it could be used to detect the ambient temperature. Alternatively, the center of the temperature-sensitive resistor exhibited a resistance of only about 100 ohms, where the heating of the electric power to a higher temperature than the ambient temperature resulted in heat conduction. The porous structure of the elastomer caused the thermal conductivity to increase under external pressure, which reduced the temperature of the center band, and thus the pressure could be converted into a voltage change on the CTD circuit. This sensor can be used to minimize the size of a single sensor by reducing the size of the center sensor and the heating temperature, thus achieving a higher spatial resolution (Fig. 3I). Consequently, this design can be used to sense not only the temperature of water, but also measure wind stimuli.
Another sensor was prepared integrating four different data detections, which consisted of two sensing layers sandwiched between PDMS doped with porous silver nanoparticles (Fig. 3J).54 Each sensing layer included two sensing elements made of concentric rings of chromium/platinum (Cr/Pt). The center sensing element was electrically heated to a higher temperature and became a hot film, and the peripheral annular sensing element became a cold film. The detection of pressure by this sensor was also realized by detecting the changes in the thermal conductivity of PDMS. The upper and lower hot films had different temperatures, and there was a temperature difference, and when subjected to pressure, the thermal conductivity of PDMS changed, and the temperature of the bottom hot film was reduced more. Simultaneously, when an object touched the upper sensing layer, the upper thermal film could detect the thermal conductivity of the object, and the double-layer sensing structure made the detected thermal conductivity more accurate. The upper and lower cold films were used to detect the object temperature and the ambient temperature, respectively (Fig. 3K). Experiments demonstrated that this sensor has high sensitivity in detecting pressure and temperature (Fig. 3L). Also, it was demonstrated that the robot arm integrated with this sensor could discriminate between objects of different shapes and sizes or materials when gripping them.
Accordingly, for the detection of chemical hazards, an artificial intelligence (AI)-driven human–robot interaction multimodal sensing robot system (M-Bot) was proposed.55 The standard chemical explosive represented by TNT or OP nerve agent simulant (paraoxon-methyl) present on the surface of an object was contacted and detected by e-skin on robotics (Fig. 4A). The detection of TNT was realized by electrodes modified by Pt nanoparticles/graphene (Fig. 4B), which have excellent electrocatalytic properties compared to conventional carbon and simply graphene electrodes. The detection principle is that the conversion of p-NO2 to p-NH2 is catalyzed and detected by negative differential pulse voltammetry (nDPV). The sensitivity of the sensor reached 0.95 μA cm−2 ppm−1, with a detection limit of 10.0 ppm. When TNT was detected using this electrode integrated into a robotic arm, accurate and stable results was obtained within 3 min through gelatin-based hydrogels. Besides, a gold nanoparticle electrode modified with a Zr-based metal–organic framework (MOF-808) was employed for the detection and analysis of organic phosphates. The non-enzymatic reduction of OP was detected by nDPV with a detection sensitivity of 1.4 μA cm−2 ppm−1 and a detection limit of 4.9 ppm (Fig. 4C). Similar to the TNT detector, the dry-phase OP analysis integrated into the robotic arm could be completed within 3–4 min.
Given that the detection of organophosphates is generally common in agriculture, some sensors for organophosphates have been designed in wearable devices that can be attached to the skin56 and worn on the finger57 for real-time alerts. However, in practice, people are still exposed to the risk. In this case, a better measure is to isolate the sensor from human skin58 or even eliminate the possibility of contact. Haptic feedback is needed during contact to avoid damage to the robot or other contact objects from excessive force. Another type of detection of organophosphates was integrated into a robotic arm together with a pressure sensor, and the system could simultaneously detect pressure and organophosphorus pesticides separately.59 The chemical sensor in this system (Fig. 4D and E) was fabricated by immobilizing an organophosphate hydrolase (OPH), which is highly specific for organophosphorus compounds. Employing the square-wave voltammetry (SWV) or amperometric technique, organophosphate could be detected after it was catalyzed to p-nitrophenol products (Fig. 4F). The choice of SWV also further improved the selectivity for OP threat field screening. In addition, electrochemical detection methods have made continuous detection possible.
In addition to converting chemical signals into electrical signals reflecting the content of chemical substances by electrochemical methods, surface-enhanced Raman spectroscopy (SERS) can be utilized for detection through image recognition by converting chemical signals into physical signals. A new multifunctional platform was fabricated by homogenously coating one-dimensional (1D) silver nanostructures (AgNPs) on elastomeric substrates. Cellulose nanocrystals are a reducing agent and stabilizer, which not only reduce silver nitrate to silver nanoparticles (AgNPs), but also enable AgNPs to form a 1D nanostructure, thus facilitating the construction of a conductive network.
The detection of neurotransmitters can also be accomplished with chemical sensors. A chemical sensor for neurotransmitters in tissue mimicry named NeuroString was presented to control multichannel and multiplexed monoamine sensing in experimental mice, measuring the serotonin kinetics in the gut, while avoiding unwanted stimulation (Fig. 4H) and perturbed peristaltic movements (Fig. 4G).60 Graphene was chosen as the electrode material, while to avoid cracking of the graphene monolayer at less than 5% strain, a laser-induced graphene nanofiber network was embedded in a polystyrene-block-poly(ethylene-ran-butylene)-block-polystyrene (SEBS) elastomer matrix to achieve high levels of softness and stretchability, while preserving the unique electrochemical properties of the nanomaterials (Fig. 4I).
Given that it is important and necessary to monitor pathogenic biohazards such as SARS-CoV-2, a robot system named M-bot contained not only chemical sensors, but also unlabeled SARS-CoV-2 virus detectors.55 With the CoV-2 spike 1(S1) protein on the printed multiwalled carbon nanotube (CNT) electrode (Fig. 5A and B), this sensor could detect SARS-CoV-2 specifically and precisely. The S1 protein bound to the antibody, which caused clogging of the electrode surface with a decrease in signal based on the electroactive redox probe (Fe3+/Fe2+). The sensitivity of the assay was in the order of parts per billion (ppb). Although non-specific adsorption may increase the likelihood of detection errors during hydrogel assays, semi-quantitative data acquired in real time can still provide timely and effective feedback to the user and alert them of the presence of biohazards to a certain extent (Fig. 5C).
In addition to the specific detection of viruses by antibodies, another effective method of detection was utilized based on fluorescence-based bioprobes.65 Nanobiosensors based on fluorescent bioprobes are highly sensitive and have the advantages of being biodegradable, environmentally friendly, less costly, and more integrable than some traditional analytical instrumentation technologies. A biosensor that specifically detected a widely used chemical inducer, isopropyl β-D-1-thiogalactopyranoside (IPTG), was carried in an all-soft biohybrid robot (Fig. 5D).62 The fabrication of this sensor was accomplished by directly integrating the cells into the soft material carrier medium used as the robot shell. By utilizing a porous PDMS–NaHCO3 membrane with a pore size of less than 0.5 μm, rich in elastic properties and optical transparency, it was ensured that the cells were blocked inside the medium without preventing the entry of chemicals, and simultaneously the signal feedback and flexibility of the robotic arm were ensured (Fig. 5E). Genetically engineered cells detecting IPTG undergo a genetic effect and express the green fluorescent protein (GFP). This fluorescent protein is further activated by excitation from a light source, and subsequently the fluorescent signal is converted into an electrical signal using a phototransistor, and the concentration of IPTG is inferred from the final electrical signal obtained (Fig. 5F).
Bioelectrical electrical signals, such as ECG signals, are a very important piece of information about the health status and are critical for the diagnosis and treatment of heart disease. Incorporating ECG acquisition into next-generation healthcare robots can play a very important role in the early detection and diagnosis of heart disease and reduce severe hospitalization. In this case, although most designs integrate ECG detection into wearable devices,66 it should be considered that 24/7 detection does not meet everyone's needs. An equally easy design for ECG signal detection is to design the sensor on the surface of a glove (Fig. 5G). This design can also be easily ported to a robotic arm at a later date.67 Considering the need to design the ECG acquisition sensor on a glove, and even later on a robotic arm, where the aberration formed on the skin by hand movement is usually less than 15%, a study designed ECG electrodes with a corrugated meandering model, where the electrodes had a near-perfect distribution of stresses in all tensile directions (Fig. 5H). By keeping the glove in contact with the chest, it was observed that the dry ECG electrodes on the glove could detect high-quality ECG signals even at 15% aberration (Fig. 5I).
In recent years, there has been an increasing amount of research on controlling robots using human electrophysiological signals. Electroencephalography (EEG) reflects the overall electrical activity of brain neural tissue on the surface of the cerebral cortex and is an indicator that can best reflect the activity of the brain. There are two main methods for collecting EEG, i.e., invasive and non-invasive brain–machine interfaces. The former mainly collects EEG signals on the scalp but often faces challenges such as low signal quality, high noise, and significant motion artifacts. Alternatively, the latter is more precise, with lower noise and a higher signal-to-noise ratio, but it requires surgical procedures on the subjects, making it difficult to implement. Electrooculography (EOG) and electromyography (EMG) are electrical signals targeting specific tissue structures. EOG is a bioelectric signal caused by the potential difference between the cornea and retina of the eye, and it varies depending on the eye movements. Therefore, an eye-dominant system can be constructed using EOG, such as a spelling system or a wheelchair movement system, to assist individuals with special needs. EMG reflects the superimposition of action potentials from multiple muscle fibers and can overall reflect the muscle activity within a specific region. In general, hand movements correspond to the contraction of the corresponding skeletal muscles. By focusing on collecting EMG signals from specific muscle groups, it is possible to reconstruct the muscle activity of the subject, which can easily correspond to specific motor actions using artificial intelligence techniques. With advancements in electrode technology and artificial intelligence, small, flexible, rapid, and highly accurate systems will be designed.
Artificial intelligence (AI) is driving a new direction in the field of e-skin, particularly in enhancing human–machine interaction. Traditional e-skin sensing has been limited to sensors with few channels and relatively simple modalities. However, with ongoing advancements in electrode and sensing technologies, the emergence of features such as multi-channel sensing and large-area high-resolution sensing necessitates improved algorithms to handle high-throughput data. Moreover, multimodal sensing is becoming increasingly prevalent, requiring novel algorithms to integrate different types of data. Machine learning algorithms, relying on extensive datasets, demonstrate excellent performance in regression and classification tasks. These two major categories of tasks encompass the majority of human–machine interaction issues. Researchers have effectively addressed challenges such as knowledge transfer, ontology perception, and intelligent decision-making by utilizing and innovating machine learning algorithms. This provides an efficient and viable solution for addressing complex practical problems in the realm of human–machine interaction.68
Strain is an important physical quantity that can reflect the degree of bending at human joints and the magnitude of force applied to a specific body part. By directly measuring the bending of various joints in the human body, information about physical movements can be realized. With this information, robots can follow human movements in a very short response time. In this field, resistive materials are widely applied.6,69,70 When a material is subjected to strain, its deformation leads to changes in resistivity. Once there is a certain linear relationship between the stretching of the material and its resistance, the degree of bending and stretching can be assessed quantitatively by measuring the changes in resistance. Currently, there is a trend towards directly printing conductive polymers on flexible substrates such as PDMS, Ecoflex, and PU, or transferring lithographically patterned conductive polymers to substrates to create electronic tattoos. These sensors exhibit good skin conformity, repeatability, and linearity.
In 2014, a transparent and stretchable human–machine interaction system consisting of a person's arm and a robotic arm was proposed.71 A resistive motion sensor and an electrotactile stimulation feedback device were placed on the person's arm, while a pressure sensor was placed on the robotic arm. The movement of the person's arm was recorded to control the robotic arm. When the robotic arm encountered an obstacle, the system provided the subject with electric stimulation as closed-loop feedback. However, in this experiment, only one sensor was used, indicating that the robotic arm had limited degrees of freedom and restricted mobility. Another recent study used a multilayered electronic transfer tattoo (METT) to accurately sense strain and achieve high-precision control of a robotic hand (Fig. 6A and B).72 METT consisted of two layers of resistive sensors and one layer of a heater, with a substrate separating each layer. In this work, the authors employed the layer-by-layer fabrication technique and placed 11 electronic tattoos in the first layer and 4 electronic tattoos in the second layer, corresponding to 15 degrees of freedom in the hand. During the fabrication process, the completed METT (with printed metal–polymer conductors) was directly transferred onto the hand or a glove using a transfer printing method. Through signal analysis, precise and short-term control of the multi-degree-of-freedom robotic hand was achieved (Fig. 6C).
EMG signals are the summation of action potentials from multiple motor units within muscle fibers. By measuring non-invasive surface electromyography (sEMG) signals, it is difficult to directly reflect the whole picture of the activated muscles. The sEMG signals represent the overall electrical activity of the muscles in the electrode sensing area over a period of time. Therefore, sEMG alone cannot serve as the gold standard for measuring muscle movement. However, there are differences and repeatability in the electromyographic changes in the same muscle region when performing non-continuous simple gestures. Therefore, EMG is considered as the movement signal of a specific muscle group. With multi-channel EMG acquisition, the movement information from different muscles can be obtained simultaneously. This improves the accuracy of classification tasks and recognition through different channels. Currently, gesture recognition tasks using EMG signals typically employ sensors with more than 4 channels, where sensors with 8 to 64 channels have been presented in the literature.
The EMG signals are relatively weak, with a value of usually less than 1 mV. Therefore, in the design of the amplification circuit, the signals need to be amplified at least 1000 times. In this case, given that the amplification factor is large, the noise caused by movement significantly affects the data quality. This, to mitigate common-mode noise, a differential amplifier was used, with the ground of the amplifier connected to the skin. The energy of the EMG signals was concentrated in the range of 20–250 Hz, and thus filtering the signals below 20 Hz and above 250 Hz could significantly improve the signal-to-noise ratio.
Additionally, researchers are not only focused on gesture classification tasks but also seeking breakthroughs in studying the force exerted during movements and EMG-based property analysis.73 For example, when performing the same grasping motion, the force used to grasp a piece of paper differs from the force used to hold a cup of water. Therefore, deriving force information from EMG signals is an interesting topic. Recent works used integrated EMG amplitude to calculate the force exerted during movements, which was obtained through signal processing. Moreover, some groups utilized deep learning techniques, treating the EMG signals as the input data and the force information the as label, to develop models that predict the force intensity.74 In 2010, a large-area epidermal electrode for collecting electrophysiological signals was reported (Fig. 6D and E).75 Photolithography was used to pattern the electrodes, and then the epidermal electrode with eight channels and 16 electrodes attached to the subject's forearm (Fig. 6F). Through signal classification algorithms, an average accuracy of 89% was achieved in classifying simple gestures. In another study, patients wore surface electrodes to control the rehabilitation of the exoskeleton.76 Exoskeleton movements were intuitively performed by the subject using classified EMG signals measured from reinnervated muscles in real-time, helping for bilateral rehabilitation. In 2019, a study on controlling a robotic arm based on muscle synergies in coordinated movements using the recurrent log-linearized Gaussian mixture network (R-LLGMN) algorithm was reported.77 The algorithm first learned the EMG signals for individual fundamental finger motions and regarded these motions as muscle synergies. Subsequently, various finger motions were expressed as combinations of these synergies. The algorithm was tested in terms of correctly decomposing known complex gestures into fundamental gestures and decomposing unknown gestures into fundamental gestures. The results showed the high accuracy and universality of the algorithm in handling coordinated movements. Moreover, EMG can also be used to control rehabilitation robotic arms, drones, and other applications.
Non-invasive EEG signals are typically collected from the scalp. However, because the cranium is a bad conductor, the signal amplitudes are usually weak, typically less than 100 μV. Furthermore, the signals are greatly influenced by noise and require complex filtering to obtain meaningful information. Similar to sEMG, surface EEG signals reflect the overall brain neural activity in a specific region over a certain period, rather than the excitatory state of individual neurons. Brain activity can be inferred by analyzing EEG signals from specific brain regions. For example, motor-related brain activity is concentrated in the central frontal region, while language-related brain areas include Broca's area. By analyzing the EEG activity in specific brain regions, the subject's current state of movement was be determined. Alternatively, eye movement signals (EOG) are relatively strong, with a maximum amplitude of about 5 mV. The signal collected from eye movements is often interfered with by EEG and EMG signals. However, in signal processing, the interference from EMG signals can be eliminated by filtering out high-frequency signals. The EEG signal itself is weak, to the extent that it often fails to reach the threshold of the filter and gets filtered out.
EEG and EOG signals are commonly collected to assist in patient treatment or to enable individuals with limited mobility to perform actions similar to that of able-bodied individuals. Brain–machine interfaces (BMI) are a popular research topic, where patients typically wear EEG devices to control external devices such as wheelchairs, exoskeletons, and even virtual games by decoding the subject's brain activity.78–80 EOG signals can quantify vertical and horizontal eye movements, enabling the control of wheelchairs using eye movements. EOG is often used in conjunction with EEG to accomplish specific tasks, such as the hybrid EEG/EOG-based hand exoskeleton system, which helps individuals with limb paralysis independently perform hand movements (Fig. 6G).81,82 In this system, the participants used a non-invasive brain/neural hand exoskeleton (B/NHE) that translates brain electric signals associated with the intention to grasp into actual hand-closing motions driven by the exoskeleton, and EOG signals related to voluntary horizontal eye movements (horizontal oculoversions) into hand-opening motions driven by the exoskeleton (Fig. 6H and I). The patient's EEG signals controlled the motion of the robotic hand, while the intensity of the EOG signals controlled the opening and closing of the hand. Analyzing EEG and EOG signals could assist patients in controlling the behavior and switch of the robotic hand, providing high stability and robustness. Additionally, a spelling system controlled entirely by EOG was proposed. This spelling system utilized two eye movements (single blink or wink) for target selection, and the average time for detecting a single target was lower compared to most EEG-based systems.83 The average classification accuracy and information transfer rate of this system were 93.6% and 43.8 bits per min, respectively.
Invasive brain–machine interfaces (BMI) can be done at different scales. One of the shallowest scales is electrocorticography (ECoG), where invasive electrodes are placed on the cortical surface inside the skull. The signal amplitudes range from 0.01–5 mV. The electrodes implanted in the brain cortex can capture local field potentials (LFPs), which are smaller than 1 mV. LFPs reflect the activity of surrounding neurons and do not directly indicate the activity of individual neurons. Compared to non-invasive EEG, invasive BMI offers a higher signal-to-noise ratio and more precise localization. Invasive brain-electrode measurements allow for more accurate capture of brain electrical potential changes, enabling control of external devices or language input.
Invasive neural interfaces can assist individuals with long-term limb paralysis in manipulating physical devices. For example, a 96-channel intracortical silicon microelectrode array was implanted in the participant's motor cortex arm area (Fig. 6J and K).84 Through five years of training and algorithm updates, the goal was achieved. The participant was able to control a robotic arm with their thoughts to grasp a cup, bringing it to her mouth for drinking. Subsequently, this experiment was validated in a clinical setting (Fig. 6L).85 Similarly, another study used a microelectrode array in the motor cortex area of the brain and placed high density percutaneous electrodes in the right upper and lower arm of the participant.86 After recording the brain signals with the detection device, the system translated the electrical activity of the cortex into commands, producing electrical stimulation to the muscles for flexion, extension, and grasping actions. The results showed that the BMI combined with functional electrical stimulation (FES) could assist patients with arm paralysis due to spinal cord injury in engaging in daily activities and improving their quality of life. Similarly, a spelling system based on invasive EEG greatly helped paralyzed patients recover.87
One application of the printing technique is used to build soft systems.90,91 A recent study used conductive inkjet printing on a flexible substrate to construct a human–machine interaction system and employed scalable inkjet printing to fabricate flexible electronic circuits (Fig. 7A).55 A type of nanomaterial-based ink, combining silver nanowires with PDMS, was developed to fabricate pressure sensors. Then, the authors also printed silver ink on PI to fabricate electrodes for collecting surface electromyography (sEMG) signals from muscle activity. These two sensors were integrated into a closed-loop feedback system with a robotic component. Furthermore, inkjet printing and electrode modification techniques were used to create a biosensor capable of detecting toxic and harmful chemicals. Another study demonstrated a large-area stretchable pattern by screen printing silver nanowires (NWs) ink on elastic poly(dimethylsiloxane) (PDMS).92 The pattern exhibited an ultralow sheet resistance of 1.9 ohm s−1, together with strong stability and mechanical repeatability. It endured over 1000 bending and stretching cycles with only 10% strain, showing promising applications. Transfer printing is an interesting printing method that typically involves inverting patterned electrodes onto a specific interface, such as adhesive PEDOT (Fig. 7B).91 The inverted side is then applied to the human skin, and the PEDOT layer is peeled away, transferring the electronic skin to the skin. A typical transfer printing study proposed a Cartan curve-inspired transfer process on a hemispherical substrate to transfer electronic tattoos to the human body for tasks such as multichannel electrocardiography and sign language recognition.
In the photolithography method, the selective exposure of a polymer coating surface to light followed by the use of specific solvents to dissolve the selected areas is employed to generate patterns. Photolithography is a complex physical and chemical process that allows for high-resolution production and the creation of various intricate flat patterns. Besides its crucial role in the semiconductor industry, photolithography also has important applications in diverse fields such as micro-capacitor fabrication, biosensing devices, optical device production, microneedle manufacturing, and LED production.93,94
In the field of micro-supercapacitor (MSC) fabrication, a method utilizing photolithography was reported. Firstly, a composite of a high-density single-wall carbon nanotube (SWNT) network and photoresist were patterned on a substrate. Subsequently, a carbonization process converted the photoresist into amorphous carbon, forming SWNT/carbon current collectors.95 The MSCs demonstrated good conductivity and superior electrochemical performance. In another study, photolithography and oxygen plasma etching were used to create a three-layer structure of PI–Au–PI by repeatedly spin-coating PI (Fig. 7E).96 By patterning the surface contours through photolithography and etching, a dynamically programmable interface driven by the Lorentz force generated by a magnetic field was achieved. This interface could rapidly be transformed into the desired shape within a short period. In the context of a soft implantable drug delivery device (SID) (Fig. 7F), photolithography was employed to fabricate the front and back ends. Photolithography and wet etching were used to pattern a Cu layer and epoxy layer on the front, as well as an Al layer on the back end of the SID using the same method.97
The basic principle of laser ablation is to focus a small, low-power laser beam with high beam quality on a tiny spot, creating a high power density at the focal point. This intense energy causes the material to evaporate instantly, forming small structures such as holes and grooves. Unlike photolithography, laser ablation does not require complex operation processes and offers advantages such as high precision and high speed. Furthermore, laser ablation can not only pattern the substrate material but also induce modifications in graphene films to enhance their properties. In practical ablation processes, parameters such as wavelength, power, and pulse duration are adjustable and important. However, it is important to choose a specific wavelength that corresponds to high absorption in the target material, given that the power and peak intensity are closely related to the patterning throughput. Laser operations can be categorized in the time domain as continuous wave (CW) and pulsed. CW lasers emit a constant laser intensity, while pulse lasers emit laser pulses at a fixed frequency. Pulse lasers can have pulse durations in the femtosecond range, and the use of short-pulse lasers enables higher spatial resolution in ablation.
Laser technology is widely used in the field of flexible material manufacturing and can be used for the fabrication of bio-sensors, soft electronic devices and e-skin.98–103 In a recent study, three-dimensional integrated electronic devices made of elastomers were reported, where laser ablation and controlled soldering were used to create vertical interconnect accesses (VIA). The pattern designs of the electrodes, heaters, and VIA connectors were all fabricated using the laser ablation technique (Fig. 7C).104 It is worth noting that in the fabrication of VIAs, the researchers used a dye to change the color of silicon and employed a 1064 nm laser that was less absorbed by copper to remove the PI and black silicone from the copper. In another study on implantable optoelectronic systems, direct laser ablation was used to engrave the top and bottom copper layers (17.5 μm thick) of the device (Fig. 7D).105 Further advance work reported direct laser writing of 3D structures at micron resolution using a two-photon lithography fabrication process. This method was used to fabricate a 16-channel array with a 300 μm pitch for capturing electrophysiological signals in the brains of mice and birds.
Textile technology is a cost-effective method for the large-scale production of electrodes. Traditional textiles utilize tools such as needles and sewing machines to transform materials such as cotton, wool, and synthetic fibers into clothing and wearable devices. These textiles are characterized by their affordability, ease of production, and ability to be laundered. Currently, fabric is emerging as a promising approach for sensor fabrication, giving rise to luminescent textiles, pressure-sensitive textiles, and sound-sensitive textiles. These materials serve as excellent choices for creating sensors and human–machine interfaces. Furthermore, large-area arrangements of organic transistors and wearable electronic textile can also be realized through textile manufacturing.70,106,107
Using textiles to fabricate screens is a creative idea, where a 6 m-long and 25 cm-wide display textile was reported in a recent work, incorporating 5 × 105 electroluminescent units (EL) (Fig. 7G).108 Each EL unit was formed by the contact between luminescent warp threads and transparent conductive weft threads, while cotton yarn served as the structural framework of the fabric. This luminescent textile functioned as a display, allowing the real-time visualization of the wearer's status when paired with a brainwave sensor. It also enabled communication through the integration of a keyboard made from pressure-sensitive fabric. Another significant endeavor involved the creation of a textile utilizing piezoelectric fibers capable of converting mechanical vibrations into electrical signals (Fig. 7H).109 High-modulus Twaron yarns and cotton yarns oriented at right angles were designed to mimic the structure of the tympanic membrane. In this fabric, a single strand of a piezoelectric elastomeric fiber transducer was incorporated, resulting in a synergistically coupled fabric. This textile was capable of capturing faint audio signals.
Low-cost manufacturing is a promising technological pathway. Leveraging technologies such as laser, printing, and textiles, it is possible to construct sensor units that meet human health requirements on a large scale and in an arrayed manner at a low cost. Furthermore, breakthroughs are anticipated in terms of higher information density, improved interactivity, and extended detection durations. In the future, as technology continues to advance and new medical and physiological challenges are addressed in engineering, the advantages of low-cost manufacturing will contribute to the accessibility to healthcare, the extension of monitoring periods, and the personalization of rehabilitation, benefiting the wider population.
Intelligent robots equipped with sensors still hold substantial untapped market potential, which is hindered by their high production costs. Currently, various methods have been developed for the fabrication of sensor. Traditional micro and nano lithography techniques are ideal choices for achieving higher performance and precision. However, the high requirements of materials and processing equipment make lithography an expensive option. Thus, novel low-cost techniques such as 3D printing, inkjet printing, screen printing, and laser engraving have been employed for device fabrication to improve the production efficiency and reduce costs during large-scale manufacturing.
Equally important is the processing and utilization of data, which can be achieved by optimizing computational models. Sensor design should possess foresight by enhancing the speed of signal processing through appropriate analysis of the data received. The human brain excels at rapidly analyzing and processing incoming signals, not only due to the exceptional performance of receptors and the musculoskeletal system but also because it possesses a unique way of data processing, allowing it to efficiently and accurately handle received information. This forms a significant pathway for optimizing control algorithms by imitating the brain's approach to data analysis and processing. Currently, several machine learning techniques have been applied to data processing, such as acquiring motion intentions through bioelectrical signals, recognizing object surface features through tactile perception, and analyzing multi-modal data. These algorithms significantly enhance the speed of data analysis, enabling robots to respond more swiftly.
The ultimate goal for intelligent robots is system integration, whether for exploring unknown environments or acting as prosthetics or exoskeletons for individuals with disabilities. This necessitates the formation of a complete system where a device can capture signals reflecting various physical, chemical, and biological characteristics, decode and analyze them, and provide appropriate feedback. This feedback can range from displaying information reflected by signals to controlling the device mechanically, thermally, or even through electrical stimulation, forming a closed loop. Challenges lie in the optimization of hardware integration for sensors and actuators, signal communication and conversion, as well as decoding and processing data. Undoubtedly, some advancements have been realized, but there are significant challenges and opportunities for development in the future.
Footnote |
† These authors contributed equally to this work. |
This journal is © The Royal Society of Chemistry 2024 |