Yan-bo Guoa,
Yu-lin Liub,
Qi-lai Chen*c and
Gang Liu*a
aDepartment of Micro/Nano Electronics, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China. E-mail: gang.liu@sjtu.edu.cn
bSchool of Materials Science and Engineering, Xiangtan University, Xiangtan 411105, China
cSchool of Materials, Sun Yat-sen University, Shenzhen, 518107, China. E-mail: chenqlai@mail.sysu.edu.cn
First published on 26th September 2022
Brain-inspired neuromorphic computing has become one of the critical technologies to overcome the bottleneck of von Neumann architecture. It is a vital step to construct a brain-like neuromorphic computing system at the hardware level by utilizing artificial synaptic devices. Compared with electronic synaptic devices, optoelectronic synaptic devices have the advantages of low power consumption, low crosstalk, and high bandwidth. Artificial optoelectronic synapses, analogous to retinal structure, can directly respond to and process light signal information to mimic the neuromorphic visual system. As high-level nerve impulses, both generated and regulated, emotions affect the strength and persistence of memory. Ambient illumination can provide visual perception to distinguish the size, color, and other characteristics of objects as well as affect the nonvisual functions of individuals, such as emotional states, thereby affecting learning and memory function. Herein, an artificial optoelectronic synapse composed of ITO/TiO2−x/p-Si was proposed. A variety of biologically dependent synaptic plasticity relating to learning and memory, including short-term synaptic plasticity, long-term synaptic plasticity, and learning-forgetting-relearning multifunctional advanced synaptic activity, was successfully simulated. A 3 × 3 artificial optoelectronic synapse array based on 9 devices was constructed to mimic the functions of visual learning and memory affected by internal emotion and ambient illumination. The proposed artificial optoelectronic synapse will exhibit great potential in visual and image information perception and memory.
The factors that affect biological visual perception and memory are diverse and complex. Here, we discuss two influencing factors, intrinsic and extrinsic. Emotion as an intrinsic factor is a characteristic mental activity of higher-grade animals, which belongs to the nerve impulses projected by nerve nuclei such as cingulate gyrus and amygdala to the various higher cortex.20,21 The main function of the amygdala is to generate emotions corresponding to various external information transmitted to the neocortex of the brain. Information about emotions can affect the activity state of the higher cortex to complete multi-component intricate functions such as perception, complex movement, learning, and memory.22 Memory relies heavily on emotions and is consistent with internal emotions. Events that trigger strong emotional responses leave long-term memory.23 Furthermore, ambient illumination as an intrinsic factor produces visual effects by stimulating the rod photoreceptor cells and cone photoreceptor cells in the retina, which provides a visual perception guarantee for distinguishing the size, color, and spatial orientation of objects, and simultaneously participates in learning, memory, and emotion regulation through various ways.24,25 Most studies have shown that the physiological characteristics and neuronal activity of the nucleus reuniens (Re) of the midline thalamus can be affected by bright light and contribute to the regulation of neuronal activity and spatial memory in the hippocampus (HPC), thereby affecting synaptic plasticity and playing a prominent role in learning and memory processing.26 For instance, phototherapy can significantly improve neurodegeneration in Alzheimer's disease, which enhances synaptic function.27 Interestingly, ambient illumination as well as has direct or indirect effects on the physical and mental states of emotional cognition and physiological rhythm through non-visual forming system due to the presence of intrinsic light-sensitive ganglion cells (ipRGCs) in the retina.28,29 Emotion changes further have an impact on memory and learning behavior. Phototherapy is also used to treat many mental health problems, including seasonal affective disorder and chronic depressive disorder.30 That is, the internal emotional state and ambient illumination interact to regulate and affect the individual visual perception, memory, and behavior. Here, the effects of emotion and ambient illumination on visual perception and memory are discussed for the first time based on optoelectronic synaptic devices.
In this paper, light-stimulated synaptic devices (optoelectronic synapses) based on ITO/TiO2−x/p-Si all-metal oxides were successfully fabricated. The adaptive optics information detection and excellent visual memory capability were exhibited on a single optoelectronic synaptic device combining the perception and synaptic function. Several critical biological synaptic behaviors were simulated by perceiving the different amplitude, duration, and quantity information of light signals, realizing the conversion of short-term synaptic plasticity (STP) to long-term synaptic plasticity (LTP) and multi-learning advanced synaptic behaviors of learning-forgetting and relearning. Most importantly, based on such visual memory properties, a 3 × 3 synapse array was developed to emulate internal emotions and ambient illumination effects, which are mapped into image “H”. This more realistically simulates the human visual perception and visual memory based on the artificial optoelectronic synaptic system, which illustrated the potential application prospects in the visual information processing and construction of advanced machine vision memory systems.
The surface topography information and cross-sectional of the TiO2−x thin films were examined by scanning electron microscope (SEM) (Fig. 1b). It can be seen from the surface topography that the grain size is small and the grains will form clusters. The thickness of the as-prepared TiO2−x film is ∼95 nm from the cross-sectional image, which is consistent with that measured in step-height image of AFM (Fig. S2a†). The roughness of ∼0.7 nm shown in the three-dimensional morphology (Fig. S2b†) illustrates the smoothness of the film surface. The XPS survey spectrum shown in Fig. S3† reveals that the samples mainly contain Ti, O, and C element. Fig. 1c shows the XPS spectra of Ti 2p. The Ti 2p peaks at 464.22 eV and 458.42 eV can be attributed to Ti 2p 1/2 and Ti 2p 3/2, respectively, which is consistent with the previous reports.40,41 The O1 peak can generally be decomposed into two sub-peaks at binding energy of 529.92 eV and 531.82 eV corresponding to lattice oxygen and non-lattice oxygen, respectively (Fig. 1d).42,43 The lower binding energy peak is attributed to Ti–O bonds, while the higher binding energy peak is associated with oxygen vacancies (defects). These defects play an important role in fabricated optoelectronic synaptic devices. Moreover, the XRD analysis images (Fig. S4†) also illustrate that the sample films have many defect states.
Humans acquire external information mainly through vision, so simulating the human visual system is crucial for the development of artificial intelligence. The synaptic transmission properties were mimicked on the proposed two-terminal optoelectronic synaptic devices more approximate to biological synapse structure.44,45 The TiO2−x optoelectronic synaptic devices were fabricated on a p-Si substrate, and top and bottom electrodes can be treated as the presynaptic and postsynaptic neurons, which are shown in Fig. 2a. Fig. 2b illustrated a typical light-induced excitatory postsynaptic current (EPSC) of the optoelectronic synaptic device under 1 V read voltage. The wavelength of the light pulse is 365 nm, and the intensity is 3.0 mW cm−2. More specifically, the value of EPSC is increased to 1.75 nA with an obvious current amplitude of ∼0.6 nA when the light is turned on for 3 s. The gradual decay property in EPSC (the fitted curve shown in Fig. S5†) indicates that short-term plasticity (STP) plays a central role in the realization of visual learning and memory.46 Paired-pulse facilitation (PPF) as a typical characteristic of STP reflects the capability of biological synapses for encoding temporal information in auditory and visual signals.47 The optoelectronic synaptic device exhibits an obvious PPF phenomenon as stimulated by two successive 365 nm light pulses with an interval time (Δt) of 2 s, which is shown in Fig. 2c. The amplitude of the EPSC inspired by the second pulse (A2) is significantly larger than the first one (A1). Furthermore, the PPF index is defined as 100% × (A2 − A1)/A1. Fig. 2d illustrates that the PPF index decreases with increasing pulse interval time, implying a gradual weakening of the effect between consecutive pulses. PPF index reaches a maximum of nearly 137% with an interval time of 2 s. The functional relationship of PPF index and interval time can be fitted with a biexponential function:48
PPF = C1 exp(−t/τ1) + C2 exp(−t/τ2) | (1) |
The important information in short-term memory (STM) is easily forgotten over time. While long-term memory (LTM) through continuous practice and stimulation is a state in which synaptic efficiency can be maintained for a long time, which is the basis for enhancing learning and memory ability.49,50 Fig. 3a shows the gradual increase in the level and the retention time of memory as the light width of the light pulse increases. The value of ΔG (ΔG = Glight − Gresting, Glight corresponds to the maximum conductance under light pulse and Gresting is the resting conductance in the dark state before the light pulse) increases from ∼0.18 nS to ∼1.5 nS when the optical pulse width increases from 0.5 s to 10 s. It is worth noting that the transition from STP to LTP can also be turned by modulating the light intensity and the number of light pulses, as shown in Fig. 3b and c, which can significantly enhance and consolidate visual learning and memory. Fig. 3d is a schematic diagram showing the biological multiple memory model in the human brain. The memory can be strengthened by enhancing or continuously repeating the input of external information stimulation, turning SM (Sensory memory) into STM and further into LTM.51 When learning new events, partial memory information will be gradually forgotten over time, as shown in Fig. 3e. The 31 light pulses were required to reach the conductance level of ∼5.8 nS from the conductance level of ∼3.8 nS, and this process takes 150 s. While, it only takes 21 light pulses to achieve the same conductance level when relearning from the same conductance lever of ∼3.8 nS, and the required time is reduced to 100 s. This phenomenon of conductance change illustrates the realization of the learning-forgetting-relearning process for external information on the proposed devices. This process is consistent with the Ebbinghaus forgetting curve, which describes the discipline of the human brain for forgetting new things.52
Human memory can be regulated and influenced by internal emotions and ambient illumination. Changes in memory are consistent with internal emotions. The higher the internal emotion, the higher the level of memory. The same goes for memory retention time. Furthermore, the brightness of the ambient illumination can also affect learning and memory ability. In this work, the memory levels and retention characteristics (response conductance) of the same event (input optical pulses information) under different internal emotion states (different bias voltages) and ambient illumination (different brightness) are mapped to images “H” identification (predefined into a 3 × 3 array). Seven optoelectronic synaptic devices were stimulated with light pulses to obtain response current. The other two optoelectronic synaptic devices were tested for response current without light stimulation. The detailed setting methods are shown in Fig. S6 and S10.† The fabricated optoelectronic devices exhibit excellent sensitivity to the event memory under different emotional states and different ambient illumination, which more realistically simulate the memory properties of human visual information.
The light intensity of the input optical pulse (event) information is 3.0 mW cm−2, the pulse width is 2 s, the pulse interval is 2 s, and the number of pulses is 8. The different internal emotion states are triggered by different biases, which are set to 0.5 V, 0.7 V, and 1.0 V, respectively. As shown in Fig. 4, the fabricated optoelectronic synaptic devices exhibit the behavior of internal emotions modulating visual memory. The visual memory based on relatively low emotion (resting conductance of ∼0.12 nS) at 0.5 V (Fig. 4a) was insufficient for “H” image recognition. The memory of the image almost disappears as the decay time increases to 200 s. The detailed current response curves were shown in Fig. S7.† As the bias voltage increases to 0.7 V, the value of the resting response conductance increases to ∼0.54 nS, indicating the raise of internal emotion (Fig. S8†). At the initial time, the image “H” can be recognized (Fig. 4d), but the image cannot be recognized as the decay time increases to 200 s (Fig. 4f). It is illustrated that insufficient emotional promotion has a weak effect on memory retention. While the response current can clearly identify the image “H” under relatively higher internal emotion with 1.0 V bias (Fig. 4g). The resting response conductance increases to ∼1.47 nS under 1.0 V bias (Fig. S9†). And memory similarly decays slowly as decay time increases. The response current still had an image-recognizable level at a decay time of 200 s under higher internal emotion (Fig. 4i). This phenomenon verifies the regulatory effect of internal emotion on visual memory.
Fig. 4 The change of visual memory of the optoelectronic synapse array with decay time at 0 s, 100 s and 200 s under three levers of internal emotions. |
Additionally, the level of visual memory can be affected by ambient illumination. The spectral distribution, brightness, and duration of ambient illumination have a certain degree of influence on memory and emotion. The regulation of ipRGCs on the individual non-visual effects is mainly accomplished by melanopsin, which has photosensitive properties. Studies have found that melanopsin is most sensitive to the blue light spectrum with wavelengths in the range of 460–480 nm.53,54 Therefore, a light source with a wavelength of 465 nm is adopted as ambient illumination in this paper. The same event was memorized after 20 s of ambient illumination. The proposed optoelectronic synaptic devices were in the same internal emotion. Fig. S10† shows the specific setting form in this process. As shown in Fig. 5a, d and g, the memory abilities for the same event were slightly improved with the increase of the brightness of the ambient illumination. While the memory retention ability is significantly enhanced with the ambient brightness (Fig. S11–S13†). The image “H” can still be distinctly recognizable when reaching the same forgetting time of 200 s. Obviously, the brighter the ambient illumination, the more pronounced the memory effect. In addition, the “H” image can be discernible at a decay time of 500 s with the light intensity of 4.0 mW cm−2, which clearly showed the characteristics of long-term memory (Fig. S14†). Ambient illumination has a non-visual effect on emotional states. The resting conductance, the emotional state, increases with the increasing brightness of the ambient illumination. At the same time, the maximum conductance triggered by the same event was also increased, indicating the enhancement of memory level. Moreover appropriately prolonging the duration time of ambient illumination can also enhance the level of memory-related proteins.55 The increase in resting response conductance and the maximum conductance under light pulses were observed on the optoelectronic synaptic device as the illumination duration increased from 5 s to 50 s (Fig. S15†). Thus, memory under lower emotions can be complemented by prolonged duration or enhanced brightness of the ambient illumination.
Fig. 5 The change of visual memory of the optoelectronic synapse array with decay time at 0 s, 100 s and 200 s under different brightness of the ambient illumination. |
Human visual memory was realistically mimicked in a designed 3 × 3 optoelectronic synapse array by adjusting the level of internal emotions and the brightness of the ambient illumination. It shows that the memory level can be significantly affected by internal emotions, and the memory maintenance ability is relatively more affected by the brightness of the ambient illumination. This behavior effectively simulates superior visual perception and visual memory performance. A detailed comparison between previously reported photoelectronic artificial synapse devices and our ITO/TiO2−x/p-Si devices is provided in Table S1.† Herein, the image information detection and memory function that simulates the artificial visual system are achieved on our relatively simple device structure. More importantly, the effects of internal emotion and ambient illumination on visual perception and memory are discussed for the first time based on an optoelectronic synaptic array, which may lend it to a wider range of applications in complex conditions.
Footnote |
† Electronic supplementary information (ESI) available. See https://doi.org/10.1039/d2ra02749f |
This journal is © The Royal Society of Chemistry 2022 |