Keeping an “eye” on the experiment: computer vision for real-time monitoring and control

Abstract

This work presents a generalizable computer vision (CV) and machine learning model that is used for automated real-time monitoring and control of a diverse array of workup processes. Our system simultaneously monitors multiple physical outputs (e.g., liquid level, homogeneity, turbidity, solid, residue, and color), offering a method for rapid data acquisition and deeper analysis from multiple visual cues. We demonstrate a single platform (consisting of CV, machine learning, real-time monitoring techniques, and flexible hardware) to monitor and control vision-based experimental techniques, including solvent exchange distillation, antisolvent crystallization, evaporative crystallization, cooling crystallization, solid–liquid mixing, and liquid–liquid extraction. Both qualitative (video capturing) and quantitative data (visual outputs measurement) were obtained which provided a method for data cross-validation. Our CV model's ease of use, generalizability, and non-invasiveness make it an appealing complementary option to in situ and real-time analytical monitoring tools and mathematical modeling. Additionally, our platform is integrated with Mettler-Toledo's iControl software, which acts as a centralized system for real-time data collection, visualization, and storage. With consistent data representation and infrastructure, we were able to efficiently transfer the technology and reproduce results between different labs. This ability to easily monitor and respond to the dynamic situational changes of the experiments is pivotal to enabling future flexible automation workflows.

---

Introduction

Automation is revolutionizing laboratory experimentation, liberating scientists from performing tedious tasks while simultaneously accelerating the pace of research and discovery.¹ Recent advancements have given rise to the concept of Self-Driving Labs (SDLs), which combine artificial intelligence (AI) with automated robotic platforms, enabling autonomous reaction planning, execution, and analysis.^2,3 SDLs go beyond standard automation by addressing specific problems (e.g., dosing solvent when volume is below a threshold) rather than being limited to executing predetermined tasks (e.g., dosing 10 mL of solvent every 20 minutes). This approach allows for a shift from mere automation towards autonomy. To date, SDLs have primarily been applied in the realm of small organic molecules^4,5 and materials synthesis.^6,7

Workup processes aim to isolate a pure, desired product through selective separation from other reaction components, including co-product(s), excess reagent(s), impurities, and solvent. An effective workup is essential for ensuring product consistency, improving manufacturing efficiency, and maintaining cost-effectiveness, especially in large-scale synthesis and process development. The automation of workup processes poses challenges due to its specificity to the composition and physical properties of the product.⁸ Recently, SDLs have begun automating separation processes, with a particular focus on liquid–liquid separation.^9–12 Despite these advancements, a universal automated purification system does not yet exist.¹³ Consequently, many workup processes continue to be carried out manually, presenting a significant opportunity for automation development. The goal is to expand SDL capabilities to encompass a broader range of workup tasks, moving towards the realization of a universal automated workup system.

The manual execution of reaction workups can be attributed to the specialized equipment, intricate procedures, and the high degree of accuracy and precision required (e.g., achieving perfect phase splits, identifying distillation end points, etc.). Moreover, workup procedures entail a series of dynamic and complex steps (e.g., extraction and distillation) that, ideally, are monitored in real-time using process analytical technology (PAT), such as ReactIR, NMR, and HPLC. These techniques provide microscopic-level process understanding and facilitate on-the-fly adjustments although their wide-spread deployment can be space and cost prohibitive. As a result, such real-time monitoring techniques are rarely incorporated in current SDLs employed in chemical procedures. Going further, workup procedures inherently rely on macroscopic visual observations such as color changes, liquid levels, crystal formation, etc. These visual cues play a pivotal role in contextualizing experimental outcomes, aiding decision-making (e.g., determining when to proceed with the next step), and detecting outliers or defects. Current automation approaches, however, rarely utilize visual cues for decision-making. These ‘blind’ automation approaches lead to the loss of valuable data which can increase experimentation time. Therefore, it is imperative to incorporate visual observations into automating workup processes to enhance efficiency and precision within these procedures, ultimately optimizing the overall workflow in SDLs.

Computer vision (CV) allows for capturing, processing, and analyzing visual inputs (i.e., digital images). This cost-effective and non-invasive technique can be seamlessly integrated into experimental workflows, providing spatially and temporally resolved data. Traditional CV algorithms have predominantly relied on image analysis techniques based on color, grayscale, or edge detection for visual recognition and reasoning. While these algorithms excel at extracting information from visual images, they fail when confronted with unideal visual conditions, such as glare, unexpected solid formations, or changes in lighting.^9,14,15 Furthermore, classical CV systems often focus on specific tasks, reducing images to one or two quantifiable variables. However, such reductionist approaches limit the “full-scene” understanding that can be achieved. Recent advancements in CV models have showcased improved utility by incorporating real-time quantification capabilities.^16–20 Additionally, advancements in machine learning (ML) have facilitated the development of more flexible CV methods, overcoming the limitations associated with task-specific approaches. Convolutional neural networks (CNNs) in particular can achieve near-human classification accuracy when trained with large datasets, enabling comprehensive analysis of diverse objects.^21,22 Thus far, CNNs have mainly focused on identification and classification rather than variable quantification.^23,24 A CV system that combines both image analysis (i.e., quantification) and classification capabilities (via CNNs) would serve as a significant milestone toward the advancement of automated systems for chemical Research and Development (R&D).

Previously, our group has focused on bringing together flexible hardware, data-rich monitoring techniques, and CV for liquid-level monitoring and control.²⁵ The resulting system was aptly named HeinSight, reflecting its ability to provide valuable visual insights into the ongoing experiments. HeinSight relied solely on image analysis (edge detection) to monitor a single output, liquid level, within a defined region of interest. This allowed for monitoring and controlling various applications dependent on liquid level data, such as continuous preferential crystallization (CPC), continuous filtration, and evaporative solvent swap. In our ongoing efforts to advance automated process development, we now introduce HeinSight2.0—a generalized vision system that combines image analysis and classification to achieve simultaneous monitoring of multiple visual cues across the entire reactor viewing window, as shown in Fig. 1a. Operating in tandem with a flexible hardware platform, our system can monitor and, if necessary, respond in real-time to visual cues. These fundamental vision-based targets hold universal relevance across various chemical experiments, making our system adaptable to a wide range of tasks. The strength of our model lies in its flexibility, achieved through the concurrent monitoring of all visual outputs, with a sub-selection being subjected to analysis based on the experimental requirements. This approach enables the utilization of a single model for diverse experiments, with the results further bolstered by cross-validation using secondary data, thereby providing deeper experimental insights.

Fig. 1 (a) Universal vision-based outputs in workups monitored by HeinSight2.0; (b) overview of interrelated components of HeinSight2.0: classification outputs (from CNN), quantification outputs (from image analysis), and process variables from iControl; (c) integrations and applications of HeinSight2.0 CV system.

The core automated lab reactor (ALR) module utilized in this study was the EasyMax™ 102, manufactured by Mettler Toledo.²⁶ This hardware platform incorporates iControl software (version 6.1) control with an integrated data environment, enabling the monitoring, recording, and controlling of various process variables.²⁷ By aggregating real-time data from the ALR (e.g., reactor temperature, stir rate, dosed solvent volume, etc.) and the HeinSight2.0 model (e.g., reactor volume, turbidity, etc.), comprehensive data streams are simultaneously generated for immediate decision-making (Fig. 1b). Furthermore, the ALR's viewing window allows for simultaneous image capture, providing qualitative measurements that cross-validate the quantifiable trends being monitored. Overall, the integration of HeinSight2.0 with iControl enables unsupervised automation through visual feedback control.

Within HeinSight2.0, CNN is employed for object boundary determination to enable classification and yield the following outputs: homogeneity, solid formation, and the presence of solid residue above the liquid level. Subsequently, image analysis techniques are utilized to derive additional visual outputs from the identified bounding boxes and segments, including volume, color, and turbidity (refer to the ESI† for further details). By combining iControl and computer vision analysis, our approach offers a macroscopic complementary analysis to microscopic in situ PAT tools and mathematical models, thereby increasing data richness from each experiment. This integration of visual data reporting within iControl also represents a crucial step towards standardized data collection, sharing, and repositories, as demonstrated by the facile technology transfer between the Hein Lab and Pfizer (Fig. 1c).

Throughout this paper, multiple visual cues were analyzed over time during each of our case studies. We have selected a single visual output depending on each specific application (the “primary output”) and corroborated its results with other collected outputs (“secondary outputs”). Liquid level monitoring was used to monitor an evaporative solvent exchange distillation and an antisolvent crystallization, employing automated feedback to maintain constant reactor volumes. The importance of monitoring multiple outputs was then demonstrated by monitoring an evaporative crystallization. Automated feedback was again demonstrated by performing a cooling crystallization while monitoring the solution homogeneity to inform heating/cooling changes. Turbidity was used to measure solid–liquid settling kinetics, while the presence of a precipitate (“solid”) was used to determine the agitation speed necessary for uniform suspension of solid. Lastly, multi-phase liquid detection was demonstrated by measuring the settling time in liquid–liquid extraction.

Experimental setup

Experiments were performed in a Mettler-Toledo EasyMax™ 102 using a 100 mL glass reactor equipped with an overhead stirrer, temperature probe, and SP-50 dosing unit, as shown in Fig. 2. Execution of experimental events (e.g., dose solvent, stir reaction, change temperature) and recoding of process variables (e.g., reaction temperature, rate of solvent dose) are handled via iControl running on an adjacent laptop. The Razer Kiyo Streaming Webcam was chosen for its affordability and commercial availability. It offers high-quality 1080p resolution at a rapid image acquisition of 30 FPS and features a built-in adjustable brightness ring light for illumination. This specific camera model was consistently used for all data collection, including training, testing, and model inference to prevent any potential domain shift that could degrade model's performance. The webcam was placed inside a 3D-printed enclosure that served to hold the camera in a fixed location, align the camera with the reactor window and block peripheral light to provide consistent illumination during experimental operation (see ESI† for enclosure 3D-print files and assembly guide). This measure was crucial because CV systems can be significantly impacted by changes in lighting conditions. While data augmentation techniques can help to some extent by introducing variety into the training data (e.g., by adjusting brightness levels), they might not cover all possible lighting scenarios. Additionally, lighting changes can create shadows, highlights, glare, and other variations that are often challenging for computer vision algorithms to interpret accurately. A such, standardizing lighting conditions, achieved through the enclosure and webcam, was essential for accurate system performance. The HeinSight2.0 model was run to provide real-time image analysis and data extraction. Data acquired by our model was imported live into iControl via an ExcelSheet (iC Data Share configuration) for trend visualization and, where necessary, real-time feedback for workflow control.

Fig. 2 Experimental workflow setup of HeinSight 2.0, illustrating the integration of EasyMax hardware components (dosing unit, overhead stirrer, temperature probe, and reaction vessel), webcam (within an enclosure), and software components ((CV model and iControl) with real-time trend visualization).

Model development

Model choice and structure

There are two methods for identifying and classifying regions of interest in images (e.g., solids, solution, residue): instance segmentation and semantic segmentation. We chose to use an instance segmentation CNN model due to its superior capability to distinguish regions of the same class (e.g., recognizing different homogenous liquid layers as would be present in liquid–liquid extraction). By utilizing an instance segmentation approach, we can effectively separate the tasks of segmentation (identifying object regions) from classification (categorizing objects), thereby enabling segmentation for objects beyond the training set. To achieve this instance segmentation, we implemented a Mask Region-based CNN²⁸ (Mask R-CNN) architecture, the leading method for object instance segmentation,²⁹ using a pre-trained model (R101-FPN) from the detectron2 library.³⁰

Model training

The model was trained on the COCO (Common Objects in Context) dataset,³¹ a vast resource for object detection and segmentation. While it provided a general understanding of visual elements, it however lacked contextualized images of chemical reactions. To address this gap, we fine-tuned the model using images from our setup, capturing various chemical and physical processes in videos. These images were manually annotated through the SuperAnnotate online tool.³² See ESI† for further details on model development and training.

We provide the HeinSight2.0 pre-trained model along with a step-by-step guide for easy use in the SI. The open-access HeinSight2.0 Python package enables users, even those without coding experience, to leverage the model for both single and multi-outputs tasks outlined in this study. It also offers a seamless entry point for transfer learning, making it ideal for exploring other applications within the chemistry domain by starting with lower initial data thresholds.

Results and discussion

Our focus was to enable the autonomous execution of crucial synthetic steps, particularly emphasizing workup, and isolation methods such as distillation, crystallization, and liquid–liquid extraction. Aside from column chromatography, which is undesirable in process chemistry, the above-mentioned methods encompass the primary unit operations involved in compound isolation after a synthetic operation. However, their automation is hindered by the aforementioned challenges discussed in the introduction. To showcase the performance and adaptability of HeinSight2.0, we have chosen a series of progressively challenging case studies for each of these key isolation strategies.

Case study #1: liquid level (solvent exchange distillation)

Distillation is commonly used to concentrate a reaction by removing volatile solvents or reagents.³³ Usually, operators visually check the vessel's fill level to determine the endpoints of distillation. Stopping the process too early can impact subsequent purification due to high levels of remaining volatile components, while allowing it to continue for too long poses safety risks of overheating the vessel and decomposing the target compounds. As such, accurate control of the vessel level and heating would be highly beneficial.

A more complex, but related unit operation involves distillation paired with the addition of a new working solvent to achieve solvent exchange,³⁴ as shown in Fig. 3a. This process is deployed when subsequent processing steps require a different solvent, either to carry out a new synthetic transformation or to drive crystallization and isolation of the target molecule. Proper management of distillation and solvent dosing rates is crucial to avoid overfilling or underfilling the vessel.

Fig. 3 (a) Illustration of the experimental setup of solvent exchange distillation. (b) Solvent exchange distillation for a reaction slurry containing a mixture of EDCI–HCl, and triethylamine in acetone, charging 10 mL of butanone when the threshold volume is reached. (c) Constant volume antisolvent crystallization via solvent exchange distillation of acetaminophen in MeCN. For both (b) and (c), the internal temperature of the reactor (red) was captured via the EasyMax temperature probe, while the reactor volume (purple) was returned from the CV system. The apparent spread in volume recorded by the CV system was due to variations in the liquid height due to fluctuation from mechanical agitation and boiling, which both changed the 2D cross sections presented to the camera; see ESI† for detailed measurement and control workflow. The volume trend data were visually emphasized by adding manual markers to guide the eye.

We selected two process examples that present realistic scenarios requiring visual control and posing challenges to our CV system due to additional visual elements. The first example was the slurry solvent exchange distillation (Fig. 3b), which involved solid residue formation and adherence to the reactor wall. The second example was constant volume antisolvent crystallization (Fig. 3c), which included the transformation from a homogeneous to a heterogeneous system, solid precipitating out of solution, and residue adhering to the reactor wall. Earlier CV systems, which are often based on image analysis by edge detection, fail to accurately capture the reactor liquid level in the presence of complex and changing image elements (e.g., glare and partial obstruction of view caused by solid residue buildup). Therefore, these tests were conducted to demonstrate HeinSight2.0's ability to adapt to complex dynamic changes in the experiment.

Initially, we honed the system's ability to maintain a minimum reactor volume for simple solvent exchange conditions without complex visual elements (see ESI† for details). Our first challenge involved a solvent exchange distillation from a slurry of EDCI·HCl and triethylamine in acetone, mimicking a typical endpoint in a carbodiimide-promoted amide coupling reaction (Fig. 3b). Acetone was distilled from the slurry, and butanone was added to enable solvent exchange for product isolation. The CV system recorded the volume and automatically introduced butanone when the reactor volume dropped below a specified threshold, ensuring the maintenance of a minimum reactor volume.

This example demonstrated that HeinSight2.0 could serve as a visual PAT for monitoring and controlling distillation without human oversight. Both the primary output (volume) and the secondary output (temperature) provided valuable information. The goal of maintaining a threshold volume was successfully achieved, as evidenced by the visual record and reactor volume trend. The rate of distillation was determined from the slope of the volume trend, which progressively decreased as the composition shifted in favor of the higher boiling butanone. HeinSight2.0 was also able to adapt to dynamic changes, adjusting to the non-linear dosing times required as the composition of the solvent system changed. Adapting to such nonlinear events is challenging without real-time feedback, usually resulting in overfilling or drying of the reactor.

We next attempted another challenging example by coupling distillation with antisolvent crystallization, as illustrated in Fig. 3c. This workflow is closely related to solvent exchange distillation and is commonly used to isolate products at the end of a synthetic sequence.³⁵ Here, the addition of the makeup solvent led to supersaturation, resulting in nucleation and crystallization as the distillation progressed. This posed a complex visual challenge for the CV system due to deposition of solid residue and solid precipitating. Once again, our primary output (volume) and secondary output (temperature) provided the main factors for reactor control and integration. Additionally, solid formation was critical to inform the onset of nucleation.

A solution of 0.17 M of Acetaminophen in 50 mL acetonitrile (MeCN) was heated to begin distillation (Fig. 3c). A charge of toluene via the dosing pump was added when HeinSight2.0 detected that the threshold volume was reached. Unlike the previous example, the rate of distillation did not appear to change as significantly during the first few solvent recharges. Throughout, HeinSight2.0 effectively monitored the reactor volume even though solid and residue began to form and obstruct the viewing window. A dramatic change in the rate of distillation occurred at the fifth dose of toluene, leading to rapid crystallization of acetaminophen. Assuming the distillate likely contained mostly MeCN, this point also corresponded to when the solvents had mostly exchanged and the solution became toluene-rich, thus driving crystallization due to the reduced solubility of acetaminophen in toluene. HeinSight2.0 correctly detected the onset of crystallization at the fifth dose through the appearance of “solid” class, as shown in image 4 of Fig. 3c. For clarity the solid trend was not shown in the figure and replaced with a pink shade. The increase in temperature rate at the fifth dose further supported the point of nearly complete solvent exchange.

Through these tests, our non-invasive, flexible CV system has proven its ability to provide autonomous oversight for solvent exchange distillation processes. HeinSight2.0 represents a significant advancement over its predecessor (HeinSight), integrating image analysis and classification technology. This integration expands its capabilities beyond liquid level monitoring to include variables like solution homogeneity, solid presence, and residue formation. It also enhances accuracy, allowing it to determine liquid levels even in the presence of additional visual cues like solids and residues, which posed challenges for the previous model. While we demonstrated our system ability via a solvent exchange distillation, however, the underlying principle of controlling the liquid level can be exploited in other unit operations requiring liquid level oversight.

Case study #2: homogeneity and turbidity (crystallization)

Crystallization stands as one of the most extensively employed techniques for achieving purification and isolation of target compounds. To successfully carry out a crystallization, it is crucial to have detailed knowledge of key phase behavior and physical properties, including solubility and the primary nucleation limit. Determining the metastable zone width (MSZW), which denotes the point at which the free energy barrier for crystal nucleation is surpassed, can prove to be a challenging and time-consuming task. While solubility measurements and modelling can offer some estimations, even slight variations in process variables such as stirring rate, reactor geometry, or impurity profile can significantly impact the likelihood of primary nucleation. Hence, prior to embarking on compound isolation through crystallization, it is often necessary to directly investigate the MSZW within the intended reactor system to ascertain the reliability of the crystallization process.

One simple method to determine the metastability limit at a constant temperature, which can be below the boiling point of the solvent, involves driving supersaturation through the evaporation of solvent with a gas purge of the reactor headspace. Unlike the previous examples involving distillation, real-time feedback was not necessary to perform the evaporative crystallization of acetaminophen, as data extraction could be completed at the experiment's completion. However, the nucleation of solids within the solution, deposition of residue on flask walls, and changing reactor volume all poses a challenging test case for HeinSight2.0 with multiple changing visual elements, as shown in Fig. 4a.

Fig. 4 (a) Illustration of the experimental setup for evaporative crystallization of acetaminophen in MeCN. (b) Monitoring evaporation of MeCN over time to observe acetaminophen nucleation. The reactor volume (purple), turbidity (yellow), and solid (pink) were captured by the CV system. (c) Illustration of the experimental setup for cooling crystallization case study. (d) Feedback loop using real-time data to determine solubility and MSZW of acetaminophen in MeCN. The internal temperature of the reactor (red) was captured via the EasyMax temperature probe, while the turbidity (orange) and homogeneity (purple and yellow) were returned from the HeinSight2.0 model.

In this experiment, the jacket temperature was maintained at 30 °C while a stream of compressed air was blown overhead of a solution of 0.13 M of acetaminophen in 50 mL MeCN. As the flow of compressed air was kept constant, we anticipated that the rate of evaporation would be invariant over most of the process. The CV system initially indicated a steady evaporation rate of 0.77 mL min⁻¹, as evidenced by the volume trend shown in Fig. 4b. However, once the volume decreased below 23.5 mL, there was a significant increase in the evaporation rate, reaching a constant 6.75 mL min⁻¹ until the solvent was completely evaporated. In addition, the formation of a solid residue on the reactor walls (“residue”) and a solid precipitate in the solution (“solid”) was captured qualitatively (via images) and quantitatively (via trends) by the CV system. The precipitate formation, indicated by an increase in the “solid” trend, was observed at 50 minutes when the reactor volume dropped below 26 mL, corresponding to the nucleation point at a constant temperature.

Furthermore, the total dryness was observed at 69 min when the volume of the solution and its turbidity value dropped to zero. Repeating the experiment resulted in a similar evaporation rate and volume for the onset of precipitation, supporting the estimation of the metastable limit for our system (see ESI† for details). More interestingly, HeinSight2.0 was able to discriminate between the crystals formed in the supernatant solution (i.e., solid) vs. the solids building up on the side of the flask (i.e., residue). This was possible due to the non-maximum suppression (NMS) applied to the CNN algorithm.³⁶ In cases where there are multiple overlapping instances in the same region of interest, NMS enables the label with the highest confidence value to receive the score while the remaining instances are suppressed, see ESI† for details. By ensuring that only the most confident label is retained for each region of interest, labelling accuracy is improved.

The dynamic variations in these visual elements serve to highlight the capability of HeinSight2.0 to simultaneously monitor multiple outputs. These changes in visual characteristics offer valuable insights into the experiment, shedding light on the underlying physical properties directly from the intended reactor system. This demonstrates the effectiveness of our simple and inexpensive approach to process monitoring.

We next applied HeinSight2.0 to monitor and control a cooling crystallization. This technique is by far the most applied means of driving supersaturation to isolate purified crystalline products. MSZW measurements on the actual process reactor are usually performed using PAT tools such as spectroscopic (FTIR,³⁷ Raman³⁸), or imaging (FBRM,³⁹ PVM⁴⁰). While these tools provide accurate in situ microscopic measurements of the solution and/or solid phase of crystallizations, they are often invasive, expensive, and specialized. Our non-invasive and cost-effective CV system has the potential to provide complementary macroscopic measurements by simply visually interrogating any reactor of interest, serving a broad range of applications.

By combining our feedback control based on visual cues with the temperature control of an EasyMax, we sought to perform temperature cycles on a solution of acetaminophen in MeCN to measure its solubility and MSZW, as depicted in Fig. 4c. The binary homogeneity determination was the primary output used by HeinSight2.0 to detect the onset of nucleation and dissolution (via cloud point and clear point determination, respectively). The quantitative turbidity trend served as a secondary output to corroborate the data. The results of the cooling crystallization are demonstrated in Fig. 4d.

Starting with a 0.31 M acetaminophen solution in 40 mL of MeCN at 50 °C, the EasyMax jacket was cooled at a constant rate (10 °C min⁻¹) until a heterogeneous solution was detected via the homogeneity variable. After waiting for 5 min, the jacket was then heated at the same rate until dissolution was detected. After waiting another 5 min, the cycle was then repeated to verify the solubility and MSZW measurements. The results showed that the clear point of the solution remained constant at 30 °C regardless of the cycling event, consistent with the fact that it is a thermodynamic property. However, the cloud point, which is a kinetic property, slightly changed with cycling at 13 °C for the first cycle and 15 °C for the second cycle. This observation was further supported by the changes in the steady turbidity trend following each nucleation event. Additionally, qualitative analysis (images 3 and 7) demonstrated disparities in the brightness of the crystals, further emphasizing the differences in the turbidity measurements. The reported turbidity value was calculated from the 2D cross section captured by the camera, so the perceived opacity of the solution affected the numerical result. Stochastic factors like number of nucleation events and material sticking to the camera's sight region could also influence the absolute value. Thus, binary delineation of homogeneity served as the primary output for feedback control during the transition between clear and cloudy states.

When the experiment was repeated at a different cooling rate (3 °C min⁻¹), the cloud point changed as a function of the cooling rate (13 °C at 10 °C min⁻¹ and 21 °C at 3 °C min⁻¹) while the clear point remained constant (30 °C at both 10 °C min⁻¹ and 3 °C min⁻¹), supporting the idea that the clear point is deterministic while the cloud point is stochastic,^41–43 see ESI† for details. The automated feedback control employed in this experiment allowed for rapid, autonomous solubility and MSZW measurements like existing clear point/cloud point approaches (e.g., Crystal16®). We envision that by adding a dosing unit to the existing workflow, automated solubility and MSZW curves could be obtained across a range of concentrations and temperatures.

Case study #3: solid and turbidity (solid–liquid mixing)

Solid–liquid mixing plays a crucial role in various industrial processes, including crystallization, heterogeneous reactions, and suspension polymerization.⁴⁴ Understanding mechanical mixing behavior can be critical for either maintaining uniform mixing (“effective agitation”) or settling solids to remove them from a liquid suspension (“setting kinetics”), as shown in Fig. 5a. It is key to understand the sensitivity of mixing performance on the process to assure successful transfer across different scale-up vessels. Assessing solid–liquid mixing through experimental means, such as measuring cloud height,⁴⁵ axial concentration gradient,⁴⁶ or particle velocity,⁴⁷ can be time-consuming, costly, and impractical for large reactors. In industrial settings, computational tools like VisiMix and Dynochem are utilized, employing mathematical equations derived from reactor databases to evaluate mixing performance at different scales. However, these simulations require extensive input data, including geometry details, boundary conditions, and material properties, making data collection a time-consuming and resource-intensive process. For specific mixing challenges, Computational Fluid Dynamics (CFD) modelling offers a more in-depth understanding of the process.⁴⁸ Given the inherently visual nature of determining solution homogeneity, we sought to demonstrate how HeinSight2.0 could readily inform mechanical mixing studies.

Fig. 5 (a) Illustration of the experimental setup for agitation case studies. (b) Monitoring the presence of solids to determine the minimum stirring speed for effective agitation of a slurry of acetaminophen in MeCN. The stir rate of the reactor (grey) was captured via the EasyMax overhead stirrer, while the turbidity (orange) and solid (pink) were returned from the CV system. (c) Monitoring turbidity to determine the settling time for a slurry of acetaminophen in MeCN.

We selected solid detection and turbidity measurements as our system's primary outputs for evaluating mixing efficiency with mechanical agitation. As a first test, a supersaturated solution of acetaminophen (0.46 M) was prepared in 50 mL MeCN at room temperature without stirring (Fig. 5b). The stir rate was increased by 20 rpm every three min to find the minimum effective agitation speed while monitoring turbidity and the presence of solids at the bottom of the reactor. While no solid suspension could be observed below 160 rpm, the onset of solid suspension was visible from 160 to 200 rpm (shown in the turbidity trend). Interestingly, although turbidity reached its maximum value upon stirring at 200 rpm, solids were still detected at the bottom of the flask which did not disappear until 220 rpm, highlighting the importance of analyzing multiple outputs. Other secondary outputs supported this analysis, including homogeneous solution identification and solution color (see ESI† for details).

The opposite experiment was carried out to determine settling time, with a stirred slurry of 0.48 M acetaminophen in 50 mL MeCN at room temperature being monitored for the change in turbidity (Fig. 5c). The effect of various stir rates (800, 600, and 400 rpm) on the time to reach complete solid settling was investigated. As expected, lower stir rates gave shorter settling times, and our system was able to detect this change mainly from the turbidity output and was supported by other indices (presence of solid and homogeneity – see ESI† for details). The success of our system's multi-outputs monitoring illustrates its ease of plug-and-play implementation, capturing the global solution behavior in a non-invasive means while remaining simple to implement.

Case study #4: multiple-phase detection (liquid–liquid extraction)

Liquid–liquid extraction (LLE) is a common unit operation in synthetic isolation owing to its scalability and cost-effectiveness that arises from the preferential solubility of target compounds in two immiscible liquid phases. However, automated LLE is challenging due to complex mixtures and potential issues such as emulsions formation, stable dispersions, and rag layers that increase separation times and slow process scale-up. As such, designing and optimizing LLE is a labor-intensive trial-and-error process, requiring extensive experimentation to determine conditions selections (e.g., solvent identity).

Evaluating phase separation, which encompasses separation time, emulsion formation, and phase split quality, is typically a dynamic process requiring visual comparison over a timescale. As such, imaging tools have been employed to classify liquid–liquid biphasic separation. Some researchers have determined the phase separation by monitoring the position of a colored float.⁴⁹ However, this approach may be prone to issues related to chemical incompatibility of the float with particular solvent combinations, or variations in float buoyancy over a wide range of solution concentrations. Non-invasive edge detection has been used to determine biphasic separation but can fail when the solutions become colored or turbid. Recently, robust high throughput image analysis algorithms have been developed to specifically automate LLE workflow.⁵⁰

Our CV model's multi-outputs approach can be used to extract quantitative and qualitative features from multiple layers in a LLE system, distinguishing between homogeneous and heterogeneous regions, and accurately measuring the volume of each layer over time, as shown in Fig. 6a. This provides critical process information such as separation time, agitation effectiveness for phase mixing, and phase split quality.

Fig. 6 (a) Illustration of settling kinetic behavior of the separation of two immiscible solutions. (b) Monitoring liquid level to determine the separation time of a mixture of DCM and water. The stir rate of the reactor (grey) was captured via the EasyMax overhead stirrer, while the volumes 1 and 2 (purple and pink) were returned from the CV system. (c) Scheme of (–)TBZ and (–)CSA sodium salt separation in DCM and water. (d) Monitoring the liquid level to determine the separation time of a mixture of (–)TBZ and (–)CSA sodium salt in DCM and water.

A well-agitated mixture of immiscible liquids comprising a specific volume of water (20 mL), brine (5 mL), and DCM (35 mL) containing citric acid, was automatically monitored to observe changes in the volume of each phase as agitation was halted, as shown in Fig. 6b. The effect of a 1000 rpm stir rate on the time required for the mixture to reach full separation was investigated. The separation time of 0.18 min was primarily determined based on the volume output of each phase, which was further supported by other secondary outputs, including color and turbidity (refer to the ESI† for detailed information). In comparison to mechanical or density-based colored sensors commonly used for image analysis in LLE, our non-invasive approach can accommodate various chemical compositions.

A more challenging example was realized by applying our liquid–liquid extraction protocol to monitor a salt break and purification of enantiopure tetrabenazine (TBZ) as its amine-free base. This workflow is a critical intermediate step in the synthesis of Valbenazine drug,⁵¹ and follows a crystallization-induced asymmetric transformation (CIAT) to convert racemic TBZ to the desired enantiomer, which is isolated as the pure (–)TBZ(–)CSA salt.⁵² Advancing the synthesis requires charging a reactor with isolated crystals, suspension in a mixture of DCM and water while charging sodium hydroxide to neutralize the acid, then recovery of the free amine from the organic layer, as shown in Fig. 6c. However, this seemingly simple procedure raises several challenges. To optimize the process mass intensity (PMI) of the overall route a minimum volume of working solvents is desired, forcing the extraction to operate at a very high concentration. The amphiphilic nature of both camphorsulfonic acid and TBZ increases the micelle forming potential of the solvents and promotes emulsion. Reducing the agitation can partially mitigate this effect but compromises efficient mass transfer leading to very long processing times. Thus, we need to identify the highest possible agitation rate that still allows facile phase separation without creating an emulsion and effectively neutralize the acid.

The process involves charging (–)TBZ(–)CSA salt into the reactor (1.557 g) followed by a minimum of DCM to create a homogenous solution. A NaOH solution (0.5 M) was added followed by water to give an equivalent volume to the organic layer. This biphasic mixture was agitated for 10 min, ensuring both phases are well mixed as demonstrated by having a single homogenous layer (Fig. 6d). Stirring was then arrested and the phases were allowed to separate, allowing the separation kinetics to be resolved. This cycle was repeated, increasing the rate of stirring, which resulted in progressively longer settling times with higher agitation rates. Stirring at 250–350 rpm was found to be optimal as the mixture displayed reliable and rapid phase split. An agitation rate of 450 rpm led to a longer phase split time. While agitation 650 rpm and higher had a stable emulsion that would not separate without intervention.

Summary

HeinSight2.0 showcases its agility through multi-outputs detection. Treating each visual feature as an independent task enables the seamless mixing and matching of available visual cues, as illustrated in Fig. 7. This flexibility facilitates effortless reconfiguration, allowing a single CV system to pivot seamlessly between diverse applications outlined above. Leveraging modularity, we establish a versatile CV system for workup processes, where the primary output forms the backbone, while other outputs complement results and offer in-depth mechanistic insights.

Fig. 7 Conceptual illustration of how multi-outputs monitoring enables modular automation for diverse workflows.

Conclusions

In summary, HeinSight2.0 represents a significant advancement over its predecessor, seamlessly integrating image analysis and CNN technology to enable the simultaneous detection and quantification of various events within the EasyMax 102 automated lab reactor. Through integration with iControl, this CV system not only facilitates holistic observations over time but also enables triggered operations based on visual cues.

Currently, we are enhancing HeinSight2.0's adaptability by employing transfer learning to extend its capabilities to various multi-scale reactor types and enable high-throughput experimentation. Our vision is for this system to seamlessly integrate with PAT tools and mathematical models, enhancing process intensification technology with visual observations. Looking ahead, we anticipate integrating HeinSight2.0 CV system with other AI tools like large language models (e.g., ChatGPT) and voice activation, creating a framework that combines inputs from various modalities to solve complex AI problems. This collaborative ecosystem, facilitated by Hugging Face,⁴⁷ will enable the integration of diverse AIs, streamlining the development and optimization of chemical processes and leading to a SDL for workup processes.

Author contributions

Rama El-khawaldeh: conceptualization, methodology, validation, investigation, data curation, visualization, writing (original draft). Mason Guy: investigation, data curation. Finn Bork: conceptualization, validation, data curation. Nina Taherimakhsousi: methodology, data curation. Kris N. Jones: conceptualisation, resources, writing (review and editing). Joel M. Hawkins: conceptualisation, resources, writing (review and editing). Lu Han: data curation, validation, writing (review and editing). Robert P. Pritchard: data curation, validation. Blaine A. Cole: data curation. Sebastien Monfette: conceptualisation, supervision, resources, writing (review and editing), project administration; validation. Jason E. Hein: conceptualisation, formal analysis, resources, data curation, writing (review and editing), supervision, project administration, funding acquisition.

Data availability

HeinSight2.0 code is available at: https://gitlab.com/heingroup/heinsightv2. HeinSight2.0's model, training/testing data, and 3D enclosure.stl files are found at https://drive.google.com/drive/folders/1M1zpiaThiVlq9--AjoLQ_K6QHCkjoXjb.

Conflicts of interest

There are no conflicts to declare.

Acknowledgements

The authors gratefully acknowledge Ms Paloma Prieto, Dr Andrew J. Kukor, and Dr Joshua S. Derasp for guidance and conversation in the preparation of this manuscript. The following Pfizer colleagues are also acknowledged for useful discussions and assistance with the internalization of the CV system: Russell Algera, Elvis Eugene, Sheri Hedquist, Nahian Khan, Carolyn Mastriano, Jason Mustakis, Kakasahed Nandiwale, Giselle Reyes, and Chase Salazar. The authors further thank Mettler-Toledo Autochem for their generous donation of process analytical equipment (Easy Max 102). Student fellowships were provided by the Natural Sciences and Engineering Research Council of Canada (CGS-M & CGS-D to R. E.; PGS-D to M. G.) and the University of British Columbia (Killam Doctoral Scholarship to R. E.). Research support for this work was provided by Pfizer Global Research, the University of British Columbia, the Canada Foundation for Innovation (CFI-35883) and the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2021-03168, Discovery Accelerator Supplement).

Notes and references

1. K. Thurow and S. Junginger, Devices and Systems for Laboratory Automation, Wiley, 1st edn, 2022,  DOI:10.1002/9783527829446.
2. M. Abolhasani and E. Kumacheva, The Rise of Self-Driving Labs in Chemical and Materials Sciences, Nat. Synth., 2023, 2(6), 483–492,  DOI:10.1038/s44160-022-00231-0.
3. A. Vriza, H. Chan and J. Xu, Self-Driving Laboratory for Polymer Electronics, Chem. Mater., 2023, 35(8), 3046–3056,  DOI:10.1021/acs.chemmater.2c03593.
4. M. Christensen, L. P. E. Yunker, F. Adedeji, F. Häse, L. M. Roch, T. Gensch, G. dos Passos Gomes, T. Zepel, M. S. Sigman, A. Aspuru-Guzik and J. E. Hein, Data-Science Driven Autonomous Process Optimization, Commun. Chem., 2021, 4(1), 112,  DOI:10.1038/s42004-021-00550-x.
5. B. Burger, P. M. Maffettone, V. V. Gusev, C. M. Aitchison, Y. Bai, X. Wang, X. Li, B. M. Alston, B. Li, R. Clowes, N. Rankin, B. Harris, R. S. Sprick and A. I. Cooper, A Mobile Robotic Chemist, Nature, 2020, 583(7815), 237–241,  DOI:10.1038/s41586-020-2442-2.
6. B. P. MacLeod, F. G. L. Parlane, T. D. Morrissey, F. Häse, L. M. Roch, K. E. Dettelbach, R. Moreira, L. P. E. Yunker, M. B. Rooney, J. R. Deeth, V. Lai, G. J. Ng, H. Situ, R. H. Zhang, M. S. Elliott, T. H. Haley, D. J. Dvorak, A. Aspuru-Guzik, J. E. Hein and C. P. Berlinguette, Self-Driving Laboratory for Accelerated Discovery of Thin-Film Materials, Sci. Adv., 2020, 6(20), eaaz8867,  DOI:10.1126/sciadv.aaz8867.
7. K. Abdel-Latif, R. W. Epps, F. Bateni, S. Han, K. G. Reyes and M. Abolhasani, Self-Driven Multistep Quantum Dot Synthesis Enabled by Autonomous Robotic Experimentation in Flow, Adv. Intell. Syst., 2021, 3(2), 2000245,  DOI:10.1002/aisy.202000245.
8. C. P. Breen, A. M. K. Nambiar, T. F. Jamison and K. F. Jensen, Ready, Set, Flow! Automated Continuous Synthesis and Optimization, Trends Chem., 2021, 3(5), 373–386,  DOI:10.1016/j.trechm.2021.02.005.
9. M. O'Brien, P. Koos, D. L. Browne and S. V. Ley, A Prototype Continuous-Flow Liquid–Liquid Extraction System Using Open-Source Technology, Org. Biomol. Chem., 2012, 10(35), 7031,  10.1039/c2ob25912e.
10. A.-C. Bédard, A. Adamo, K. C. Aroh, M. G. Russell, A. A. Bedermann, J. Torosian, B. Yue, K. F. Jensen and T. F. Jamison, Reconfigurable System for Automated Optimization of Diverse Chemical Reactions, Science, 2018, 361(6408), 1220–1225,  DOI:10.1126/science.aat0650.
11. A. D. Clayton, A. M. Schweidtmann, G. Clemens, J. A. Manson, C. J. Taylor, C. G. Niño, T. W. Chamberlain, N. Kapur, A. J. Blacker, A. A. Lapkin and R. A. Bourne, Automated Self-Optimisation of Multi-Step Reaction and Separation Processes Using Machine Learning, Chem. Eng. J., 2020, 384, 123340,  DOI:10.1016/j.cej.2019.123340.
12. S. Steiner, J. Wolf, S. Glatzel, A. Andreou, J. M. Granda, G. Keenan, T. Hinkley, G. Aragon-Camarasa, P. J. Kitson, D. Angelone and L. Cronin, Organic Synthesis in a Modular Robotic System Driven by a Chemical Programming Language, Science, 2019, 363(6423), eaav2211,  DOI:10.1126/science.aav2211.
13. W. Gao, P. Raghavan and C. W. Coley, Autonomous Platforms for Data-Driven Organic Synthesis, Nat. Commun., 2022, 13(1), 1075,  DOI:10.1038/s41467-022-28736-4.
14. S. Eppel and T. Kachman, Computer Vision-Based Recognition of Liquid Surfaces and Phase Boundaries in Transparent Vessels, with Emphasis on Chemistry Applications, 2014.
15. Q. Liu, B. Chu, J. Peng and S. Tang, A Visual Measurement of Water Content of Crude Oil Based on Image Grayscale Accumulated Value Difference, Sensors, 2019, 19(13), 2963,  DOI:10.3390/s19132963.
16. H. Barrington, A. Dickinson, J. McGuire, C. Yan and M. Reid, Computer Vision for Kinetic Analysis of Lab- and Process-Scale Mixing Phenomena, Org. Process Res. Dev., 2022, 26(11), 3073–3088,  DOI:10.1021/acs.oprd.2c00216.
17. C. Yan, M. Cowie, C. Howcutt, K. Wheelhouse, N. Hodnett, M. Kollie, M. Gildea, M. Goodfellow and M. Reid, Computer Vision for Understanding Catalyst Degradation Kinetics, 2022,  DOI:10.26434/chemrxiv-2022-n0wf3.
18. C. Yan, C. Fyfe, C. Jamieson and M. Reid, Computer Vision as a New Paradigm for Monitoring of Solution and Solid Phase Peptide Synthesis, ChemRxiv, 2023, preprint, DOI:  DOI:10.26434/chemrxiv-2023-tp5n9.
19. N. Bugeja, C. Oliver, N. McGrath, J. McGuire, C. Yan, F. Carlysle-Davies and M. Reid, Teaching Old Presumptive Tests New Digital Tricks with Computer Vision for Forensic Applications, Digital Discovery, 2023, 2(4), 1143–1151,  10.1039/d3dd00066d.
20. S. Smolders, H. Sheng, M. P. Mower, A. Potdar and J. Dijkmans, Automated Image-Based Color Analysis as an Accessible and Widely Applicable PAT Tool.
21. K. He, X. Zhang, S. Ren and J. Sun, Deep Residual Learning for Image Recognition, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE: Las Vegas, NV, USA, 2016, pp. 770–778,  DOI:10.1109/CVPR.2016.90.
22. A. Krizhevsky, I. Sutskever and G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Commun. ACM, 2017, 60(6), 84–90,  DOI:10.1145/3065386.
23. N. Taherimakhsousi, B. P. MacLeod, F. G. L. Parlane, T. D. Morrissey, E. P. Booker, K. E. Dettelbach and C. P. Berlinguette, Quantifying Defects in Thin Films Using Machine Vision, npj Comput. Mater., 2020, 6(1), 111,  DOI:10.1038/s41524-020-00380-w.
24. N. Taherimakhsousi, M. Fievez, B. P. MacLeod, E. P. Booker, E. Fayard, M. Matheron, M. Manceau, S. Cros, S. Berson and C. P. Berlinguette, A Machine Vision Tool for Facilitating the Optimization of Large-Area Perovskite Photovoltaics, npj Comput. Mater., 2021, 7(1), 190,  DOI:10.1038/s41524-021-00657-8.
25. T. Zepel, V. Lai, L. P. E. Yunker and J. E. Hein, Automated Liquid-Level Monitoring and Control Using Computer Vision, ChemRxiv, 2020. DOI:  DOI:10.26434/chemrxiv.12798143.v1.
26. M.-T. I. I. reserved, all rights. EasyMax 102 Advanced Thermostat system, https://www.mt.com/gb/en/home/products/L1_AutochemProducts/chemical-synthesis-reactor-systems/EasyMax-Synthesis-Reactor.html, accessed 2023-02-12.
27. M.-T. I. I. reserved, all rights. iControl, https://www.mt.com/ca/en/home/products/L1_AutochemProducts/automated-reactor-in-situ-analysis-software/icontrol.html, accessed 2023-11-22.
28. K. He, G. Gkioxari, P. Dollár, R. Girshick and R.-C. N. N. Mask, preprint, arXiv January 24, 2018, arXiv:1703.06870, accessed 2023-03-31.
29. S. Eppel and A. Aspuru-Guzik, Generator Evaluator-Selector Net for Panoptic Image Segmentation and Splitting Unfamiliar Objects into Parts.
30. Benchmarks—detectron2 0.6 documentation, https://detectron2.readthedocs.io/en/latest/notes/benchmarks.html, accessed 2023-03-31.
31. T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick and P. Dollár, Microsoft COCO: Common Objects in Context, arXiv, February 20, 2015, preprint, arXiv:1405.0312, accessed 2023-03-31.
32. The ultimate training data platform for AI|SuperAnnotate, https://www.superannotate.com/, accessed 2023-03-31.
33. R. F. Wilcox, Distillation in the Pharmaceutical Industry, in Handbook of Downstream Processing, ed. Goldberg, E., Springer Netherlands, Dordrecht, 1997, pp 417–455,  DOI:10.1007/978-94-009-1563-3_16.
34. Y.-E. Li, Y. Yang, V. Kalthod and S. M. Tyler, Optimization of Solvent Chasing in API Manufacturing Process: Constant Volume Distillation, Org. Process Res. Dev., 2009, 13(1), 73–77,  DOI:10.1021/op800152n.
35. W. Genck, Make The Most of Antisolvent Crystallization, Chemical Processing. https://www.chemicalprocessing.com/processing-equipment/powder-solids/article/11372856/make-the-most-of-antisolvent-crystallization, accessed 2023-04-03.
36. R. Girshick, arXiv, Fast R-CNN, preprint, September 27, 2015, arXiv:abs/1504.08083, accessed 2023-03-31.
37. N. E. Leadbeater, In Situ Reaction Monitoring of Microwave-Mediated Reactions Using IR Spectroscopy, Chem. Commun., 2010, 46(36), 6693–6695,  10.1039/c0cc01921f.
38. C. Starbuck, A. Spartalis, L. Wai, J. Wang, P. Fernandez, C. M. Lindemann, G. X. Zhou and Z. Ge, Process Optimization of a Complex Pharmaceutical Polymorphic System via In Situ Raman Spectroscopy, Cryst. Growth Des., 2002, 2(6), 515–522,  DOI:10.1021/cg025559k.
39. D. Acevedo, W.-L. Wu, X. Yang, N. Pavurala, A. Mohammad and T. F. O'Connor, Evaluation of Focused Beam Reflectance Measurement (FBRM) for Monitoring and Predicting the Crystal Size of Carbamazepine in Crystallization Processes, CrystEngComm, 2021, 23(4), 972–985,  10.1039/d0ce01388a.
40. L. Dang, H. Yang, S. Black and H. Wei, The Effect of Temperature and Solvent Composition on Transformation of β- to α-Glycine As Monitored in Situ by FBRM and PVM, Org. Process Res. Dev., 2009, 13(6), 1301–1306,  DOI:10.1021/op900134w.
41. L.-T. Deck and M. Mazzotti, Conceptual Validation of Stochastic and Deterministic Methods To Estimate Crystal Nucleation Rates, Cryst. Growth Des., 2023, 23(2), 899–914,  DOI:10.1021/acs.cgd.2c01133.
42. L. Goh, K. Chen, V. Bhamidi, G. He, N. C. S. Kee, P. J. A. Kenis, C. F. Zukoski and R. D. Braatz, A Stochastic Model for Nucleation Kinetics Determination in Droplet-Based Microfluidic Systems, Cryst. Growth Des., 2010, 10(6), 2515–2521,  DOI:10.1021/cg900830y.
43. S. Toschev, A. Milchev and S. Stoyanov, On Some Probabilistic Aspects of the Nucleation Process, J. Cryst. Growth, 1972, 13–14, 123–127,  DOI:10.1016/0022-0248(72)90073-5.
44. Handbook of Industrial Mixing: Science and Practice, ed. Paul, E. L., Atiemo-Obeng, V. A. and Kresta, S. M., Wiley-Interscience, Hoboken, N.J, 2004.
45. M. T. Hicks, K. J. Myers and A. Bakker, Cloud Height In Solids Suspension Agitation, Chem. Eng. Commun., 1997, 160(1), 137–155,  DOI:10.1080/00986449708936610.
46. P. Ayazi Shamlou and E. Koutsakos, Solids Suspension and Distribution in Liquids under Turbulent Agitation, Chem. Eng. Sci., 1989, 44(3), 529–542,  DOI:10.1016/0009-2509(89)85030-4.
47. J. Janzon and H. Theliander, On the Suspension of Particles in an Agitated Vessel, Chem. Eng. Sci., 1994, 49(20), 3522–3526,  DOI:10.1016/0009-2509(94)00149-9.
48. G. Micale, F. Grisafi, L. Rizzuti and A. Brucato, CFD Simulation of Particle Suspension Height in Stirred Vessels, Chem. Eng. Res. Des., 2004, 82(9), 1204–1213,  DOI:10.1205/cerd.82.9.1204.44171.
49. R. J. Ingham, C. Battilocchio, D. E. Fitzpatrick, E. Sliwinski, J. M. Hawkins and S. V. Ley, A Systems Approach towards an Intelligent and Self-Controlling Platform for Integrated Continuous Reaction Sequences, Angew. Chem., Int. Ed., 2015, 54(1), 144–148,  DOI:10.1002/anie.201409356.
50. J. Daglish and A. J. Blacker, Determining Phase Separation Dynamics with an Automated Image Processing Algorithm, Org. Process Res. Dev., 2023, 27(4), 627–639,  DOI:10.1021/acs.oprd.2c00357.
51. D. E. Grigoriadis, E. Smith, S. R. J. Hoare, A. Madan and H. Bozigian, Pharmacologic Characterization of Valbenazine (NBI-98854) and Its Metabolites, J. Pharmacol. Exp. Ther., 2017, 361(3), 454–461,  DOI:10.1124/jpet.116.239160.
52. A. J. Kukor, N. Depner, I. Cai, J. L. Tucker, J. C. Culhane and J. E. Hein, Enantioselective Synthesis of (−)-Tetrabenazine via Continuous Crystallization-Induced Diastereomer Transformation, Chem. Sci., 2022, 13(36), 10765–10772,  10.1039/d2sc01825j.

---

Footnote

† Electronic supplementary information (ESI) available: Code availability https://gitlab.com/heingroup/heinsightv2, HeinSight2.0 model and data availability https://drive.google.com/drive/folders/1M1zpiaThiVlq9--AjoLQ_K6QHCkjoXjb. See DOI: https://doi.org/10.1039/d3sc05491h

---