Lehan
Yao
a,
Hyosung
An
ab,
Shan
Zhou
a,
Ahyoung
Kim
a,
Erik
Luijten
cdef and
Qian
Chen
*aghi
aDepartment of Materials Science and Engineering, University of Illinois, Urbana, IL 61801, USA. E-mail: qchen20@illinois.edu
bDepartment of Petrochemical Materials Engineering, Chonnam National University, Yeosu, 59631, Korea
cDepartment of Materials Science and Engineering, Northwestern University, Evanston, IL 60208, USA
dDepartment of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, IL 60208, USA
eDepartment of Chemistry, Northwestern University, Evanston, IL 60208, USA
fDepartment of Physics and Astronomy, Northwestern University, Evanston, IL 60208, USA
gDepartment of Chemistry, University of Illinois, Urbana, IL 61801, USA
hMaterials Research Laboratory, University of Illinois, Urbana, IL 61801, USA
iBeckman Institute for Advanced Science and Technology, University of Illinois, Urbana, IL 61801, USA
First published on 24th October 2022
Nanoscale morphology of functional materials determines their chemical and physical properties. However, despite increasing use of transmission electron microscopy (TEM) to directly image nanomorphology, it remains challenging to quantify the information embedded in TEM data sets, and to use nanomorphology to link synthesis and processing conditions to properties. We develop an automated, descriptor-free analysis workflow for TEM data that utilizes convolutional neural networks and unsupervised learning to quantify and classify nanomorphology, and thereby reveal synthesis–nanomorphology relationships in three different systems. While TEM records nanomorphology readily in two-dimensional (2D) images or three-dimensional (3D) tomograms, we advance the analysis of these images by identifying and applying a universal shape fingerprint function to characterize nanomorphology. After dimensionality reduction through principal component analysis, this function then serves as the input for morphology grouping through unsupervised learning. We demonstrate the wide applicability of our workflow to both 2D and 3D TEM data sets, and to both inorganic and organic nanomaterials, including tetrahedral gold nanoparticles mixed with irregularly shaped impurities, hybrid polymer-patched gold nanoprisms, and polyamide membranes with irregular and heterogeneous 3D crumple structures. In each of these systems, unsupervised nanomorphology grouping identifies both the diversity and the similarity of the nanomaterial across different synthesis conditions, revealing how synthetic parameters guide nanomorphology development. Our work opens possibilities for enhancing synthesis of nanomaterials through artificial intelligence and for understanding and controlling complex nanomorphology, both for 2D systems and in the far less explored case of 3D structures, such as those with embedded voids or hidden interfaces.
For these reasons, it is important to synthetically control the nanomorphology of materials, and to do so in a predictable manner; a goal that has been hindered by challenges in characterizing and analyzing them in an automated, standardized, and high-throughput manner. Ensemble methods, including light/X-ray/neutron scattering, can resolve averaged characteristic dimensions of nanomaterials,31–35 but not their nanomorphology and heterogeneities. TEM, on the other hand, can capture nanomorphology either as projected images in two dimensions (2D) or as tilt-series and reconstructed three-dimensional (3D) volumes with nanometer resolution. Several analysis methods have been developed to extract nanomorphology from TEM data sets, including the recent application of machine learning (ML)-based convolutional neural networks (CNNs) to identify and segment NP features.36,37 Unsupervised ML algorithms, e.g., k-means,38 the Gaussian mixture model (GMM),39 and density-based spatial clustering of applications with noise (DBSCAN),40 have also been used to categorize materials according to their morphology. However, these efforts concern 2D TEM images of simple nanomorphologies, which can be described by a small set of shape descriptors chosen by humans (e.g., size and aspect ratio), and of simple samples containing one chemical component, often only inorganic materials. What remains greatly underexplored is the regime of complex nanomorphologies of real-world materials. These can be heterogeneous, containing multiple types of shapes in one sample; they can be irregularly shaped, not sufficiently captured by conventional shape descriptors and making it difficult for humans to objectively choose the appropriate descriptors; they can be hybrid with multiple components, including organic ones that have low contrast under TEM; and they can have 3D features, with multiple features overlaying or agglomerated in 2D projections and requiring a larger parameter space to describe the shape comprehensively. Here we develop and implement a unified ML-based nanomorphology analysis and classification workflow that is robustly applicable to three systems representing real-world materials containing all the above-mentioned complexity aspects. Using this workflow, we are able to elucidate the synthesis–nanomorphology relationship in these systems at a highly quantitative level. As illustrated in Scheme 1, our workflow starts from the segmentation of samples—i.e., recognizing the shape contours of the features—from TEM images using CNNs. Recent work by us and others has used CNNs with a single output channel to recognize NPs in TEM images by solving a pixel-wise binary classification problem.36,37 To account for complex materials systems, in this work we incorporate multiple channels in the output layer of the CNN, with each channel associated with a different physical meaning during training. Such a multi-channeled CNN has been shown to distinguish different atoms in scanning transmission electron microscopy images.41 Here we demonstrate that it accurately separates overlaying NPs and differentiates between inorganic and organic components in a single particle. Next, the workflow utilizes a universal fingerprint function42 to characterize the nanomaterial morphology based on the segmented contours, which applies to arbitrary, irregular, and even 3D shapes not studied before. This fingerprint function provides a comprehensive description of the nanomaterial morphology based on the spatial coordinates of the segmented shape contours. No empirical decisions are needed on which specific shape descriptor is relevant. Lastly, using this fingerprint function as input, followed by dimensional reduction via principal component analysis (PCA), we perform GMM clustering to classify the corresponding nanomorphology instances into groups with similar features to quantitatively elucidate the synthesis–nanomorphology relationships. We apply this workflow to three complex nanomaterials systems, namely (i) gold tetrahedral NPs that are traditionally considered difficult to synthesize and that are typically accompanied by irregularly shaped impurity NPs; (ii) gold nanoprisms decorated with amorphous polymer patches; and (iii) polyamide membranes with 3D void structures used for water purification and molecular separation. Each system represents a morphological complication that previous methods have been unable to solve, and collectively the three systems cover a spectrum of sample features: high-contrast inorganic and low-contrast organic components; 2D and 3D nanomorphologies; separated as well as overlaying features. This approach allows us to relate nanomorphologies to synthesis conditions in a high-throughput and high-fidelity manner.
Our choice of the fingerprint function applies to diverse nanomorphologies and utilizes all the coordinates of the shape contours. Inspired by previous work,42 we describe a shape contour by the length d(θ) of the distance vector with orientation θ connecting a point along the shape contour and the centroid of the contour as a function of θ (Scheme 1). Subsequently, the discrete points along the shape contour are converted to a continuous one-dimensional function with range θ∈(−180°,180°). This definition applies directly to convex shapes but can result in complications for concave shapes (Fig. S5†). We overcome this by redefining d(θ) as the sum of the segment lengths of the distance vector that reside within the shape contours, thus extending this concept to arbitrary 2D contours, including concave ones (Fig. S5†). We show that using this definition, the diverse nanomorphologies present in the three systems studied here are sufficiently described for quantitative comparison and classification. Moreover, we extend this fingerprint for the analysis of 3D nanomorphologies obtained from TEM tomographs, where a 2D function d(θ,φ) in the spherical coordinate system is used to describe a 3D surface contour consisting of variable numbers of meshes5 within a well-defined domain of θ∈(0°,180°) and φ∈(0°,360°).
Classification, the last step in our workflow, relates nanomorphology to synthesis conditions. Previous classification methods used specific shape descriptors such as particle area and aspect ratio as inputs.38–40 Such descriptors only selectively retain partial information embedded in shape contours, and can vary across shapes. For example, edge length is typically used to describe triangular nanoprisms and nanocubes, whereas the major axis length is more suitable for nanorods. In contrast, the contour fingerprint function used here captures most of the shape information, unselectively, for all shapes. Choosing an interval of 1° for θ to achieve sufficiently high spatial resolution, we obtain a fingerprint of a 2D shape that is effectively a feature vector of 360 elements. To make this vector usable in our classification approach, we perform a dimensional reduction through PCA, which projects high-dimensional data onto a low-dimensional space while maximizing their variance, thus retaining the most important (combination of) features in an unsupervised manner. We find that usually the first few principal components (PCs) of the shape fingerprint contain a large fraction of the total variance (Fig. S6–S8†), i.e., faithfully represent the original data. GMM is then applied to the dimension-reduced feature vectors for nanomorphology classification. In our systems, the nanomorphologies are classified into more than two groups. To determine the optimal number of groups, we use the Bayesian information criterion (BIC) and the Akaike information criterion (AIC),43 which quantitatively evaluate the separation of data points into groups while preventing the model from becoming too complicated due to the assignment of too many groups (ESI and Fig. S6–S8†). We now discuss the insights obtained from this workflow for three representative systems.
Indeed, as shown in Fig. 1a, gold tetrahedral NPs that are synthesized following a seeded growth method45 pack densely on a TEM grid (Fig. 1b). Conventional threshold-based segmentation recognizes such overlaying projections as a single connected structure. Post-processing algorithms such as watershed transformation can further segment them into individual NPs by drawing lines to cut through the “neck” region between two touching NPs,36,39 but this method is more suitable for rounded shapes than for NPs with polygonal projections (Fig. S9†). Moreover, the watershed algorithm separates NPs without recovering the overlaying parts of the contours, so that for two overlaying NPs neither of the segmented contours reflect the true particle shape (Fig. S9†). For this reason, some recent work opted to simply filter out the overlaying contours,38 which can potentially lead to a bias in yield analysis. For example (Fig. 1c), well-facetted tetrahedral NPs tend to overlay more frequently than irregularly shaped impurities, so that discarding overlaying features would underestimate the yield.
Fig. 1 Identification of overlaying gold tetrahedral NPs and yield analysis. (a) Schematic of gold tetrahedral NPs synthesis through the seeded growth method. Different reaction conditions give rise to different product shapes, accompanied by impurities. (b) TEM image and schematic of as-synthesized tetrahedral NPs assembled into overlaying structures after deposition on the TEM grid. (c) TEM image containing tetrahedral NPs (shaded in yellow) and impurities (shaded in blue gray). (d) Workflow of shape fingerprint extraction from raw TEM images. Two output channels from U-Net identify the non-overlaying parts of NPs (gray, Channel 1) and the overlaying of NPs (white, Channel 2) from the input TEM images. Combining closed regions in each channel yields the reconstructed individual NP contours. The shape fingerprints d(θ) of both tetrahedra and impurities are then extracted from the reconstructed contours. (e) Full set of shape fingerprints collected, including tetrahedral NPs (colored) and impurities (gray). Different colors indicate classes identified by the GMM (cf. panel (f)). (f) Full set of shape fingerprints projected onto their first two principal components (PCs). Insets show the average NP shape of the classes identified by the GMM. (g) Average edge length (triangular symbols) and truncation (circular symbols) as a function of seed amount. Circular charts show the fraction of each GMM class at the three different seed amounts (24.0 μL, 39.7 μL, 44.0 μL; see ESI† for more details). Percentages indicates the yield of non-impurity (colored) classes. Scale bars: 50 nm in panels (b) and (c); 25 nm in panel (d). |
We address this issue using the trained U-Net with three output channels. In this approach, a pixel in the TEM image is classified to belong to (i) a non-overlaying part of a NP (Channel 1); (ii) an overlaying part of NPs (Channel 2); or (iii) the background (Channel 3) (Fig. 1d). Combining the segmented contour in Channels 1 and 2 makes it then possible to restore the full contours of overlaying tetrahedral NPs and impurities without bias (Fig. 1d, see ESI and Fig. S9 and S10† for details and evaluations). Shape fingerprint functions (Fig. 1e) are calculated from the contours.
Classification of the shape fingerprints from ∼2700 NPs synthesized at three different conditions separates impurities from tetrahedral NPs and identifies subgroups of tetrahedral NPs with distinctive edge lengths and truncations (Fig. S6e and f†). We use this approach to assess the effect of the amount of spherical gold seed NPs in the synthesis. Most shape fingerprints exhibit three peaks (Fig. 1e), corresponding to the triangular projection of tetrahedral NPs. The remaining fingerprints, which exhibit randomly distributed features, are classified as impurities. PCA-based dimensionality reduction of the shape fingerprints (Fig. 1f) shows that the first two PCs recover 85% of the total variance, indicating successful dimensionality reduction (Fig. S6a†). Lastly, an unsupervised GMM with six centers (suggested by BIC, Fig. S6d†) is used for shape classification. Three major clusters in the PC space are identified, corresponding to three different fingerprints (Fig. 1e, colored curves). Relating each of the three groups to the corresponding averaged NP shape in real space (Fig. 1f inset), we find that they correspond to tetrahedral NPs of different sizes and extents of truncation. Moreover, we find (Fig. S6e and f†) that PC 1 correlates with the particle edge length and PC 2 with the extent of truncation, suggesting that, without a priori judgement or selection, these two shape descriptors identify the principal differences between the products across seed amounts. These observations are consistent with previous studies, where truncation of tetrahedral NPs was shown to be synthetically determined by the competition of gold deposition and oxidative etching, which can be changed by the concentration of added precursor, surfactant, and seed.50,51
In addition to the three major groups of tetrahedral NPs, the GMM also recognizes impurities among the products. The yield of desired tetrahedral NPs displays a non-monotonic trend with the seed amount (Fig. 1g). In addition, the edge length of the tetrahedral NPs (with impurities eliminated) decreases from 46 nm to 38 nm (Fig. 1g) as the seed amount increases given the same growth condition, consistent with earlier hypotheses in the synthesis literature;45,52,53 in the presence of more seeds, the amount of growth species per seed is reduced, resulting in smaller NPs. Lastly, the extent of truncation, defined as the distance between the corner and triangle vertices extended from three edges (cf. ESI and Fig. S11†), also changes with the seed amount (Fig. 1g).
Using our trained three-channeled U-Net, we can robustly segment the inorganic core and polymer patches without having to adjust the intensity threshold. The subsequent classification of the segmented contours allows us to discern asymmetric versus symmetric polymer patch expansion as a function of the ligand concentration used in the synthesis. The occurrence of different patches is analyzed both for the global ensemble of prisms and in terms of their distribution on the prisms.
The patchy gold prisms are synthesized following our previous method by incubating as-synthesized gold prisms in a solution of thiol ligand (2-naphthalenethiol, 2-NAT) and block copolymer (polystyrene-b-poly(acrylic acid), PS-b-PAA) (Fig. 2a).21 The 2-NAT ligands adsorb onto the highly curved prism tips via thiol–gold bonds to render the tip area hydrophobic, and PS-b-PAA polymers then adsorb onto ligand-coated area via hydrophobic attraction. By controlling the ligand concentration (denoted as the ratio of 2-NAT concentration to the optical density of the prism suspension at its maximum extinction wavelength, α), we control the accessible tip area that polymers can adsorb onto, resulting into various patch patterns on the prism surface.21 As shown in Fig. 2b and c, whereas the gold cores have a distinct contrast against the background, polymer patches and background have a large overlapping range of intensity. Therefore, simply applying a threshold to single out polymer patches results in incorrect segmentation. While pretreatments such as Gaussian filtering can improve the segmentation, such methods sacrifice image spatial resolution and nanoscale morphology details.36 In contrast, our modified U-Net with three output channels predicts both the gold core and polymer patches in a threshold-free manner at pixel-wise accuracy as high as 99. 5% (Fig. 2d and ESI†). Furthermore, the U-Net is also proven to work with varying material contrast (cf. ESI and Fig. S12†). About 1000 contours of segmented polymer patches (∼300 contours for gold cores) are collected from the TEM images of the patchy prisms synthesized at different α conditions.
Choosing the centroid of the gold prisms as the coordinate center, we obtain shape fingerprints of both the polymer patches and gold cores. Given the threefold symmetry of the triangular prism core, we focus on the fingerprints of the patches on each tip with θ∈(−60°,60°). The position θ = 0° in the patch fingerprints is found from the local maximum of d(θ) in the fingerprints of gold cores, which is the orientation of the tip (Fig. 2e). The fingerprints of the cores also serve as reference for determining the patch patterns, i.e., the orientation and position of the patches relative to the core.
Combined with BIC, GMM classifies the patches into six groups based on their shape fingerprints after PCA (Fig. 2f and g and Fig. S7†). The average patch shapes (Fig. 2g, insets) illustrate that important parameters determining the patch shape grouping are the patch coverage and the symmetry of individual patches relative to the prism tip. The patch coverage is defined as the angular range θ of the gold core covered by a polymer patch, which determines, for example, the effective valency of the patchy NPs during assembly.57,61–66 As shown in Fig. 2g and S7g,† patch coverage increases from groups 1 to 6. Regarding the symmetry of patches, the fingerprints of groups 1, 4, and 6 show patch distributions symmetric around θ = 0°, whereas the fingerprints of groups 2, 3 and 5 suggest asymmetric patch growth. This symmetry breaking in patch growth was not captured in our earlier work using specific patch shape descriptors.21
Ligand concentration alters the fractions of the six patch groups. As α increases, the fractions of groups with higher patch coverage increase (Fig. 2h), consistent with the ligand island expansion mechanism proposed in our earlier work. The 2-NAT ligands expand their coverage on the prism tip with increasing α due to strong ligand–ligand attraction, thereby recruiting more polymers to the tip.21 Regarding the patch symmetry along the tip, the groups representing symmetric (groups 1, 4, and 6) and asymmetric (groups 2, 3 and 5) patches dominate alternatingly as α increases. At the lowest ligand concentration, α = 0.12 mM, the dominant patch shape (group 1) is symmetric. As α increases to 0.25 mM and 0.5 mM, the asymmetric groups 2 and 3 become the dominant patch shapes. Comparison of the fingerprints of groups 1, and 2, 3 (Fig. 2g) suggests that the ligands tend to extend along one side of the prism at this ligand concentration. At even higher α (α = 1.0 and 2.5 mM), the symmetric patch shapes (groups 4 and 6) take over. We speculate that starting from a small symmetric patch on the tip, the initial random polymer chain adsorption on one side of the tip promotes more polymer adsorption on the same side to maximize polymer–polymer contacts, which gives rise to asymmetric patch growth. On the other hand, asymmetric patches have a higher surface area exposed to the solvent than the symmetric ones, which yields less favorable polymer–solvent interaction. The two effects compete as α increases, leading to a switch between symmetric and asymmetric patch coverages before the three patches fully merge into a single domain.
Lastly, we examine the occurrence of different patch combinations on prisms. Fig. 2i shows the ten most frequently observed combinations, illustrating a tendency toward higher patch coverage with increasing α. Thus, utilizing our ML-assisted TEM image segmentation and analysis workflow allows us to uncover and quantify the complexity in the morphology of hybrid NPs. These insights into the synthesis mechanism of the hybrid NPs can guide the design of other hybrid nanomaterials, such as core–shell NPs, Janus particles, and heterostructures in optoelectronic devices.54,67,68
TEM-based electron tomography is used to image polyamide membranes in 3D at sub-nm voxel resolution. Experimentally, tilt series of 2D projections of the membranes are collected under TEM and reconstructed into 3D volumes (Fig. 3b). We isolate the crumples in each membrane to serve as input of our analysis and classification workflow. For the fingerprint function, we consider the polar angle θ and azimuthal angle φ in the spherical coordinate system to determine a distance vector in 3D (Fig. 3b). To capture the structural features of the membrane, we measure the distance d(θ,φ) as the segment length of the distance vector of orientation (θ,φ) that resides within the membrane volume, excluding void spaces (Fig. 3b, red line). Higher values of d(θ,φ) suggest that more membrane mass is distributed along the vector direction. This conversion “collapses” the 3D crumples into 2D shape fingerprints maps. For example, the shape fingerprint (Fig. 3c) of the crumpled membrane in Fig. 3b describes a morphology with most of its mass distributed below its centroid, suggesting a bowl-like crumple with concavity facing upwards.
Using PCA of the fingerprints, we choose the first three PCs with a cumulative variance of 73% (Fig. S8†) and utilize GMM to classify the crumples into four groups: “dome”, “dimple”, “lay”, and “pancake”, each named after their apparent shapes (Fig. 3d and e). Previous studies have shown that the crumples exhibit different effective moduli depending on the morphology groups they belong to.72 Despite the variety of crumples observed in the membrane (Fig. 3a), they consistently fall into these four groups; crumples within each group show high quantitative similarity in their fingerprints (Fig. 3f and g; Fig. S13†) and 3D tomographs (Fig. 3g), confirming the robustness of this classification for recognizing similar shapes in a diverse collection of nanomorphologies. Specifically, group 1 has high d(θ,φ) regions spanning the top part of the fingerprint maps and vanishing d(θ,φ) regions at the bottom, which corresponds to dome-shaped crumples with an opening at the bottom. Group 2 has low d(θ,φ) regions as a basin in the center of the map (θ ≈ 90° and φ ≈ 180°), indicating the existence of a concave surface on the vertical side of the crumples, which we refer to as standing dimples. Group 3 shows a similarly concave surface as well. But unlike group 2, the center position of the concave region in group 3 is off the horizontal center of the crumple, resulting in a V-shaped region of vanishing d(θ,φ) values in the map. The corresponding 3D crumple shape resembles a sofa, which we denote as “lay”. Group 4 shows high d(θ,φ) regions clustered in a belt at θ = 90°, while the d(θ,φ) values near the vertical Poles (θ = 0°, 180°) are low. This type of fingerprint corresponds to a “pancake” with most of its mass positioned at the middle plane, θ = 90°.
The four groups of crumples identified by the GMM show distinct curvature-related features. Considering different combinations of local Gaussian curvature G and mean curvature H, we can distinguish the mesh vertices on the crumple surface as four surface curvature structures: tip (G > 0,H > 0), tube (G ≈ 0,H > 0), saddle (G < 0,H ≈ 0), and valley (G > 0,H < 0) (cf. ESI and Fig. S8a†).5,73 As illustrated in Fig. 4a, for group 4 (“pancake”), the long perimeter results in the highest fraction belonging to “tube”, unlike for other groups. Due to the mostly convex shape of domes, the tip dominates in this group while dimple and lay have more balanced fractions between tip and tube. Dimples and lays are very similar in terms of all G and H curvatures, and as a result we regarded them as the same group in our previous report.5 Nevertheless, our 3D shape fingerprint method successfully differentiates dimples and lays in terms of the orientation of the concavity of their surface relative to the substrate, an aspect not recognized by the conventional average curvature descriptors.
Fig. 4 Morphological features of polyamide membrane crumple groups and their fractions under different reaction conditions. (a) Fractions of different surface elements in each crumple group. The surface elements are defined in the Supporting information and Fig. S8a.† (b) Fractions of each crumple group in membranes synthesized at different monomer concentrations. Colors denote the crumple group and match those used in panel (a). The units of MPD and TMC concentrations are in w/v%. |
Finally, we examine the fractions of crumple groups under synthesis at different concentrations of two monomers, m-phenylenediamine (MPD) and trimesoyl chloride (TMC). Fig. 4b illustrates that the four crumple groups are observed under all five reaction conditions, suggesting that the morphologies identified by GMM are common across synthesis conditions. However, the fractions of each crumple group vary for different reaction conditions. At a fixed MPD concentration of 1 w/v%, increasing the TMC concentration yields more domes and fewer pancakes. Given that the average size of dome crumples is smaller than that of pancakes, we infer that a high TMC concentration generally leads to smaller crumples. This classification of the crumples of a polyamide membrane demonstrates the applicability of our workflow to the characterization of 3D, irregular, and heterogeneous nanomaterials.
MATLAB code examples of shape fingerprint extraction for tetrahedral NPs, patchy nanoprisms, and polyamide membrane crumples are available at: https://github.com/chenlabUIUC/ShapeFingerprintExtraction.
Footnote |
† Electronic supplementary information (ESI) available: Details of synthesis, sample preparation, and electron tomography imaging methods; full workflow of the machine-learning-based image segmentation, fingerprint extraction, and unsupervised classification with figures; additional figure of neural network architecture. (PDF). See DOI: https://doi.org/10.1039/d2nr03712b |
This journal is © The Royal Society of Chemistry 2022 |