Mario
Piris
*ab
aDonostia International Physics Center (DIPC), Euskal Herriko Unibertsitatea (UPV/EHU), 20018 Donostia, Spain. E-mail: mario.piris@ehu.eus
bIKERBASQUE, Basque Foundation for Science, 48013 Bilbao, Spain
First published on 10th October 2024
In recent years, Natural Orbital Functional (NOF) theory has gained increasing significance in quantum chemistry, successfully addressing one of the field's most challenging problems: providing an accurate and balanced description of systems with strong electronic correlation. The quest for NOFs that strike the delicate balance between computational tractability and predictive accuracy represents a holy grail for researchers. Today, NOFs provide an alternative formalism to both density functional and wavefunction-based methods, with their appeal rooted in a wonderfully simple conceptual framework. This perspective outlines the basic concepts, strengths and weaknesses, and current status of NOFs, while offering suggestions for their future development.
Density functional theory (DFT) was established1 by Hohenberg and Kohn (HK), with practical implementations relying on the Kohn–Sham (KS) formulation.2 In the KS approach, the unknown kinetic energy functional as a functional of the electron density is avoided by introducing a reference system of non-interacting electrons described by a single Slater determinant. This difference highlights issues, particularly from the correlation kinetic energy, posing a significant challenge in current density functionals. While local density and generalized gradient approximations are commonly used, they often fail to describe strong electron correlations. Hybrid-exchange functionals and range-separated interactions have been practical solutions to this problem, showing improved performance, although failures persist.3,4 Today, there is a plethora of empirical and non-empirical parametrized density functionals,5 but this extensive parametrization often limits their applicability to specific systems and phenomena, deviating from a physically grounded modeling approach.6,7
The exact solution of the time-independent non-relativistic Schrödinger equation can be achieved by the full configuration interaction (FCI) method, which accounts for all possible electronic configurations based on a one-particle basis, the spin-orbitals. However, FCI scales exponentially with the number of orbitals, making methods based on approximate wavefunctions more practical for solving the many-body problem, despite the progress made with the density matrix renormalization group technique.8 Currently, the coupled-cluster wavefunction with single, double, and perturbative triple excitations9 is considered the “gold standard” in quantum chemistry, with efficient and nearly linear-scaling implementations.10,11 Nevertheless, this method still demands significant computational resources as the system size increases and is unsatisfactory for systems with strong static correlation due to its single Hartree–Fock reference character.
There is no universal consensus on the distinction between static and dynamic electron correlation, and the transition between them is gradual. Static correlation generally involves a limited number of close-lying orbitals, usually the frontier orbitals near the Fermi level. The wavefunction in such cases is a mixture of configuration state functions, each with appreciable weight. In contrast, dynamic correlation involves a very large number of configurations, each with a small weight. The most interesting chemistry—such as systems with partially broken bonds, transition states for bond breaking, transition metal compounds, compounds of strongly electronegative first-row atoms, large conjugated systems, electronically excited states, etc.—requires a balanced treatment of both static and dynamic correlation. Presently, the most reliable approaches to address this are multireference methods, such as the complete active space self-consistent field (CASSCF)12 and its combination with second-order perturbation theory (CASPT2).13 Unfortunately, multireference methods still suffer from high computational costs and sensitivity to active space selection, especially for a large number of strongly correlated orbitals. Furthermore, while incorporating dynamic correlation through perturbative corrections improves the accuracy of absolute energies, it does not improve the quality of the reference. Achieving results that include all electron correlation at once requires a comprehensive optimization process, which becomes computationally prohibitive for multireference-perturbative methods.
Reduced density matrix (RDM)-based methods offer alternative formalisms to both density functional and wavefunction-based approaches. It has long been known14 that the energy of a system with at most two-body interactions is an exact linear functional of the two-particle RDM (2RDM), the only functional we know with certainty. Today, realistic variational 2RDM calculations are possible,15 but they remain computationally expensive. A dramatic reduction in computational scaling can be achieved by further reducing the complexity of the fundamental variable, specifically by using the one-particle reduced density matrix (1RDM).
The existence of a functional of the 1RDM16–18 is already implicit in the original HK theorem.19 However, the existence of the functional alone does not allow us to determine its analytical expression in a general manner. While the Levy formulation17 can reveal the explicit form of the exact functional for certain model systems,20,21 it remains impractical for computational purposes. This limitation has driven the development of approximate functionals, most of which are formulated in the natural orbital (NO) representation,22 where the 1RDM is diagonal. The spectral decomposition of the 1RDM allows the functional to be expressed in terms of the NOs and their occupation numbers (ONs), thereby defining it as a natural orbital functional (NOF).
This perspective is intended for a general audience, emphasizing the fundamental and practical aspects of NOF approximations. Rather than delving into detailed computational procedures and specific system results, it provides insights into the current state of the field and outlines future directions. For a comprehensive historical overview of the formulation and evolution of NOFs, readers are referred to the recent review article by the author.23
(1) |
Assuming we can determine the universal functional Vee[Γ], it becomes necessary to impose certain restrictions on the allowable 1RDMs to ensure a physically valid ground-state energy.24 These restrictions require that the 1RDM originate from an N-particle quantum state that is antisymmetric with respect to the exchange of identical electrons, a property known as N-representability. A crucial aspect of N-representability is its connection to either a pure or ensemble state. In 1963, Coleman established25 the conditions for ensuring the ensemble N-representability of Γ, and these are relatively straightforward to implement. In contrast, the more recent systematic approach to deriving pure-state N-representability conditions for the 1RDM,26,27 has proven extremely complex to implement.
For a considerable time, it was widely believed that pure and ensemble universal functionals were equivalent within their shared domain of pure-state N-representable 1RDMs.18,28 However, recent research29 has shown that the ensemble functional is actually the lower convex envelope of the pure functional. Fortunately, the equivalence of pure and ensemble energy functionals on the set of v-representable 1RDMs was confirmed recently.30 Consequently, we can focus on physical electronic systems in their ground states under an external potential v(r), where pure and ensemble functionals are indistinguishable, and consider only ensemble constraints during the energy minimization process to satisfy N-representability conditions.
Disappointingly, analytically determining Vee[Γ] remains a formidable challenge. Despite significant efforts, this goal has yet to be achieved, necessitating the use of approximations for Vee. To approximate the unknown energy functional, one can use the exact functional form (1) in the NO representation, with a 2RDM constructed using a reconstruction functional D[ni, nj, nk, nl], namely,
(2) |
The approximate functional (2) clearly retains dependence on the 2RDM,31 meaning it is necessary to ensure not only the N-representability of the 1RDM but also the N-representability of the functional itself.32 Specifically, the reconstructed D[ni, nj, nk, nl] must satisfy the same N-representability conditions as an unreconstructed D (ref. 33) to guarantee the existence of a compatible N-electron system for the functional.
It has often been assumed, incorrectly, that approximate functionals do not encounter N-representability issues because the 1RDM conditions were believed to be sufficient. Although some functionals, like Müller's,34 appear well-formulated and accurate for certain systems, they often violate fundamental N-representability conditions. This has exposed that many currently used approximate functionals35–37 fail to meet functional N-representability criteria.38–40
Two procedures can be envisioned to derive Eel[{ni, ϕi}]: the so-called top-down and bottom-up methods.41 In the top-down approach, an approximate N-particle wavefunction is proposed in the NO representation, with the expansion coefficients explicitly expressed in terms of the ONs. This method directly yields an approximate functional that is N-representable by construction. Although this approach is straightforward, its success has been limited to a single case beyond the Hartree–Fock approximation: PNOF5.42,43
In contrast, the bottom-up approach involves proposing D[ni, nj, nk, nl] without relying on an N-particle state. The functional is then constructed by incrementally incorporating the set of N-representability conditions33 into the proposed 2RDM reconstruction. Since the complete set of N-representability conditions for the 2RDM involves higher-order RDMs, only those conditions directly related to the 2RDM itself can be effectively applied. Specifically, the (2,2)-positivity conditions are used, which require the two-electron density matrix, the electron–hole density matrix, and the two-hole density matrix to be positive semidefinite. Consequently, we can never be entirely certain that the functional is fully N-representable, with the exception of PNOF5, which was derived using this method before its generating wavefunction was identified. However, the other functionals obtained via the bottom-up method can be considered approximately N-representable, as only some of the necessary N-representability conditions are imposed. The more conditions imposed on the 2RDM, the closer the resulting energy (2) is to the exact energy.
Given the implicit dependence of approximate functionals on the 2RDM, it is more accurate to classify Eel[{ni, ϕi}] as a NOF rather than as a 1RDM functional. Ultimately, these functionals are only comprehended within the NO representation, including even the well-established functional derived from the Löwdin–Shull wavefunction44 with a chosen phase combination, which accurately describes closed-shell two-electron systems. In this context, NOFs can be regarded as approximate energy expressions derived from an approximate quantum state, with certainty when using the top-down method and with high confidence when employing the bottom-up method. A significant consequence of this is that energy is not invariant under unitary transformations of the orbitals, resulting in the absence of a generalized Fockian when working with approximate functionals.45 This property can only be guaranteed by the exact 1RDM functional, which can be defined in any representation, not just that of the NOs.
A systematic application of the bottom-up approach, using an explicit two-index approximation46 of the two-particle cumulant,47,48 has resulted in the development of a series of JKL-only functionals known as PNOFi (i = 1–7),49–55 where J, K, and L represent Coulomb, exchange, and exchange-time-inversion integrals, respectively.56 These functionals have been particularly successful in capturing static electron correlation,57 especially those based on electron pairing,58 which closely match results from accurate wavefunction-based methods for small systems by also accounting for dynamic electron correlation within intrapair interactions.
The missing dynamic electron correlation can be addressed through perturbative corrections.59–62 However, while this approach primarily improves energies relative to reference NOF values, it does not enhance the quality of the NOs and ONs. Fully correlated NOs and ONs requires a more comprehensive strategy, such as the global NOF (GNOF).63 GNOF achieves a balanced description of dynamic and static correlations, and its strong agreement with accurate wavefunction-based methods in both relative and absolute energies further underscores its reliability.
Minimizing a NOF requires adhering to the orthonormality constraints for the NOs, while the ONs must meet both ensemble N-representability conditions and Löwdin normalization. For NOFs based on electron pairing, additional conditions are imposed, which inherently ensure normalization. This setup results in a constrained optimization problem, typically tackled by optimizing the energy separately with respect to ONs and NOs, since simultaneous optimization of all variables has been shown to be inefficient for approximations.64 Currently, open-source codes like DoNOF65,66 efficiently handle this problem and support GPU acceleration via implementations in modern programming languages such as Python and Julia. These codes provide various features, including geometry optimization, natural and canonical representations of molecular orbitals, computation of ionization potentials and electric moments, perturbative corrections for dynamic correlation, NOF-based ab initio molecular dynamics, and excited state calculations.
NOFs effectively address a broader spectrum of systems72–76 than density functionals, which are limited by their single-reference nature, frequently achieving chemical accuracy.63,77 They are particularly effective in complex scenarios such as bond-breaking and bond-forming reactions,78–81 as well as in transition metal chemistry.82,83 PNOFs based on electron pairing have been especially successful in capturing static electron correlation84 and are the only functionals to date capable of accurately determining the correct number of electrons in the fragments resulting from homolytic dissociations,85–87 thereby eliminating delocalization errors.88
Despite these advantages, the application of NOFs faces several challenges, with computational expense being a significant one. A primary issue is the need to optimize NOs at the molecular orbital level, unlike density functional calculations where optimized orbitals are derived from solving the Kohn–Sham equations in the atomic orbital basis. Optimizing NOs at the molecular level requires a four-index transformation to convert atomic orbital electron repulsion integrals into molecular orbital integrals. Although this transformation is only performed once during occupancy optimization, it must be repeated each time the orbitals change, making NO optimization a particularly time-consuming step in the energy minimization process.
To enhance efficiency, various strategies have been employed. Parallelizing key sections of the code responsible for the four-index transformation has led to significant performance improvements. For certain NOFs, summing over NO indices separately has reduced computational costs by enabling feasible integral-direct formalism calculations,89 decreasing memory demands, and facilitating the use of efficient techniques such as integral screening and the fast multipole method. Additionally, the resolution of identity approximation has been employed66 to further reduce arithmetic and memory scaling, enabling the application of NOF approximations to larger molecular systems of general chemical interest.83
Another key factor influencing computational efficiency is the optimization procedure itself, despite recent efforts.90,91 Over the past two decades, various implementations with varying degrees of effectiveness have emerged. While advances in optimizing ONs have improved efficiency,92 optimizing NOs still requires substantial improvements to make NOF-based methods competitive with density functional approximations. The iterative diagonalization procedure45 represented a significant advancement by generating a generalized pseudo-Fockian, but its efficiency is constrained by the lack of an exact prescription for its diagonal elements and the absence of effective techniques to accelerate convergence in the final stages.
An established alternative to orbital optimization through diagonalization is orbital rotations.93 In this approach, orthonormal orbitals are rotated relative to each other using an orthogonal transformation defined by a skew matrix, which specifies the independent rotation parameters. Consequently, the energy becomes a function of these rotation parameters, with its gradients dictated by the off-diagonal elements of the pseudo-Fockian. This eliminates the need for diagonal elements, enabling the use of trust-region methods,94,95 which provide accurate nonlinear approximation models in each iteration. However, as system size increases, these conventional techniques become increasingly costly. Given the advantages of each approach, combining methods can accelerate the optimization of NOs. Yet, to achieve robust convergence at a reasonable cost, it is evident that we must look beyond traditional techniques and leverage innovative optimization methods. Additionally, to further reduce the scaling of orbital optimization, solving the problem in the atomic orbital basis, as is usually done in self-consistent field methods, is essential.
Another difficulty to consider in any functional theory is spin. Since the electronic Hamiltonian commutes with the total spin squared (S2) and one of its components, typically Sz, its eigenvectors must also be eigenvectors of these spin operators. Some methodologies to address this challenge use spin-unrestricted approaches, which bypass the requirement for approximate functionals to fully exhibit all symmetries. In these methodologies, the number of spin-up and spin-down electrons is fixed, but spin contamination is allowed. Relaxing the constraint on total spin conservation can improve size consistency and accuracy in energy predictions for molecular dissociations, among other scenarios. Nevertheless, spin-unrestricted methodologies often involve non-physical interpretations.
In the absence of an external field, total spin invariance implies that the ground state of a many-electron system forms a multiplet, a mixed quantum state encompassing all possible Sz values. One approach to addressing the spin issue is to focus solely on the high-spin component of the multiplet. Extensions of NOF theory to high-spin multiplet states have been reported,96–98 demonstrating some positive outcomes for atoms and molecules. However, no approximation currently exists that can accurately describe each component of the multiplet individually.
A more comprehensive strategy, without focusing on the specifics of each component, is to treat the multiplet as a whole.99,100 This approach centers on the mixed-spin state with the highest multiplicity (2S + 1 = NI + 1), corresponding to the quantum number S = NI/2, where NI denotes the number of unpaired electrons. A key feature of this formulation is the zero average spin projection across the ensemble, which enables 2RDM reconstructions that conserve spin and allows for the application of the machinery developed for singlet states. The results from this approach, which contrasts with conventional methods that emphasize the high-spin component or break spin symmetry, are promising.101 However, in some systems, additional correlation for unpaired electrons is still necessary,82,83 and simulating individual states remains crucial, especially in cases where the multiplet components are no longer degenerate.
In the context of NOF calculations, the general formulation of analytic energy gradients with respect to nuclear motion facilitates the efficient computation of equilibrium geometries.100,102 These gradients can be obtained through straightforward evaluation, eliminating the need for linear response theory or iterative procedures. However, calculating the analytic Hessian requires knowledge of NOs and ONs at the perturbed geometry, which can only be obtained by solving coupled-perturbed equations.103 Consequently, computing second-order energy derivatives is significantly more demanding in terms of storage and computational time compared to evaluating gradients, making numerical differentiation of analytical gradients more efficient than direct analytical evaluation of the Hessian. This has led to the development of codes that use numerical Hessian calculations to find equilibrium geometries and calculate harmonic vibrational frequencies.65 Nevertheless, it remains highly desirable to develop more efficient techniques to characterize critical points on potential energy surfaces, including saddle points, especially for large systems.
The ability to compute gradients analytically also makes ab initio molecular dynamics (AIMD) simulations using NOF-based methods practical.104 The strategy involves calculating nuclear forces on-the-fly during molecular dynamics, eliminating the need for predefined interatomic potentials, which can be problematic in chemically complex systems with evolving bonding patterns. In such cases, density functional-based AIMDs are often inadequate, while multi-reference methods are costly and may introduce energy discontinuities due to instabilities from changes in active-space orbitals. In addition, NOF-based AIMD offers a distinct edge over existing methods: by tracking the real-time evolution of the NOs along with their ONs, we can observe the dynamic evolution of the electronic structure. Indeed, the unique NO representation adapts dynamically during the trajectory, adjusting to the most favorable nuclear interactions at each step, making it especially valuable for monitoring real-time changes in bonding patterns. Since a reaction mechanism involves the sequence of bond formation and breaking, the time evolution of NOs is the ideal approach for insightful descriptions of reaction mechanisms. Currently, only implementations based on the Born–Oppenheimer approximation are available; however, in the future, non-adiabatic molecular dynamics will be necessary for cases involving strong couplings between two or more potential energy surfaces, which require a quantum treatment of the nuclei.
The most significant achievements of NOFs have so far been in describing ground states, while developing a robust theoretical framework for accurately addressing excited states, both charged (EνN±1 − E0N) and neutral (EνN − E0N), remains an ongoing challenge. Research has mainly focused on the energies of vertical excitations, with only a few studies addressing adiabatic transitions for charged excitations. For neutral excitations, the problem remains unresolved, as a description of the excited state independent of the ground state is still needed. Progress has been made using time-dependent theory within the adiabatic linear response framework105–107 and extending 1RDMFT to ensemble states with fixed weights w.108,109 However, both approaches still require more efficient minimization schemes to achieve significant success.
Current vertical excitation calculations rely on the ground state and vary depending on the excitation operator used to generate either charged or neutral states. Charged excitations are typically addressed using the extended Koopmans' theorem (EKT),110,111 while neutral excitations employ Rowe's excitation operator formalism.112 In both methods, the coefficients defining these excited states are derived from the first- and second-order RDMs of the ground state. NOF-EKT provides satisfactory results for lower ionization potentials (IPs), but its accuracy diminishes for higher IPs.113–116 Recent advancements have improved IP accuracy and effectively addressed band gap opening in various systems,117,118 although some inaccuracies persist in photoemission spectra. Conversely, NOF-EKT offers an unsatisfactory description of electron affinities, though these can be estimated using the inverse of the IP of the corresponding anionic species, computed at the experimental geometry of the neutral species.115 On the other hand, neutral excitations within the extended random-phase approximation (ERPA)119 have recently led to the development of various NOF-ERPA methods,120 which have produced very promising results, comparable to high-level CI methods but at significantly lower computational costs, and substantially outperform those obtained with time-dependent density functional theory. The accuracy of vertical excitation calculations is expected to improve in parallel with advancements in ground-state NOFs, as well as with enhanced implementations to better handle cases involving degeneracy of ONs.
A promising research direction is the extension of NOFs to periodic systems. Previously, some attempts were made using very basic functionals for polymers121 and solids,122,123 where translational symmetry allowed NOs to be represented as Bloch states. Other studies have focused on model systems such as the Hubbard model124 or systems consisting solely of hydrogen atoms.68–70 The most significant achievement to date is demonstrating the viability of NOFs in cases where density functionals struggle, particularly in treating strong correlations, including Mott insulators. Consequently, there is a heightened focus on developing NOF-based methods that can effectively tackle the design and analysis of strongly correlated materials.125
One unresolved issue is the treatment of dispersion interactions. Although some attempts have been made in the past,50JKL-only functionals face significant challenges with these systems. Employing non-JKL functionals with more complex dependencies seems necessary,126 but this approach increases the computational cost of NOFs and detracts from their inherent simplicity. Current alternatives include incorporating perturbative corrections,61 using empirical treatments similar to those in DFT, or potentially improving functionals through machine learning. The latter has already shown promise in 1RDM-based electronic structure calculations,127 in DFT,128,129 and in 1RDMFT for bosons.130 By training on ab initio data, more accurate approximations could be achieved. Nevertheless, a key challenge for these functionals is generalization. Alternatively, machine learning could be employed as a phenomenological predictive tool, directly forecasting observables from large datasets or computational results by establishing intricate connections. However, this approach does not facilitate the rationalization of structure–property relationships from first principles, highlighting the important distinction that prediction does not equate to understanding.131
Finally, an encouraging avenue of research is the integration of NOFs into quantum computing algorithms. Quantum computing already holds the potential for significant advantages over classical methods in electronic structure calculations. However, current quantum processors have limited capabilities, so NOFs could help minimize the quantum resources required. Reducing the cost of quantum simulations for many-electron problems has been demonstrated with 2RDM-based algorithms.132,133 Since 1RDM measurements require sampling of significantly fewer elements than 2RDM measurements, NOFs are expected to result in substantial savings in circuit executions.
Purists argue that current NOFs are not the exact functional of the 1RDM and lack key properties, such as invariance under unitary transformations of the orbitals. Given the current state of knowledge, finding an analytical expression for the exact 1RDM functional seems unlikely. However, we can aim to develop a sufficiently universal NOF approximation for physical electronic systems that captures as many properties as possible, including those related to N-representability. Our goal is to obtain the right answers for the right reasons. Today, functionals like GNOF have already demonstrated the ability to handle arbitrary electronic systems with any spin value, independent of the external potential, while providing a balanced description of both dynamic and static correlations.
NOF-based methods can be considered “black box” approaches because they do not require the definition of an active space. However, their main drawback remains the computational cost in comparison to density functional calculations. To achieve robust convergence at a reasonable cost, we must go beyond traditional techniques, employing innovative optimization methods and, if possible, using the atomic orbital representation, as done in conventional self-consistent field methods. With the advent of artificial intelligence, it is highly likely that more accurate approximations will be achieved for a wide range of electronic systems. Nevertheless, it is important to remember that we are primarily theorists, not mere predictors, even though our models must still be capable of providing predictive insights.
This journal is © The Royal Society of Chemistry 2024 |