In the field of materials design, machine learning techniques have been used to develop data-driven models for optimizing material properties. The performance of the model can be sensitive to the choice of optimization algorithm, making optimizer selection a challenging task. In recent years, there has been a transition in optimization from hand-designed to learned features, with the learning to optimize (L2O) approach attempting to learn the optimization conditions itself. While many studies in the field of L2O have demonstrated good performance on a specific task distribution, they often suffer from poor generalization to other distributions.
In materials design, the availability of training data can be limited due to the expense of prototyping or detailed simulations. This can make it difficult to achieve convergence of the learned features, which may require large amounts of data. To address these issues, we propose a new approach called "Learning to Choose Optimizers" (L2CO). Our method allows a meta-learner to select from a range of static, well-established optimizers at test time. This enables the system to adapt to different task distributions and enhance generalization performance by switching to a different optimizer as needed. In this proof-of-concept study, we train our model offline on a diverse set of benchmark loss-functions and apply a range of gradient-based, population-based, and probabilistic model-based optimizers. We take inspiration from recommendation systems to identify the most appropriate optimizer for an unseen test problem, and in the online stage, the meta-learner is able to choose and switch optimizers during training.
The potential of this approach is demonstrated by comparing its performance to classical optimizers and established L2O-architectures on benchmark loss-functions and simple material design studies. The results suggest that L2CO has the potential to be a useful tool in the field of materials design, and we make our code and documentation available as open-source resources using the f3dasm framework.