Bayesian optimization has recently been proposed as a framework for automatically tuning the hyperparameters of machine learning models and has been shown to yield state-of-the-art performance with impressive ease and efficiency. In this paper, we explore whether it is possible to transfer the knowledge gained from previous optimizations to new tasks in order to find optimal hyperparameter settings more efficiently. Our approach is based on extending multi-task Gaussian processes to the framework of Bayesian optimization. We show that this method significantly speeds up the optimization process when compared to the standard single-task approach. We further propose a straightforward extension of our algorithm in order to jointly minimize the average error across multiple tasks and demonstrate how this can be used to greatly speed up k-fold cross-validation. Lastly, we propose an adaptation of a recently developed acquisition function, entropy search, to the cost-sensitive, multi-task setting. We demonstrate the utility of this new acquisition function by leveraging a small dataset to explore hyperparameter settings for a large dataset. Our algorithm dynamically chooses which dataset to query in order to yield the most information per unit cost.
translated by 谷歌翻译
Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of developing automatic approaches which can optimize the performance of a given learning algorithm to the task at hand. In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). The tractable posterior distribution induced by the GP leads to efficient use of the information gathered by previous experiments, enabling optimal choices about what parameters to try next. Here we show how the effects of the Gaussian process prior and the associated inference procedure can have a large impact on the success or failure of Bayesian optimization. We show that thoughtful choices can lead to results that exceed expert-level performance in tuning machine learning algorithms. We also describe new algorithms that take into account the variable cost (duration) of learning experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization on a diverse set of contemporary algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.
translated by 谷歌翻译
Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations. It relies on querying a distribution over functions defined by a relatively cheap surrogate model. An accurate model for this distribution over functions is critical to the effectiveness of the approach, and is typically fit using Gaussian processes (GPs). However, since GPs scale cubically with the number of observations, it has been challenging to handle objectives whose optimization requires many evaluations, and as such, massively parallelizing the optimization.In this work, we explore the use of neural networks as an alternative to GPs to model distributions over functions. We show that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically. This allows us to achieve a previously intractable degree of parallelism, which we apply to large scale hyperparameter optimization, rapidly finding competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.
translated by 谷歌翻译
深度神经网络(DNNS)和数据集的增长不断上升,这激发了对同时选择和培训的有效解决方案的需求。许多迭代学习者的高参数优化方法(HPO)的许多方法,包括DNNS试图通过查询和学习响应表面来解决该问题的最佳表面来解决此问题。但是,这些方法中的许多方法都会产生近视疑问,不考虑有关响应结构的先验知识和/或执行偏见的成本感知搜索,当指定总成本预算时,所有这些都会加剧识别表现最好的模型。本文提出了一种新颖的方法,称为迭代学习者(BAPI),以在成本预算有限的情况下解决HPO问题。 BAPI是一种有效的非洋流贝叶斯优化解决方案,可以说明预算,并利用有关目标功能和成本功能的先验知识来选择更好的配置,并在评估期间(培训)做出更明智的决策。针对迭代学习者的不同HPO基准测试的实验表明,在大多数情况下,BAPI的性能比最先进的基线表现更好。
translated by 谷歌翻译
由于其数据效率,贝叶斯优化已经出现在昂贵的黑盒优化的最前沿。近年来,关于新贝叶斯优化算法及其应用的发展的研究激增。因此,本文试图对贝叶斯优化的最新进展进行全面和更新的调查,并确定有趣的开放问题。我们将贝叶斯优化的现有工作分为九个主要群体,并根据所提出的算法的动机和重点。对于每个类别,我们介绍了替代模型的构建和采集功能的适应的主要进步。最后,我们讨论了开放的问题,并提出了有希望的未来研究方向,尤其是在分布式和联合优化系统中的异质性,隐私保护和公平性方面。
translated by 谷歌翻译
Several recent advances to the state of the art in image classification benchmarks have come from better configurations of existing techniques rather than novel approaches to feature learning. Traditionally, hyper-parameter optimization has been the job of humans because they can be very efficient in regimes where only a few trials are possible. Presently, computer clusters and GPU processors make it possible to run more trials and we show that algorithmic approaches can find better results. We present hyper-parameter optimization results on tasks of training neural networks and deep belief networks (DBNs). We optimize hyper-parameters using random search and two new greedy sequential methods based on the expected improvement criterion. Random search has been shown to be sufficiently efficient for learning neural networks for several datasets, but we show it is unreliable for training DBNs. The sequential algorithms are applied to the most difficult DBN learning problems from [1] and find significantly better results than the best previously reported. This work contributes novel techniques for making response surface models P (y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements.
translated by 谷歌翻译
贝叶斯优化是一种过程,允许获得黑盒功能的全局最佳功能,并且在超参数优化等应用中有用。在目标函数的形状上估计的不确定性估计是引导优化过程的工具。但是,如果客观函数违反基础模型(例如,高斯)的假设,这些估计可能是不准确的。我们提出了一种简单的算法,可以通过目标函数校准后部分布的不确定性作为贝叶斯型优化过程的一部分。我们表明,通过提高校准后分布的不确定性估计,贝叶斯优化使得更好的决策并以较少的步骤到达全球最佳。我们表明,该技术提高了贝叶斯优化对标准基准函数和超参数优化任务的性能。
translated by 谷歌翻译
Bayesian Optimization is a useful tool for experiment design. Unfortunately, the classical, sequential setting of Bayesian Optimization does not translate well into laboratory experiments, for instance battery design, where measurements may come from different sources and their evaluations may require significant waiting times. Multi-fidelity Bayesian Optimization addresses the setting with measurements from different sources. Asynchronous batch Bayesian Optimization provides a framework to select new experiments before the results of the prior experiments are revealed. This paper proposes an algorithm combining multi-fidelity and asynchronous batch methods. We empirically study the algorithm behavior, and show it can outperform single-fidelity batch methods and multi-fidelity sequential methods. As an application, we consider designing electrode materials for optimal performance in pouch cells using experiments with coin cells to approximate battery performance.
translated by 谷歌翻译
贝叶斯优化(BO)已成为许多昂贵现实世界功能的全球优化的流行策略。与普遍认为BO适合优化黑框功能的信念相反,它实际上需要有关这些功能特征的域知识才能成功部署BO。这样的领域知识通常表现在高斯流程先验中,这些先验指定了有关功能的初始信念。但是,即使有专家知识,选择先验也不是一件容易的事。对于复杂的机器学习模型上的超参数调谐问题尤其如此,在这种模型中,调整目标的景观通常很难理解。我们寻求一种设定这些功能性先验的替代实践。特别是,我们考虑了从类似功能的数据中,使我们可以先验地进行更紧密的分布。从理论上讲,我们与预先训练的先验表示对BO的遗憾。为了验证我们在现实的模型培训设置中的方法,我们通过训练在流行图像和文本数据集上的数以万计的近状态模型配置来收集了大型多任务超参数调谐数据集,以及蛋白质序列数据集。我们的结果表明,平均而言,我们的方法能够比最佳竞争方法更有效地定位良好的超参数。
translated by 谷歌翻译
Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible. On the other hand, bandit-based configuration evaluation approaches based on random search lack guidance and do not converge to the best configurations as quickly. Here, we propose to combine the benefits of both Bayesian optimization and banditbased methods, in order to achieve the best of both worlds: strong anytime performance and fast convergence to optimal configurations. We propose a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian neural networks, deep reinforcement learning, and convolutional neural networks. Our method is robust and versatile, while at the same time being conceptually simple and easy to implement.
translated by 谷歌翻译
信息理论的贝叶斯优化技术因其非洋流品质而变得越来越流行,以优化昂贵的黑盒功能。熵搜索和预测性熵搜索都考虑了输入空间中最佳的熵,而最新的最大值熵搜索则考虑了输出空间中最佳值的熵。我们提出了联合熵搜索(JES),这是一种新的信息理论采集函数,它考虑了全新的数量,即输入和输出空间上关节最佳概率密度的熵。为了结合此信息,我们考虑从幻想的最佳输入/输出对条件下的熵减少。最终的方法主要依赖于标准的GP机械,并去除通常与信息理论方法相关的复杂近似值。凭借最少的计算开销,JES展示了卓越的决策,并在各种任务中提供了信息理论方法的最新性能。作为具有出色结果的轻重量方法,JES为贝叶斯优化提供了新的首选功能。
translated by 谷歌翻译
贝叶斯优化是黑匣子功能优化的流行框架。多重方法方法可以通过利用昂贵目标功能的低保真表示来加速贝叶斯优化。流行的多重贝叶斯策略依赖于采样政策,这些策略解释了在特定意见下评估目标函数的立即奖励,从而排除了更多的信息收益,这些收益可能会获得更多的步骤。本文提出了一个非侧重多倍数贝叶斯框架,以掌握优化的未来步骤的长期奖励。我们的计算策略具有两步的lookahead多因素采集函数,可最大程度地提高累积奖励,从而测量解决方案的改进,超过了前面的两个步骤。我们证明,所提出的算法在流行的基准优化问题上优于标准的多尺寸贝叶斯框架。
translated by 谷歌翻译
大多数机器学习算法由一个或多个超参数配置,必须仔细选择并且通常会影响性能。为避免耗时和不可递销的手动试验和错误过程来查找性能良好的超参数配置,可以采用各种自动超参数优化(HPO)方法,例如,基于监督机器学习的重新采样误差估计。本文介绍了HPO后,本文审查了重要的HPO方法,如网格或随机搜索,进化算法,贝叶斯优化,超带和赛车。它给出了关于进行HPO的重要选择的实用建议,包括HPO算法本身,性能评估,如何将HPO与ML管道,运行时改进和并行化结合起来。这项工作伴随着附录,其中包含关于R和Python的特定软件包的信息,以及用于特定学习算法的信息和推荐的超参数搜索空间。我们还提供笔记本电脑,这些笔记本展示了这项工作的概念作为补充文件。
translated by 谷歌翻译
Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that addresses these issues in isolation. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA, spanning 2 ensemble methods, 10 meta-methods, 27 base classifiers, and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show classification performance often much better than using standard selection/hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.
translated by 谷歌翻译
贝叶斯优化(BO)已成为黑框函数的顺序优化。当BO用于优化目标函数时,我们通常可以访问对潜在相关功能的先前评估。这就提出了一个问题,即我们是否可以通过元学习(meta-bo)来利用这些先前的经验来加速当前的BO任务,同时确保稳健性抵抗可能破坏BO融合的潜在有害的不同任务。本文介绍了两种可扩展且可证明的稳健元算法:稳健的元高斯过程 - 加工置信度结合(RM-GP-UCB)和RM-GP-thompson采样(RM-GP-TS)。我们证明,即使某些或所有以前的任务与当前的任务不同,这两种算法在渐近上都是无重组的,并且证明RM-GP-UCB比RM-GP-TS具有更好的理论鲁棒性。我们还利用理论保证,通过通过在线学习最大程度地减少遗憾,优化分配给各个任务的权重,从而减少了相似任务的影响,从而进一步增强了稳健性。经验评估表明,(a)RM-GP-UCB在各种应用程序中都有效,一致地性能,(b)RM-GP-TS,尽管在理论上和实践中都比RM-GP-ucb稳健,但在实践中,在竞争性中表现出色某些方案具有较小的任务,并且在计算上更有效。
translated by 谷歌翻译
Bayesian Optimization(BO)是一种优化昂贵对评估黑匣子功能的采样有效的方法。大多数BO方法忽略了评估成本如何在优化域中变化。然而,这些成本可以是高度异质的并且通常提前未知。这发生在许多实际设置中,例如机器学习算法或基于物理的仿真优化的超参数调整。此外,那些确认成本异质性的现有方法并不自然地适应总评估成本的预算限制。这种未知的成本和预算限制的组合引入了勘探开发权衡的新维度,其中关于成本的学习成本本身。现有方法没有原因地理由以原则的方式对此问题的各种权衡,经常导致性能不佳。我们通过证明,每单位成本的预期改进和预期改善,可以使这两个最广泛使用的采购职能在实践中的预期改进和预期的索赔可以是任意劣等的。为了克服现有方法的缺点,我们提出了预算的多步预期改进,是一个非近视收购函数,以概括为异质和未知评估成本的古典预期改进。最后,我们表明我们的采集功能优于各种合成和实际问题的现有方法。
translated by 谷歌翻译
黑匣子优化需要指定搜索空间以探索解决方案,例如解决方案。 D维紧凑空间,此选择对于以合理的预算获得最佳结果至关重要。不幸的是,在许多应用中确定高质量的搜索空间可能具有挑战性。例如,当在给出有限的预算时调整机器学习管道的机器学习管道时,必须在不包括潜在有前途的地区之间进行平衡,并将搜索空间保持足够小以易于发动。这项工作的目标是激励 - 通过调整深度神经网络的示例应用程序 - 预测预算条件的搜索空间质量的问题,以及提供基于应用于a的实用程序功能的简单评分方法概率响应表面模型,类似于贝叶斯优化。我们表明我们所呈现的方法可以在各种情况下计算有意义的预算条件分数。我们还提供实验证据,即精确的分数可用于构建和修剪搜索空间。最终,我们认为评分搜索空间应该成为深度学习实验工作流程中的标准实践。
translated by 谷歌翻译
贝叶斯优化提供了一种优化昂贵黑匣子功能的有效方法。它最近已应用于流体动力学问题。本文研究并在一系列合成测试函数上从经验上比较了常见的贝叶斯优化算法。它研究了采集函数和训练样本数量的选择,采集功能的精确计算以及基于蒙特卡洛的方法以及单点和多点优化。该测试功能被认为涵盖了各种各样的挑战,因此是理想的测试床,以了解贝叶斯优化的性能,并确定贝叶斯优化表现良好和差的一般情况。这些知识可以用于应用程序中,包括流体动力学的知识,这些知识是未知的。这项调查的结果表明,要做出的选择与相对简单的功能不相关,而乐观的采集功能(例如上限限制)应首选更复杂的目标函数。此外,蒙特卡洛方法的结果与分析采集函数的结果相当。在目标函数允许并行评估的情况下,多点方法提供了更快的替代方法,但它可能需要进行更多的客观函数评估。
translated by 谷歌翻译
贝叶斯优化已被证明是优化昂贵至尊评估系统的有效方法。然而,根据单一观察的成本,一个或多个目标的多维优化可能仍然是昂贵的。多保真优化通过包括多个更便宜的信息来源,例如数值模拟中的低分辨率近似来解决这个问题。用于多保真优化的采集功能通常基于勘探重算法,这些算法难以与多种目标的优化结合。在这里,我们认为预期的超越改善政策可以在许多情况下作为合适的替代品起作用。我们通过两步评估或在单个采集函数内纳入评估成本,额外的保真相关目标。这允许同时多目标和多保真优化,这允许以分数成本准确地建立帕累托集和前部。基准显示成本降低了一个数量级或更多的顺序。因此,我们的方法允许极其膨胀的黑盒功能进行静态优化。在现有的优化贝叶斯优化框架中实现了本方法简单且直接,可以立即扩展到批量优化。该技术还可用于组合不同的连续和/或离散保真度尺寸,这使得它们特别相关地与等离子体物理,流体动力学和许多科学计算分支中的模拟问题相关。
translated by 谷歌翻译
我们考虑使用昂贵的功能评估(也称为实验)的黑匣子多目标优化(MOO)的问题,其中目标是通过最小化实验的总资源成本来近似真正的帕累托解决方案。例如,在硬件设计优化中,我们需要使用昂贵的计算模拟找到权衡性能,能量和面积开销的设计。关键挑战是选择使用最小资源揭示高质量解决方案的实验顺序。在本文中,我们提出了一种基于输出空间熵(OSE)搜索原理来解决MOO问题的一般框架:选择最大化每单位资源成本的信息的实验,这是真正的帕累托前线所获得的信息。我们适当地实例化了OSE搜索的原理,以导出以下四个Moo问题设置的高效算法:1)最基本的EM单一保真设置,实验昂贵且准确; 2)处理EM黑匣子约束}在不执行实验的情况下无法进行评估; 3)离散的多保真设置,实验可以在消耗的资源量和评估准确度时变化; 4)EM连续保真设置,其中连续函数近似导致巨大的实验空间。不同综合和现实世界基准测试的实验表明,基于OSE搜索的算法在既有计算效率和MOO解决方案的准确性方面改进了最先进的方法。
translated by 谷歌翻译