Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal, like the final fitness values of multiple trials. For many benchmarks, however, a trial can also terminate once it reaches a pre-specified target value. When only some trials reach the target value, two variables characterize a trial's outcome: the time it takes to reach the target value (or not) and its final fitness value. This paper describes a simple way to impose linear order on this two-variable trial data set so that traditional non-parametric methods can determine the better algorithm when neither dominates. We illustrate the method with the Mann-Whitney U-test. A simulation demonstrates that U-scores are much more effective than dominance when tasked with identifying the better of two algorithms. We test U-scores by having them determine the winners of the CEC 2022 Special Session and Competition on Real-Parameter Numerical Optimization.

While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.

自然语言处理中的性能，特别是针对问题解答任务的性能，通常是通过将模型最自信（主要）预测与黄金答案（地面真相）进行比较来衡量的。我们证明，即使对于失败的示例，量化模型的距离也有多大的预测答案也很有用。我们将示例的黄金等级（GR）定义为其最自信的预测的等级，该预测与地面真理完全匹配，并显示了为什么这种比赛总是存在的。对于我们分析的16个变压器模型，大多数辅助预测空间中的金色答案大多数都非常接近最高等级。我们将次要预测称为在降低置信度概率顺序中排名高于0的预测。我们展示了如何使用GR来分类问题并可视化他们的难度范围，从持续的成功到持续的极端故障。我们在整个测试集中得出了一个新的聚合统计量，称为黄金等级插值中位数（GRIM），该统计量量化了失败的预测与模型最佳选择的接近度。为了开发一些直觉并探索这些指标的适用性，我们使用了斯坦福大学问题答案数据集（Squad-2）和一些来自拥抱面枢纽的流行变压器模型。我们首先证明了严峻的与F1和精确匹配（EM）分数没有直接相关。然后，我们计算和可视化各种变压器体系结构的这些分数，通过群集进行失败的预测来探测其在错误分析中的适用性，并比较它们与其他训练诊断（例如EM和F1分数）的关系。我们最终提出了各种研究目标，例如扩大这些指标的数据收集及其在对抗培训中的可能使用。

In this study, we consider simulation-based worst-case optimization problems with continuous design variables and a finite scenario set. To reduce the number of simulations required and increase the number of restarts for better local optimum solutions, we propose a new approach referred to as adaptive scenario subset selection (AS3). The proposed approach subsamples a scenario subset as a support to construct the worst-case function in a given neighborhood, and we introduce such a scenario subset. Moreover, we develop a new optimization algorithm by combining AS3 and the covariance matrix adaptation evolution strategy (CMA-ES), denoted AS3-CMA-ES. At each algorithmic iteration, a subset of support scenarios is selected, and CMA-ES attempts to optimize the worst-case objective computed only through a subset of the scenarios. The proposed algorithm reduces the number of simulations required by executing simulations on only a scenario subset, rather than on all scenarios. In numerical experiments, we verified that AS3-CMA-ES is more efficient in terms of the number of simulations than the brute-force approach and a surrogate-assisted approach lq-CMA-ES when the ratio of the number of support scenarios to the total number of scenarios is relatively small. In addition, the usefulness of AS3-CMA-ES was evaluated for well placement optimization for carbon dioxide capture and storage (CCS). In comparison with the brute-force approach and lq-CMA-ES, AS3-CMA-ES was able to find better solutions because of more frequent restarts.

尽管在机器学习的方法论核心中是一个问题，但如何比较分类器仍未达成一致的共识。每个比较框架都面临着（至少）三个基本挑战：质量标准的多样性，数据集的多样性以及选择数据集选择的随机性/任意性。在本文中，我们通过采用决策理论的最新发展，为生动的辩论增添了新的观点。我们最终的框架基于所谓的偏好系统，通过广义的随机优势概念对分类器进行排名，该概念强大地绕过了繁琐的，甚至通常是自相矛盾的，对聚合的依赖。此外，我们表明，可以通过解决易于手柄的线性程序和通过适应的两样本观察随机化测试进行统计测试来实现广泛的随机优势。这确实产生了一个有力的框架，可以同时相对于多个质量标准进行分类器的统计比较。我们在模拟研究和标准基准数据集中说明和研究我们的框架。

基准和性能分析在理解迭代优化启发式（IOHS）的行为中发挥着重要作用，例如本地搜索算法，遗传和进化算法，贝叶斯优化算法等。然而，这项任务涉及手动设置，执行和分析实验单独的基础，这是艰苦的，可以通过通用和设计精心设计的平台来缓解。为此，我们提出了Iohanalyzer，一种用于分析，比较和可视化IOH的性能数据的新用户友好的工具。在R和C ++中实现，Iohanalyzer是完全开源的。它可以在Cran和GitHub上获得。 Iohanalyzer提供有关固定目标运行时间的详细统计信息以及具有实际值的Codomain，单目标优化任务的基准算法的固定预算性能。例如，在多个基准问题上的性能聚合是可能的，例如以经验累积分布函数的形式。 Iohanalyzer在其他性能分析包上的主要优点是其高度交互式设计，允许用户指定对其实验最有用的性能测量，范围和粒度，以及不仅分析性能迹线，还可以分析演变动态状态参数。 Iohanalyzer可以直接从主基准平台处理性能数据，包括Coco平台，JOVERRAD，SOS平台和iohExperenter。提供R编程接口，供用户更倾向于对实现的功能进行更精细的控制。

无论是在功能选择的领域还是可解释的AI领域，都有基于其重要性的“排名”功能的愿望。然后可以将这种功能重要的排名用于：（1）减少数据集大小或（2）解释机器学习模型。但是，在文献中，这种特征排名没有以系统的，一致的方式评估。许多论文都有不同的方式来争论哪些具有重要性排名最佳的特征。本文通过提出一种新的评估方法来填补这一空白。通过使用合成数据集，可以事先知道特征重要性得分，从而可以进行更系统的评估。为了促进使用新方法的大规模实验，在Python建造了一个名为FSEVAL的基准测定框架。该框架允许并行运行实验，并在HPC系统上的计算机上分布。通过与名为“权重和偏见”的在线平台集成，可以在实时仪表板上进行交互探索图表。该软件作为开源软件发布，并在PYPI平台上以包裹发行。该研究结束时，探索了一个这样的大规模实验，以在许多方面找到参与算法的优势和劣势。

Grid search and manual search are the most widely used strategies for hyper-parameter optimization. This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid. Empirical evidence comes from a comparison with a large previous study that used grid search and manual search to configure neural networks and deep belief networks. Compared with neural networks configured by a pure grid search, we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time. Granting random search the same computational budget, random search finds better models by effectively searching a larger, less promising configuration space. Compared with deep belief networks configured by a thoughtful combination of manual search and grid search, purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets, and superior performance on one of seven. A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that for most data sets only a few of the hyper-parameters really matter, but that different hyper-parameters are important on different data sets. This phenomenon makes grid search a poor choice for configuring algorithms for new data sets. Our analysis casts some light on why recent "High Throughput" methods achieve surprising success-they appear to search through a large number of hyper-parameters because most hyper-parameters do not matter much. We anticipate that growing interest in large hierarchical models will place an increasing burden on techniques for hyper-parameter optimization; this work shows that random search is a natural baseline against which to judge progress in the development of adaptive (sequential) hyper-parameter optimization algorithms.

In the field of derivative-free optimization, both of its main branches, the deterministic and nature-inspired techniques, experienced in recent years substantial advancement. In this paper, we provide an extensive computational comparison of selected methods from each of these branches. The chosen representatives were either standard and well-utilized methods, or the best-performing methods from recent numerical comparisons. The computational comparison was performed on five different benchmark sets and the results were analyzed in terms of performance, time complexity, and convergence properties of the selected methods. The results showed that, when dealing with situations where the objective function evaluations are relatively cheap, the nature-inspired methods have a significantly better performance than their deterministic counterparts. However, in situations when the function evaluations are costly or otherwise prohibited, the deterministic methods might provide more consistent and overall better results.

自动化的机器学习（AUTOML）过程可能需要通过不仅机器学习（ML）组件及其超参数的复杂配置空间进行搜索，还需要将它们组合在一起，即形成ML管道。如果该管道配置空间过大，那么固定时间预算可实现的优化效率和模型精度可实现。一个关键的研究问题是，通过利用其历史表现来完成各种ML任务（即元知识），避免对ML管道的不良评估是否可能既可能又实用。以前的经验以分类器/回归器准确性排名的形式来自（1）（1）在历史自动运行期间进行的大量但无尽的管道评估数量，即“机会性”元知识，或（2）全面的交叉 - 通过默认超参数（即“系统”的元知识，对分类器/回归器的验证评估。使用AUTOWEKA4MCPS软件包进行了许多实验，表明（1）机会性/系统的元知识可以改善ML的结果，通常与元知识的相关性以及（2）配置空间扣除在不太保守的情况下是最佳的（2）也不是激进的。但是，元知识的效用和影响急性取决于其发电和剥削的许多方面，并保证了广泛的分析；这些通常在汽车和元学习文献中被忽视/不足。特别是，我们观察到对数据集的“挑战”的强烈敏感性，即选择预测因子的特异性是否会导致性能明显更好。最终，确定这样定义的“困难”数据集对于生成信息丰富的元知识基础和理解最佳搜索空间降低策略至关重要。

We present a new algorithm for Bayesian network structure learning, called Max-Min Hill-Climbing (MMHC). The algorithm combines ideas from local learning, constraint-based, and search-and-score techniques in a principled and effective way. It first reconstructs the skeleton of a Bayesian network and then performs a Bayesian-scoring greedy hill-climbing search to orient the edges. In our extensive empirical evaluation MMHC outperforms on average and in terms of various metrics several prototypical and state-of-the-art algorithms, namely the PC, Sparse Candidate, Three Phase Dependency Analysis, Optimal Reinsertion, Greedy Equivalence Search, and Greedy Search. These are the first empirical results simultaneously comparing most of the major Bayesian network algorithms against each other. MMHC offers certain theoretical advantages, specifically over the Sparse Candidate algorithm, corroborated by our experiments. MMHC and detailed results of our study are publicly available at http://www.dsl-lab.org/supplements/mmhc paper/mmhc index.html.

Metaheuristics are popularly used in various fields, and they have attracted much attention in the scientific and industrial communities. In recent years, the number of new metaheuristic names has been continuously growing. Generally, the inventors attribute the novelties of these new algorithms to inspirations from either biology, human behaviors, physics, or other phenomena. In addition, these new algorithms, compared against basic versions of other metaheuristics using classical benchmark problems without shift/rotation, show competitive performances. In this study, we exhaustively tabulate more than 500 metaheuristics. To comparatively evaluate the performance of the recent competitive variants and newly proposed metaheuristics, 11 newly proposed metaheuristics and 4 variants of established metaheuristics are comprehensively compared on the CEC2017 benchmark suite. In addition, whether these algorithms have a search bias to the center of the search space is investigated. The results show that the performance of the newly proposed EBCM (effective butterfly optimizer with covariance matrix adaptation) algorithm performs comparably to the 4 well performing variants of the established metaheuristics and possesses similar properties and behaviors, such as convergence, diversity, exploration and exploitation trade-offs, in many aspects. The performance of all 15 of the algorithms is likely to deteriorate due to certain transformations, while the 4 state-of-the-art metaheuristics are less affected by transformations such as the shifting of the global optimal point away from the center of the search space. It should be noted that, except EBCM, the other 10 new algorithms proposed mostly during 2019-2020 are inferior to the well performing 2017 variants of differential evolution and evolution strategy in terms of convergence speed and global search ability on CEC 2017 functions.

我们对两个单目标和两个多目标的全局全局优化算法进行了全面的全局灵敏度分析，作为算法配置问题。也就是说，我们研究了超参数对算法的直接效果和与其他超参数的效果的影响的影响质量。使用三种敏感性分析方法Morris LHS，Morris和Sobol，可以系统地分析协方差矩阵适应进化策略，差异进化，非主导的遗传算法III和多目标进化算法的可调型矩阵适应性进化策略，基于框架的分解，基于框架揭示，基于框架的遗传算法，超参数对抽样方法和性能指标的行为。也就是说，它回答了等问题，例如什么超参数会影响模式，它们的互动方式，相互作用的互动程度以及其直接影响程度。因此，超参数的排名表明它们的调整顺序，影响模式揭示了算法的稳定性。

我们查看模型可解释性的特定方面：模型通常需要限制在大小上才能被认为是可解释的，例如，深度5的决策树比深度50中的一个更容易解释。但是，较小的模型也倾向于高偏见。这表明可解释性和准确性之间的权衡。我们提出了一种模型不可知论技术，以最大程度地减少这种权衡。我们的策略是首先学习甲骨文，这是培训数据上高度准确的概率模型。 Oracle预测的不确定性用于学习培训数据的抽样分布。然后，对使用此分布获得的数据样本进行了可解释的模型，通常会导致精确度明显更高。我们将抽样策略作为优化问题。我们的解决方案1具有以下关键的有利属性：（1）它使用固定数量的七个优化变量，而与数据的维度（2）无关，它是模型不可知的 - 因为可解释的模型和甲骨文都可能属于任意性模型家族（3）它具有模型大小的灵活概念，并且可以容纳向量大小（4）它是一个框架，使其能够从优化领域的进度中受益。我们还提出了以下有趣的观察结果：（a）通常，小型模型大小的最佳训练分布与测试分布不同； （b）即使可解释的模型和甲骨文来自高度截然不同的模型家族，也存在这种效果：我们通过使用封闭的复发单位网络作为甲骨文来提高决策树的序列分类精度，从而在文本分类任务上显示此效果。使用字符n-grams； （c）对于模型，我们的技术可用于确定给定样本量的最佳训练样本。

我们介绍了强大的子组发现的问题，即，找到一个关于一个或多个目标属性的脱颖而出的子集的一组可解释的描述，2）是统计上的鲁棒，并且3）非冗余。许多尝试已经挖掘了局部强壮的子组或解决模式爆炸，但我们是第一个从全球建模角度同时解决这两个挑战的爆炸。首先，我们制定广泛的模型类别的子组列表，即订购的子组，可以组成的单次组和多变量目标，该目标可以由标称或数字变量组成，并且包括其定义中的传统Top-1子组发现。这种新颖的模型类允许我们使用最小描述长度（MDL）原理来形式地形化最佳强大的子组发现，在那里我们分别为标称和数字目标的最佳归一化最大可能性和贝叶斯编码而度假。其次，正如查找最佳子组列表都是NP-Hard，我们提出了SSD ++，一个贪婪的启发式，找到了很好的子组列表，并保证了根据MDL标准的最重要的子组在每次迭代中添加，这被显示为等同于贝叶斯一个样本比例，多项式或子组之间的多项式或T检验，以及数据集边际目标分布以及多假设检测罚款。我们经验上显示了54个数据集，即SSD ++优于先前的子组设置发现方法和子组列表大小。

多模式的多目标问题（MMOP）通常在现实世界中出现，而决策空间中遥远的解决方案对应于非常相似的目标值。为了获得MMOP的所有溶液，已经提出了许多多模式多模式的多模型进化算法（MMEAS）。目前，很少有研究涵盖了最近提出的大多数代表性MMEAS，并进行了比较。在这项研究中，我们首先回顾了过去二十年中相关作品。然后，我们选择了12种利用不同多样性维护技术的最先进的算法，并比较了它们在现有测试套件上的性能。实验结果表明，不同类型的MMOP上不同技术的优势和劣势，从而为如何在特定情况下选择/设计MMEAS提供指导。

排名和分数是判断使用的两个常见数据类型，以表达对象集合中对质量的偏好和/或质量的看法。存在许多模型以单独研究每种类型的数据，但没有统一的统计模型同时捕获两个数据类型，而不首先执行数据转换。我们提出了Mallows-Binomial模型来缩短这种差距，它通过量化的参数来与二项式分数模型相结合，这些差距通过量化的参数来量化对象质量，共识等级和法官之间的共识水平。我们提出了一种有效的树搜索算法来计算模型参数的精确MLE，分析和通过模拟研究模型的统计特性，并通过模拟将我们的模型应用于来自授予面板审查的实例，从而将其分数和部分排名的拨款。 。此外，我们展示了如何使用模型输出来排序对象的信心。拟议的模型被证明是从分数和排名中明智地结合信息，以量化对象质量并衡量具有适当统计不确定性的相互达成的共识。

Performance debugging in production is a fundamental activity in modern service-based systems. The diagnosis of performance issues is often time-consuming, since it requires thorough inspection of large volumes of traces and performance indices. In this paper we present DeLag, a novel automated search-based approach for diagnosing performance issues in service-based systems. DeLag identifies subsets of requests that show, in the combination of their Remote Procedure Call execution times, symptoms of potentially relevant performance issues. We call such symptoms Latency Degradation Patterns. DeLag simultaneously searches for multiple latency degradation patterns while optimizing precision, recall and latency dissimilarity. Experimentation on 700 datasets of requests generated from two microservice-based systems shows that our approach provides better and more stable effectiveness than three state-of-the-art approaches and general purpose machine learning clustering algorithms. DeLag is more effective than all baseline techniques in at least one case study (with p $\leq$ 0.05 and non-negligible effect size). Moreover, DeLag outperforms in terms of efficiency the second and the third most effective baseline techniques on the largest datasets used in our evaluation (up to 22%).

The performance of individual evolutionary optimization algorithms is mostly measured in terms of statistics such as mean, median and standard deviation etc., computed over the best solutions obtained with few trails of the algorithm. To compare the performance of two algorithms, the values of these statistics are compared instead of comparing the solutions directly. This kind of comparison lacks direct comparison of solutions obtained with different algorithms. For instance, the comparison of best solutions (or worst solution) of two algorithms simply not possible. Moreover, ranking of algorithms is mostly done in terms of solution quality only, despite the fact that the convergence of algorithm is also an important factor. In this paper, a direct comparison approach is proposed to analyze the performance of evolutionary optimization algorithms. A direct comparison matrix called \emph{Prasatul Matrix} is prepared, which accounts direct comparison outcome of best solutions obtained with two algorithms for a specific number of trials. Five different performance measures are designed based on the prasatul matrix to evaluate the performance of algorithms in terms of Optimality and Comparability of solutions. These scores are utilized to develop a score-driven approach for comparing performance of multiple algorithms as well as for ranking both in the grounds of solution quality and convergence analysis. Proposed approach is analyzed with six evolutionary optimization algorithms on 25 benchmark functions. A non-parametric statistical analysis, namely Wilcoxon paired sum-rank test is also performed to verify the outcomes of proposed direct comparison approach.

Monte Carlo Tree Search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarise the results from the key game and non-game domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.