Hydrocarbon prospect risking is a critical application in geophysics predicting well outcomes from a variety of data including geological, geophysical, and other information modalities. Traditional routines require interpreters to go through a long process to arrive at the probability of success of specific outcomes. AI has the capability to automate the process but its adoption has been limited thus far owing to a lack of transparency in the way complicated, black box models generate decisions. We demonstrate how LIME -- a model-agnostic explanation technique -- can be used to inject trust in model decisions by uncovering the model's reasoning process for individual predictions. It generates these explanations by fitting interpretable models in the local neighborhood of specific datapoints being queried. On a dataset of well outcomes and corresponding geophysical attribute data, we show how LIME can induce trust in model's decisions by revealing the decision-making process to be aligned to domain knowledge. Further, it has the potential to debug mispredictions made due to anomalous patterns in the data or faulty training datasets.

Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one.In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.

了解黑盒机器学习模型对于广泛采用至关重要。学习全球可解释的模型是一种方法，但是与他们一起实现高性能是具有挑战性的。另一种方法是使用本地解释的模型来解释个人预测。对于本地可解释的建模，已经提出了各种方法，并且确实使用了常用，但是它们的保真度低，即它们的解释不能很好地近似预测。在本文中，我们的目标是推动高保真性的本地解释建模。我们提出了一个新颖的框架，使用实例的亚采样（LIMIS）进行局部解释的建模。 Limis利用策略梯度选择少数实例，并使用这些选定的实例将黑框模型提炼成一个低容量的本地解释模型。培训是通过衡量本地可解释模型的保真度直接获得的奖励来指导的。我们在多个表格数据集上显示了LIMIS接近匹配黑框模型的预测准确性，从忠诚度和预测准确性方面大大优于最先进的本地解释模型。

In the last years many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness sometimes at the cost of scarifying accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, delineating explicitly or implicitly its own definition of interpretability and explanation. The aim of this paper is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.

人工智能（AI）和机器学习（ML）在网络安全挑战中的应用已在行业和学术界的吸引力，部分原因是对关键系统（例如云基础架构和政府机构）的广泛恶意软件攻击。入侵检测系统（IDS）使用某些形式的AI，由于能够以高预测准确性处理大量数据，因此获得了广泛的采用。这些系统托管在组织网络安全操作中心（CSOC）中，作为一种防御工具，可监视和检测恶意网络流，否则会影响机密性，完整性和可用性（CIA）。 CSOC分析师依靠这些系统来决定检测到的威胁。但是，使用深度学习（DL）技术设计的IDS通常被视为黑匣子模型，并且没有为其预测提供理由。这为CSOC分析师造成了障碍，因为他们无法根据模型的预测改善决策。解决此问题的一种解决方案是设计可解释的ID（X-IDS）。这项调查回顾了可解释的AI（XAI）的最先进的ID，目前的挑战，并讨论了这些挑战如何涉及X-ID的设计。特别是，我们全面讨论了黑匣子和白盒方法。我们还在这些方法之间的性能和产生解释的能力方面提出了权衡。此外，我们提出了一种通用体系结构，该建筑认为人类在循环中，该架构可以用作设计X-ID时的指南。研究建议是从三个关键观点提出的：需要定义ID的解释性，需要为各种利益相关者量身定制的解释以及设计指标来评估解释的需求。

本文研究了与可解释的AI（XAI）实践有关的两个不同但相关的问题。机器学习（ML）在金融服务中越来越重要，例如预批准，信用承销，投资以及各种前端和后端活动。机器学习可以自动检测培训数据中的非线性和相互作用，从而促进更快，更准确的信用决策。但是，机器学习模型是不透明的，难以解释，这是建立可靠技术所需的关键要素。该研究比较了各种机器学习模型，包括单个分类器（逻辑回归，决策树，LDA，QDA），异质集合（Adaboost，随机森林）和顺序神经网络。结果表明，整体分类器和神经网络的表现优于表现。此外，使用基于美国P2P贷款平台Lending Club提供的开放式访问数据集评估了两种先进的事后不可解释能力 - 石灰和外形来评估基于ML的信用评分模型。对于这项研究，我们还使用机器学习算法来开发新的投资模型，并探索可以最大化盈利能力同时最大程度地降低风险的投资组合策略。

如今，人工智能（AI）已成为临床和远程医疗保健应用程序的基本组成部分，但是最佳性能的AI系统通常太复杂了，无法自我解释。可解释的AI（XAI）技术被定义为揭示系统的预测和决策背后的推理，并且在处理敏感和个人健康数据时，它们变得更加至关重要。值得注意的是，XAI并未在不同的研究领域和数据类型中引起相同的关注，尤其是在医疗保健领域。特别是，许多临床和远程健康应用程序分别基于表格和时间序列数据，而XAI并未在这些数据类型上进行分析，而计算机视觉和自然语言处理（NLP）是参考应用程序。为了提供最适合医疗领域表格和时间序列数据的XAI方法的概述，本文提供了过去5年中文献的审查，说明了生成的解释的类型以及为评估其相关性所提供的努力和质量。具体而言，我们确定临床验证，一致性评估，客观和标准化质量评估以及以人为本的质量评估作为确保最终用户有效解释的关键特征。最后，我们强调了该领域的主要研究挑战以及现有XAI方法的局限性。

局部性的好处是石灰的主要前提之一，这是解释黑盒机器学习模型的最突出方法之一。这种强调依赖于一个假设，即我们在本地观察实例附近的越多，黑框模型变得越简单，并且我们可以用线性替代物模拟它越准确。尽管如此，我们的发现似乎是合乎逻辑的，表明，借助石灰的当前设计，当解释过于本地时，即当带宽参数$ \ sigma $趋于零时，替代模型可能会退化。基于此观察，本文的贡献是双重的。首先，我们研究带宽和培训附近对石灰解释的忠诚度和语义的影响。其次，基于我们的发现，我们提出了\史莱姆，这是一种调和忠诚度和位置的石灰的扩展。

这项研究通过对三种不同类型的模型进行基准评估来调查机器学习模型对产生反事实解释的影响：决策树（完全透明，可解释的，白色盒子模型），随机森林（一种半解释，灰色盒模型）和神经网络（完全不透明的黑盒模型）。我们在五个不同数据集（Compas，成人，德国，德语，糖尿病和乳腺癌）中使用四种算法（DICE，WatchERCF，原型和GrowingSpheresCF）测试了反事实生成过程。我们的发现表明：（1）不同的机器学习模型对反事实解释的产生没有影响； （2）基于接近性损失函数的唯一算法是不可行的，不会提供有意义的解释； （3）在不保证反事实生成过程中的合理性的情况下，人们无法获得有意义的评估结果。如果对当前的最新指标进行评估，则不考虑其内部机制中不合理的算法将导致偏见和不可靠的结论； （4）强烈建议对定性分析（以及定量分析），以确保对反事实解释和偏见的潜在识别进行强有力的分析。

本文通过分析每月降雨数据和应用机器学习算法，包括Logistic回归，K-Collect邻居，决策树，随机林和支持向量机，对印度喀拉拉邦的洪水预测模型提供洪水预测模型。虽然这些模型在特定年份的洪水发生的高精度预测中，但它们没有定量和定性地解释预测决定。本文展示了如何学习背景特征，这有助于预测决定，进一步扩展以解释可解释的人工智能模块的开发。所获得的结果证实了解释器模块在喀拉拉邦历史洪水月降雨数据上揭示的研究结果的有效性。

近年来，关于机器学习，AI伦理和算法审核的公平性的讨论增加了。许多实体已经开发了框架指南，以建立公平和问责制的基线标题。但是，尽管讨论增加了，但在实践中仍然很难执行算法和数据审核。许多开源审核工具都可以使用，但是用户并不总是知道这些工具，它们对它们有用或如何访问它们。模型审核和评估并不经常强调机器学习的技能。也有法律原因积极采用这些工具，这些工具超出了对机器学习中更公平的渴望。在我们高度联系的全球社会中，有积极的公众感知和善意问题。对这些工具的更高认识以及积极利用它们的原因可能对AI和机器学习产品的程序员，数据科学家，工程师，研究人员，用户和消费者的整个连续性有所帮助。对于每个人来说，重要的是要更好地了解输入和输出差异，它们的发生方式以及可以在机器和深度学习中促进命运（公平，问责制，透明和道德）的能力。自由访问开源审计工具的能力消除了在机器学习的最基本水平上公平评估的障碍。本文旨在强化迫切需要实际使用这些工具，并为此提供动力。本文突出显示的示例性工具是带有软件或代码碱存储库的开源工具，可立即在全球任何人使用。

The advances in Artificial Intelligence are creating new opportunities to improve lives of people around the world, from business to healthcare, from lifestyle to education. For example, some systems profile the users using their demographic and behavioral characteristics to make certain domain-specific predictions. Often, such predictions impact the life of the user directly or indirectly (e.g., loan disbursement, determining insurance coverage, shortlisting applications, etc.). As a result, the concerns over such AI-enabled systems are also increasing. To address these concerns, such systems are mandated to be responsible i.e., transparent, fair, and explainable to developers and end-users. In this paper, we present ComplAI, a unique framework to enable, observe, analyze and quantify explainability, robustness, performance, fairness, and model behavior in drift scenarios, and to provide a single Trust Factor that evaluates different supervised Machine Learning models not just from their ability to make correct predictions but from overall responsibility perspective. The framework helps users to (a) connect their models and enable explanations, (b) assess and visualize different aspects of the model, such as robustness, drift susceptibility, and fairness, and (c) compare different models (from different model families or obtained through different hyperparameter settings) from an overall perspective thereby facilitating actionable recourse for improvement of the models. It is model agnostic and works with different supervised machine learning scenarios (i.e., Binary Classification, Multi-class Classification, and Regression) and frameworks. It can be seamlessly integrated with any ML life-cycle framework. Thus, this already deployed framework aims to unify critical aspects of Responsible AI systems for regulating the development process of such real systems.

自动错误通常涉及培训数据和学习过程，调试机器学习模型很难。如果我们没有关于模型如何实际工作的线索，这变得更加困难。在这项调查中，我们审查了利用解释的论文使人类提供反馈和调试NLP模型。我们称这个问题解释为基础的人类调试（EBHD）。特别是，我们沿着EBHD的三个维度（错误上下文，工作流程和实验设置）分类和讨论现有工作，编译EBHD组件如何影响反馈提供商的调查结果，并突出可能是未来的研究方向的打开问题。

交互式机器学习（IML）应使智能系统能够从其最终用户进行交互式学习，并迅速变得越来越重要。尽管它将人类置于循环中，但相互作用主要是通过错过上下文信息的相互解释来执行的。此外，Caipi等当前的模型IML策略仅限于“破坏性”反馈，这意味着它们仅允许专家阻止学习者使用无关的功能。在这项工作中，我们提出了一个新颖的互动框架，称为文本域的语义互动学习。我们将将建设性和上下文反馈纳入学习者的问题将其作为找到一个架构的任务，以找到（a）在人与机器之间实现更多的语义对齐，并且（b）同时有助于维持输入域的统计特征。根据有意义的更正生成用户定义的反例。因此，我们介绍了一种称为smanticpush的技术，该技术可有效地将人类对人类的概念校正转换为非排除培训示例，以便将学习者的推理推向所需的行为。在几个实验中，我们表明我们的方法在预测性能以及下游多级分类任务中的局部解释质量方面显然优于Caipi（一种最先进的IML策略）。

Explainability has been widely stated as a cornerstone of the responsible and trustworthy use of machine learning models. With the ubiquitous use of Deep Neural Network (DNN) models expanding to risk-sensitive and safety-critical domains, many methods have been proposed to explain the decisions of these models. Recent years have also seen concerted efforts that have shown how such explanations can be distorted (attacked) by minor input perturbations. While there have been many surveys that review explainability methods themselves, there has been no effort hitherto to assimilate the different methods and metrics proposed to study the robustness of explanations of DNN models. In this work, we present a comprehensive survey of methods that study, understand, attack, and defend explanations of DNN models. We also present a detailed review of different metrics used to evaluate explanation methods, as well as describe attributional attack and defense methods. We conclude with lessons and take-aways for the community towards ensuring robust explanations of DNN model predictions.

Post-hoc explanations of machine learning models are crucial for people to understand and act on algorithmic predictions. An intriguing class of explanations is through counterfactuals, hypothetical examples that show people how to obtain a different prediction. We posit that effective counterfactual explanations should satisfy two properties: feasibility of the counterfactual actions given user context and constraints, and diversity among the counterfactuals presented. To this end, we propose a framework for generating and evaluating a diverse set of counterfactual explanations based on determinantal point processes. To evaluate the actionability of counterfactuals, we provide metrics that enable comparison of counterfactual-based methods to other local explanation methods. We further address necessary tradeoffs and point to causal implications in optimizing for counterfactuals. Our experiments on four real-world datasets show that our framework can generate a set of counterfactuals that are diverse and well approximate local decision boundaries, outperforming prior approaches to generating diverse counterfactuals. We provide an implementation of the framework at https://github.com/microsoft/DiCE. CCS CONCEPTS• Applied computing → Law, social and behavioral sciences.

越来越多的电子健康记录（EHR）数据和深度学习技术进步的越来越多的可用性（DL）已经引发了在开发基于DL的诊断，预后和治疗的DL临床决策支持系统中的研究兴趣激增。尽管承认医疗保健的深度学习的价值，但由于DL的黑匣子性质，实际医疗环境中进一步采用的障碍障碍仍然存在。因此，有一个可解释的DL的新兴需求，它允许最终用户评估模型决策，以便在采用行动之前知道是否接受或拒绝预测和建议。在这篇综述中，我们专注于DL模型在医疗保健中的可解释性。我们首先引入深入解释性的方法，并作为该领域的未来研究人员或临床从业者的方法参考。除了这些方法的细节之外，我们还包括对这些方法的优缺点以及它们中的每个场景都适合的讨论，因此感兴趣的读者可以知道如何比较和选择它们供使用。此外，我们讨论了这些方法，最初用于解决一般域问题，已经适应并应用于医疗保健问题以及如何帮助医生更好地理解这些数据驱动技术。总的来说，我们希望这项调查可以帮助研究人员和从业者在人工智能（AI）和临床领域了解我们为提高其DL模型的可解释性并相应地选择最佳方法。

可解释的人工智能（XAI）是提高机器学习（ML）管道透明度的有前途解决方案。我们将开发和利用XAI方法用于防御和进攻性网络安全任务的研究越来越多（但分散的）缩影。我们确定3个网络安全利益相关者，即模型用户，设计师和对手，将XAI用于ML管道中的5个不同目标，即1）启用XAI的决策支持，2）将XAI应用于安全任务，3）3）通过模型验证通过模型验证xai，4）解释验证和鲁棒性，以及5）对解释的进攻使用。我们进一步分类文献W.R.T.目标安全域。我们对文献的分析表明，许多XAI应用程序的设计都几乎没有了解如何将其集成到分析师工作流程中 - 仅在14％的情况下进行了解释评估的用户研究。文献也很少解开各种利益相关者的角色。特别是，在安全文献中将模型设计师的作用最小化。为此，我们提出了一个说明性用例，突显了模型设计师的作用。我们证明了XAI可以帮助模型验证和可能导致错误结论的案例。系统化和用例使我们能够挑战几个假设，并提出可以帮助塑造网络安全XAI未来的开放问题

由于算法预测对人类的影响增加，模型解释性已成为机器学习（ML）的重要问题。解释不仅可以帮助用户了解为什么ML模型做出某些预测，还可以帮助用户了解这些预测如何更改。在本论文中，我们研究了从三个有利位置的ML模型的解释性：算法，用户和教学法，并为解释性问题贡献了一些新颖的解决方案。

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases. CCS CONCEPTS• Computing methodologies → Machine learning; Supervised learning by classification; • Human-centered computing → Interactive systems and tools.