数据隐私是可以感知环境,与人类交流并在现实世界中采取行动的体现代理的核心问题。在帮助人类完成任务的同时,代理商可以观察和处理用户的敏感信息,例如房屋环境,人类活动等。在这项工作中,我们介绍了隐私的体现的代理人学习,以实现视觉和语言导航的任务( VLN),其中一个体现的代理通过按照自然语言说明进行导航房屋环境。我们将每个房屋环境视为本地客户,除了与云服务器和其他客户端共享本地更新,并提出了一个新颖的联合视觉和语言导航(FIDVLN)框架,以保护培训和预培训期间的数据隐私勘探。特别是,我们提出了一种分散的培训策略,以将每个客户的数据限制在其本地模型培训中,并采用联合的预探测方法,以进行部分模型聚合,以改善模型的通用性,使其对看不见的环境。 R2R和RXR数据集的广泛结果表明,在我们的FEDVLN框架下,分散的VLN模型在集中式培训的同时,在保护可见的环境隐私的同时,取得了可比的结果,并且联合预探索明显超过了集中式预探测,同时保留了不见了的环境隐私。
translated by 谷歌翻译
Federated embodied agent learning protects the data privacy of individual visual environments by keeping data locally at each client (the individual environment) during training. However, since the local data is inaccessible to the server under federated learning, attackers may easily poison the training data of the local client to build a backdoor in the agent without notice. Deploying such an agent raises the risk of potential harm to humans, as the attackers may easily navigate and control the agent as they wish via the backdoor. Towards Byzantine-robust federated embodied agent learning, in this paper, we study the attack and defense for the task of vision-and-language navigation (VLN), where the agent is required to follow natural language instructions to navigate indoor environments. First, we introduce a simple but effective attack strategy, Navigation as Wish (NAW), in which the malicious client manipulates local trajectory data to implant a backdoor into the global model. Results on two VLN datasets (R2R and RxR) show that NAW can easily navigate the deployed VLN agent regardless of the language instruction, without affecting its performance on normal test sets. Then, we propose a new Prompt-Based Aggregation (PBA) to defend against the NAW attack in federated VLN, which provides the server with a ''prompt'' of the vision-and-language alignment variance between the benign and malicious clients so that they can be distinguished during training. We validate the effectiveness of the PBA method on protecting the global model from the NAW attack, which outperforms other state-of-the-art defense methods by a large margin in the defense metrics on R2R and RxR.
translated by 谷歌翻译
事实证明,演讲者的追随者模型在视觉和语言导航中有效,在该导航中,扬声器模型用于合成新的说明,以增强追随者导航模型的培训数据。但是,在以前的许多方法中,生成的指令未直接训练以优化追随者的性能。在本文中,我们介绍\ textsc {foam},a \ textsc {fo} llower- \ textsc {a} ware speaker \ textsc {m} odel,它不断更新给定关注的反馈,以便生成的指令可以是更多的指令。适合当前追随者的学习状态。具体而言,我们使用BI级优化框架优化了扬声器,并通过评估标记数据的跟随器来获得其训练信号。房间对房间和房间的室内数据集中的实验结果表明,我们的方法可以超越跨设置的强大基线模型。分析还表明,我们生成的指示的质量比基线更高。
translated by 谷歌翻译
视觉和语言导航(VLN)是一个任务,代理在人类指令下的体现室内环境中导航。以前的作品忽略了样本难度的分布,我们认为这可能会降低他们的代理表现。为了解决这个问题,我们为VLN任务提出了一种基于课程的基于课程的培训范式,可以平衡人类的先验知识和特工关于培训样本的学习进度。我们开发课程设计原则,并重新安排基准房间到室(R2R)数据集,以使其适用于课程培训。实验表明,我们的方法是模型 - 不可知的,可以显着提高当前最先进的导航剂的性能,概括性和培训效率而不会增加模型复杂性。
translated by 谷歌翻译
视觉和语言导航(VLN)任务要求代理根据语言说明浏览环境。在本文中,我们旨在解决此任务中的两个关键挑战:利用多语言指令改进教学路径接地,并在培训期间看不见的新环境中导航。为了应对这些挑战,我们提出了明确的:跨语性和环境不可屈服的表示。首先,我们的经纪人在室内室内数据集中学习了三种语言(英语,印地语和泰卢固语)的共享且视觉上的跨语言表示。我们的语言表示学习是由视觉信息对齐的文本对指导的。其次,我们的代理商通过从不同环境中最大化语义对齐的图像对(对象匹配的约束)之间的相似性来学习环境不足的视觉表示。我们的环境不可知的视觉表示可以减轻低级视觉信息引起的环境偏见。从经验上讲,在房间 - 室内数据集中,我们表明,当通过跨语性语言表示和环境 - 非局部视觉表示形式概括地看不见的环境时,我们的多语言代理在所有指标上都比强大的基线模型进行了巨大改进。此外,我们表明我们学到的语言和视觉表示可以成功地转移到房间和合作的视觉和二元式导航任务上,并提出详细的定性和定量的概括和基础分析。我们的代码可从https://github.com/jialuli-luka/clear获得
translated by 谷歌翻译
Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions. However, given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. Pretraining on large text and image-text datasets from the web has been extensively explored but the improvements are limited. We investigate large-scale augmentation with synthetic instructions. We take 500+ indoor environments captured in densely-sampled 360 degree panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory using Marky, a high-quality multilingual navigation instruction generator. We also synthesize image observations from novel viewpoints using an image-to-image GAN. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets, and contains a wider variety of environments and viewpoints. To efficiently leverage data at this scale, we train a simple transformer agent with imitation learning. On the challenging RxR dataset, our approach outperforms all existing RL agents, improving the state-of-the-art NDTW from 71.1 to 79.1 in seen environments, and from 64.6 to 66.8 in unseen test environments. Our work points to a new path to improving instruction-following agents, emphasizing large-scale imitation learning and the development of synthetic instruction generation capabilities.
translated by 谷歌翻译
视觉导航要求代理商遵循自然语言说明以达到特定目标。可见的环境和看不见的环境之间的巨大差异使代理商概括良好的挑战。先前的研究提出了数据增强方法,以明确或隐式地减轻数据偏见并提供概括的改进。但是,他们试图记住增强的轨迹,并在测试时忽略在看不见的环境下的分布变化。在本文中,我们提出了一个看不见的差异,预期视力和语言导航(戴维斯),该差异通过鼓励测试时间的视觉一致性来概括为看不见的环境。具体来说,我们设计了:1)半监督框架戴维斯(Davis),该框架利用类似的语义观测来利用视觉一致性信号。 2)一个两阶段的学习程序,鼓励适应测试时间分布。该框架增强了模仿和强化学习的基本混合物与动量形成对比,以鼓励在联合训练阶段和测试时间适应阶段对类似观察的稳定决策。广泛的实验表明,戴维斯在R2R和RXR基准上实现了与先前最先进的VLN基线相比,取得了模型不合命源性的改进。我们的源代码和数据是补充材料。
translated by 谷歌翻译
视觉和语言导航(VLN)是一种任务,即遵循语言指令以导航到目标位置的语言指令,这依赖于在移动期间与环境的持续交互。最近的基于变压器的VLN方法取得了很大的进步,从视觉观测和语言指令之间的直接连接通过多模式跨关注机制。然而,这些方法通常代表通过使用LSTM解码器或使用手动设计隐藏状态来构建反复变压器的时间上下文作为固定长度矢量。考虑到单个固定长度向量通常不足以捕获长期时间上下文,在本文中,我们通过显式建模时间上下文来引入具有可变长度存储器(MTVM)的多模式变压器,通过模拟时间上下文。具体地,MTVM使代理能够通过直接存储在存储体中的先前激活来跟踪导航轨迹。为了进一步提高性能,我们提出了内存感知的一致性损失,以帮助学习随机屏蔽指令的时间上下文的更好关节表示。我们在流行的R2R和CVDN数据集上评估MTVM,我们的模型在R2R看不见的验证和测试中提高了2%的成功率,并在CVDN测试集上减少了1.6米的目标进程。
translated by 谷歌翻译
在视觉和语言导航(VLN)中,按照自然语言指令在现实的3D环境中需要具体的代理。现有VLN方法的一个主要瓶颈是缺乏足够的培训数据,从而导致对看不见的环境的概括不令人满意。虽然通常会手动收集VLN数据,但这种方法很昂贵,并且可以防止可扩展性。在这项工作中,我们通过建议从HM3D自动创建900个未标记的3D建筑物的大规模VLN数据集来解决数据稀缺问题。我们为每个建筑物生成一个导航图,并通过交叉视图一致性从2D传输对象预测,从2D传输伪3D对象标签。然后,我们使用伪对象标签来微调一个预处理的语言模型,作为减轻教学生成中跨模式差距的提示。在导航环境和说明方面,我们生成的HM3D-AUTOVLN数据集是比现有VLN数据集大的数量级。我们通过实验表明,HM3D-AUTOVLN显着提高了所得VLN模型的概括能力。在SPL指标上,我们的方法分别在Reverie和DataSet的看不见的验证分裂分别对艺术的状态提高了7.1%和8.1%。
translated by 谷歌翻译
联合学习是一种数据解散隐私化技术,用于以安全的方式执行机器或深度学习。在本文中,我们介绍了有关联合学习的理论方面客户次数有所不同的用例。具体而言,使用从开放数据存储库中获得的胸部X射线图像提出了医学图像分析的用例。除了与隐私相关的优势外,还将研究预测的改进(就曲线下的准确性和面积而言)和减少执行时间(集中式方法)。将从培训数据中模拟不同的客户,以不平衡的方式选择,即,他们并非都有相同数量的数据。考虑三个或十个客户之间的结果与集中案件相比。间歇性客户将分析两种遵循方法,就像在实际情况下,某些客户可能会离开培训,一些新的新方法可能会进入培训。根据准确性,曲线下的区域和执行时间的结果,结果的结果的演变显示为原始数据被划分的客户次数。最后,提出了该领域的改进和未来工作。
translated by 谷歌翻译
愿景 - 语言导航(VLN)任务要求代理逐步导航,同时感知视觉观察并理解自然语言指令。大数据偏置,这是由小数据量表和大型导航空间之间的视差比率引起的,使得VLN任务具有挑战性。以前的作品提出了各种数据增强方法来减少数据偏差。但是,这些作品不会明确降低不同房间场景的数据偏差。因此,该代理将覆盖所见的场景,并在看不见的场景中实现较差的导航性能。为了解决这个问题,我们提出了随机环境混合(REM)方法,它通过混合环境作为增强数据生成交叉连接的房屋场景。具体而言,我们首先根据每个场景的房间连接图选择键视点。然后,我们交叉连接不同场景的关键视图,以构建增强场景。最后,我们在交叉连接场景中生成增强的指令路径对。基准数据集的实验结果表明,我们的增强数据通过REM帮助代理商会降低所见和看不见的环境之间的性能差距,提高整体性能,使我们的模型成为标准VLN基准的最佳现有方法。该代码已发布:https://github.com/lcfractal/vlnrem。
translated by 谷歌翻译
高效联合学习是在边缘设备上培训和部署AI模型的关键挑战之一。然而,在联合学习中维护数据隐私提出了几种挑战,包括数据异质性,昂贵的通信成本和有限的资源。在本文中,我们通过(a)通过基于本地客户端的深度增强学习引入突出参数选择代理的上述问题,并在中央服务器上聚合所选择的突出参数,(b)分割正常的深度学习模型〜 (例如,CNNS)作为共享编码器和本地预测器,并通过联合学习训练共享编码器,同时通过本地自定义预测器将其知识传送到非IID客户端。所提出的方法(a)显着降低了联合学习的通信开销,并加速了模型推断,而方法(b)则在联合学习中解决数据异质性问题。此外,我们利用梯度控制机制来校正客户之间的梯度异质性。这使得训练过程更稳定并更快地收敛。实验表明,我们的方法产生了稳定的训练过程,并与最先进的方法相比实现了显着的结果。在培训VGG-11时,我们的方法明显降低了通信成本最高108 GB,并在培训Reset-20时需要7.6美元的通信开销,同时通过减少高达39.7 \%$ 39.7 \%$ vgg- 11.
translated by 谷歌翻译
Federated learning achieves joint training of deep models by connecting decentralized data sources, which can significantly mitigate the risk of privacy leakage. However, in a more general case, the distributions of labels among clients are different, called ``label distribution skew''. Directly applying conventional federated learning without consideration of label distribution skew issue significantly hurts the performance of the global model. To this end, we propose a novel federated learning method, named FedMGD, to alleviate the performance degradation caused by the label distribution skew issue. It introduces a global Generative Adversarial Network to model the global data distribution without access to local datasets, so the global model can be trained using the global information of data distribution without privacy leakage. The experimental results demonstrate that our proposed method significantly outperforms the state-of-the-art on several public benchmarks. Code is available at \url{https://github.com/Sheng-T/FedMGD}.
translated by 谷歌翻译
旨在为通用机器人铺平道路的边界研究,视觉和语言导航(VLN)一直是计算机视觉和自然语言处理社区的热门话题。 VLN任务要求代理在不熟悉的环境中按照自然语言说明导航到目标位置。最近,基于变压器的模型已在VLN任务上获得了重大改进。由于变压器体系结构中的注意力机制可以更好地整合视觉和语言的模式内和模式信息。但是,当前基于变压器的模型中存在两个问题。 1)模型独立处理每个视图,而无需考虑对象的完整性。 2)在视觉模态的自我注意操作期间,在空间上遥远的视图可以彼此交织而无需明确的限制。这种混合可能会引入额外的噪音而不是有用的信息。为了解决这些问题,我们建议1)基于插槽注意的模块,以合并来自同一对象的分割的信息。 2)局部注意力掩模机制限制视觉注意力跨度。所提出的模块可以轻松地插入任何VLN体系结构中,我们将复发的VLN-Bert用作基本模型。 R2R数据集的实验表明,我们的模型已达到最新结果。
translated by 谷歌翻译
Federated learning (FL) has been proposed as a privacy-preserving approach in distributed machine learning. A federated learning architecture consists of a central server and a number of clients that have access to private, potentially sensitive data. Clients are able to keep their data in their local machines and only share their locally trained model's parameters with a central server that manages the collaborative learning process. FL has delivered promising results in real-life scenarios, such as healthcare, energy, and finance. However, when the number of participating clients is large, the overhead of managing the clients slows down the learning. Thus, client selection has been introduced as a strategy to limit the number of communicating parties at every step of the process. Since the early na\"{i}ve random selection of clients, several client selection methods have been proposed in the literature. Unfortunately, given that this is an emergent field, there is a lack of a taxonomy of client selection methods, making it hard to compare approaches. In this paper, we propose a taxonomy of client selection in Federated Learning that enables us to shed light on current progress in the field and identify potential areas of future research in this promising area of machine learning.
translated by 谷歌翻译
联邦学习一直是一个热门的研究主题,使不同组织的机器学习模型的协作培训在隐私限制下。随着研究人员试图支持更多具有不同隐私方法的机器学习模型,需要开发系统和基础设施,以便于开发各种联合学习算法。类似于Pytorch和Tensorflow等深度学习系统,可以增强深度学习的发展,联邦学习系统(FLSS)是等效的,并且面临各个方面的面临挑战,如有效性,效率和隐私。在本调查中,我们对联合学习系统进行了全面的审查。为实现流畅的流动和引导未来的研究,我们介绍了联合学习系统的定义并分析了系统组件。此外,我们根据六种不同方面提供联合学习系统的全面分类,包括数据分布,机器学习模型,隐私机制,通信架构,联合集市和联合的动机。分类可以帮助设计联合学习系统,如我们的案例研究所示。通过系统地总结现有联合学习系统,我们展示了设计因素,案例研究和未来的研究机会。
translated by 谷歌翻译
联合学习(FL)和分裂学习(SL)是两种新兴的协作学习方法,可能会极大地促进物联网(IoT)中无处不在的智能。联合学习使机器学习(ML)模型在本地培训的模型使用私人数据汇总为全球模型。分裂学习使ML模型的不同部分可以在学习框架中对不同工人进行协作培训。联合学习和分裂学习,每个学习都有独特的优势和各自的局限性,可能会相互补充,在物联网中无处不在的智能。因此,联合学习和分裂学习的结合最近成为一个活跃的研究领域,引起了广泛的兴趣。在本文中,我们回顾了联合学习和拆分学习方面的最新发展,并介绍了有关最先进技术的调查,该技术用于将这两种学习方法组合在基于边缘计算的物联网环境中。我们还确定了一些开放问题,并讨论了该领域未来研究的可能方向,希望进一步引起研究界对这个新兴领域的兴趣。
translated by 谷歌翻译
Federated learning has recently been applied to recommendation systems to protect user privacy. In federated learning settings, recommendation systems can train recommendation models only collecting the intermediate parameters instead of the real user data, which greatly enhances the user privacy. Beside, federated recommendation systems enable to collaborate with other data platforms to improve recommended model performance while meeting the regulation and privacy constraints. However, federated recommendation systems faces many new challenges such as privacy, security, heterogeneity and communication costs. While significant research has been conducted in these areas, gaps in the surveying literature still exist. In this survey, we-(1) summarize some common privacy mechanisms used in federated recommendation systems and discuss the advantages and limitations of each mechanism; (2) review some robust aggregation strategies and several novel attacks against security; (3) summarize some approaches to address heterogeneity and communication costs problems; (4)introduce some open source platforms that can be used to build federated recommendation systems; (5) present some prospective research directions in the future. This survey can guide researchers and practitioners understand the research progress in these areas.
translated by 谷歌翻译
语音情感识别(SER)处理语音信号以检测和表征表达的感知情绪。许多SER应用系统经常获取和传输在客户端收集的语音数据,以远程云平台进行推理和决策。然而,语音数据不仅涉及在声乐表达中传达的情绪,而且还具有其他敏感的人口特征,例如性别,年龄和语言背景。因此,塞尔系统希望能够在防止敏感和人口统计信息的意外/不正当推论的同时对情感构建进行分类的能力。联合学习(FL)是一个分布式机器学习范例,其协调客户端,以便在不共享其本地数据的情况下协同培训模型。此培训方法似乎是安全的,可以提高SER的隐私。然而,最近的作品表明,流动方法仍然容易受到重建攻击和会员推论攻击等各种隐私攻击的影响。虽然这些大部分都集中在计算机视觉应用程序上,但是使用FL技术训练的SER系统中存在这种信息泄漏。为了评估使用FL培训的SER系统的信息泄漏,我们提出了一个属性推理攻击框架,其分别涉及来自共享梯度或模型参数的客户端的敏感属性信息,分别对应于FEDSGD和FADAVG训练算法。作为一种用例,我们使用三个SER基准数据集来统一地评估我们预测客户的性别信息的方法:IEMocap,Crema-D和MSP-EXPLA。我们表明,使用FL培训的SER系统可实现属性推理攻击。我们进一步确定大多数信息泄漏可能来自SER模型中的第一层。
translated by 谷歌翻译
A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a naturallanguage navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visuallygrounded navigation instructions, we present the Matter-port3D Simulator -a large-scale reinforcement learning environment based on real imagery [11]. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -the Room-to-Room (R2R) dataset 1 .1 https://bringmeaspoon.org Instruction: Head upstairs and walk past the piano through an archway directly in front. Turn right when the hallway ends at pictures and table. Wait by the moose antlers hanging on the wall.
translated by 谷歌翻译