With the increasing ability of large language models (LLMs), in-context learning (ICL) has become a new paradigm for natural language processing (NLP), where LLMs make predictions only based on contexts augmented with a few training examples. It has been a new trend exploring ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress, challenges, and future work in ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques of ICL, including training strategies, prompting strategies, and so on. Finally, we present the challenges of ICL and provide potential directions for further research. We hope our work can encourage more research on uncovering how ICL works and improving ICL in future work.

Autonomous robotic surgery has advanced significantly based on analysis of visual and temporal cues in surgical workflow, but relational cues from domain knowledge remain under investigation. Complex relations in surgical annotations can be divided into intra- and inter-relations, both valuable to autonomous systems to comprehend surgical workflows. Intra- and inter-relations describe the relevance of various categories within a particular annotation type and the relevance of different annotation types, respectively. This paper aims to systematically investigate the importance of relational cues in surgery. First, we contribute the RLLS12M dataset, a large-scale collection of robotic left lateral sectionectomy (RLLS), by curating 50 videos of 50 patients operated by 5 surgeons and annotating a hierarchical workflow, which consists of 3 inter- and 6 intra-relations, 6 steps, 15 tasks, and 38 activities represented as the triplet of 11 instruments, 8 actions, and 16 objects, totaling 2,113,510 video frames and 12,681,060 annotation entities. Correspondingly, we propose a multi-relation purification hybrid network (MURPHY), which aptly incorporates novel relation modules to augment the feature representation by purifying relational features using the intra- and inter-relations embodied in annotations. The intra-relation module leverages a R-GCN to implant visual features in different graph relations, which are aggregated using a targeted relation purification with affinity information measuring label consistency and feature similarity. The inter-relation module is motivated by attention mechanisms to regularize the influence of relational features based on the hierarchy of annotation types from the domain knowledge. Extensive experimental results on the curated RLLS dataset confirm the effectiveness of our approach, demonstrating that relations matter in surgical workflow analysis.

Large pretrained language models have shown surprising In-Context Learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input without additional parameter updates. Despite the great success in performance, the working mechanism of ICL still remains an open problem. In order to better understand how ICL works, this paper explains language models as meta-optimizers and understands ICL as a kind of implicit finetuning. Theoretically, we figure out that the Transformer attention has a dual form of gradient descent based optimization. On top of it, we understand ICL as follows: GPT first produces meta-gradients according to the demonstration examples, and then these meta-gradients are applied to the original GPT to build an ICL model. Experimentally, we comprehensively compare the behavior of ICL and explicit finetuning based on real tasks to provide empirical evidence that supports our understanding. The results prove that ICL behaves similarly to explicit finetuning at the prediction level, the representation level, and the attention behavior level. Further, inspired by our understanding of meta-optimization, we design a momentum-based attention by analogy with the momentum-based gradient descent algorithm. Its consistently better performance over vanilla attention supports our understanding again from another aspect, and more importantly, it shows the potential to utilize our understanding for future model designing.

Data heterogeneity across clients in federated learning (FL) settings is a widely acknowledged challenge. In response, personalized federated learning (PFL) emerged as a framework to curate local models for clients' tasks. In PFL, a common strategy is to develop local and global models jointly - the global model (for generalization) informs the local models, and the local models (for personalization) are aggregated to update the global model. A key observation is that if we can improve the generalization ability of local models, then we can improve the generalization of global models, which in turn builds better personalized models. In this work, we consider class imbalance, an overlooked type of data heterogeneity, in the classification setting. We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization by combining the uniformity and semantics of class prototypes. FedNH initially distributes class prototypes uniformly in the latent space and smoothly infuses the class semantics into class prototypes. We show that imposing uniformity helps to combat prototype collapse while infusing class semantics improves local models. Extensive experiments were conducted on popular classification datasets under the cross-device setting. Our results demonstrate the effectiveness and stability of our method over recent works.

Score-based modeling through stochastic differential equations (SDEs) has provided a new perspective on diffusion models, and demonstrated superior performance on continuous data. However, the gradient of the log-likelihood function, i.e., the score function, is not properly defined for discrete spaces. This makes it non-trivial to adapt \textcolor{\cdiff}{the score-based modeling} to categorical data. In this paper, we extend diffusion models to discrete variables by introducing a stochastic jump process where the reverse process denoises via a continuous-time Markov chain. This formulation admits an analytical simulation during backward sampling. To learn the reverse process, we extend score matching to general categorical data and show that an unbiased estimator can be obtained via simple matching of the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.

在连续空间中，已经对大都市杂货（M-H）算法进行了充分的研究，但在离散空间中缺乏类似的理解。最近，事实证明，一个本地平衡的建议（LBP）是渐进的最佳选择，但最佳缩放问题仍然开放。在本文中，我们首次确定离散空间中M-H的效率也可以以独立于目标分布的渐近可接受率来表征。此外，我们从理论和经验上验证了LBP和Randy Walk Metropolis（RWM）的最佳接受率分别为$ 0.574 $和0.234美元。这些结果还有助于确定LBP是渐近的$ o（n^\ frac {2} {3}）$比RWM相对于模型尺寸$ n $更有效。了解最佳接受率的知识使人们可以在离散空间中自动调整提案分布的邻域大小，直接类似于连续空间中的尺寸控制。我们从经验上证明，这种适应性M-H采样可以在离散空间中的各种目标分布（包括训练深度能量模型）中的各种目标分布中进行稳健改进采样。

最近，培训预培训方法在以任务为导向的对话框（TOD）系统中表现出了很大的成功。但是，大多数现有的预培训模型用于TOD专注于对话的理解或对话生成，但并非两者兼而有之。在本文中，我们提出了Space-3，这是一种新型的统一的半监督预培训的预训练的对话模型，从大规模对话CORPORA中学习有限的注释，可以有效地对广泛的下游对话任务进行微调。具体而言，Space-3由单个变压器中的四个连续组件组成，以维护TOD系统中的任务流：（i）对话框编码模块编码对话框历史记录，（ii）对话框理解模块以从任一用户中提取语义向量查询或系统响应，（iii）一个对话框策略模块，以生成包含响应高级语义的策略向量，以及（iv）对话框生成模块以产生适当的响应。我们为每个组件设计一个专门的预训练目标。具体而言，我们预先培训对话框编码模块，使用跨度掩码语言建模，以学习上下文化对话框信息。为了捕获“结构化对话框”语义，我们通过额外的对话注释通过新颖的树诱导的半监视对比度学习目标来预先培训对话框理解模块。此外，我们通过将其输出策略向量与响应响应的语义向量之间的L2距离最小化以进行策略优化，从而预先培训对话策略模块。最后，对话框生成模型由语言建模预先训练。结果表明，Space-3在八个下游对话框基准中实现最新性能，包括意图预测，对话框状态跟踪和端到端对话框建模。我们还表明，在低资源设置下，Space-3比现有模型具有更强的射击能力。

联合超分辨率和反音调映射（SR-ITM）旨在提高具有分辨率和动态范围具有质量缺陷的视频的视觉质量。当使用4K高动态范围（HDR）电视来观看低分辨率标准动态范围（LR SDR）视频时，就会出现此问题。以前依赖于学习本地信息的方法通常在保留颜色合规性和远程结构相似性方面做得很好，从而导致了不自然的色彩过渡和纹理伪像。为了应对这些挑战，我们建议联合SR-ITM的全球先验指导的调制网络（GPGMNET）。特别是，我们设计了一个全球先验提取模块（GPEM），以提取颜色合规性和结构相似性，分别对ITM和SR任务有益。为了进一步利用全球先验并保留空间信息，我们使用一些用于中间特征调制的参数，设计多个全球先验的指导空间调制块（GSMB），其中调制参数由共享的全局先验和空间特征生成来自空间金字塔卷积块（SPCB）的地图。通过这些精心设计的设计，GPGMNET可以通过较低的计算复杂性实现更高的视觉质量。广泛的实验表明，我们提出的GPGMNET优于最新方法。具体而言，我们提出的模型在PSNR中超过了0.64 dB的最新模型，其中69 $ \％$ $ $较少，3.1 $ \ times $ speedup。该代码将很快发布。

在混合完成的多任务，多域和多模式数据上进行预训练仍然是视力感知预训练的开放挑战。在本文中，我们提出了GPPF，这是一个普遍的感知预训练框架，预先培训任务级的动态网络，该网络是由在标签的多任务和多域数据集上的各层知识“乐高”组成的。通过检查人类在复杂环境中学习的先天能力，我们识别并将三个关键要素转移到深网上：（1）同时暴露于每个批次中的各种交叉任务和跨域信息。 （2）由知识共享驱动的单独的乐高单元中的分区知识存储。 （3）用于训练和下游任务的乐高单元子集的稀疏激活。值得注意的是，由于其在输入形状，损失功能，输出格式，数据分布等方面的差异，不同视觉任务的联合培训是不平凡的。因此，我们创新地开发了插件的多任务培训算法，该培训算法是支持单个迭代多个任务（SIMT）同时培训。 Simt用大型多任务多任务数据集为预训练的基础奠定了基础，并且被证明对于我们的GPPF实验中的稳定培训至关重要。令人兴奋的是，详尽的实验表明，我们的GPPF-R50型号在GPPF-15M中的8个预训练预培训任务的强大基线上取得了显着改善，并在22个下游任务中收获了一系列SOTA，并具有相似的计算预算。我们还验证了GPPF对SOTA视觉变压器的概括能力，并具有一致的改进。这些可靠的实验结果充分证明了我们新颖的GPPF框架提供的有效的知识学习，存储，共享和转移。

与传统方法相比，学到的图像压缩已在PSNR和MS-SSIM中取得了非凡的速率延伸性能。但是，它遭受了密集的计算，这对于现实世界的应用是无法忍受的，目前导致其工业应用有限。在本文中，我们将神经体系结构搜索（NAS）介绍到具有较低延迟的更有效网络，并利用量化以加速推理过程。同时，已经为提高效率而做出了工程努力。使用PSNR和MS-SSIM的混合损失以更好的视觉质量进行了优化，我们获得的MSSIM比JPEG，JPEG XL和AVIF在所有比特率上都高得多，而JPEG XL和AVIF之间的PSNR则获得了PSNR。与JPEG-Turbo相比，我们的LIC的软件实施实现了可比较甚至更快的推理速度，而多次比JPEG XL和AVIF快。此外，我们的LIC实施达到了145 fps的惊人吞吐量，用于编码为208 fps，用于在Tesla T4 GPU上解码1080p图像。在CPU上，我们实施的延迟与JPEG XL相当。