Cross-Spectral Neural Radiance Fields

Given an observed scene, NeRF [Mildenhall2020ECCV] allows for novel view synthesis from arbitrary vantage points. This is achieved by training a neural network, i.e. a Multi Layer Perceptron (MLP), on a set of sparse images collected from different viewpoints. The MLP parametrises the Radiance Field of the scene, i.e. a function of continuous 5D values $(x,y,z,\theta ,\varphi )$, where $x=(x,y,z)$ are 3D coordinates in space and $(\theta ,\varphi )$ are viewing angles, function of camera pose $\Pi$. The direction can be also expressed as a 3D Cartesian unit vector $d$. Such a function produces a 4D output R,G,B,$\sigma$, encoding the color ($c=(R,G,B)$) and volume density $\left(\sigma \right)$ of each 3D point in the scene. Specifically, the vanilla NeRF estimates color and density by means of two MLPs as $(\sigma ,e)={\text{MLP}}^{\left(pos\right)}\left(x\right)$, $c={\text{MLP}}^{\left(rgb\right)}(e,d)$, with $\sigma$ being interpreted as the differential probability of a ray terminating at $(x,y,z)$, and $e$ being a feature embedding.

Volume Rendering. According to [Max1995Optical], the color $C\left(r\right)$ rendered from a camera ray $r\left(t\right)=o+td$ is obtained by solving the following integral:

with $T\left(t\right)$ being the accumulated transmittance from $t_n$ to $t$ along ray $r$. The value of the integral is estimated via quadrature, by sampling $[t_n,t_f]$ in N evenly-spaced bins, with $t_n$ and $t_f$ being the near and far plane respectively.

with ${\delta }_i$ being the distance between adjacent samples $t_{i+1}$ and $t_i$. This procedure turns out to be equivalent to alpha compositing, assuming ${\alpha }_i=1-\text{exp}(-{\sigma }_i{\delta }_i)$. Terms $T\left(i\right){\alpha }_i$ act as a weight $\left(w_i\right)$ for each point along the ray.

Positional Encoding with Fourier Features. Traditionally, neural networks excel at learning low-frequency representations at the expense of high-frequency ones. As shown by NeRF [Mildenhall2020ECCV], encoding 3D coordinates $x$ into a higher dimensional space allows to better recover the latter. This is achieved by applying a so-called Fourier mapping $\gamma$ to each input component $p$ independently as $\gamma \left(p\right)=(\sin \left(2^0\pi p\right),\cos \left(2^0\pi p\right),...,\sin \left(2^{L-1}\pi p\right),\cos \left(2^{L-1}\pi p\right))$.

Ray Coordinates. According to the specific scene, two main coordinate systems are used to compute 3D coordinates of points laying along rays. In case of 360° scenes framing objects with masked backgrounds, conventional world coordinate systems are used, requiring the definition of near/far bounding planes. In case of forward-facing scenes, i.e. the camera rotation between views is small or absent, Normalized Device Coordinates (NDC) [Mildenhall2020ECCV, zhang2020nerfpp] are used as standard convention, warping an infinitely deep camera frustum into a bounded $[-1,1{]}^3$ cube, where distance along the z-axis corresponds to disparity (inverse distance). This parameterization optimizes the network capacity in a way that is consistent with the geometry of perspective projection, easing the problem itself – in particular, in presence of large displacements between foreground and background [zhang2020nerfpp].

Differently from the original NeRF formulation, our goal is to obtain a Cross-Spectral neural scene representation. We assume availability of $N_m$ cameras featuring different modalities (e.g., RGB, infrared, multi-spectral), mounted on a rigid system – i.e. with fixed relative poses between cameras. Each camera with modality $m$ has a spatial resolution of $H_m\times W_m$ and $C_m$ number of channels, and it has been previously calibrated to estimate the intrinsic parameters $f_m^x,f_m^y,c_m^x,c_m^y$ (focal length and piercing point). For each camera, we acquire a set of $N_{views}$ images from different viewpoints of the same scene. Thus, we gather a total of $N_m\times N_{views}$ images per scene.

We learn the Cross-Spectral scene representation as a function to map 5D coordinates $(x,y,z,\theta ,\varphi )$ into a volume density ($\sigma$) and $N_m$ output modalities, with each modality having $C_m$ channels, for a total of ${\sum }_m{C}_m$ channels. In our setup, we assume a shared volume density across modalities: this means rays emitted by the different sensors hit the very same 3D points and frame it in the images. This assumption would not hold in case this latter hypothesis is violated (e.g., when dealing with RGB and X-rays sensors).

At each optimization iteration, we select a training image with modality $m$, iterating over all the possible modalities at each step. Then, we sample a random batch of camera rays from the set of all its pixels. For each ray $r$, we then use the volume rendering procedure described in Sec. 3.1 to estimate the response of that modality, ${\hat{C}}_m\left(r\right)$. Our loss is simply the total squared error between the rendered and true pixel values for the considered modality:

where $R$ is the set of rays in each batch, and $C_m\left(r\right)$ and ${\hat{C}}_m\left(r\right)$ are the ground truth and predicted modality for ray $r$, respectively. Since each modality is acquired from different viewpoints, for a single ray $r$ we can never compute the loss on multiple modalities. Thus, the training is carried out on the different modalities in interleaved manner.

Pose Estimation. NeRF assumes camera poses to be known beforehand during training. This information is typically retrieved by means of COLMAP [schoenberger2016sfm] on the set of RGB images based on matching between image keypoints. Since in our case we have images captured by cameras with different modalities, it is extremely hard to estimate reliable keypoints and descriptors amenable to perform matching across modalities. A possible solution could be to apply COLMAP on each modality independently. However, the estimated poses would be in different reference systems (typically the first frame of the sequence) and up to different scale factors, and would be not trivial to align all cameras in a shared reference system – since it would require, again, matching across modalities.

However, as we assume cameras to be mounted on the same rigid rig, we can exploit COLMAP only on a single modality, and learn the relative poses between sensors as latent variables optimized during training, thereby avoiding the problem of matching pixel across spectra. One may argue that relative poses could be estimated offline through calibration [zhang2000flexible]. It is however non-obvious how to perform it across distant spectra, e.g. LWIR vs RGB may need ad-hoc calibration patterns [shivakumar2020pst900]. More importantly, such poses would be metric and thus require alignment with the unknown COLMAP scale, a non-trivial problem itself.

Formally, given poses ${\Pi }_{m_{\alpha }}^i,i\in \left\{\mathrm{1..}{N}_{views}\right\}$ estimated by COLMAP on a reference modality $m_{\alpha }$ – RGB in our setup – we learn the relative poses ${\Pi }_{m_{\alpha }\to m}$ for any modality $m\ne m_{\alpha }$. These, multiplied by ${\Pi }_{m_{\alpha }}^i$, allow for obtaining poses ${\Pi }_m^i$ for any image collected by the camera of modality $m$. Relative poses are learned by back-propagating the loss from Eq. 3 up to ${\Pi }_{m_{\alpha }\to m}$ as in [wang2021nerf]

We highlight that each pose ${\Pi }_{m_{\alpha }\to m}$ is learned based on the reconstruction loss of its modality $m$ solely, without enforcing any match across modalities. We will show empirically that this is sufficient to estimate consistent poses.

NXDC: Normalized Cross Devices Coordinate. In forward-facing scenes, ray coordinates are usually expressed in Normalized Device Coordinates (NDC) [Mildenhall2020ECCV]. However, NDC assumes that all images have been acquired by the same camera. In our case, we have several cameras with marked differences such as resolution, focal length, etc. Using every camera with its own intrinsics would cause a misalignment between ray coordinates across devices, leading X-NeRF to learn a non-registered scene representation – i.e., the rendered images from the same point of view would be misaligned. Thus, we introduce Normalized Cross-Device Coordinates (NXDC). Assuming cameras looking in the $-z$ direction ¹¹1http://www.songho.ca/opengl/gl_projectionmatrix.html, 3D points (in homogeneous coordinates) are projected according to perspective projection matrix $M$, function of near/far clipping planes $n,f$ and top/right scene bounds $r,t$ at near plane $n$

The projected point is now in NDC space, where the frustum has been mapped to a [-1,1]${}^3$ cube. Given a ray $o+td$, we want to find the ray in NDC space that traces out the same point as the original ray (either at the same rate or not) – i.e., compute the ray origin $o^{\prime }$ and direction $d^{\prime }$ such that, for every sampled point with $t$, there exists a $t^{\prime }$ such that $\pi (o+td)=o^{\prime }+t^{\prime }{d}^{\prime }$. We define:

Assuming that the far scene bound is infinity, we obtain $a_z=1,b_z=2n$. If we consider the standard pinhole camera math, we can rewrite as:

where $f_x,f_y,H,W$ are the $x$ and $y$ focal lengths, height and width, respectively.

However, we employ different devices, thus we have different focal length, height and width for each camera. Simply using for each device its own parameters would lead to a different normalization across devices –i.e., different reference systems – and thus X-NeRF will not be able to learn a unified scene representation with registered modalities. To overcome this problem, we constrain ratios $\frac{f_x}{W}$ and $\frac{f_y}{H}$ to be fixed across devices. To achieve this, we select the camera modality having minimum focal/image size ratio – i.e., the largest FoV:

Following NeRF derivation ²²2https://github.com/bmild/nerf/files/4451808/ndc_derivation.pdf, we get $o^{\prime }$, $t^{\prime }$, and $d^{\prime }$ according to fixed ratios as:

In practice, this equals to padding images from sensors with lower FoV, while keeping focals unaltered. We dub the framework presented so far as X-NeRF.

Speeding-up X-NeRF. Finally, given the recent advances concerning fast training and rendering [sun2021direct, yu2021plenoxels, mueller2022instant], we also implement a faster variant of X-NeRF to bring it closer to unconstrained use in real applications. Specifically, we exploit a mixed implicit-explicit representation to speed up both phases. Following DirectVoxGO (DVGO) [sun2021direct], we implement voxel grids – actually, Multi-Plane Images (MPIs) in the case of forward-facing scenes [sun2022improved] – allowing for efficient queries in 3D space. Two structures, $M^{\text{(dens)}}$ and $M^{\text{(feat)}}$, are built respectively to encode density and feature embeddings, from which $\sigma$ is extracted by means of trilinear interpolation on the former, while color $c$ is predicted by a shallow MLP queried with features interpolated from the latter as $\sigma \left(x\right)=\text{interp}(x,M^{\text{(dens)}})$, $c(x,d)={\text{MLP}}^{\left(rgb\right)}\left(\text{interp}\right(x,M^{\text{(feat)}}),x,d)$. Both are optimized through back-propagation during training. We dub this variant of our framework X-DVGO, since it extends the DVGO framework.

Our acquisition setup consists of three devices with different spectral sensitivity: a Ximea RGB camera equipped with a Sony IMX253LQR-C 12.4 Mpx sensor; an MS camera sensitive to 10 bands within the visible spectrum, based on an IM-SM4X4-VIS2 2.2 Mpx (one MS pixel capturing the 10 bands information uses a 4$\times$4 grid of native pixels, thus reducing the spatial resolution to 1/16); the passive infrared sensor of an Azure Kinect device, with a native resolution of 1Mpx. Accordingly, our rig perceives 14 total channels across the sensed modalities. The three cameras have been mounted on a rig, as depicted in Fig. 2, to acquire 16 indoor scenes – since the Kinect IR sensor saturates outdoor – with $\sim$30 images from each sensor per scene, 5 kept out for testing and the remaining used for training. As already mentioned, in this paper we focus on scenes acquired in forward-facing settings. Examples of images acquired by our rig are shown in Fig. 2, where IR and MS images are encoded with colormaps magma and viridis, respectively (with MS being averaged over channels).

We implemented our framework using PyTorch. During training and testing, all images are normalized in $[0,1]$ over a single scene and modality, clipping intensities to the 99th percentile to filter intensity peaks in MS and IR images. Since IR images acquired by the Kinect are particularly noisy, we pre-process them by means of a $7\times 7$ bilateral filter [BILATERAL]. In the remainder of this sub-section we report implementation details concerning X-NeRF and X-DVGO, whose output layers have been extended to predict multiple modalities as described in Sec. 3.2.

X-NeRF. It is built on top of the NeRF– codebase [wang2021nerf], which replicates the vanilla NeRF except for (i) not using hierarchical sampling strategy, (ii) reducing hidden layers dimension from 256 to 128 and (iii) sampling only 128 points along each ray, following [wang2021nerf] to pursue computational efficiency. X-NeRF is trained for 5K epochs on each scene, i.e. $\sim$450k steps (150K per modality, $\sim$2.5 hours of overall training on a single 3090 RTX GPU).

X-DVGO. It is built on top of the DVGO codebase [sun2021direct], using 128 depth planes and a shallow MLP made of two hidden layers with 128 channels. Following the default settings, X-DVGO is trained for 75K steps (25K per modality, taking about 15 minutes overall on a single 3090 RTX GPU), using a total variation regularizer [rudin1994total] in addition to the rendering loss. Since we observed sub-optimal results when jointly optimizing camera poses with X-DVGO, leading to scarce alignment, we bootstrap it with camera poses learned after a few steps of X-NeRF training. However, training this latter for $<$10 minutes yields stable poses, ready for training X-DVGO.

We start by evaluating the quality of novel views rendered by X-NeRF and X-DVGO for any modality. Specifically, given a single modality – MS, for instance – we render images from the viewpoints of that camera alone – e.g., the MS camera – and measure the quality over such modality – e.g., MS predictions by the MLP. To this aim, we report the PSNR and SSIM metrics [Mildenhall2020ECCV] (the higher, the better), leaving out LPIPS – since meaningful for RGB images only.

Bimodal Cross-Spectral Radiance Field. As a first experiment, we train X-NeRF to deal with images belonging to two modalities, RGB and MS, and compare it with a vanilla NeRF trained on single modalities alone. In this case, a single relative pose between RGB and MS camera is learned during optimization. Tab. 1 collects the outcome of this evaluation. Each row corresponds to a specific model (NeRF or X-NeRF), the modalities used for training, optional use of NXDC space and the testing modality – which also bounds rendering resolution.

Considering RGB rendering, we can notice that the vanilla NeRF trained on RGB images alone achieves, overall, the best performance. This is not surprising, since the additional MS images processed by X-NeRF are at much lower resolution (about 100$\times$ smaller) and, of course, of different modality. However, the drop is moderate thanks to the MS bands partially overlapping the RGB ones. Moreover, we can appreciate how the drop is smaller when the NXDC space is used, showing how our proposal favors learning a joint representation of the two modalities by the MLP. By looking at MS rendered images, we can appreciate how X-NeRF outperforms vanilla NeRF with NXDC, rendering higher-quality images. The higher-resolution of RGB images and the partial overlap with MS ones allows for such improvement, while X-NeRF using NDC cannot exploit such advantage. Further experiments concerning NDC and XNDC are reported as supplementary material.

To summarize, X-NeRF achieve comparable performance with respect to NeRF when rendering RGB images – thanks to the NXDC space – while it can effectively exploit the learned cross-spectral representation of the scene to improve the quality of rendered MS images.

Trimodal Cross-Spectral Radiance Field. We now add a further modality to X-NeRF, training it to render jointly RGB, MS and IR images. With this setup we will conduct all the following experiments. Tab. 2 collects the outcome of this experiment. We can notice how, on any modality, the vanilla NeRF trained on the single modality alone achieves the best results on average. Again, we feel this trend to be not surprising, given the variety of content framed by the three modalities and the very different resolutions of each. However, we can observe once again how NXDC allows for the smallest drops. By looking at the individual modalities, on average X-NeRF achieves equivalent performance with respect to vanilla NeRF on MS images, while dropping on RGB and IR rendered images.

Learned Poses Analysis. We now inquire about how relative poses between RGB-MS and RGB-IR cameras are optimized by X-NeRF during training. Fig. 3 shows how MS and IR cameras centers move during training, according to the relative pose learned with respect to the RGB camera (green) after a certain number of epochs, encoded in blue (MS) and red (IR) color intensities respectively. According to colors being normalized over different epoch ranges, specifically 50, 500 and 5000, we can notice how after roughly 250 epochs the relative poses get stable and very close to those obtained after an entire training cycle. We can notice how the camera visualized in Fig. 3 (right) are placed as in the real rig shown in Fig. 2.

Speeding-up Cross-Spectral Radiance Fields. We now evaluate the rendering performance of the X-NeRF accelerated variant, namely X-DVGO. To train X-DVGO on a single scene, we bootstrap camera poses by training for 250 epochs X-NeRF on the same scene – taking about 7.5 minutes. Then, we freeze relative poses and start training X-DVGO. Tab. 3 shows a comparison between X-NeRF and X-DVGO, trained on RGB, MS and IR modalities jointly and tested to render any of the three. In general, when rendering MS and IR images the two achieve very similar performance on average, with X-DVGO achieving slightly lower PSNR and higher SSIM scores, while when rendering RGB images X-DVGO outperforms X-NeRF on both metrics while training approximately 8$\times$ times faster (i.e., 7.5 plus 15 minutes versus 2.5 hours).

We now study the impact of the bootstrapped poses on X-DVGO performance, aimed at assessing the advantages it yields in terms of time required for training. In Tab. 4 we report rendering performance by X-DVGO when trained with poses being optimized for different amounts of epochs. We compare it to X-NeRF trained for an amount of time equal to the total time required by X-DGVO (i.e., bootstrapping plus actual 25K steps of training). We can notice how poses optimized for 50 epochs only already yields render quality not that far from those by X-NeRF trained for an entire cycle (5K epochs), with only 16.5 minutes of total training. With the very same time budget, X-NeRF achieves remarkably worse results. Some improvements are achieved by X-DVGO using poses optimized for 250 epochs, while elongating the poses initialization process for more epochs does not allow for further significant improvements. X-NeRF still results inferior in rendering quality when limiting the time budget up to one hour, confirming that X-DVGO achieves a better trade-off in terms of training time/rendering quality.

To conclude, we assess how effective X-NeRF and X-DVGO are at rendering images aligned across spectra, i.e. so as to create a virtual Cross-Spectral camera. This evaluation is carried out by rendering images according to each of the three cameras viewpoints, and thus at the three different resolutions they are characterized by, which are then cropped to match the area common to the three – i.e., the one observed by the camera with the narrowest FoV, the MS camera in our case. This results in evaluating on 1694$\times$3434 images when rendering from RGB camera viewpoints, 254$\times$510 from MS viewpoints and 181$\times$363 from the IR cameras. We compute pair-wise Mutual Information (MI, the higher the better) [MUTUAL_INFORMATION] across the three modality pairs, and then average the three scores we obtain.

Tab. 5 collects the outcome of this evaluation, involving X-NeRF – without and with NXDC – and X-DVGO. We can notice how the MI across modalities is very low when X-NeRF uses the classical NDC: indeed, the MLP learns to render the three different modalities by casting rays in very different regions of the 3D space, resulting in unaligned rendered images. On the contrary, NXDC allows for learning much better aligned representations, thus achieving much higher MI scores. We can observe this effect also qualitatively, by looking at images and depth maps rendered by X-NeRF. Fig. 4 reports two samples from the Dino and Penguin scenes of our collected dataset, followed by images rendered from MS viewpoint by X-NeRF, trained with NDC or NXDC convention. We can notice how the latter allows for rendering images that are aligned across spectra, and properly models the 3D space as we can notice by observing the rendered depth maps.

X-DVGO achieves results almost equivalent to X-NeRF, resulting in slightly lower MI scores. In Tab. 6, we show average MI scores at each resolution achieved when bootstrapping X-DVGO with poses initialized for different amounts of epochs, i.e. the same reported in Tab. 4. As we observed for rendering results, after 250 epochs the improvement almost saturates. By training X-NeRF with the same time budgets, alignment slightly reduces over time to favor the rendering quality over single modalities. The supplementary material provides more qualitative results.

Despite the high quality of both rendering and alignment yielded by X-NeRF, the task we are facing and the hypotheses under which we operate – partially known camera poses and very different sensors modalities, resolutions, focals and FoVs – are very challenging, thus some failure cases occur. Specifically, for some scenes X-NeRF gets stuck into local minima and cannot align the three modalities at their best. We show in the supplementary material some examples of this occurrences, with two modalities being properly aligned and the third one resulting slightly drifted.