Visualizing the Passage of Time with Video Temporal Pyramids

Timelapse has been in existence since the late 1800’s [wikipedia2022timelapse] and is a popular way to visualize the passage of time on a small scale (e.g., a pineapple rotting [temponaut2021vid]) or a large scale (e.g., Google Earth Timelapse [google2022timelapse]). Of particular interest, Martin-Brualla et al.[martinbrualla2015timelapse] drew from internet imagery to create years-long timelapse videos, but the noise from internet photos captured by different cameras and at different times requires heavy smoothing so that shorter-term changes are not visible and long-term changes can be hard to detect.

Several techniques have been proposed to smooth out the aliasing artifacts that result from timelapse sampling after the fact. With consumer video applications (e.g., hyperlapse and timelapse) in mind, Zhang et al.[zhang2017photometric] propose a method for smoothing transitions between frames in temporally subsampled videos. Whereas their method interpolates and smooths after subsampling, our method necessitates fewer modeling assumptions because we smooth discontinuities before temporal subsampling; starting with the densely sampled input also allows us to produce smooth visualizations of any timescale. Their method is also tested only on videos that span minutes or hours of time, and are subsampled to seconds or minutes, whereas ours is designed to work with years-long video streams. Finally, their method is also more computationally expensive, operating around 0.5 frames per second, whereas ours runs at 6 frames per second. While both methods can benefit from parallelization, this is nonetheless a significant difference when considering datasets such as ours that have on the order of 1 billion frames. Details on runtime and datasets can be found in the supplemental material.

In remote sensing, it is often desirable to visualize long-term changes related to a variety of phenomena caused by humans or natural events. The data is often of low temporal frequency, sometimes only before-and-after satellite pictures or landscape photographs taken far apart in time, such as the U.S. Geological Survey repeat photography project [usgs2016repeat]. Animation techniques have been proposed which can help create a smooth transition between images. Lobo et al.[lobo2019satellite] does this by simulating plausible intermediary frames, while Harrower [harrower2001animation] provides the user with control over the spatial and temporal resolution to allow for optimal visualization of a given phenomenon. While our work shares the same goals of visualizing changes happening over long timescales, we work with datasets with high temporal resolution and do not rely on interpolation to smooth transitions. That said, interpolation methods such as Lobo et al.[lobo2019satellite] could be complementary to ours as a possible way to fill in segments of missing data from our datasets.

Most existing techniques for video visualization are designed for videos no longer than a few hours, and their end goals often differ significantly from ours. In fact, over ten years ago, Borgo et al.[borgo2011vidviz] published a survey of different video visualization techniques; while our work is related to many of the techniques they describe, the authors clearly assumed the use of relatively short videos. This section highlights some of the most closely related work in this area.

Various works adapt the frame rate, or temporal sampling rate, of a video over time based on its content [zhou2014time, joshi2015hyperlapse]. The most closely related technique is Computational Timelapse [bennett2007timelapse], which uses temporal differences in video frames to dynamically speed up the frame rate when little is changing and slow it down when more changes are occurring. While this is effective as an automatic fast-forward tool, it requires a chosen output video length; furthermore, long-term changes appear choppy and broken up due to sudden changes in the frame rate. Several works have proposed various non-axis-aligned manipulations of space-time cubes such as videos; Rav-Acha et al.[rav-acha05evolving] explore this idea from in a graphics/vision context, while Bach et al.give a thorough visualization-oriented review of the possibilities in this space. These techniques are generally incompatible with cuboids with significantly longer time extent, such as months-long videos.

Video summarization techniques take another approach—rather than maintaining chronological and spatial continuity, these techniques attempt to find frames or clips that encapsulate the activity in a video. These techniques generally approach the problem by automatically detecting “noteworthy” frames or clips either by using unsupervised saliency-based approaches [pritch2008nonchronological] or by using example-based learning [zhang2016video, rochan2018video]. These techniques tend to focus on the real-time timescale and treat longer-term changes as noise; they also make automated decisions that may remove content of interest even at the real-time timescale.

Video fast-forward techniques are closely related to the ideas supporting timelapse videos and often use frame-skipping. The disadvantages of timelapse were explored by Hoferlin et al.[hoferlin2012ffvis]. Their evaluation of fast-forward techniques also included an interesting experiment with the use of temporal blending—similar in spirit to our temporal filtering method. However, their filtering approach does not generalize to more than one timescale.

A few prior works have considered multiple timescales of activity in videos. Motion Denoising [rubinstein2011motion] separates a video into short-term and long-term components. This technique produces excellent results, but handles only two timescales and is very computationally expensive, making it impractical for months-long videos, much less for years-long videos. Wehrwein et al.[wehrwein2021scene] propose a method to composite clips from different timescales into a single “scene summary,” showing, for example, people walking, clouds moving, and the sun crossing the sky all at once. This method begins with a Gaussian temporal pyramid much like the one constructed as a side-effect of our Laplacian pyramid construction, then composites salient clips from different pyramid layers together; though multiple timescales are visualized at once, the vast majority of the lower pyramid levels are discarded from the final output. In contrast, we construct visualizations that assists in exploration of the whole dataset without any assumptions about which timescales or scene elements are of interest to the viewer.

In contrast to the automated methods above, a separate category of prior work facilitates interactive browsing, exploration, and retrieval in video. Though this is more closely related to our task, most of these techniques are oriented towards retrieval of specific content rather than discovery, or towards improving the ability to scrub or seek in a shorter video.

The Video Browser Showdown [rossetto2020vbs] is a yearly contest of video browsing tools designed to locate either a specific event or clip in a video, or locate all instances of an event or action. Tools that are successful on this task (e.g., [kratochvil2020somhunter]) tend to leverage the fact that the sequence of interest is known a priori, making them less useful for discovering unknown anomalies; for similar reasons, these tools are also unlikely to generalize well for the purpose of identifying structure at longer timescales.

Several techniques have been proposed to show a visual overview of an entire video sequence, or improve scrubbing. Gutwin et al.[gutwin2019spreadloading] proposed a spread-loading scheme to load frames at varying intervals when loading a streaming video to improve the scrubbing experience. They showed that this improved users’ ability to seek to a particular point in the video quickly, but even with instantaneous availability of all frames, a months-long video would be tedious to explore by scrubbing. Barnes et al.[barnes2010tapestries] create a continuous visual overview of automatically selected keyframes to assist in scrubbing around in the video, while Jackson et al.[jackson2013panopticon] arrange short clips of a video in an animated grid so a user can shift their focus to any point in a video or watch a single thumbnail as it cycles through the entire video. These techniques work well for shorter clips, but their utility is limited by available screen size for longer-duration videos.

From a visualization perspective, Romero et al.[romero2008vizavis] also uses computer vision to analyze and visualize video volumes. In particular, their interface proposed a closely related heat map visualization called the Activity Table that is similar in spirit to our Video Spectrogram. The Activity Table displays aggregate motion computed using thresholded frame differences, closely related to the frame differences performed in the construction of our Laplacian Pyramids. Whereas they plot aggregate motion in a single (real-time) timescale across different spatial locations, we are interested in activity at multiple timescales and use the vertical axis of the heatmap to index temporal frequency instead of spatial location.

Although our work relates to the rich literature on time series visualization (e.g., [aigner2011timevis, ali2019timecluster]), video has unique properties that benefit from domain-specific techniques. One notable example from this literature is the work by Cakmak et al.[cakmak2021multiscale] which does visualize time-varying data at multiple timescales; their interface for traversing temporal scales and viewing summaries at different levels is loosely analogous to our Video Spectrogram, but is geared towards the very different domain of time-varying graph data.

The pyramid computation component of our approach is directly adapted from image pyramid techniques from the computer vision literature. Our adaptation will be described in detail below. In particular, we compute Gaussian [burt1981pyramids] and Laplacian [burt1983laplacian] pyramids along the time dimension, in contrast with their traditional application to the spatial dimensions of images. The idea of generalizing image pyramid techniques to videos is not new; a related generalization of the Gaussian pyramid to video was proposed by [finkelstein1996multiresolution] to send videos at variable resolutions in space and time over limited bandwidth network connections. Their approach resembles a Gaussian pyramid (in contrast to our Laplacian pyramid), operates across spatial and temporal dimensions, and is designed for efficient coding and variable-resolution transmission of videos under limited bandwidth. Our method aims to visualize longer-term changes with high fidelity. For the problem of human action recognition, numerous works propose variations of pyramids applied temporally for short (e.g., minutes-long) video clips [shao2014actionrecog, lan2014tsp, wang2017action, wang2017actionpooling, zheng2019action]. To our knowledge, however, our method is the first to extend the temporal Laplacian pyramid concept to extremely long-duration videos with the intent to visualize long timescales. Finally, our Video Spectrogram tool is directly inspired by the idea of time-frequency representations like the spectrogram, which are commonly used in audio visualization and processing [cohen1995time].

Image pyramids are a classical technique from the computer vision literature [burt1981pyramids, burt1983laplacian], widely used to apply image processing and computer vision algorithms at multiple scales or in a scale-invariant fashion. The most basic image pyramid is a Gaussian Pyramid [burt1981pyramids] (Figure 1 (a)), constructed by repeatedly blurring then subsampling an image. Each subsequent level of the pyramid represents what remains after a low-pass filter is applied to the prior level. Each level of a Laplacian pyramid [burt1983laplacian] (Figure 1 (b)) represents the result of a high-pass filter applied to the prior level of the Gaussian pyramid, or equivalently a band-pass filter applied to the original image. The resulting Laplacian pyramid levels contain narrow slices of spatial frequency content of the image, thereby resembling the output of a bank of band-pass filters.

Where Laplacian pyramids are traditionally used to isolate and manipulate spatial frequency content of images, we instead apply the same procedure to the temporal dimension of a video, leaving the spatial dimensions alone. A temporal analog to the Gaussian pyramid consists of videos that have been filtered and subsampled along the time dimension only. The Laplacian pyramid is also constructed analogously, by subtracting the temporally blurred video from the original. In principle, frames of the Laplacian temporal pyramid can be computed by subtracting the computed blur frames from the current level of the Gaussian pyramid before subsampling. In practice, we use the standard pyramid construction approach given by [burt1983laplacian] to avoid quantization errors and ensure that the pyramid levels can accurately reconstruct the input signal. We first filter and downsample the input, then upsample it again to match the prior level’s temporal sampling rate; this downsampled-then-upsampled signal is finally subtracted from the input video to calculate the Laplacian pyramid level.

The pyramid levels are constructed recursively as shown in Figure 2 and Algorithm 1. We begin with a long input video, which serves as the first level of the Gaussian temporal pyramid. Each subsequent level is computed in one pass through the prior level’s video, calculating blurred frames from a sliding window of prior level frames. The resulting pyramid levels are themselves videos of the same spatial resolution and covering the same real-world duration in time, but with a reduced frame count. For this reason, each level of a Gaussian temporal pyramid is similar to a timelapse video with a particular sampling rate. One key difference is that blurring across time before subsampling causes short-term motions to blur out in higher levels of the pyramid, thus eliminating aliased content that would appear in a true timelapse video. The levels of the Laplacian temporal pyramid are less intuitive to watch, as they contain only specific bands of temporal frequency content. Sample Laplacian pyramid frames are shown in Figure 3.

In a Laplacian temporal pyramid the original source video has been decomposed into multiple component videos. These components can be reassembled to create an exact copy of the source by performing the pyramid-creation steps in reverse order (see Algorithm 2). It is also possible to proceed with this reconstruction while leaving out certain Laplacian pyramid levels (specified in Algorithm 2 using the $W$ vector). Reconstructing without the last few Laplacian layers yields a smooth but slower-moving version a Gaussian blur level. We found this smooth temporal upsampling option to be quite useful for some of our videos, as it provides a way to slow down the action so more information can be absorbed by the viewer.

Each level of the pyramid represents a specific timescale. For instance, the base frame rate of most webcam video is 30 frames per second (fps), or 1/30th of a second per frame. Motions easily visible at this frame rate can be thought of as belonging in the “1/30th of a second” timescale. Higher levels of the pyramid represent changes happening at a frequency of “once per second” or “once per day” or even “once per month.”

The prior section described a straightforward generalization of the Gaussian and Laplacian image pyramids to construction of temporal pyramids. We now describe a few simple adjustments we made to adapt these pyramids for the use case of visualizing and exploring long, fixed-camera video streams.

Events that happen repeatedly stand out with clarity in our pyramid videos. The day/night cycling is the obvious example, and this is noticeable in all of our datasets. However, we found many other examples of cyclic activity that appeared in the videos as distinctive repetitive patterns. Tidal patterns were apparent in the Buxton oceanside dataset as well as the Geiranger ferry dock dataset. Geiranger shows boats rising and falling next to a dock. In the timescales below 1-day, the boats move up and down as the video progresses, and in the timescales longer than 1-day the movement becomes averaged and the video shows a ghostly blur encompassing all of the vertical positions of the boat over time. The cyclic nature of seasons becomes obvious in the datasets with multiple years, such as Hiuchi, Kutcharoko, and Smoky, where we can view the seasons changing fast enough that we understand the similarities of the cycle from year to year. The switching between white snowy winter and green leafy summer becomes a visual rhythmic pulse at the higher pyramid levels, just like the day/night changes at the lower pyramid levels.

On a smaller scale, the natural cycles of plant growth are nicely visualized in the Mad River dataset. This scene includes a deciduous tree in the foreground, as well as bushes and other trees on the edge of a river. The tree in the foreground loses its leaves and grows them again, and the bushes and other plants can be viewed getting larger in the summer and smaller in the winter. Another very interesting discovery is how the branches of the foreground tree droop at night and perk up during the day, which becomes more noticeable because it happens repetitively. This dataset seems to have its camera recording with infrared at night, which fortunately makes the tree always visible.

In addition to the pyramid videos themselves, the Video Spectrogram also seems to be especially good at visualizing cyclic events. The day/night cycle is very obvious at the 12-hour timescale, with (usually) more activity and a lighter color on the spectrogram for the half of the day which is mostly daylight, and (usually) a darker color for the less-active night. The dark and light colors on the spectrogram switch back and forth creating a distinctive pattern at that level of the pyramid for most of our datasets. In the higher levels of the multiple-year datasets, the seasonal pulsing is also clearly visualized with the spectrogram colors.

We also found other examples where the periodicity of human activity showed up clearly in the spectrogram (Figure 6). In the Bryant Park ice rink dataset, the spectrogram had lighter colors during the times the rink was busy with skaters and darker colors for the times when the ice was cleared. This seemed to happen on a cycle of about an hour and a half, presumably a planned timing. Another example is from the Bridge dataset, where the spectrogram shows a light bar every time the drawbridge goes up which happened frequently on Memorial Day in 2021. Since this was a repeating event over the course of that day, its pattern on the spectrogram was more noticeable than it was on the days where the drawbridge only went up once or twice.

The video spectrogram tool connects events at different timescales by virtue of using a common timeline. If the viewer sees a short blip of some interesting or anomalous event at a higher timescale, the user allows for pinpointing the general date/time of the anomaly and drilling down to lower levels at that same time in order to see more detail. For example, one might see a truck appear ‘out of nowhere’ in the 6-day timescale, then quickly drill down to the day it appeared and view the 5-second timescale on that day in order to see which direction the truck drove in from before it parked (Figure 7, top).

Most real-world events do not fall neatly into one discrete timescale, and this often means that an interesting event can first be discovered from the bird’s-eye perspective of a higher pyramid level and then the full extent of the occurrence can be discovered and viewed by drilling down to lower levels. We found an example of this in the Mad River dataset (Figure 7, bottom), when a bright pink object catches the eye briefly and then disappears during the 5-minute timescale on October 6, 2020, in a corner of the scene. Drilling down to the 5-minute level, the pink object appears to be a pink jacket left on a rock but it still disappears quickly. Drilling down to the 1-minute level we can see fast-moving people and we also see that the pink jacket moves from one rock to another. However, it is not until we drill down to the 5-second timescale that we can make out the group of people and their general movements. The fact that they left a bright pink jacket in one place for about 15 minutes, while they themselves moved faster, left a clue to their presence in that longer 15-minute timescale video.

The multiple timescale visualization also provides a useful and possibly educational demonstration of the role and scope of human activity in a particular scene. In the Bryant Park dataset, the lower level pyramid videos show crowds of people skating. However, the higher level pyramid videos show an eerily empty skating rink, with no humans in sight. At the 4-hour timescale and above, it is mainly the lighting changes, and certain infrastructure changes that stand out (such as a rink-side tent being erected for a while). In the Mid Mountain ski slope dataset, tire tracks and ski tracks in the snow provide a clue to human activity at the lower levels of the pyramid, but we don’t see the humans making those tracks unless we drill down. The same thing can be seen with tire tracks and footprints in the sandy beach of the Buxton dataset. With both snow and sand, the evidence of human activity is melted or eroded away fairly quickly. In contrast, the Rane construction dataset shows a scene where the long term effect of human activity is exactly the point, and in that case it is definitely instructive to drill down and see exactly which lower-timescale activities were responsible for the higher-timescale view of a building being constructed.

The pyramid videos by themselves, as well as the spectrogram tool, can be useful for surfacing anomalous events. Missing data is the most obvious anomaly to find, and sections of missing data show up as all-black frames in the videos and as solid dark colored areas of the spectrogram, as can be seen in Figure 4. Corrupted data is another kind of anomaly that shows up in the videos and the spectrogram. For instance, in the Buxton coastline dataset the high-level videos have a section that shows a static night scene which is clearly out of place since it lasts longer than a single night both in the pyramid video and on the timeline. We traced the problem back to the original footage, where a static image was looped for a while.

Many anomalies are related to the camera itself, the most common of which is a sudden change in camera angle or placement. The Mid Mountain dataset includes a few months during ski season where the camera zooms or moves closer to the ski slope. The Buxton beach video includes a section where the camera faces out to the ocean instead of along the coastline. Sometimes camera errors can be seen, such as a day in the Hiuchi dataset which included camera footage of an office ceiling when normally the scene is outdoors in the mountains. That event occurred directly after a long period of missing data, so we suspect the camera was being repaired before being reinstalled outdoors. When there is a rainstorm or snowstorm, the video will often show raindrops on the camera lens for a short while at the lower timescales. The most entertaining camera-related anomalies we found occured when birds perch in front of the camera (Buxton) or a spider builds a web on it (Mad River).

In contrast with standard timelapse, the pyramid videos are visually less cluttered and they can be upsampled to adjust the rate of change to be easier to absorb. Anomalous events are more distinctive against this backdrop and are thus fairly easy to spot. For instance, when watching the Buxton dataset video for the 4-hour timescale, we noticed that a railing suddenly appears directly underneath the camera. At that level we could localize it to May 14, 2020, using the Video Spectrogram. We went directly to the 5-minute timescale for May 14, 2020, but the railing was there at the beginning of the day, so we switched to May 13, 2020, and drilled down further. We could see the railing put into place in the real-time original video for May 13, 2020. Even then it was very fast and rather anti-climactic since there was no visual of the person putting it in place (see Figure 8).

Our pyramid videos provide a window into the reality of long-term dynamics without sacrificing much verisimilitude. There is some level of blurring that occurs at the upper levels of the pyramid, since it has averaged a lot of small changes over time. We also see the blending of day and night scenes. These factors mean we sacrifice some precision and spatial resolution at the longer timescales; however, we found we were still able to discern larger patterns. For instance, in our Mid Mountain dataset, the gradual pattern of snow melt on a ski slope over the course of days or weeks stands out with clarity at the 6-day timescale. In our Buxton oceanside dataset, at the 1-month timescale, the water’s edge can be seen slowly changing its position relative to the beach, slowly rising and retreating much slower than the tides.

Another way these pyramid videos contribute to understanding of long-term dynamics is through the knowledge that anything showing up in a particular timescale must have generally stayed in the same place for a long enough time, related to the timescale. In the 1-hour timescale, a car driving by would not show up but a car parked in one spot for at least an hour would show up, and the duration of its appearance in the video would correspond to how long it stayed parked.

For a few datasets, we constructed standard timelapse videos by subsampling at different rates and compiling the resulting frames together into videos for each timescale. At the top levels of the pyramid this resulted in extremely short timelapse videos (less than one second), which were thus not very informative or interesting. However, going down the pyramid levels, once the videos were at least a few seconds long they did provide a good baseline for comparison with our pyramid videos. We found consistent results among all of the datasets which are summarized below.

Our method has some important limitations. One is the requirement that the camera viewpoint be fixed. This ensures that changes in the video are due to the scene, rather than camera motion; however, if camera angle changes are infrequent then the spectrogram is minimally affected, and in fact our method is useful for discovering unusual camera events such as a change in viewpoint as discussed in subsection 4.3.

Another limitation is that the top two or three levels of the pyramid are usually not very informative because there are not enough frames available for any changes to register when stitching the frames together. In a pyramid built from one year’s worth of data, the 90-day timescale will only have 4 representative frames. However, in a pyramid built from 8 year’s worth of data, the 90-day timescale will include 32 representative frames. We can discern changes over 32 frames much more easily than over 4 frames.

We also noticed that as the pyramid levels progress higher, the edges of all scene elements tend to become slightly less sharp with each new level. This is probably caused by very small camera movements that register as brief ‘whole scene changes’ with an effect in the pyramid that compounds as levels are built recursively. This effect could possibly be mitigated by the addition of a video stabilization preprocessing step. Simple feature matching-based image alignment techniques [szeliski2007image] could be used to align the frames to minimize movement due to camera shakiness. A similar feature matching technique could be used to manage camera viewpoint changes as well. We found that the noise present in our datasets was small enough that these techniques were not necessary, but they could be used to boost visual quality if desired. They could also help our method generalize to more datasets, such as those with automated and periodic changes in viewpoint, or fixed-viewpoint cameras that exhibit noticeable motion due to wind.

Our temporal pyramid computes a very simple, low-level measure of intensity change from one frame to the next. In the spirit of Viz-a-Vis [romero2008vizavis], we intend to explore more sophisticated types of analyses that can be aggregated into heatmaps to show more high-level and/or task-specific measures of activity. For example, optical flow could be used to measure motion rather than per-frame intensity change at multiple timescales (similar to [wehrwein2021scene]). In scenes with specific object categories of interest (e.g., people, cars etc.), object detection or crowd counting techniques could extract more meaningful trends which could then be visualized in a similar time-frequency spectrogram.

Although our method was not designed with the intent of video anomaly detection, it could provide the basis for some new techniques in that area. One of the many existing video anomaly detection methods [nayak2021anomaly] could possibly be applied to upper level pyramid videos in order to quickly and automatically surface unusual events at those timescales, which might yield insight upon drill-down to lower levels. For instance, detecting the origin of an unattended bag, after the fact, would be likely made easier with the aid of a temporal pyramid.

Another direction for future work is to explore different types of visualizations for our spectrogram, other than a heatmap. For example, a circular or radial representation might be useful for visualizing periodic events. We would also like to find ways to more easily compare different days with each other (or different years, months, etc.), even if the days chosen for comparison are far removed from each other in time.