Averaging Time-Lapse Imagery

Lately I’ve been creating some long-term time-lapse stills and animations out of photos I’ve taken with an ordinary handheld camera.  In this post, I’ll explain what I’ve been doing to try to get decent-looking results.  I’ll also share a Photoshop script I cobbled together to automate much of the process, which would otherwise be prohibitively time-consuming.  Be forewarned that there are some large animated GIFs ahead—depending on time of day, you may want to go grab a beer or a cup of coffee while they load.


Introduction

Time-lapse photography is pretty cool, and time-lapse photography covering long spans of time is especially cool—after all, who wouldn’t appreciate watching a familiar scene cycle rapidly through the seasons of the year with (say) the leaves turning color, falling, and growing back in a matter of seconds?  But this sort of thing is generally assumed to require a certain minimum of gear: a tripod for stability, an intervalometer to trigger shots at programmed intervals, a neutral density filter, a powerful battery.  On those grounds, it may not seem like something the average person could undertake casually.  Moreover, all that gear traditionally gets tied up for however long it takes to capture each individual time-lapse sequence.  That’s a disincentive for tackling unusually long durations, such as a whole year, and a real impediment in the way of capturing multiple long-term scenes simultaneously.

But for the past fifteen months, I’ve been taking time-lapse photo sequences using nothing but a handheld Panasonic DMC-ZS8.  And I haven’t needed to leave even that in place anywhere for long periods of time.  Instead, I’ve been identifying spots where I can put my camera back in (almost) exactly the same position time and time again, day after day, week after week, month after month.  The first such spot I hit upon was the window looking out of my office at home: I place my camera on the top rail of the lower sash, slide it along with the lens square against the pane until it runs up against the jamb, and then take a picture of the scenery outside.  I’ve also been using the corners of railings and ledges and steps and benches, and the tops of square posts, and other analogous physical reference points as makeshift pseudo-tripods.  This approach has something in common with a “time study,” where someone sets up a fixed reference point such as a tripod out in a field, revisits it to capture images at longer intervals, and then fades or cuts between the results to simulate gradual change.  But I’ve been revisiting my photo spots frequently enough to yield “true time lapse.”  This approach takes some discipline and perseverance, but it doesn’t take any special purchases or planning.  If you wanted to do something similar, you could start right now, as long as you have a camera of any kind—even an iPhone.

Below is a screenshot of some thumbnails from one of my image sequences: a view of my yard at home from the right forward-facing corner of the back deck, with the camera body lined up to the right angle of the wooden railing.  I’ve been trying to take a photo from this same spot twice a day every day, once in the morning (around 7 AM) and once in the early evening (around 5:30 PM).  I don’t always succeed, since sometimes I’m out past dark or away from town altogether, and every now and then I take more than two pictures per day.  But there were 632 total images in the sequence, covering a one-year period running from late July 2015 through late July 2016, at the time I tapped it to prepare some illustrative examples for this blog post.

back-porch-forward2-source-image-folderIt’s easy in this way to collect lots of different image sequences covering impressive spans of time.  But is this worth doing?  You might think the results of such an informal, technically primitive approach to time-lapse photo capture would be crude and messy.  And taken strictly in their raw form, I guess they are.  With digital processing, however, I’ve found I can produce smooth video animations and decent still images from photos taken under such conditions.  Below, I’ll describe how I’ve been accomplishing this through a three-step process consisting of sizing, alignment, and averaging.

Much of this process could also be used to advantage on long-term image sequences captured in more conventional ways.  But what excites me about it is its potential for democratizing long-term time-lapse photography, turning this into something just about anyone can do to good effect.  If you have a camera and a fixed vantage point where you can reliably position it again and again, you can capture the source images.  And if you have a computer with Photoshop, you can process them exactly as I’ll be describing and illustrating.  A year from now, you too could have a presentable time lapse of your back yard!


Sizing

I’ve always intended to take my sequence pictures at the Panasonic DMC-ZS8’s highest resolution setting, which is 4320×3240 pixels.  After all, I’d like to be able to zoom in on details—say, by cropping to a particular plant growing over time.  However, 4320×3240-pixel images are too large for me to process comfortably in quantity: the memory-intensive alignment step would take forever at that scale.  Moreover, there were a couple times over the past year when I accidentally reset the image size on my camera and didn’t notice for a while that I’d done it.  From August 29th through October 8th, all my images ended up captured at half the usual dimensions, or 2560×1920.  And it was even worse for the period from December 29th through January 4th, when my camera was set to capture at a measly 640×480.  So the first thing I found I needed to do was to resize all my source images to a consistent and manageable scale.  For the backyard-scene sequence I’m going to use to illustrate the process, I resized the width of all my source images to 850 pixels before processing, a number I chose in hopes of leaving sufficient headroom for a target video frame size of 720×480 after alignment.  Here’s what the images looked like at this initial stage:

https://archive.org/download/animated-gifs-of-time-lapse-sequence/back-porch-forward2.gif

I dumped the contents of my folder for this image sequence into the resizing process without first vetting it on an image-by-image basis, and there are definitely some anomalies in it.  For instance, frames 302-304 were all taken before sunrise on the morning of the first snowfall of the year (January 10th), which I wanted to be sure to document; in experimenting sleepily with shutter speed, I ended up with one pitch-black frame, which I accidentally left in the folder, so that it ended up in the project along with both of the more successful images.  I also took multiple photos later that day, after the sun came up—hey, I was excited finally to have some snow in the scene!—so January 10th is definitely over-represented as a whole.  I’m currently working on a method of compensating for time-base inconsistencies such as these, which I may cover in a future post.  For now, I’ll just observe that the anomalies skip past the eye so quickly (at 0.03 second per frame here, or 33 1/3 frames per second) that they probably don’t have much impact on the overall effect.

And with that in mind, I’d say our most conspicuous problem is the distracting frame-to-frame jitter, which comes from slight differences in the position of my handheld camera from shot to shot.


Alignment

The next thing we want to do, then, is align the images with each other.  I suppose we could try to accomplish this with video stabilization tools, but I’ve opted to do all my processing on still images in Photoshop CS5.  The alignment step doesn’t take much effort to set in motion, but it does take a lot of computer memory and processing time.

  1. Go to Scripts > Load Files into Stack; then browse to and select all the source images in the series, which will be imported as layers.
  2. Select all the layers.
  3. Go to Edit > Auto-Align Layers, with Projection set to “Auto” (the default).  Photoshop will now attempt to align the images with each other.  Depending on the quantity and size of images, this step can take a good many hours.  I sometimes leave the process running overnight.

Each image ends up individually skewed, which increases its canvas size both horizontally and vertically and introduces transparent areas around the edges.  Overlaying all the images into a single document then increases the size of our canvas even more.  In the case of the backyard-scene sequence, I started with my source images resized to 850×638 and ended up after auto-alignment with a canvas size of 1016×767, about a 20% increase.  Below is an uncropped animated GIF of all the photos after they’ve been auto-aligned with each other, except for one single frame that I realigned manually (a night view that confused the auto-align algorithm).

https://archive.org/download/animated-gifs-of-time-lapse-sequence/back-porch-forward2-aligned.gif

This step already delivers a big improvement on what we had before, compensating nicely for the use of a migratory handheld camera rather than one mounted continuously in place.  Indeed, it might already give us the exact effect we want if we’re looking to foreground frenetically sped-up motion.  But now we run into another dilemma.  For outdoors scenes, we’ll tend at this point to see some distracting sunlight flicker and branch jostle, if I can coin a couple terms whose meanings should be self-evident.  And my goal in this example was to focus not on the aspects of the backyard scene that vary from day to day, such as fluctuations in sunlight, but on more gradual changes linked to the grand cycle of the seasons.  There’s a philosophical question worth posing here.  Is the point of a time-lapse video just to make things look as though they’re happening much faster than they actually did, or is it to expose long-term processes of change that are so slow we can’t ordinarily perceive them unfolding?  I’d think it will often be the latter.  And if that’s so, then flicker and jostle could arguably be not signal, but noise—unless, of course, it’s the flicker and jostle themselves that we’re aiming to expose.

Flicker and jostle turn up routinely in the most highly-regarded of time-lapse videos.  Take One Year in 2 Minutes by Eirik Solheim, who arranged a computer-contolled camera to take a picture of a scene once every thirty minutes for a whole year, selected all the images captured between 9 AM and 2 PM, and placed them into a timeline.  It’s a fine video, and well worth your while to watch, but the sunlight flicker and branch jostle are at least as conspicuous as the seasonal transitions.  We find something comparable in the four-month sequences of Forrest Pound’s Watershed: long-term change comes accompanied by flickering light and other comparatively rapid phenomena, such as cows popping in and out of view at the blink of an eye.  Meanwhile, Samuel Orr’s A Forest Year incorporates time-lapse segments with a lot of rapid flicker and jostle alongside segments with other kinds of pacing, to good artistic effect.

My point is that, taken in their raw form, long-term outdoor time-lapse sequences will inevitably flicker and jostleIt’s just in their nature.  These effects could be valued positively as hallmarks of authenticity, or maybe as drawing attention to the time-lapse process itself.  Leaving them in would be a legitimate aesthetic choice, and I don’t mean to knock any of the examples I’ve cited above for doing so.  But personally I’d like to have the option of reducing or removing flicker and jostle in order to bring out the longer-term processes they mask.  The strategy I’ve been trying out for this purpose entails averaging successive and overlapping groups of images together—not just two (as with simple dissolves) or three (as with overlapping dissolves), but groups of five, or a dozen, or fifty.


Averaging

The best way to average a group of layers in Photoshop CS5 is to ensure they’re all visible, select them, convert them into a Smart Object, and then go to Layers > Smart Objects > Stack Mode and select either “Mean” or “Median.”  Here’s the median of all 632 layers of my sample image sequence:

back-porch-forward2-all-632-layers-medianAnd here, for comparison, is the mean of the same 632 layers.

back-porch-forward2-all-632-layers-meanI tend to prefer the median, which comes out a bit sharper than the mean, maybe because it’s less responsive to outlier pixel values.  On the other hand, the mean does have a mistier, more ethereal look that might sometimes be desirable.  The stark appearance of the edges comes from averaging auto-aligned images with transparent areas left around them.  If we were to fill the transparent areas with white before averaging, we’d instead get a mildly cropped vignette with the edges soft and rounded.  Here’s the median processed that way:

back-porch-forward2-all-632-layers-median-vignetteIt’s easy to go through the steps of averaging a group of images in Photoshop by hand once, or twice, or a few times.  And you don’t need to do anything very complicated to get stills like the averages I’ve shown above, as long as you’ve captured a lot of pictures from almost the exact same spot and have the computing power and/or patience to auto-align them.

For a serious time-lapse animation project based on averaging, however, we need to go through these same steps hundreds upon hundreds of times in a row based on successive but overlapping groups of layers (such as 1+2+3, 2+3+4, 3+4+5, 4+5+6, and so on), and to save or export the results each time in an orderly form that allows us to regather them conveniently into an animated timeline afterwards.  With that in mind, I decided to try and design an automated action to do the lion’s share of the work for me.  (In Photoshop, an action is a set of “recorded” steps that can be “played back” at will.)

Here’s what I came up with.

First, to prepare the results of the previous step—alignment—for automated incremental averaging, I did a couple things that might not seem to make much sense at first, but bear with me:

  1. Create an empty layer at the very bottom of the stack.
  2. Make all the layers visible by clicking on the column of “eyes” and selecting Show/Hide all other layers.
  3. Click on the “eye” next to the empty layer at the bottom of the stack to make it invisible.

Then I recorded the following steps as an action:

  1. Alt+. — selects the top layer
  2. Shift+Alt+[ — adds the second layer from the top to the selection; then repeat this step however many times is needed to select the number of layers we want to average into each output frame.  To generate output frames that each average fourteen successive source images, for example, we’d repeat Shift+Alt+[ thirteen times.
  3. Layer > Smart Objects > Convert to Smart Object
  4. Layer > Smart Objects > Stack Mode > Median — averages all the selected images.
  5. Control+click “new layer” icon — creates a new layer immediately below the averaged Smart Object.
  6. Fill the new layer with paintbucket tool.  This masks the remaining layers that haven’t yet been combined into averages; otherwise we’d see them peeking distractingly around the edges.
  7. Alt+. — re-selects the top layer
  8. Control+Shift+E — merges all visible layers.  This leaves us with just two layers, instead of hundreds of them: the one containing the average, and the one invisible empty layer at the bottom of the stack.
  9. Scripts > Export Layers to Files, with “Visible Layers Only” checked — this exports only the single merged visible layer containing the average with a filename based on the name of the layer, which is conveniently inherited from the name of the top layer in the selection we turned into a Smart Object.  Note that this command isn’t available if only one layer is present, which is why we needed that mysterious empty layer at the bottom.
  10. Step backward — undo merge visible
  11. Step backward — undo paintbucket fill
  12. Step backward — undo create layer
  13. Step backward — undo median stack mode
  14. Step backward — undo convert to Smart Object; and we’re right back where we started.
  15. Alt+. — selects the top layer.  (That’s enough if we want to move through the source images in one-image increments, but if we instead want to skip ahead in larger increments—by twos or threes or whatever—we need to add one Shift+Alt+[ per skipped image.)
  16. Layer > Delete > Layer — deletes the selected top layer (or alternatively the top two layers, or three, etc.).  The idea is to leave things so that when the action repeats, it will commence one layer further down the original stack than before (or two layers further down, or three, etc.).

I called this recorded action Averager.

Now I needed to repeat it hundreds upon hundreds of times.  Unfortunately, Photoshop actions don’t offer any means of doing that automatically.  I suppose I could have sat there clicking “play” again and again, but that would have gotten pretty tedious pretty quickly.  So I was happy to discover the “Iterate Actions” plug-in by Dr. Woohoo! (as with Yahoo!, the exclamation point is part of the name).  This handy plug-in lets you automatically repeat a given Photoshop action a designated number of times.  There’s a video tutorial about it here, but to cut to the chase, the plug-in can be downloaded here in a zip file, together with installation instructions.  With this plug-in installed, we open the Photoshop file containing all the aligned images—prepared as described above—and go to Window > Extensions > Iterate Actions, which will bring up a window for the plug-in.  Clicking the icon next to “Loop Action” now sets things in motion.  A popup asks what action we want to play; we enter Averager.  Then a second popup asks: “How many times would you like to play this action?”  To play it the correct number of times, we enter 1+((f-a)/i) rounded down to the next integer, where f = total quantity of source images, a = quantity of source images averaged per output frame, and i = increment by which we advance past source images between output.  Then let ‘er rip.

We end up with a folder full of files with the same filenames as the original source images.  Each one is an average mixing the image that originally had that name with a given number of images immediately following it.

Let’s take a look at the results of averaging some different quantities of source images, with the output frames organized into animated GIFs as before.

In my first example, each frame is an average of five source images (corresponding to roughly two and a half days), advancing in one-image increments (corresponding to roughly half a day each).  The animation is still distractingly flickery, but less so than before.  Here and further below, I’ve left the transparency around the skewed source images during processing rather than filling them in white, since we end up seeing slightly more of the scene that way.
https://archive.org/download/animated-gifs-of-time-lapse-sequence/back-porch-forward2-averaged-by-5s.gif

In the next example, each frame is an average of fourteen source images (corresponding to roughly one week), advancing in one-image increments (corresponding to roughly half a day each).  (Download at full 1016×767 scale here [187 MB].)
https://archive.org/download/animated-gifs-of-time-lapse-sequence/back-porch-forward2-averaged-by-14s.gif

In example number three, each frame is an average of twenty-eight source images (corresponding to roughly two weeks), advancing in one-image increments (corresponding to roughly half a day each).  By now the animation is pretty darn smooth.  (Download at full 1016×767 scale here [197 MB].)
https://archive.org/download/animated-gifs-of-time-lapse-sequence/back-porch-forward2-averaged-by-28s.gif

In our fourth example, each frame is an average of sixty source images (corresponding to roughly one month), advancing in three-source-image increments (corresponding to roughly a day and a half each).  I’ve kept the same frame rate as before, so the action unfolds a lot more rapidly: you can see the branches spring up when they’re unburdened by the weight of their leaves in the fall, and then sag under the strain of fresh greenery and pears in the spring.  (Download at full 1016×767 scale here [55.2 MB].)
https://archive.org/download/animated-gifs-of-time-lapse-sequence/back-porch-forward2-averaged-by-60s-incr3.gif

In place of jitter and flicker, the averaged versions of the sequence show some blurring, which I consider a savvy trade.  We’re making each frame a more representative average of a segment of timeline at the expense of time resolution, a principle I’ve already written about here.  If the leaves look blurry, well, at this accelerated speed I think they would legitimately have been “vibrating” enough to blur.

I’ve left my animated GIFs uncropped, but remember that I originally resized the width of my source images to 850 pixels with a target video frame size of 720×480 in mind.  Things ultimately worked out pretty well on that front; witness the “real” 720×480 video below.  If you can’t stomach the pixelation in the embedded version, you can download a higher-quality AVI file here (843 MB).

I came up independently with this idea of averaging overlapping sets of frames in long-term time lapse animations, but after some key-phrase searching I’ve found a few other examples of this strategy in use, and of course there could always be more (if you know of any, please drop me a line).  One of them is a YouTube video called “One year timelapse” by Radek Grec.  “I’ve been taking picture of this green in my neighbourhood every single day for one year,” he writes in the opening of the video.  “And here’s the result….”  He doesn’t explain just what he did to process his pictures, but it looks to me as though he might have averaged his daily photos in groups of a week or so and then created overlapping dissolves between the group averages—not quite what I’ve been doing, but probably easier to implement.  In any case, I see telltale signs of multiple-image averaging, such as a blurred triple view of a tree trunk (during March):

radek-grec-timelapse-detailAnother YouTube example, called “Our backyard one year timelapse,” comes with the explanation: “There’s a still image taken every day at noon. Every image is faded in several frames to the next image to make the time-lapse smooth.”  It’s not actually all that smooth, but this time the idea is explicitly stated.  Jorge Luis Reyes Ortiz discusses some more sophisticated methods of compensating for sunlight flicker in a long-term time lapse of a construction site here, but I’ve only found his “before” version to watch online.

The technique of time-lapse mining from Internet photos also couples an averaging strategy with a long time scale, although it’s so much more complex and ambitious than what I’m trying to do that I’d consider it a different animal altogether.  Ricardo Martin-Brualla, David Gallup, and Steve M. Seitz have been harvesting vast quantities of online photos of views of popular landmarks, warping these to a consistent perspective, and then creating time-lapse animations from them spanning multiple years.  They write that they tried straightforward median averaging but found it didn’t suit their data set: “We found that large temporal windows are needed to reduce flicker but also result in oversmoothed transitions.”   They resorted to inpainting occluded pixels (which doesn’t seem relevant to the kind of project I’m describing) and regularizing pixel values (which does seem relevant).

 


Creating a Script

It’s clearly possible to create smooth long-term time-lapse videos from image sequences captured with a run-of-the-mill handheld camera through post-production alignment and averaging.  But I wanted to make it convenient as well, and that took some more doing.

At first, I had to go in and record or delete steps in my Averager action whenever I wanted to change its settings: the quantity of images being averaged into each output frame, the number of source images advanced through per output frame, and so on.  But that got annoying to have to do.  So I tried setting up a bunch of steps representing multiple options with nested actions (e.g., “select top 5 layers,” “select top 10 layers,” “select top 20 layers”) and using the checkboxes next to these steps to enable and disable them.  That was an improvement, but still rather clunky and inflexible.  It soon became clear that if I wanted a conveniently dynamic interactive tool for running the sequence of steps I’d come up with, I’d need to create a script for it, and that of the scripting options available in Photoshop, JavaScript was preferable because it could work on multiple platforms.

The last time I’d done anything along these lines was as a kid back in the 1980s programming games on my TRS-80 in BASIC, so I found the prospect of creating a script in JavaScript a little daunting.  Fortunately, there’s a handy tool called Script Listener that will record code for whatever you’re doing in Photoshop, much like recording an action.  Using that, I was able to obtain a JavaScript equivalent of my Averager action.  And I developed things further from there, first by adding in the iterations, and then by adding additional optional steps to the beginning and end of the action (such as a resizing step, an auto-alignment step, a centering step to accommodate sources that won’t auto-align, and so on), often borrowing and modifying scraps of code other people had posted online.  At first, I handled the various inputs through a sequence of prompts the user had to click through one by one, but eventually I figured out how to combine them into a pop-up user interface with text-entry boxes, checkboxes, and radio buttons.  I also made a few changes to the process itself:

  • Moving layers out of the way to the bottom of the stack as they’re processed, rather than deleting them, so that the source document has its original state more or less restored at the end. (I also tried hiding layers after they’d been processed, but that seemed to take a lot longer.)
  • Providing an option to open and resize all source images in a folder before pasting them into to the main document as layers, rather than using the standard “Load Files into Stack…” script (which requires adding them all as layers at full size—a memory-intensive proposition—and then resizing them).

The script I ended up with isn’t elegant by any means.  But it works!  Or at least it does for me—you can download it in a zip file here if you like, but I make no guarantees about it running successfully on any system other than my own.  I’ve bundled it with another script called Averager Lite, which simply averages all the currently selected layers in an open document, copies the result to the clipboard, and pastes it as a new layer in a second document.  If you try either of these scripts out, please let me know how it goes.

averager-screenshotA few tips for using Averager 1.0:

  • Be sure to check “Export output frames” if you want Photoshop to generate any; otherwise it won’t.
  • Check “Center source images” if your source images are dissimilar and you think auto-alignment might fail.
  • Be sure your chosen output directory already exists; Photoshop won’t create it for you if it doesn’t.
  • If you’re importing your source images but not resizing them, make sure to specify a default canvas size that will fit them.

Things to Come

I hope the prospect of creating decent-looking stills and animations through alignment and averaging, as outlined above, might motivate some people to shoot sequences of daily photos of their surroundings who wouldn’t otherwise see much point in doing this.

Moreover, there’s no reason sequences like these need to be taken by a single photographer.  If there were some suitable vantage point chosen in the middle of a big city, such as the top of a post or the corner of a railing, dozens of people could take pictures from it whenever they pass by and then upload them to some agreed-upon place in a collaborative effort.  That would free the project from the vagaries of any one person’s schedule, vacations, and so forth, or even of any one person’s lifespan—this might conceivably go on for a century or more, as long as the physical reference point were to survive in place long enough.  (Unfortunately, I’ve already had seemingly permanent fixtures shift position or disappear on me.)  Such a project could resemble time-lapse mining from online photos in the breadth of its source base, but in this case the photos would be taken intentionally and regularly from the same vantage point, simplifying the process and avoiding the need for warping and interpolation.  And it would allow for time-lapse sequences set in public places where conventional gear designed for this purpose couldn’t be conveniently or safely left over the long term.

Meanwhile, I’ve been trying out some further ideas myself which I hope to cover in future posts.

  • I’ve taken the plunge into 3D, and you can already see some of the results here.  The main challenge is aligning the left and right images appropriately with each other.
  • I’m working on a modified version of my Averager script that will group source photos for averaging based on filenames auto-generated to reflect the dates and times when they were taken, rather than just averaging consistent quantities of them.
  • I’m trying to develop a less memory-intensive approach to auto-alignment, which is the biggest bottleneck in the process as it stands right now.  My idea is to align each image separately to a locked template, rather than all images to each other all at once.
  • I’ve been gathering archived public-domain webcam images to experiment with, such as the ones from the National Park Service available here.
  • I’m starting to apply these same principles to historical source materials including government weather maps, magazine covers, and front pages of newspapers.

Stay tuned for more developments as time permits.

 

Advertisements

3 thoughts on “Averaging Time-Lapse Imagery

  1. Pingback: Long-Term Time Lapses in 3D | Griffonage-Dot-Com

  2. Pingback: Improvements in Face Averaging | Griffonage-Dot-Com

  3. Pingback: Time-Based Image Averaging | Griffonage-Dot-Com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s