Back in August 2016, I blogged about a hypothetical piece of software I wished were available: one that could conveniently and efficiently average large numbers of images by warping them to a set of user-defined control points (and without being limited to forward-looking human faces). Last year, I finally wrote some code of my own in MATLAB to do just that. I’ve been calling it Chronomorph because I originally designed it to extend the range of possibilities for time-based image averaging, but it can be used to average images of any kind as long as there’s a one-to-one correspondence between similarly-arranged features. Note that the process of assigning control points is entirely manual and doesn’t involve any machine learning.
Want to try it out? You can download an installer for a Windows 10 executable here. Use at your own risk; no warranties implied.
Below is one average I created from start to finish using the standalone executable. It’s based on forty-one images of automobiles taken from advertisements published in 1916, with just nine control points (front and rear axles, front and rear top of body, front and rear bottom of body, bottom and top of steering column, back of rear fender). The version on the top is a mean average, and the version on the bottom is a median average. In both cases, I used the simple “warp” setting. Adding more control points in strategic locations could reduce blurriness. Imagine this as just one frame in a video illustrating the evolution of automobile design from the 1890s to the present and you’ll get a sense for where this sort of thing could lead.
So how does Chronomorph work? Well, the first step is to create a template, or a model to follow when marking up each image:
- Upon opening the program, you’re prompted to select a location for saving your work.
- Then click “Start” under “New Template.”
- Navigate to the image you want to use for creating your template.
- Left-click on points in the image to define them as control points (or right-click to back up in case of mistakes).
- Click “Save” to name and save your template.
After defining and saving a template, you can then click the “Apply” button to cycle through a selected group of source images, which will each appear in turn on the left side of a screen, with the template shown on the right; there are also buttons provided for cropping and/or flipping them as necessary. As each control point is highlighted in turn on the template, you left-click on the corresponding point in the source image (or right-click to back up). Each marked-up image is saved as a .mat file in a folder assigned the same name as the template. To create an average, you then click on “Average” and navigate to the set of .mat files you want to process. At this point there are a number of adjustable parameters:
- Choosing similarity, affine, or projective transformations; or warping (based on an existing piece of code called warpImage.m, modified as described here); or sim+warp (a similarity transform followed by warping).
- Averaging a selected group of images all at once into a single still image, or averaging them with a sliding window to create an image sequence or video (with ability to control the “detail” and “shape” settings conjointly or independently).
- Taking the mean or median average of the pixel values, and then maximizing contrast or leaving it unchanged.
Each .mat file corresponding to a source image contains a structure array, imagedata, with fields:
- imagedata.I = the source image
- imagedata.name = the name assigned to the source image
- imagedata.points = the control points, where imagedata.points(a,b,c) gives the value of control point number a with its horizontal (b=1) or vertical (b=2) coordinate and a status marker, where c=1 if the control point is active, c=0 if it was never assigned, and c=2 if a value was assigned but is currently inactive.
The template .mat file is similarly structured but contains template.I, template.name, and template.points instead.
You can also open an existing template, save it again (with a new name) after adding additional control points to it, and update a set of existing .mat files by adding whatever new control points they’re missing.
Now I’d like to share an example that’s been a bit more fully developed. This one involves human faces in profile on coins, which is the specific subject I was blogging about trying to average when I first brought up the idea for this software.
I began by choosing a picture of the obverse of a large cent and creating a template based on it, defining a number of control points by clicking on each of them in turn.
Then I generated still averages for each year of United States coinage from the initial patterns of 1792 through the regular issues of 1797, including all denominations issued during the year, the possibilities in this period being half cent, cent, half dime, dime, quarter dollar, half dollar, dollar, quarter eagle, half eagle, and eagle. I tried to achieve a reasonable mix among these, although I haven’t been strict about representing each denomination equally—it was easier to find pictures of some combinations of year and denomination online than others.
First, here are averages with simple warping, which do a good job of preserving the rim of the coin but present rather blurry faces for 1794 and 1795 in particular.
Finally, here’s a video made up of averages generated with a sliding time window. This time I’ve limited my scope to large cents and half cents, but I’ve extended the time coverage to span the years 1792 through 1810. The averages are “sim+warp” means, with detail handled in groups of fifteen and shape handled in groups of forty, and then with the result re-averaged in groups of twenty (both shape and detail). Towards the end of the sequence, we run into a dilemma, since the bust of Liberty on Classic Head cents faces left rather than right as in all the previous designs. Here I’ve flipped all the Classic Head cent images to face right so that I can average them with the preceding Draped Bust cents, but that means that the final frames of the video are arguably backwards, and they’d remain so through the end of large cent production in 1857 if I were to extend the video out that far. Complications, complications!
Vogue Cover Faces, 1937-2017
It’s been over a year now since I first showed a Chronomorph-generated video in public, during a presentation on “Unlocking Sounds of the Past” which Dario Robleto and I gave at Northwestern University under the auspices of the Block Museum on April 25, 2018 (you can watch a video of the whole thing online). In a sense, this animated average is less exciting than the newer examples I’ve shared above because it deals with forward-looking human faces, and there’s already plenty of other software available for averaging those. But I think it’s still an interesting piece, and it helps illustrate further what Chronomorph can do.
This video presents a time-based average of faces that appeared on the cover of Vogue magazine from 1937 through 2017. Specifically, it’s a median average with a sliding twenty-five-face window (and, if memory serves, with the averaging done twice in a row). The effect is rather uncanny as different parts of the face twitch into new positions independently of one another. Overall, the parade of changing fashions approaches what I was hoping to achieve with my “Fashionable Face” project. But an even more thought-provoking parameter here might be facial expression, which seems to shift through a number of distinctly different moods. Reflections of the times, somehow?