Improvements in Face Averaging

The historical face-averaging experiments I’ve been blogging about here for the past couple years have depended on efficient, decent-quality methods of averaging faces in the first place.  I’ve recently figured out some ways to improve my results and streamline the process of getting them, and in this post I’d like to describe my latest revised techniques.

For review, here’s a list of the main face-averaging strategies I’ve covered on this blog previously (here, here, here, here, here, and here):

  • Averaging groups of faces that represent different moments in time (e.g., yearbook photos, images in fashion magazines, paintings from different decades or centuries).
  • Displaying time-based face averages over time as animations, sometimes smoothed by creating overlapping averages from groups of other averages, by morphing between averages, or both.
  • Averaging groups of contemporaneous portraits of specific historical individuals.
  • Selectively flipping the source images in a project to orient all faces in the same direction, facing left or right.
  • Creating stereoviews from pairs of averages based on the same data sets but contrived to face at different angles.
  • Using image-morphing software to average full profiles or other types of source material which dedicated face-averaging software can’t handle.

Below is a representative example of the sort of thing I’ve been experimenting with: an averaged face based on just over 150 source images drawn from Vogue magazines of early 1994 (for juxtaposition with results from other years).  The version on the left typifies the results I’ve been able to achieve in the past.  The version on the right shows what I can do now.

1994-vogue-comparisonThe difference isn’t so much in the face itself as in what’s around it, especially the hair; but since I’m interested in exposing historically contingent details, that’s pretty important, and it’s as though these parts of the image have had a veil lifted from them or emerged from a dense fog.  Meanwhile, I’ve gained the ability to create sequences of these averages based on overlapping time windows far more efficiently than before using a Photoshop script I originally pulled together for processing other kinds of long-term time lapse animation.  Taken in tandem, these two developments have made the sort of large-scale face-averaging work I want to do a lot more feasible.

The face-averaging software I’ve been using to create full-to-half-profile composites is Abrosoft FaceMixer.  I still don’t know of anything else out there that would serve my purposes as well as it does.  However, even its “Deluxe Edition”—which is what you need if you want to be able to average more than twenty-five source images at a time—has some shortcomings that have become ever clearer to me over time.  And Abrosoft itself doesn’t seem to be pursuing any further development of either this program or their flagship product FantaMorph: they’ve never replied to any of my emails to info@fantaface.com, which have now started to bounce, and as of this writing their home page boasts quaintly about FantaMorph being “compatible with upcoming Windows 8.”  Hence I’m not optimistic about any improvements getting made to the FaceMixer program itself, and I’ve refocused my attention on devising new ways of harnessing its raw results for processing in other tools.

When you import an image into FaceMixer, you can have the program automatically try to identify one or more faces in it (step one) and then assign warp points to whatever faces it finds (step two).  For better results, however, I’ve found that it’s necessary to carry out these steps by hand, or at least to tweak the automated results.  Here’s how the process works.

facemixer-processing-sampleOn the left is a source image (which I may have flipped horizontally beforehand for consistency).  Following that is the “Edit Face” screen where you can move, expand, contract, and rotate a simple grid to identify a face within the image.  The software crops the image to the square and ignores anything outside of it, which in this case includes part of the ear.  Next is the “Edit Dot” screen where you can adjust individual warp points.  Moving one warp point nudges other warp points around as well, which makes this process a matter of imperfect compromise rather than precision (note the single outlier warp point along the hairline near the eyebrow in the example shown above, which I gave up on trying to fix).  And I’m never sure what to do with the warp points corresponding to an occluded ear—generally I just try to put them as close to the cheek as I can without overlapping it.  For the final composite, each source image is warped to align it with all the other source images, as seen on the right.  This example came out pretty well, barring the smeared ear and the jog in the hairline (an artifact created by that single outlier warp point along the hairline which I couldn’t seem to get right without messing up other points).  Sometimes the process goes quickly, but sometimes it doesn’t, and on the whole it can be pretty time-consuming.

Unfortunately, there’s no satisfactory means within FaceMixer of saving faces for long-term reuse once they’ve been prepared in this way.  It’s true that the Deluxe Edition allows you to save prepared faces in a “Face Library,” assign meaningful names to them in place of their auto-generated numbers (although I’d have preferred auto-assigned names derived from the source image filenames), and organize them into directories.  But the source images aren’t part of what’s saved—only the warp points, with a reference to the source image location.  If anything changes in the file path of a group of source images, which is likely to happen as drives fail or need to be reorganized, you have to relocate and navigate back to each one individually, which takes a lot of time and is prone to error.  That means there’s really no good means of backing up your efforts and reorganizing them for consistency or convenience over a number of months or years.  There’s also no convenient way to combine larger groups of prepared faces together in different groupings as I often want to do (say, all faces from 1950-55, and then 1951-56, and then 1952-57, with fifty or a hundred faces prepared and saved for each year).  If you want to add faces from the Face Library to a new project, you can’t simply select a group of them and drag them in all at once; instead, you have to select every individual face manually in turn and then add it by clicking an icon down towards the bottom of the screen.

But I did hit upon a way to export each individual face as warped to fit the average shape in a given project, which I thought might let me archive, group, and average these outside of FaceMixer.  The procedure is to go to the Face Mixing page; then, without changing the Shape setting (which I’ve shown locked in the screenshot below, although it’s not necessary to lock it), adjust the Feature percentage of each face to 100% in turn, save it as a Candidate (a term FaceMixer uses for tentative composite results), and then export all the Candidates.

exporting-warped-facesI knew the results would be warped specifically to the average shape of the faces in whatever specific project I was working on, but I figured that if a project were to contain a large enough quantity of faces, its average shape ought to be nearly interchangeable with the average shapes from other projects based on similar data sets, assuming reasonable consistency in facial expression, angle, and so forth.  On that basis, I thought it might be possible to store up warped faces from multiple FaceMixer projects; combine and average them together using some more flexible piece of software, such as Photoshop; and still obtain passable results from them.  With that in mind, I set out to try a few experiments.

In Photoshop, the best way to average a group of images is to load them all into a stack, create a Smart Object from them, and then set the stack mode to one of two averaging options: Mean or Median.  I decided to try both to see what would happen.  Below I’ve shown the mean average generated in Photoshop from warped versions of the official portraits of all fifty-two Miss America contestants for 2016-17 on the left, and the median average generated from the same warped images on the right.  (My first face-averaging time lapse animation was “The Face of Miss America, 1921-2014,” so this could fit into an update or continuation of that project.)

miss-america-2016-17-mean-median2FaceMixer itself can only generate mean averages, like the one on the left.  But the median average is a lot sharper; look especially at the detail in the hair.  So this emerged as another potential advantage of exporting individual warped faces from FaceMixer and then averaging them in Photoshop: we can choose among different methods of averaging, which can make a big difference.

Why did Abrosoft decide to go with mean averages in FaceMixer rather than median averages?  Probably because they expected most users to average only a few faces at a time.  They explain in their FAQ that the Standard Edition ($29.95) lets users average up to eight faces; the Professional Edition ($49.95) increases that to twenty-five; and the Deluxe Edition ($99.95), which I use, doesn’t impose any limit, although I’ve found it will run out of memory and crash beyond a certain not entirely predictable point.  With lower quantities of source images, the mean average generally gives an appealing impression of soft focus, while the median average just tends to look noisy or harsh or blotchy, and to make flaws in alignment more conspicuous.  To illustrate the point, I’ve generated mean and median averages from the sample data set of seven “girls” supplied with the FaceMixer software for demonstration purposes.  The mean is on the left below, and the median on the right.

abrosoft-girls-mean-and-medianIt’s only with larger quantities of source images that the mean average can take on an objectionably blurry appearance, as though we were dealing with a severely restricted depth of focus, and the crisper look of the median average becomes advantageous.  I’m not sure quite where the quantity threshold lies, but my time-based face-averaging experiments mostly seem to lie on the far side of it.

Below is a set of examples I created based on 109 images from Vogue published during the first half of 1954—part of the Fashionable Face project I described here.  The mean average is shown on the left, and the median average is shown in the middle; results have had auto-contrast and auto-color applied.

1954-vogue-mean-and-median3As with the Miss America examples shown earlier, the median average is sharper and presents a discernible hairdo (albeit a somewhat frizzy-looking one) where the mean has only a hazy blur.  It does have one weakness relative to the mean average, though: the low contrast between lips and teeth, which makes it look as though the subject is biting her tongue.  We’re combining source images with teeth alternately bared and hidden, and the median average hasn’t handled that situation as satisfactorily as the mean average has.  Indeed, “tongue-biting syndrome” seems to be a chronic flaw of median averages.  But we can always copy the teeth from the mean average into the median average, as I’ve done to create the composite on the right.  And we could mix and combine the means and medians in any number of other ways besides.

Now let’s increase the quantity of source images even further.  Here are the mean and median averages of 241 images from the March 2014 issue of Vogue, processed in the same way as the examples from 1954.

2014-mar-vogue-mean-and-median2And the mean and median averages of 241 images from Vogue during the first half of 2004:

2004-vogue-mean-and-median2In these last two examples, the mean averages give an uncanny impression of a disembodied head floating in a thick mist, while the median averages anchor the faces to hazy but appreciable—and different—hair-shapes.  The example I showed at the start of this blog post presents a pair of averages from the same project, based on 154 images from Vogue over the first half of 1994.  I’d definitely go with medians here rather than means, now that this is an option.

Now let’s return to my original plan of exporting warped faces from FaceMixer as a means of storing them up for re-use over the long term.  Last winter, I cropped a bunch of color images of female faces from the Media History Digital Library covering the years 1913-1941 with the idea of preparing each year’s pictures for averaging as opportunity permitted and then eventually incorporating all the results into a time-based animation.  I got through the years 1914, 1919, 1924, and 1929, but then left things sitting, partly because I wasn’t sure how to save my work reliably.  So by way of experiment, I’ve now exported the warped faces from these four FaceMixer projects.  Below are the results of the individual projects for 1914, 1919, 1924, and 1929 (from right to left) as averaged in Photoshop with the mean average on the top and the median average on the bottom (results have had auto-contrast, auto-color, and auto-tone applied).

1914-1919-1924-1929-median-and-meanBut how do things go when we try to combine faces exported from different projects?  To try out one scenario, I loaded all the warped source images for 1914, 1919, 1924, and 1929 as layers into a single Photoshop document, in chronological order, and then ran my Averager 1.0 script on it, set to generate median averages in groups of fifty while advancing twenty source images per frame.  Here’s the result, combining exported images warped to the Shapes of four different projects:

1914-1919-1924-1919-median-test-animationJudging from this quick-and-dirty experiment, I’m optimistic that separately warped and exported source images will combine well enough for animation, one potential problem area being the mouth (this appears to be an animated version of tongue-biting syndrome).  And that has exciting implications in turn.  As I’ve exported warped images from my earlier work on the Fashionable Face project, I’ve been assigning filenames to them that sort naturally into chronological order, so whenever I like I should be able to run my iterative averaging script on the folder of warped sources and automatically generate a sequence of overlapping averages based on my current data set.  Previously, this processing step was labor-intensive enough not to be undertaken lightly.  Now it should be trivially easy to create draft animations showing where the project stands at any point.

Meanwhile, here are all 229 source images exported from my FaceMixer projects for 1914, 1919, 1924, and 1929, averaged into a single pair of still images:

1914-1919-1924-1929-warped-sources-mean-and-medianThese results could be sharper than they are, but they still look reasonably acceptable under the circumstances.  Granted, the averaged subject suffers from tongue-biting syndrome, but that could be retouched if we’re concerned about it.

Another advantage of exporting individual warped images from FaceMixer is that we can adjust them individually in a variety of ways before averaging.  Parts of images that end up looking grotesquely distorted after warping tend to get “averaged out” in the end result, but they could also be selectively erased, blurred, or otherwise removed (although I’m not sure the difference would be worth the time and effort).  Also, whenever the face-selection square on the “Edit Face” screen extends beyond the borders of a source image, FaceMixer fills in the missing area with black, so if there are a number of such images in a project, one or more edges of the resulting average end up conspicuously darkened, as can be seen above in many of the averages from 1914-1929.  But we have the option of filling these areas with some other color.  Below are averages of 115 images from Harper’s Bazaar from early 1978, first created with the warped images “as is” on the left, and then on the right with the black border areas filled using typical colors selected from each of the sources with an eyedropper tool.  The mean average at upper left is what I was previously restricted to creating and is limited pretty much to the face itself with a hazy, dark background.

1978-harpers-mean-and-median-edges-as-is-vs-filledThe combination of color-filling the black border areas and substituting the median for the mean makes a big difference.  Once again, the median suffers from tongue-biting syndrome, but we can combine median and mean results:

1978-harpers-teeth-comboIt’s easy enough to make that last adjustment (of the teeth) manually for a single still image: just overlay the median average as a new layer over top of the mean average in Photoshop, erase the center portion of the mouth from the top layer, and flatten the result.  Carrying this out the hundreds or thousands of times needed for an animation would be another story, but I can probably come up with a script to do it automatically, as long as the area to be replaced is reasonably consistent.  We’ll see.

So far I’ve been leaving the Shape adjustment in FaceMixer alone.  The default is for faces to be warped and overlaid to the average Shape of all the faces in the project, with the warp point values of each face contributing an equal percentage to the whole.  However, you can also adjust the percentages to weight individual source faces more or less heavily.  Unfortunately, I’ve found that if you tinker with the Shape and Feature settings simultaneously, FaceMixer starts getting buggy and will generate a blank frame roughly one time in five if you try to export a result or add it to the list of Candidates.  This limits what we can conveniently do with the Shape adjustment feature.  However, FaceMixer still displays a result onscreen in these instances, so it’s at least possible to capture screenshots of results even if they can’t reliably be exported or saved as Candidates in the usual way.  Below is an animation I created from the Miss America data set for 2016-17 by capturing screenshots of the composite result with the Shape percentage of one of the source images set to each incremental value between 0% and 50% and then averaging these in overlapping groups of ten and advancing forward by three images between frames:

miss-america-2016-17-mean-animationAnd here’s an anaglyph of median averages created from results with the same source image set to two different Shape percentage values.  You’ll need red-cyan glasses to perceive the stereoscopic effect.

miss-america-2016-17-median-anaglyph1FaceMixer can’t handle faces portrayed in full profile, so for averaging those I’ve been using image morphing software, specifically FotoMorph, which is a far more time-consuming approach and doesn’t scale well at all.  The technique I described here entails generating animated morph sequences between pairs of images, selecting the midpoint frames as the averages, and then averaging those averages the same way in turn to create composites based on four sources, or eight, or sixteen, or thirty-two, or sixty-four, and so on.  So, for instance, I averaged these details of four casually chosen nineteenth-century paintings—

profile-4-nineteenth-century-paintings-sources—into a pair of two-source averages, and then averaged those in turn into a four-source average:

profile-4-nineteenth-century-paintings1Here too I have an update to share in terms of technique.  In my earlier blog post about this technique, I lamented “the need to average images grouped by powers of two (2, 4, 8, 16, 32, etc.).”  But it’s since occurred to me—and now seems obvious in retrospect—that we can in fact average any quantity of images, weighted equally, if we select frames other than the midpoint.  Let’s say we want to add a fifth image (already cropped and flipped horizontally) into the four-image average shown above:

fifth-image-smallWhat we need is an output image that is 4/5 the earlier composite and 1/5 the new source.  If we were to create a morph sequence with six frames, starting with the earlier composite and ending with the new source, we’d simply select the second frame.  FotoMorph lets us select a rate of 10, 15, 20, 25, or 30 frames per second for durations at increments of 1/4 second up to 60 seconds; and if we select 25 frames per second for 1/4 second, we get a six-frame export (it should technically be 6.25 frames, but in practice the software appears to treat it as exactly six).  So here are our six frames, with fractions showing what proportion of each frame represents the new source:

fifth-image-sequenceThe second frame is our target, with each of five source images weighted at exactly one-fifth:

five-images-sequence_02This is a hassle, of course, but at least it should be possible to average any desired quantity of source images in this way.

So those are the latest techniques I’ve hit upon for face averaging.  In this post, I’ve focused on explaining and illustrating the techniques themselves without worrying too much about the value of my subject matter.  Once I’ve had a chance to export more of the individual warped source images from my existing FaceMixer projects, however, I hope to be able to share some more interesting results.


Addendum: As I mentioned above, I’m not optimistic about any improvements being made to the FaceMixer software itself, but just in case, here’s a list of recommended changes and new functionalities I’d recommend:

  • Provide an option for saving prepared faces in the Face Library bundled together with copies of their source images, so that deleting or moving the source images from their original locations won’t affect these.
  • Let users select and add multiple faces from the Face Library to a project all at once, rather than only one at a time; or ideally to merge multiple projects.
  • Offer a choice between Mean and Median Averages.
  • Include a provision for aligning all the faces in a project to one or more preset Templates, rather than to Shapes calculated based on the faces in a given project at particular percentages; and allow the user to save the current Shape setting of a project as a new preset Template, or to average multiple Templates into a new Template.
  • Enable the user to export multiple averages based on Templates representing successive angles for animations “panning” from side to side or pairs of horizontally separated angles for stereoviews.
  • On the Edit Dot screen, provide a dynamic display of the result as warped to a preset Template, so that the user can see how each adjustment affects the output.  Also allow the user to “lock” warp points once they’re correctly placed so that moving other warp points won’t cause them to shift.
  • Fix the problems that cause larger data sets (e.g., 250+ moderately sized face images) to crash the program, and that cause complex combinations of Shape and Feature percentage settings intermittently to generate blank frames.
  • Allow the user to vary the canvas size surrounding the facial features (e.g., the size of the square relative to the face on the “Edit Face” screen) rather than invariably cropping to a set distance from them.
  • If the face-selection square on the “Edit Face” screen extends beyond the borders of a source image, fill the missing area with something other than black, such as transparency or a fill of the average color of the whole image.
  • Allow users to create custom warp point grids for subjects other than faces.
Advertisements

2 thoughts on “Improvements in Face Averaging

  1. Pingback: Time-Based Image Averaging | Griffonage-Dot-Com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s