Averaging Faces in Profile, and Other Things

Face averaging has now become a relatively common procedure, with an assortment of software programs specifically designed to carry it out.  But all past efforts I’ve seen have had one significant limitation: they assume faces will be shown looking forward, and they can’t accommodate faces shown in full profile.  That’s frustrating if we want to use face averaging as a historical technique, since many images that would otherwise be prime candidates for averaging happen to be profiles, including the portraits found on coins such as these (courtesy of CGB, via Wikimedia Commons):

profile-coins2In this post, I’ll describe a technique for averaging faces in profile, and I’ll share some results based on both ancient and modern sources.  As for the sharing part, let’s go ahead and start that right now. There’s no particular reason anyone would want to average the specific four coin portraits shown above, but just to give you an idea of what kind of results to expect, here’s an average of the two coins on the left (left), the two coins on the right (right), and all four coins (in the middle).

averages-of-miscellaneous-coinsThe technique I’m using to create these averages is somewhat time-consuming (at least, based on the tools I have available to me).  But I don’t think it needs to be, and I’ll also describe the characteristics I believe a piece of software would need in order for us to be able to carry it out more conveniently and efficiently.

Now, it’s not only faces in full profile that have fallen outside the face-averaging mainstream.  Most past experiments with face averaging have attempted to align and analyze source images so that the result faces straight ahead, as in a passport photo.  That has two drawbacks.  First, passport photos aren’t the most aesthetically pleasing of portraits, although that may not matter in some contexts (e.g., a psychology experiment studying the correlation between objective averageness and subjective attractiveness).  Second, source images collected in the wild tend to show subjects facing at least a little to the left or the right, inconsistently as to direction.  These might average out to facing forward when superimposed “as is,” but the details of the opposing sides of faces will then either cancel each other into a blur or end up shown simultaneously, with both ears “sticking out” in semi-profile view.  To show what I mean, the blue box below outlines an average I created of ten portraits of U. S. senators from 1856 together with my ten source images, five facing to the left and five facing to the right.

1856-senators-harmonized-and-unharmonizedWith this problem in mind, I’ve been flipping images horizontally as needed to make them all face the same direction before averaging them.  The red box above outlines an average I created from the same ten source images as the other one, but with five of them flipped to face in the same direction as the other five—the bottom row of five portraits is simply the top row flipped in its entirety.  Because I’ve been doing this flipping as a matter of course, I’ve mostly ended up in the past with averages that face not straight ahead, but a little off to one side, sometimes approaching a half profile.

sample-oblique-compositesEven this minor departure from the facing-straight-ahead or “full face” average isn’t particularly common.  In searching about online, I’ve found examples of obliquely-facing averages created by only a few other people, which I’ll take the liberty of quoting here for review.  The average shown below on the left is by Martin Nenoff; the one in the middle is by Jun’ichiro Seyama, who has created a number of similar images; and the one on the right is from an advertisement for Seyama’s “Average Face!” app, designed to let other people do the same thing for themselves.

oblique-facing-averagesNenoff and Seyama both seem to be taking the aesthetic side of face averaging seriously here, even if they’re working in the comparatively unimaginative “fantasy female face” genre.  That said, I don’t know whether either of them flipped any of their source images to match the others, or whether they just chose images that all faced the same direction originally, so they may or may not be using the same flipping technique I use.  The same is true of Pierre Tourigny’s averages of female faces from hotornot.com, which all face a little rightward—a hallmark of his work in general, judging from other examples.

tourigny-averagesIn any event, there aren’t any real technical obstacles to keep us from applying face-averaging tools designed with forward-facing faces in mind to faces turned partway towards one side or another, up to a half profile or perhaps even beyond.  The same principles apply to both cases: you just need to identify the pupils and corners of the eyes, the corners of the mouth, certain key points on the nose, and so forth, and then to align these points across multiple images.  The software I’ve been using for face averaging—the deluxe edition of Abrosoft’s FaceMixer—can handle obliquely-facing faces without difficulty, even if this sometimes stretches the facial grid template to limits at which it doesn’t seem entirely comfortable.

abrosoft-obliqueWhen it comes to full profiles, though, this kind of facial grid template just isn’t up to the task.  What we most want to do is align corresponding points along the edges of the profiles—following the contours of the nose and chin—and on that score the underlying logic of the software is no help at all.  Don’t bother trying the experiment shown below.  It doesn’t work, at least not as a means of averaging multiple full profiles with each other.

abrosoft-profileAnd so I’ve turned instead to image morphing tools, which tend to be a lot more flexible about the kinds of source images we can feed into them, but also to require more intensive manual work, since they don’t come with ready-made facial grid templates.

Image morphing tools are designed primarily for creating animations in which one source image morphs seamlessly into another over a number of frames, often for comic effect.  Here’s a typical result obtained by morphing between this image (by Hillary for Iowa) and this image (by Michael Vadon):

clinton-trump-morphWhat we’re interested in here is the middle frame, which is an average of the two source images.

clinton-trump-morph-middle-frameOne inconvenient aspect of this approach to averaging is that it requires processing images in pairs.  That’s not to say that it limits you to averaging a maximum of two source images: you could always average two sources, and then another two sources, and combine the results to get an average of all four sources, and then increase the quantity of sources step by step to eight, or sixteen, or more.  But standard morphing tools only allow you to average two images at a time, which can be time-consuming and tedious.  Even so, this kind of pair-by-pair cumulative morphing has been used successfully for face-averaging experiments before, notably by Dr. Martin Gruendl at the University of Regensburg, as described here.  Gruendl’s own averages face straight forward, but we can also use the same approach to average faces shown in profile.  That’s the “one weird trick” I’ll be covering here, in the parlance of clickbait peddlers.

Since it’s August right now, I’ve chosen as my first subject the namesake of the current month: Augustus, the first Roman Emperor.  Here are images of two coins with two different contemporaneous profile portraits of him, borrowed from an online Roman Coins Database:

augustus-morph-1zTo average them, I opened them into a morph sequence in FotoMorph.

augustus-morph-1yThis tool requires that we manually define points in the two images that correspond to each other: first we click in one image to define a point, and then we drag the yellow dot that appears in the other image to the corresponding spot.  In this case, I defined about twenty-five points.

augustus-morph-dotsThat yielded this animated morph sequence:

augustus-morph-1aBut we’re really interested only in the midway point: the average frame that falls exactly halfway between the two source images.  With that in mind, we want to be sure to render the animation in a way that creates an odd number of frames, such as fifteen, so that the midway point corresponds to a single frame—for fifteen frames, that would be the eighth frame.  Here’s frame eight excerpted from a fifteen-frame export of our morph sequence:

augustus-morph-1mWe can stop here if we like, maybe retouching the image to taste like this:

augustus-morph-1nnOr we could mix the results of the two-source average with the results of another two-source average to get a four-source average, and we could then mix the results of that four-source average with the results of another four-source average to get an eight-source average—

augustus morph 8 sources—and we could keep this process up for as long as we like, doubling the number of source images each time to 16, 32, 64, 128, and beyond.  For purposes of illustration, though, I’ve stopped here at eight.  Below you can see a neatened-up version of the eight-source average of Augustus coin portraits:

augustus morph 8 sources nnThe hair is a bit of a mess because there’s no one-to-one correspondence among the source waves and curls, and the base is a little wonky too, but otherwise this average appears to represent the source “type” pretty well.  I’d say it looks more typical of the group as a whole than any of the individual coin portraits does.

Of course, this technique can be applied to images on more recent coins as well.  Here’s an average of the four portraits of Queen Elizabeth II used on standard British coinage over the past fifty years:

elizabeth-ii-averaged-coin-portraitAnd here’s the average of the presidential portraits on the U. S. penny, nickel, dime, and quarter as of 2005, with Lincoln’s face flipped to face left like the rest of them.

penny+nickel+dime+quarterWhat else can we do with this technique?  Well, it lends itself nicely to the “fantasy female face” genre, if we want to go there. Below are two four-source averages created in that spirit (at left and right), with an eight-source combination of both (in the middle).

profile-modern-composite2In the tradition of the genre, I’m not identifying my source images, which are individually copyrighted by other people.  I think face averaging is plenty transformative, but I could still see thorny issues arising out of intellectual property rights if anyone decided there was money at stake.  But if you’d like a fun challenge, you could always see if you can identify my eight source images.  This shouldn’t actually be too difficult, since I simply googled phrases such as “in profile” and chose the first eight suitable pictures I found.

Sometimes the creators of averages like these opt to leave the telltale artifacts of the process unaltered—such as ghostly wisps of hair—maybe because they consider this to be more honest or authentic, or because they want to draw attention to how the images were produced, or because they don’t feel like processing their results further.  But here’s a retouched and neatened version of my eight-source average, just for good measure (note that this step further disguises the source images, which would now be just about impossible to identify):

profile-modern-2eSo yeah, the profile-averaging technique I’m using can create passable stock “fantasy female faces.”  But as popular as such images seem to be, I’d like to think there are other more meritorious profile-averaging tasks out there for us to tackle.  For instance, we can select four paintings of faces shown in profile from the latter half of the nineteenth century (here, here, here, and here)—

profile-4-nineteenth-century-paintings-sources—and average them to create a “new” face in profile that I’d argue looks even more quintessentially Victorian than the sources do:

profile-4-nineteenth-century-paintings1This particular average is based on a fairly small data set, and I wouldn’t dream of trying to pass it off as the “average Victorian painted female profile.”  However, by combining enough source images into an average such as this one to make it truly representative of a given period—say, 32 or 64 of them—we could draw full-profile pictures into the project I introduced in my earlier post on “Face Averaging and Art History.”  All that would be needed is time and patience.

But wait, there’s more!  I’ve focused here so far on averaging faces shown in full profile because that’s the specific problem I set out to solve, but the same cumulative morphing technique could be used to average images of any kind whatsoever, as long as they share a structure with one-to-one correspondences between points.  To continue our coin-averaging theme, here’s an average of eight different specimens of the English “short cross” penny (1180-1247), presenting an idealized view of the reverse cross design:

average-8-short-cross-pennies2Or for a more elaborate example, let’s take the iconic Portrait of Henry VIII created in 1536-37 by Hans Holbein the Younger.  This portrait formed part of a mural that was lost in the Palace of Whitehall fire of 1698, but it remains famous today thanks to a number of surviving copies.  Of course, none of the individually painted copies is an exact duplicate of Holbein’s original; the backgrounds in particular vary considerably.  But if we average several of the copies, we might be able to approach the appearance of the lost original by reinforcing details where the copies agree and factoring out details where they don’t.  Below is an average I created from four of the best copies (found here, here, here, and here) using the cumulative morphing technique.

henry-viii-averageI’d like to conclude by describing a kind of specialized software tool I’d like to have for doing this kind of work.  If you’re not into the nuts and bolts of the process, you may now be dismissed!

Right now, I’m aware of programs that can (1) morph between two images of any kind, like FotoMorph, and (2) average any quantity of images of forward-looking faces, like FaceMixer.  What I haven’t yet found, though, is any convenient tool for averaging more than two images of subjects other than forward-looking faces.

Let me spell out the sort of thing I have in mind, in the way I’d imagine it ideally working.  To start a new project, I would first create a template defining a set of points relevant to a particular image type.  The example shown below defines around eighty-five points for a face in full profile.

profile-points-template2Then I would open the first source image in another window adjacent to the one showing the template (or it could be the second source image, if the first one is being used as the basis for the template itself).  Each point defined in the template would be highlighted in turn according to a fixed sequence (which could itself be edited and revised); and for each one, I would identify the corresponding point in the source image by mousing over it and clicking.  Once all the reference points had been identified in a given source image, I would move on to the next one, and so on until all the images in the data set had been duly marked up.  If I decided along the way that a point defined in the template was superfluous, I could delete it.  Alternatively, if I decided to define a new reference point midway through a project, I could add it to the template, and then I would be prompted to cycle through each of the images marked up so far, identifying where the newly-defined point ought to go in each one.  Finally, the average positions assigned to the reference points would be calculated as a basis for Delaunay triangulation, and each “triangle” would be extracted, reshaped, and blended from all the source images to produce an average.

profile-template-triangulation2Does such a thing exist?  I’ve seen a hint that it might be possible to contrive something equivalent in MATLAB, but I’m not sure how conveniently, or what kind of learning curve would be involved.  Anyhow, some tool along these lines could save those of us trying to average miscellaneous images a lot of time, while simultaneously freeing us from the need to average images grouped by powers of two (2, 4, 8, 16, 32, etc.).  If the software doesn’t yet exist, maybe someone looking for a worthy project might consider creating it.

Advertisements

2 thoughts on “Averaging Faces in Profile, and Other Things

  1. Pingback: Improvements in Face Averaging | Griffonage-Dot-Com

  2. Pingback: Time-Based Image Averaging | Griffonage-Dot-Com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s