Face Averaging as a Historical Technique

News media were recently abuzz with reports of a study in which the “average” faces of women from forty-one different countries had just been discovered by combining and averaging large numbers of real individual photos.  In fact, it wasn’t so simple: as the Huffington Post observes, this work had actually been done a couple years ago, and it was conceived more as an art project than as a “study.”  But it was news to me, and I thought it was pretty interesting to compare the averaged faces as they varied from country to country.

And then I started wondering: what would happen if we substituted variation over time for variation in geography?  Why not create “average” faces from different periods and compare them with each other?  Had anyone ever done anything like that before?

Looking around the web, I found an interesting composite of five hundred photographs of the same person, repurposed from a self-portrait-a-day project; a fusion of the faces of fifty-seven girlfriends from the show Seinfeld; a comparative study of averaged photographs of “attractive and unattractive faces”; a composite face of the “average gamer” based on over 2,000 source images; and some eerie composite faces built on images extracted from popular movies by face-recognition software.  All very interesting.  But the only historical applications of face averaging I could find were pairs of averaged faces of modern versus Golden Age actors and actresses.

There’s no reason why we should be limited to creating averages for two moments in time, and I wanted to try something more ambitious.  The first idea that occurred to me was to work with yearbook photographs—a seemingly ideal source for large numbers of comparable pictures of people’s faces, conveniently organized by year.  And since I’m at Indiana University Bloomington, I thought I’d make an experiment using some of our own yearbooks.  The Arbutus has been published here every year since 1895, and each one contains photographs of everyone (more or less) in the senior class.  Most volumes up through 1925 are available in digital form at decent resolution via Archive.org, so I downloaded several (1901, 1908, 1913, 1919, and 1925) and manually cropped out all the individual student photos in whatever category seemed to make the most sense in each case (e.g., limited to Arts and Sciences whenever other schools were separated out into different sections of the yearbook).

Then there was the matter of what software to use.  Some sites offer online face averaging through a web browser, such as this one and this one; but none of them seemed to do quite what I wanted, and I also wanted more control over the process.  That led me to Abrosoft’s FaceMixer (note that to average unlimited numbers of faces, it’s necessary to choose the “deluxe” version of the program).

So here are the “average” faces of the men and women of the Indiana University classes of 1901, 1908, 1913, 1919, and 1925:

arbutus-1901+1908+1913+1919+1925But it doesn’t stop here, since the averages of sequential moments in time can also be placed onto a timeline and “played” (or “educed,” to use my preferred term).  Here’s a preliminary and imperfect animation of the “average” face of female students at Indiana University as photographed between 1901 and 1925, based on the five years shown above, with other frames interpolated through tweening:

arbutus-preliminary-animationI consider this animation preliminary and imperfect first of all because it’s based on some of my earliest experiments using FaceMixer, while I was still learning how to use it; and secondly, because it would be possible to average every single year between 1901 and 1925, rather than just a few select points along the way to serve as keyframes.  In fact, we could create separate averages for every year between 1895 and 2014!  That would take a lot of work (scanning unscanned yearbooks, cropping out and sorting images, etc.)—would anyone out there like to help?  Or maybe I could just join Classmates.com and download vast numbers of datable, digitized, and conveniently cropped yearbook photos that way.  For now this animation illustrates the potential of the idea but doesn’t really do it justice.

That said, we can already see some likely patterns of change here—for example, the shift in facial expression.  In 1895, it was rare for anyone to smile for a yearbook photo.  In 2014, pretty much everyone smiles.  The idea that we should “smile for the camera” is a cultural convention that arose sometime between those two dates.  And between 1901 and 1925 we can see the faintest start of the transition; if we created an animation spanning the whole period from 1895 to 2014, one thing I’m sure we’d see is the onset of a broad grin.  Maybe someone could find another way to study the change in cultural conventions that constrain the facial expressions people adopt when they’re being photographed, but it would probably be more difficult and less vivid than this method.

One question that’s worth considering is how many faces are needed to create a meaningful average—what’s just enough, and what’s overkill?  Do we need to include all the photos in a yearbook, for example, or could we just pick a random sample?  To illustrate the implications of using different numbers of examples, here are the faces of the Topps baseball card series of 1952, 1953, and 1954 (from top to bottom) as averaged respectively from 25, 50, and 100 examples (from left to right)—based on images of cards for sale on a dealer’s site:

topps1952+1953+1954-averages-of-25+50+100It’s worth noting that the variation seen from year to year has less to do with a change in how baseball players looked than in different artistic approaches taken for each year’s card series.  As for the numbers of examples, the 25-example averages are already pretty representative, but they do differ in some features from the 100-example averages; note for instance that the eye color for 1954 changes from light blue to something darker as more data gets factored in.  The Indiana University yearbook examples shown earlier were based on numbers of examples ranging from 32 women and 72 men (for 1901) to 222 women and 149 men (for 1925).  I suspect there would be diminishing returns going much beyond 100 examples, although the features might smooth out a little more.

For this technique to work, then, I believe we want portraits that are taken on some regular basis (such as once a year) of around 50-100 people who are all comparable in some way.  So in brainstorming about other experiments to try, I thought: how about a beauty pageant?  Photographs of most national contestants for Miss America since 1998 were available on the official pageant website, and I managed to grab 1997 as well via the Wayback Machine’s archived version of the site.  The image below on the left shows the average of all eighteen winners of the Miss America competition for the years 1997 through 2014, while the image on the right shows the average of all 934 national contestants in the pageants for those same years.  (To avoid confusion, I’ll refer to Miss America pageants by the winner’s regnal year; the national pageant typically occurs early that year or late the previous year, with state competitions occurring earlier in the previous year.)

miss-america-averaged-1997to2014-winners+all-contestantsHere we can see the winners and the contestants alike converging on a remarkably similar “ideal” facial appearance, bearing in mind that the image on the right is based on over fifty times more data than the one on the left.  So far, these images tell us nothing about possible changes in average facial appearance over time—that is, whether the Miss America “look” developed in any particular way between 1997 and 2014 or remained consistent over the whole eighteen years.  However, we can also average all the contestants from each year separately and then compare the results year by year.  If we compare the averages of any two adjacent years—such as 1997 and 1998, 1998 and 1999, 1999 and 2000, and so forth—the differences between them are pretty minor.  On the other hand, if we compare 1997 and 2014, the beginning and end years, we find the results looking noticeably different from one another:

miss-america-averaged-1997and2014And what better way to display change over time than by arranging the information on a timeline and “playing” it?  The animated GIF below shows a gradual transformation in the average facial appearance of Miss America contestants from 1997 through 2014 at a rate of three-fifths of a second per year:

miss-america-averaged-animation-1997to2014If I had portraits of all national Miss America contestants back to 1921, I could create a similar animation for that whole time period.  I don’t, and it would probably be tough to gather them.  However, it’s not difficult to find images of the winners of the competition.  So to carry out an experiment with a longer time scale, I hunted down two different portraits of each Miss America from 1921 to the present (to avoid giving any one portrait undue weight), with four for Mary Campbell (who won in both 1922 and 1923) and one each for Vanessa Williams and Suzette Charles (both 1984).  In the first animation shown below, each image is an average of the winners of twenty-five consecutive pageants, beginning with the first twenty-five and ending with the most recent twenty-five:

miss-america-25-yr-averages-1921-2014This second animation is based on composites of the winners of ten consecutive pageants each, so we get more finely-grained detail:

miss-america-10-yr-averages-1921-2014These results exceeded my expectations, and I think they illustrate certain historical patterns (shifting ideals of beauty, styles of makeup, etc.) more vividly than any other approach I know about.  I’ve also uploaded them as videos in higher resolution to YouTube.

Another idea that caught my interest was to average the faces of some well-defined group of politicians over time, such as all members of the United States Congress.  Here for a base-line are the average faces of all current senators (as of June 2014), divided by gender and by party (Democrats on the left and Republicans on the right, excluding the two Independents), with each senator represented by five different images:senators-2014There are many more male senators to average out than female ones, which I assume is why the top two images have converged more fully on an “ideal” face than the bottom two.  I’ll let others decide whether there’s any significance in the differences between the Democrats and the Republicans.  But has the appearance of U. S. Senators changed over time (apart from the gender issue)?  To find out, I went looking for handy sources of historical senatorial portraits and quickly found two: Portraits of United States Senators (1856) and Biographies of the Present Senators of the United States (1892).  But there was a slight problem.  In all my previous experiments, a majority of the portraits showed their subjects facing pretty much straight ahead.  But now nearly all the portraits I had to work with showed senators facing off to the left or the right, as with these examples from the 1856 volume:

senators-1856-originalSo I tried processing these portraits in two ways: once “as is,” with the original rightward or leftward orientations; and once with all portraits oriented the same way (i.e., flipped horizontally as needed).  Here are the results for 1856 (on the left) and 1892 (on the right), with the original-orientation version shown first, and then the “flipped” version:

senators-1856+1892-sidebysideI was surprised, first of all, to find the 1856 average turn out looking like a charcoal drawing—I suppose any composite based on engraved line drawings is likely to yield a similar effect.  But an even bigger surprise was the apparent difference in angular perspective, which effectively produces pairs of stereoscopic images.  I’ve successfully viewed these image pairs in 3D using a vintage stereoscope, but here are some tweened animations for handy online viewing, made using the technique described here:


It seems there are indeed perceptible changes in average senatorial appearance over time; the facial fuzziness of the 1892 image reflects a tendency towards copious facial hair in that period, for instance.  With a lot of work, I suppose it would be possible to create a time-lapse animation of the average face of U. S. Senators from the eighteenth century to today, and it would probably be rather interesting to watch.  We might even learn something from it.

In retrospect, “flipping” all images so that the subject is facing the same direction is something I should probably have been doing from the start of these experiments.  Not only does it make better use of the data, since portraits posed in different directions aren’t partly cancelling each other out, but it also tends to produce more aesthetically pleasing portraits, which I suppose is why the artists and/or photographers posed their subjects this way in the first place.  Here’s the “average” female student of the Indiana University class of 1919, re-processed with directions harmonized:

arbutus-female-class-of-1919And we’re not limited to creating averages from images of different people; we can also average multiple images of the same person.  Here, for instance, is the average of thirty-one different portraits of Queen Elizabeth I, all created during her lifetime (although at different ages):

Elizabeth-I-averageSo here we have a new portrait of Queen Elizabeth I that is arguably also somewhat authentic, insofar as it’s based on contemporaneous portraits averaged together more or less objectively.  Indeed, this may give us a better sense for what her portraiture looked like in general than any one “real” portrait would be capable of doing.  I could imagine some interesting debates arising on that point.

One caveat.  Images can be run through FaceMixer using automatic defaults (i.e., letting the program identify the faces and the locations of eyes, nose, and so forth, which it sometimes gets wrong), or the processing can be painstakingly manipulated.  I’ve paid different amounts of care and attention to the individual source faces when creating the averages shown in this blog post as I’ve grown more comfortable with the program, ranging from some of the Indiana University yearbook pictures (least and earliest effort) to Queen Elizabeth I (most and latest effort).  I imagine everything I’ve done so far could be improved upon, although I’m not sure by how much.  Even if FaceMixer makes mistakes in its automatic detection of faces and facial features, individual errors seem unlikely to make much difference—if we’re using enough data—because they’re random and so shouldn’t “accumulate” as the good data does.

In summary:

  • Face averaging can be used to display faces “typical” of longer or shorter moments in time.
  • Averaged faces from different times can be arranged on a timeline in sequence and “played” as motion pictures.
  • Face averaging might help us detect patterns of change in quantities of historical visual data that are otherwise too vast to comprehend all at once.
  • These patterns of change can involve—among other things—changes in pose and facial expression and changes in styles of artistic representation.
  • By creating two averages—one from faces oriented the same direction, and one from faces oriented in opposite directions—we can produce a stereoscopic image pair.
  • We can also average multiple images of the same historical figure to create new portraits that are arguably more representative than any one “real” contemporary portrait.

So, ladies of the Indiana University classes of 1919 and 1925: do you think we’ve exhausted the possibilities of face averaging as a historical technique yet?



9 thoughts on “Face Averaging as a Historical Technique

  1. Pingback: Face Averaging and Art History | Griffonage-Dot-Com

  2. Pingback: “Say Cheese!”: Using Face Averaging to Track the Rise of the Photo Smile | Griffonage-Dot-Com

  3. Pingback: What is Eduction? | Griffonage-Dot-Com

  4. Pingback: Historical 3D Fun With Anaglyphs | Griffonage-Dot-Com

  5. Pingback: The Fashionable Face: A Work in Progress | Griffonage-Dot-Com

  6. Pingback: Averaged Portraits of U. S. Presidents, 1789-1829 | Griffonage-Dot-Com

  7. Pingback: My Fiftieth Griffonage-Dot-Com Blog Post | Griffonage-Dot-Com

  8. Pingback: Improvements in Face Averaging | Griffonage-Dot-Com

  9. Pingback: Time-Based Image Averaging | Griffonage-Dot-Com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s