The principle is simple enough: average high school yearbook portraits, year by year, and line up the results in order so that we can see how norms changed over time. I’ve done this before, and so have other people. But I don’t think it’s ever been done in quite the way I’d now like to share. My latest face-averaging project spans over a hundred years’ worth of yearbook photos from one single high school. And it’s not just any high school, but my own high school in Valparaiso, Indiana, where I was a member of the graduating class of 1989.
First off, here’s an animated pair of averaged portraits of the senior men and women of Valparaiso High School from 1913 through 2013, incorporating a total of 23,313 yearbook photos—11,983 female and 11,330 male—with a three-year sliding time window, which is to say we’re always seeing the average of three years’ worth of portraits, but with the groupings slowly inching forward along the timeline.
We can also use grids of still images to display the same information a bit differently, and to different effect. Here are averaged portraits of senior women for each year from 1913 through 2012, progressing from left to right and then from top to bottom, incorporating 11,802 source photos. (You can click on most of the still images in this post to load them in full resolution, usually 1000 pixels square per portrait.)
And here’s a display of the same kind for the senior men of 1913-2012, incorporating 11,149 source photos:
Unlike the year, which is a meaningful organizing unit for yearbooks and high schools alike, the decade is an arbitrary distinction here—both the ten-year unit in general and the specific ten-year groupings that reflect our year-numbering conventions: 1920 through 1929 as the “twenties,” 1930 through 1939 as the “thirties,” and so on. There’s no reason why we should expect the “sixties” (literally 1960-1969) to constitute a more historically coherent unit than, say, 1965-1974, or why cultural trajectories ought to align naturally with ten-year periods or have ten-year lifespans. When it comes to multiple-year averages, I prefer an approach that suggests continuous gradual change, as in the video below, generated this time with a ten-year sliding time window (which is after all no less defensible than a seven-year window, a thirteen-year window, or whatever other window size we might choose).
But just for good measure, here’s a grid of still portraits of senior men averaged by decade from 1920-1970. Averaging this many source images takes a lot of computing time, and I just haven’t gotten around to processing equivalent ten-year averages for 1980, 1990, or 2000 yet.
Quite a few high school yearbooks have been digitized and posted online at this point, but it’s still challenging to find collections that are both complete across a long span of years and convenient to access and process. It’s a happy coincidence for me that the yearbook for my own high school is one of the very few that fits the bill. The Porter County Public Library System digitized the yearbooks in its collections for several local high schools, including a long run of yearbooks from Valparaiso High School which it uploaded to Archive.org at the start of 2017, ranging in date from 1904 through 2012.
The 1904 yearbook doesn’t contain any photographs, but individual portraits of members of the senior class were routinely included from 1905 onward. The sole exception is the 1909 yearbook, which instead has full-body portraits of members of the senior class posed in groups of three or four. The collection on Archive.org is missing only a single yearbook—the one for 1912—from the whole period 1904-2012. One of the senior portrait pages of the 1908 yearbook is also torn, with a loss of two portraits. But all published individual senior portraits for the hundred-year span of time from 1913 to 2012 seem to be available. Moreover, I bought a copy of the 2013 yearbook on eBay and scanned the portrait pages myself (at higher resolution than the material at Archive.org) to extend coverage by one additional year, bringing the total to 101 continuous years—or, to put things more vividly into perspective, one year for each puppy in 101 Dalmatians. Of course, as of this writing there should also be six more recent yearbooks out there for the years 2014-2019, which aren’t a part of my current dataset.
At first, the Valparaiso High School yearbook was called simply the Annual or High School Annual, but in 1915 it became the Tubican, in 1916 the Vee Aich Ess, and in 1917 the Valenian, a name that stuck and has continued to the present day, with formal numbering starting with the so-called “third issue” in 1919. That’s why the 2013 yearbook is labeled “Vol. 97” rather than some higher number.
Before I go into how I’ve processed the Valparaiso High School yearbook photos, I’d like to describe two other projects centered on averaging yearbook portraits, one by artist Jason Salavon and one by computer vision researcher Shiry Ginosar and her colleagues.
Jason Salavon is well known for using custom software to create works in a fine arts context, often by digitally averaging numerous images in a category such as wedding pictures, photos of children with Santa, or Playboy centerfolds. In 1998, he produced a four-panel artwork called The Class of 1988 & The Class of 1967 by averaging the male and female portraits for his own high school graduating class and that of his mother, who had gone to the same school. He seems to have aligned the heads and torsos before averaging but not to have tried to align other features such as eyes and noses and mouths (which is more computationally demanding), with the result the faces themselves came out as nearly featureless blurs. This blurriness is a stylistic hallmark of Salavon’s work in general, and it makes for an artistically provocative effect, presenting views that are at once intimately familiar and uncannily alien. On the other hand, the results arguably illuminate the corpus on which they’re based less richly than they could. They highlight some common features of the yearbook portraits—the ones that happen to be easiest to draw out, from a technical standpoint—at the expense of others. There’s nothing wrong with that: Salavon plainly values the unique affective visual experience a blurry average can deliver, so bringing the averages more sharply into focus might have defeated his artistic purpose. But the choice resembles the one a photographer might make in capturing a galloping horse as a blur rather than in a form where the exact positions of its hooves can be seen frozen in time.
In 2015, Shiry Ginosar et al. published a paper called “A Century of Portraits: A Visual Historical Record of American High School Yearbooks,” presenting a project in which they had averaged high school yearbook portraits on an unprecedented scale. They began by gathering together 949 digitized yearbooks from 128 schools. After going through them and identifying the pages containing senior portraits, they reduced these to grayscale “for consistency across years” and then used a face-detection algorithm to extract 168,055 faces from them. Next, they used a software package called IntraFace to identify facial landmarks, estimate poses, and exclude all faces that weren’t forward-facing, resulting in a pared-down dataset of 37,921 faces (in 814 yearbooks from 115 schools). They then applied an affine transformation to align each face to a consistent position and used another algorithm to sort the faces by gender, resolving cases that stumped the algorithm with a crowdsourced subjective judgment call. In their own paper, they presented averages only by decade (“1900s” through “2010s”), although they’ve also made their dataset available, and Aymann Ismail of Slate produced a video of averages by individual year, skipping several years for which the project apparently had insufficient data (1907, 1917, 1918, 1920, 1921, and 1924). The approach taken by this team works well for the kinds of analysis that interest them, centered on exposing trajectories in smiling, eyewear, and hairstyle (for which they also averaged clustered subsets). Among other things, they’ve experimented with using their results to date other photographs, with promising results. But for them the averaged images seem to have a strictly functional value, in that they provide a handy means of structuring and displaying a particular kind of scientific data. The images should be informative, but beyond that aesthetic concerns don’t intrude.
Salavon prioritizes art over science, and Ginosar et al. prioritize science over art; and in both cases their priorities impact what their averages end up looking like. But I care about both aesthetics and accuracy—about providing useful information and affective impact. Separating these concerns feels downright alien to me. Maybe that’s because of my background in media preservation, a field in which the demands of accuracy and aesthetics tend to coincide. Or maybe it’s because of my background in media history, which makes me want to explore the potential of time-based image averaging by analogy with the sound recording and the motion picture—as something that could reshape the engagement of our senses with the passage of time. We’re dealing here with a novel way of seeing into the past, and I want to help probe the limits of what we can do with it.
In 2014, the year before the release of the Ginosar study, I published some sequences of averaged high school yearbook portraits I’d made using Abrosoft FaceMixer—a piece of commercial software I’d found that can warp source images before averaging so that eyes, noses, mouths, and so forth line up with each other. This software will try to auto-detect faces in input images, as well as landmarks within the faces, but it’s still necessary to do some time-consuming manual tweaking of control points to get optimal results, and even the “deluxe” version isn’t really set up to accommodate large-scale projects, in terms of both its interface design and its tendency to crash if you try to load too many images into it. All those factors limited the scale of what I was able to do with it, forcing me to be selective and strategic. FaceMixer also crops faces rather more closely than I’d prefer.
I was particularly interested in tracing the rise of the convention of smiling for yearbook portraits—one of the same topics Ginosar et al. would later address—and women seemed consistently to smile more than men, besides which they varied more obviously in their appearance over time, with starker differences in hairstyle and such. So in my yearbook experiments I focused on female portraits, cropping them and separating them out by hand and creating averages for the senior women of Kewanee (Illinois) High School for 1989-2000, as well as averages covering 1904-1939 at five-year intervals based on a combination of three schools because I couldn’t find digitized yearbooks from any single school for all of these years. The plate below is based on 1,107 source portraits, which felt like a lot at the time.
I also put together a short video based on the same material to animate the transition to a “say cheese smile”:
This was only one of several time-based face-averaging projects I took on back in 2014, but it’s fairly representative of the type of work I was trying to do at the time. For example, I made a point of flipping source faces as needed to orient them all in the same direction before further processing so that the details of partial profiles would reinforce each other instead of canceling each other out. My results accordingly ended up looking less like passport photos or mug shots and more like tastefully-posed portraits.
The next step forward came in 2017, when I wrote a MATLAB script for averaging faces that was designed to streamline and automate the process in ways that would let me tackle projects on a much larger scale than had been feasible with FaceMixer. I didn’t use machine learning to train any new models of my own, but just relied on preexisting models for identifying faces, eyes, noses, and mouths. The sequence of automated steps I came up with runs like this: (1) identifying, cropping out, and resizing faces, with a much larger surrounding area than FaceMixer offers; (2) rotating images to align eyes horizontally, which creates a variously-angled black border; (3) estimating the direction in which the subject is facing and flipping images as needed to harmonize direction; (4) locating facial landmarks and setting control points based on them; (5) warping each source image based on the average positions of control points within an averaging group; and (6) calculating the actual pixel-by-pixel averages. I’ve used versions of this script for a few different purposes, including averaging Google image search results, but I first put it to work on portraits from the Indiana University yearbook, the Arbutus, to create still images—
—and video animations:
So when I discovered the run of conveniently digitized Valparaiso High School yearbooks on Archive.org, I already had a custom tool ready to handle the mechanics of averaging them. It’s not perfect. I wish it brought the ears into focus, and lower lips are another weak point. But I think it strikes a reasonable balance between efficiency and quality. One detail I’ve continued to tweak is how I combine mean and median averages, both of which have advantages and drawbacks. Right now I’m adding the median (contrast-maximized) to three times the mean multiplied by the median (also contrast-maximized), with the mean and median both gamma corrected (but without gamma correction used for the initial averaging). I came up with that approach through trial and error and don’t have any other line of reasoning to put forward in defense of it.
My Valparaiso High School yearbook averages differ from Salavon’s yearbook averages in that:
- He allowed facial features to blur. I’ve tried to bring them into focus.
- He averaged faces for two personally meaningful years: his own graduating class and his mother’s. I’ve averaged faces for over a hundred years.
And they differ from the work of Ginosar et al. in that:
- That team discarded all color information from their source images. I’m keeping it, so that it will carry through into our averages wherever it’s available.
- They excluded all portraits from their analysis that weren’t forward-facing, which turned out to be approximately 77.5% of them. In the process they had to omit 135 entire yearbooks and 13 schools, each of which apparently didn’t offer a single forward-facing portrait. I’m choosing instead to create averages that face in whatever the average direction is, and to exclude nothing but full profiles.
- They used a simple affine transform for alignment. I’m warping based on numerous control points, which is more computationally expensive but should yield sharper results as well as permitting differently-posed portraits to be aligned effectively with each other.
- Their dataset combines images from many different schools, so that averages vary both by year and by institutional/regional composition. My dataset is drawn from a single school.
I’ve only had to intervene manually at two points in my averaging process. First, I’ve gone through the yearbooks to identify the pages containing individual portraits for the senior class (and other classes). Second, after faces have been automatically detected and cropped, I’ve sorted them by gender by selecting them and dragging them by hand into “male” and “female” folders. This can sometimes be a difficult call, particularly for parts of the 1970s and 1980s. Whenever I’m unsure about a particular case, I check the student’s name; and then if I’m still unsure, I might check other portraits of the same student in adjacent yearbooks or search online for references or other pictures. I’m sure I’ve sometimes classified a picture wrongly. But when we’re dealing with a hundred pictures or more, one or two or three won’t have much effect on the result. Besides, I don’t mean to claim we should be able to make a straightforward binary gender distinction for every portrait anyway. If I have any ethical problem with my project, it’s that dividing portraits into “male” and “female” ignores any nuances of gender fluidity. But the only other option that suggests itself would be to average all portraits together irrespective of gender. Here’s what that would give us for the senior class of 2013:
When it comes to the first decade of Valparaiso High School yearbooks, the missing yearbook for 1912 is an obvious obstacle, but it’s not the only obstacle. There’s also the fact that earlier years contain comparatively few portraits, due (I guess) to smaller class sizes. Even later years sometimes have so few portraits available that my averages have ended up a bit wonky, but the issue is particularly acute early on. The problem with small datasets is that single outliers or mistakes in processing can have an outsized impact on the result. My “default” results for 1906-1911 are—shall we say—rather hit or miss.
And then when it comes to 1905—the very first year for which portraits are available—there are so few of them, and those few so variously posed besides, that my automated process wholly failed to pull anything presentable from them. But I hated to leave them unexploited, so here’s a pair of averages for that year I created with a much more labor-intensive cumulative morphing technique.
By “cumulative morphing,” I mean that I take a piece of image-morphing software (specifically FotoMorph), create animated morph sequences for pairs of images (which involves defining and adjusting dozens of corresponding control points), and then create new sequences between points partway through those sequences, until I’ve factored in each of my source images in equal proportion. Below are some of the animated sequences I created in the process of making the still averages for 1905 presented above.
This kind of cumulative morphing gives impressively good results, much better than anything else I’ve tried; but unfortunately it’s also extremely time-consuming and seems unfeasible for averaging thousands of images as would be needed for a project on the scale in question here.
So far we’ve only been considering the seniors, but this doesn’t have to be exclusively about them. During the Valparaiso High School yearbook’s first few decades, group portraits often appeared of the freshman, sophomore, and junior classes (and sometimes even of the senior class in addition to the seniors’ individual portraits). The 1932 yearbook was the first to contain individual portraits of the members of all four classes, but this didn’t happen again until 1947, after which point it became a consistent feature. One notable inconsistency is that the yearbooks from 1963 through 1986 don’t contain any freshman portraits because the ninth grade was reassigned to “junior high schools” in those years. The copies of the yearbooks scanned for Archive.org also have missing pages for the junior class in 1952 and the sophomore and junior classes in 1953, while one freshman portrait was torn out of the 1988 volume and two—back to back—from the 1979 volume.
I haven’t worked my way through all of the non-senior class portraits yet. However, I did finish the freshman portraits for the years 1987-2013 (i.e., after the ninth grade had been reassigned to the high school). Below is a video I’ve prepared from these in the same way as the senior-class video presented at the beginning of this post, although of course it doesn’t run for nearly as long because it doesn’t cover nearly as many years.
This video again uses a three-year sliding time window: the first frame is the average of all portraits for the first three years (1987, 1988, and 1989), and a later frame averages another three-year group shifted one year forward (1988, 1989, and 1990), and then another year forward (1989, 1990, and 1991), and so on and so on. But I also generate other averages in between, varying the proportion of available portraits from each year. In this particular case, there are three “real” transitional averages—
- 1987, 1988, 1989
- 3/4 of 1987, 1988, 1989, 1/4 of 1990
- 1/2 of 1987, 1988, 1989, 1/2 of 1990
- 1/4 of 1987, 1988, 1989, 3/4 of 1990
- 1988, 1989, 1990
—with some further interpolation between these results. The goal of this approach is to make change over time appear smooth and continuous. Fashions may not change abruptly from year to year, but the layout of yearbook pages and the styles of different photographers certainly can, such that a video cobbled together from single-year averages, even with transitions introduced between them, would appear to “bounce” or “jerk” quasi-rhythmically from year to year to year. Even videos of three-year averages “pulse” to some degree as particular years with distinctive tendencies come into and then fade out of the picture. Increasing the time window can overcome this, but only at the expense of finer chronological detail.
Year-by-year averages fare better in the form of static grids, so here are average portraits of ninth-grade men from yearbooks for 1987 through 2013:
And the ninth-grade women:
We don’t run into any significant dilemmas when it comes to averaging portraits from any one single class (seniors, juniors, etc.). The dilemmas arise when we try to combine classes. Yearbooks tend to invest more resources in senior portraits than in portraits for other classes. Witness my own photos from 1988 (as a junior) and 1989 (as a senior), shown below at accurate relative scale:
My senior portrait is a lot bigger than my junior portrait, and I’m better dressed for it, with my hair better cut and/or combed. And it wasn’t just me. There are major stylistic differences between senior and underclass portraits in general. Senior portraits were printed in color starting in 1998, but other classes didn’t follow suit until 2003. Here are averages of freshman and senior portraits for 2002:
But even before then there had been pronounced differences. Here’s a similar comparison for 1993:
The main divide seems to have been between senior portraits and all other portraits. To illustrate, here are averages of female portraits from all four classes (ranged from freshmen on the left to seniors on the right), drawn from yearbooks dating from 2010 on the top to 2013 on the bottom.
In each year, the averaged senior portraits differ conspicuously from the averaged portraits for the other three classes, which by contrast look nearly identical to one another. The seniors are posed more obliquely relative to the camera, and their heads are more tilted (as shown by the tilt of the surrounding frames after adjustment).
Next, here are averages of the equivalent male portraits, which display a similar pattern but add some differences of coloration and dress (the average senior male is shown wearing a suit and tie):
The same clustering appears when we go back before the reassignment of ninth-grade students to the high school. Here are averages of female portraits for the years 1971 through 1974, running from sophomores on the left to seniors on the right:
And here are equivalent averages of male portraits:
If we want to combine freshmen, sophomores, juniors, and seniors into a single averaging sequence, which I haven’t yet done, I see two options. One is to average together all the portraits in each yearbook. The other is to average the portraits for each class of students across multiple yearbooks—in the sense that the seniors in the 2013 yearbook were also the juniors in the 2012 yearbook, the sophomores in the 2011 yearbook, and the freshmen in the 2010 yearbook. Both approaches make sense to me, but the latter would have the advantage of factoring out any photographic styles that vary between individual yearbooks.
One final point. We’re not limited here to straightforward averages, but can also try to create caricatures and antifaces in much the same way. A caricature exaggerates differences from the norm, while an antiface inverts them to present something that differs from the norm just as much but in the opposite direction (see this blog post for details). I haven’t actually been all that impressed with the results I’ve got from applying these techniques to Valparaiso High School yearbook portraits. Still, I’ve decided to share two examples below, both of which take the 101-year average as the norm for reference: a caricature of the average senior female portrait of the 1920s and an antiface of the average senior female portrait of the 2000s. The rust-colored background in the latter represents the inversion of a blue-colored background, and both images have been minorly retouched (since the process is unforgiving towards slight inaccuracies in setting control points).
I’ll keep plugging away at this project as time permits, and maybe I’ll eventually be able to share some sequences of multiple-class averages. For now, I hope you’ve enjoyed what you’ve seen. And if you were a student at Valparaiso High School during or before 2013, your face is probably in here somewhere!