Time-Based Graphs as Moving Pictures (1786-1878)

The wavy lines of time-based graphs can easily be converted into audio, as I explained in an earlier post.  But we can also convert them into video, and in fact sound recordings were being played back as moving pictures in the 1860s, long before anyone had played one back as sound—another notable difference being that the “reproduction” was in super-slow motion.  Read on to learn more about those historical efforts by Helmholtz and other luminaries.  In the Griffonage-Dot-Com tradition, I’ll also share the results of some similar experiments of my own aimed at bringing even older graphs to life and displaying early sound recordings visually at full speed—instead of in slow motion—using the strategy of the oscilloscope.  I’m tempted to add a tongue-in-cheek warning here about “graphic” content, but I’ll resist the urge.


I.

Below is one of two copies of the first edition of Hermann Helmholtz’s Die Lehre von den Tonempfindungen (1863) at the William and Gayle Cook Music Library at Indiana University Bloomington, opened to show pages 34 and 35—a section that discusses the Scott-Koenig phonautograph, how it recorded the vibrations of sounding bodies automatically over time, and how to interpret the resulting graphs.

helmholtz-1863-two-pagesHere’s the part of the text I’d like to focus on here:

helmholtz-1863-textOr in English translation:

Such a drawing thus shows immediately at which point of its path the vibrating body found itself in any desired moment of time, and hence gives a complete picture of its movement.  If the reader wants to reproduce the movement of the vibrating point, let him cut a narrow vertical slit in a piece of paper, lay the paper over Fig. 6 or 7 so that he sees a small part of the curve through the vertical slit, and now slowly draw the book along under the paper, and the white or black point in the slit will go back and forth just as the fork originally did, only more slowly.

Figure 7 (on page 34) is an artificially constructed graph of a sine curve, meant to show the movements a tuning fork would hypothetically make.

helmholtz-fig-7On the other hand, Figure 6 (on page 33) is a facsimile of a trace made on a phonautograph by the real vibrations of a real tuning fork—a recording, in other words, that Helmholtz was suggesting his readers use to “reproduce” the recorded motion:

helmholtz-fig-6Here’s an animated GIF I’ve prepared to illustrate Helmholtz’s technique, using a piece of Figure 6:

helmholtz-fig-6-slitI think it’s reasonable to assume that Helmholtz had actually tried the technique he was describing, that it had worked, that anyone who had read his book starting in 1863 could easily have done likewise, and that at least some readers followed his suggestion.  Maybe there’s even a copy of Die Lehre von den Tonempfindungen out there with a strip of paper containing a slit lodged between pages 34 and 35, easily mistakable as a bookmark.  (Please let me know if you find one!)

In its original form, Helmholtz’s slit technique required a compromise between precision and visibility.  The narrower the slit, the more precisely it could display the position of the trace at each point in time, but the harder it would become to see.  As narrow as the virtual slit is in the animation I’ve provided above, it’s still not as narrow as I could have made it—it’s two pixels wide, and since it represents the source image at 300 dpi, that corresponds to a 1/150 inch slit cut in a sheet of paper.  If I’d made my virtual slit a single pixel wide (corresponding to 1/300 inch), I could have doubled the time precision of the display, but only at the expense of making it nearly invisible.  Imagine Helmholtz drawing his graph along beneath a razor-thin slit and you’ll appreciate the problem.

But in the magical world of the digital humanities, we’re no longer constrained by the width of a physical slit.  We can create an animation that’s a single pixel wide, giving us the maximum time resolution allowed by any given digital scan, and then expand it horizontally to any width we want—say, whatever it takes to achieve a standard 4:3 aspect ratio.  Here’s an animation of Helmholtz’s complete Figure 6 based on a single-pixel virtual slit:

helmholtz-fig-6-4x3-animationThere’s a lot of visual noise herethe result of white specks in the source printbut the eye can easily pick out the good signal: the thicker line moving gradually up and down near the middle of the frame, corresponding to the changing position of the tuning fork over time. Granted, the playback speed I’ve chosen is arbitrary and doesn’t match the original recording speed, which would have been much faster, although we don’t know exactly how fast, since Helmholtz doesn’t tell us what the frequency of the tuning fork was.  But Helmholtz himself had expected the reproduced motion to be slower than the “original,” so I believe my animation remains faithful to his vision.  Its principle is exactly the same as that of his slit technique; I’m just trying to improve on his implementation by enhancing visibility through increased width.

What we have here is, as far as I’m aware, the earliest case of a record of sound vibrations being targeted for reproduction as motion over timenot as sound, as Thomas Edison did in 1877, but as visible movement slowed down to a speed at which the eye could follow it.  This is significant for the history not only of phonography, but of cinematography as well.  At the time Die Lehre von den Tonempfindungen was published in 1863, the idea of rapidly capturing sequences of photographic images and then displaying them in order to reproduce visible movement had been suggested (by Sir John F. W. Herschel in 1860 and Henry-Désiré du Mont in 1861), but there’s no evidence that anyone had yet succeeded in accomplishing this, or had even really tried.  I’ll concede that the tuning-fork record Helmholtz put forward for reproduction doesn’t quite fit the expected profile for a motion picture: it was two-dimensional rather than three-dimensional, and it had been created mechanically, by a stylus scratching in soot, rather than photographically.  But to whatever extent the essence of cinema lies in recording the motions of objects over time and then displaying the data over time to reproduce the motion for visual apprehension, Helmholtz could have said: “been there, done that.”

And he wasn’t alone.  Franz Josef Pisko recommended the same technique in Die neueren Apparate der Akustik (1865), page 92, which I translate from the original German as follows:

If one of the phonograms printed above is covered with a paper that has a very narrow notch (or aperture) in such manner that only a very small piece of the curve is to be seen through the aperture, and the phonogram or the covering paper is slowly pulled to the right or left—then the points of the wavy line appearing through the slit imitate the motion of the body that originally sounded, from which the inscription of sound actually stems; only now the motion is slower and can therefore be grasped better with the eye. The friendly reader will hopefully make this easy and, indeed, pleasant experiment.

Alfred Mayer described a more technically elaborate version of it in his article “Researches in Acoustics, No. 5,” in the American Journal of Science 8:45 (Sept. 1874), 170-182, at page 180:

On a piece of Bristol board I drew a circle, and in one quadrant of this circle I drew 500 equidistant radii.  On these radii, as ordinates, I transferred the corresponding values of the same ordinates of the resultant of fig. 3 [a synthetic graph of a sound wave], diminished to one-fourth of their lengths.  I thus deflected the axis of curve fig. 3 into one-fourth of a circle curve; and this repeated for times on the Bristol board, rendered the curve continuous and four times recurring, as shown in fig. 7.  I now cut this curved figure out of the board and used it as a template.  I placed the latter centered on a glass disc of 20 inches in diameter.  The disc was covered on one side with opaque, black varnish, and with the template and the separated points of a pair of spring-dividers, I removed from the glass disc a sinuous band, as shown in fig. 7.  The glass disc was now mounted on a horizontal axis and placed in front of a lantern the diameter of whose condensing lens was somewhat greater than the amplitude of the curve.  The image of that portion of the curve which was in front of the condenser was now projected on a screen, and then a piece of card board having a narrow slit cut in it was placed close to the disc, with the slit in the direction of one of its radii.  On now revolving the disc I reproduced on the screen the vibratory motion of a molecule of air or of a point on the tympanic membrane, when these are acted on by the joint impulses of the first six harmonic or pendulum vibrations, forming a musical note.  On slowly rotating the disc one can readily follow the compound vibratory motion of the spot of light; but on a rapid revolution of the disc, persistence of visual impressions causes the spot to appear lengthened into a band; but this band is not equally illuminatedit has six distinct bright spots in it, beautifully showing the six inflections in the curve.

mayer-loop-1874By sticking a pin in the center of fig. 7, as an axis about which revolves a piece of paper with a fine slit, the reader can gain some idea of the complex motion which I have described.

In other words, Mayer had looped a sound curve to deliver continuous action, and he had also projected it onto a screen.  Looping was, of course, a key strategy in early moving image composition, and projection draws us solidly into the realm of pre-cinematic “screen practice”indeed, we’re dealing here with a species of movable magic lantern slide.  It’s true that Mayer used his method to set in motion a synthetically-drawn curve rather than one that had been recorded automatically from actual vibrations, but he called the effect “reproduction” anyway, and the same method would obviously have lent itself to reproducing recorded vibrations too.  You may also recognize Mayer’s looped trace as the image printed on the CD in my book Pictures of Sound.

Feel like trying out a little experiment?  If so, print out the above graphic of Mayer’s looped curve (or just take the Pictures of Sound CD if you have a copy), center it on a turntable, and revolve it under a slit held perpendicular to the direction of revolution to “reproduce the vibratory motion,” just as Mayer suggested.  If you don’t have a turntable, you can try moving the image by hand instead.

Of course, it’s possible that the technique Helmholtz described in 1863 dates back even earlier.  There had definitely been previous efforts to enhance the experience of viewing sound recordings in potentially relevant ways.  In the early 1840s, Guillaume Wertheim had studied the vibrations of rods and wires made of different metals by causing them to record themselves phonographically on revolving discs, alongside traces of a 256 Hz tuning fork, and in one of his publications (“De l’élasticité et de la cohésion des métaux,” Annales de Chimie et de Physique, 3rd Series, 12 (Nov. 1844), 385-454 and Plate I) he had illustrated a special viewing device he’d used for inspecting the results through a microscope and counting the vibrations:

wertheim-waveform-viewerThe trace left by the tuning fork looked like this (which is, incidentally, the oldest known illustration of automatically recorded sound vibrations):

wertheim-trace“By causing this tablet to turn slowly, the vibrations are counted with the greatest safety [sûreté],” Wertheim wrote.  But did he ever look through the microscope while revolving the disc, advancing his view forward along the time axis?  It’s hard to imagine how he could have avoided losing his place while counting vibrations if he hadn’t.  In some small way, then, did Wertheim’s device for viewing records of motion over time anticipate the peephole kinetoscope?


II.

The practice of displaying time-based graphs as moving images is interesting in part because it’s something people actually did in the 1860s and 1870s, and perhaps even earlier.  But I believe it’s also worthwhile to make other old graphs “come to life” in this way as a cool feat of animation in its own right, regardless of whether anyone had the idea of doing so back when the graphs were new.  And with that observation, I’d like to take us out a bit further on a limb.

Carl Ludwig was the first person ever to record internal physiological processes by automatic meansSpecifically, he captured the pulse and respiration of horses and dogs as wavy lines on sheets of paper wrapped around rotating cylinders starting in late 1846.  Below is Figure 7 from his first publication on the subject: “Beiträge zur Kenntniss des Einflusses der Respirationsbewegungen auf den Blutlauf im Aortensysteme,” Archiv für Anatomie, Physiologie und wissenschaftlichen Medicin, Jahrgang 1847 (Berlin: Veit et Comp., [1847]), 242-302, Tables X-XIV.

ludwig-fig7

The top line represents a horse’s arterial pressure, while the bottom line represents pleural pressure (increasing with exhalation and decreasing with inhalation).  And here’s an animation I made based on the same Figure 7, based on a virtual slit one pixel wide, but expanded to fill up a traditional 4:3 aspect ratio.

ludgiw-fig-7-animation1I’ve set the playback speed to match a heart rate of just under thirty beats per minute.  That isn’t exact, but Ludwig writes (p. 245) that the heart rate of the horse with the Vorschlagpuls—which I take to mean the kind of dicrotic pulse seen here—never rose above thirty, so the timing should at least be close to correct.  Each frame in my animation corresponds to a single column of pixels in a digital copy of the source image, rescaled to yield a frame rate of about 25 frames per second.  The top line depicts the horse’s pulse, while the bottom line depicts its breathing—lower during inhalation, higher during exhalation—as it stood there nearly 170 years ago with manometric sensors inserted into its artery and pleura for the purposes of the experiment.

For some time now, Dario Robleto and I have been converting early records of the human pulse into playable audio using a modified version of paleospectrophony.  As Ron Cowen wrote in his account of our work, “The Echoes of Hearts Long Silenced,” in the New York Times of December 15, 2014: “For the height of the line, which represents an increase or decrease in the amount of blood pressure, [Dr. Feaster] assigned higher and lower sound frequencies—giving the greatest volume to places in the waveform where the pressure changed abruptly, and lower volume to places where the pressure remained relatively constant. Rapid changes in pressure trace the opening and closing of heart valves.”  The resulting audio resembles the sounds we typically associate with a heartbeat.  But graphs of the pulse as such don’t directly represent sound, so I’d suggest that displaying them as visual animations is an equally legitimate strategy for bringing them to life.  Indeed, as I wrote in an earlier post (“What is Eduction?”), “we could even contrive some means of educing the rhythm as a tactile stimulus, yielding a perception equivalent to feeling a pulse with the fingertips, which might arguably be the most appropriate modality of all.  I’m picturing a contraption where pressure is variably exerted on the liquid in a tube connected to an ‘artery’ in an artificial arm that could be palpated like a real artery.”  By the same token, Ludwig’s respiration curves could be used to move a bellows or control a stream of air in some other way to simulate breath.  There are lots of possibilities here for someone adept at designing computer-controlled gadgetry.  I’m not, so I’ll stick with ordinary video for the moment.

Carl Ludwig’s kymograph records are the oldest automatically-recorded time-based graphs I’ve yet seen, but there are older time-based graphs to be found out there.  Here’s another celebrated early example: a graph of “Exports and Imports to and from DENMARK & NORWAY from 1700 to 1780,” inscribed at the bottom “Published as the Act directs, 1st May 1786, by Wm. Playfair.”

playfair-graphAnd here’s that same graph rendered as a time-lapse animation that displays Playfair’s data at a rate of about 1.6 seconds per decade.

playfair-animationThe vertical grid lines and text (“BALANCE AGAINST,” “BALANCE in FAVOUR of ENGLAND”) produce minor visual disturbances—akin to the effects of scratches and dirt on a film during projection—but the movements of the red and yellow lines and of the shaded area between them are still easy for the eye to follow.

As far as I’m aware, neither Carl Ludwig nor William Playfair ever intended their graphs to be animated as I’ve done here.  But if we limited ourselves to using historical sources strictly in the ways their creators had originally intended them to be used, there wouldn’t be much creativity to speak of in the historian’s craft.  These two animations give us another prime case of what I like to call eduction against the grainthe delightfully subversive practice of making inscriptions sensorily accessible in meaningful ways their creators never anticipated.  This is just as legitimate an approach as, say, reading a Victorian novel through the lens of twenty-first-century understandings of class, race, and gender.  The inscriptions of the past are fair game for reinterpretation and repurposing, both intellectually and materially.  The creator’s “intention” is just one potentially illuminating data point among many.


III.

I animated Ludwig’s kymograph record of the horse’s pulse and breathing at its approximate original recording speed, and the Playfair graph at a much faster speed than the eighty-year span its x-axis represents.  But Helmholtz had suggested his slit reproduction technique specifically as a means of animating phonographic traces—records of sound vibrations—and his assumption was that the reproduction in that case would need to be much slower than the original motion: “the white or black point in the slit will go back and forth just as the fork originally did, only more slowly.”  Unlike the narrowness of the slit, that seems to be a limitation we can’t transcend with modern technology.  A typical frame rate for conventional motion pictures is 24-30 frames per second, and a typical frame rate for an LCD monitor is 60 frames per second.  Using such technologies, there’s really no way for us to display a graph of the vibrations of (say) a 250 Hz tuning fork using the approach described and illustrated above.  And even if we could somehow manage to make our display rapid enough—for instance, by rotating the curve in analog form past a narrow slit—the eye wouldn’t be able to follow the motion.  As Alfred Mayer observed in 1874, all we’d see would be a blur.  The periodic movements that correspond to audible tones are simply too fast for us to discern visually.

What we can do in such instances, however, is mimic the display of an oscilloscope.  An oscilloscope is designed to trace a particular number of cycles, quickly retrace to the beginning, and repeat, yielding a wavy line that regularly gets refreshed.  I remember talking into one of these gadgets during a grade-school visit to a science museum in the late 1970s or early 1980s, but the technology was already old by then—witness this cartoon from a Westinghouse advertisement published in connection with the 1939 New York World’s Fair:

oscilloscope-cartoon-1939To demonstrate how an oscilloscopic approach can fit nineteenth-century media, let’s take the following record of the spoken vowel “ay,” captured photographically on a glass plate by Eli Whitney Blake Jr. and published in the American Journal of Science and Arts in 1878:

blake-ayHere’s an animation that displays the same trace as an oscilloscope would, three cycles at a time—I didn’t try to match the original speed of recording exactly, but what you see is at least comparably rapid.

blake-ayCool, huh?  That was a relatively simple and straightforward case.  Blake moved his photographic plate at a constant rate of speed during recording, and the vowel was uttered in a monotone, so by simply chopping the source image into segments of more or less the same width and arranging them into an animation, I was able to get a fairly stable oscilloscopic display.  And stability is the main goal—as well as the main challenge—when it come to oscilloscopes.

What else can we view using this same strategy?  Well, let’s see….

The phonautograms of Édouard-Léon Scott de Martinville are the world’s oldest automatically captured records of airborne sound.  My colleagues in the First Sounds initiative and I have been playing these back as audio for several years now, but Scott himself understood them strictly as visual representations of sound.  Like the boy in the Westinghouse cartoon, he wanted to see what a voice looks like.  With that in mind, I’ve thought it would be worthwhile to try to animate his phonautograms visibly as well as audibly.  Among other things, this would provide us with attractive synchronized visuals for accompanying the audio in film or on television.

We face one obstacle here that we didn’t face with Blake’s vowel record, though.   The drum of Scott’s phonautograph was rotated by hand at an irregular speed, so a given width on a phonautogram doesn’t correspond consistently to a given duration.  Just chopping the phonautogram into segments of equal width won’t create a stable time base.  However, Scott’s most technically proficient phonautograms, made during 1860, contain two simultaneously-made traces: one representing to the voice, and the other drawn by a 250 Hz tuning fork as a time reference.  We’ve already been using the tuning-fork trace as a pilot tone for correcting speed fluctuations in the audio we’ve extracted from phonautograms, but we can also use it as the basis for a stable oscilloscopic display.  Here, for instance, is a two-hundred-frame animation of the first 2000 tuning-fork cycles of Scott phonautogram #38, leaf [6] in the Regnault Papers, Ms. 2935, in which each frame displays ten tuning-fork cycles for 0.04 seconds (the exact length of time it represents).

regnault-6-animation-200x150-original-width-to-frame-200In this animation, the frame gets narrower where the recording speed was lower and wider where the recording speed was higher, reflecting the dimensions of the original phonautogram.  However, we also have the option of harmonizing the widths of the frames to yield a consistent 4:3 aspect ratio, like this:

regnault-6-animation-200x150-width-corrected-to-frame-200Only the upper tracecorresponding to the tuning forkis actually held stable in my two sample animations, with each frame representing ten cycles.  The voice documented in the other trace varied a lot in pitchit’s a record of dramatic speechso the lower waveform appears to drift from left to right, or right to left, depending on how its fundamental frequency relates to the frequency of the tuning fork at each moment.  But that’s to be expected.  Indeed, I believe this is exactly the effect we want.

On the other hand, the overall stability of the picture could easily be improved upon.  Rather than starting over from scratch with the full-page scan of the phonautogram, I made these animations from the cropped line-by-line files I’d prepared a few years ago when I was working to play the same images back as audio.  I didn’t try to center the waveforms vertically the same way in each frame or to ensure that the same amount of “background” was preserved around them at the top and bottomI just ran with whatever I had.  Still, I think the result provides a compelling proof of concept.  Using this technique, it’s clear we could turn each of Scott’s phonautograms into a short sound film, giving us something to hear and something to watch at the same time.


I’d like to wrap things up with two take-away points:

(1) Time-based graphs such as the ones featured above can be treated as two-dimensional motion pictures today and have been used to “reproduce” automatically recorded movements since at least the 1860s.  Any statements about the world’s first motion pictures or moving images should be framed carefully with these cases in mind.

(2) Setting records of sound in motion to “reproduce” the movements they document isn’t necessarily the same thing as playing them back as sound.  On this front, too, any statements about the first use of a sound recording for “reproduction” need to be formulated carefully.

Advertisements

2 thoughts on “Time-Based Graphs as Moving Pictures (1786-1878)

  1. Pingback: My Fiftieth Griffonage-Dot-Com Blog Post | Griffonage-Dot-Com

  2. Pingback: See the Phonautograph Turn: Motion Capture with Sound (1860) | Griffonage-Dot-Com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s