IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 1 Coded Strobing Photography Compressive Sensing of High Speed Periodic Videos Ashok Veeraraghavan Member IEEE Dikpal Reddy Student Member IEEE and Ramesh Raskar Member IEEE Abstract We show that via temporal modulation one can observe and capture a high speed periodic video well beyond the abilities of a low frame rate camera By strobing the exposure with unique sequences within the integration time of each frame we take coded projections of dynamic events From a sequence of such frames we reconstruct a high speed video of the highfrequency periodic process Strobing is used in entertainment medical imaging and industrial inspection to generate lower beat frequencies But this is limited to scenes with a detectable single dominant frequency and requires high intensity lighting In this paper we address the problem of sub Nyquist sampling of periodic signals and show designs to capture and reconstruct such signals The key result is that for such signals the Nyquist rate constraint can be imposed on the strobe rate rather than the sensor rate The technique is based on intentional aliasing of the frequency components of the periodic signal while the reconstruction algorithm exploits recent advances in sparse representations and compressive sensing We exploit the sparsity of periodic signals in the Fourier domain to develop reconstruction algorithms that are inspired by compressive sensing Index Terms Computational imaging high speed imaging compressive sensing compressive video sensing stroboscopy I I NTRODUCTION Periodic signals are all around us Several human and animal biological processes such as heart beat breathing several cellular processes industrial automation processes and everyday objects such as hand mixer and blender all generate periodic processes Nevertheless we are mostly unaware of the inner workings of some of these high speed processes because they occur at a far greater speed than can be perceived by the human eye Here we show a simple but effective technique that can turn an off the shelf video camera into a powerful high speed video camera for observing periodic events Strobing is often used in entertainment medical imaging and industrial applications to visualize and capture high speed visual phenomena Active strobing involves illuminating the scene with a rapid sequence of flashes within a frame time The classic example is Edgerton s Rapatron to capture a golf swing 13 In modern sensors it is achieved passively by multipleexposures within a frame time 36 28 or fluttering 29 We use the term strobing to indicate both active illumination and passive sensor methods Coded Strobing Schematic Time Every frame is modulated U 80 Periodic phenomenon with times with a unique binary code by opening closing the shutter unknown period P say 16 ms Ashok Veeraraghavan and Dikpal Reddy contributed equally to this work Structured sparse recovery Coded Strobing Time Domain At each pixel the periodic signal is temporally modulated with a binary code t P 16ms Binary code of length U 80 TFrame Frame Duration 40ms Coded Strobing Frequency Domain At a pixel the M observed intensity values are linear combinations of the periodic signal s sparse Fourier coefficients f fMax 4fP 2fP fP 0 fP 1 P 2fP Measure Linear Combinations 4fP fMax Structured Sparsity Enforcing Reconstruction Algorithm Fig 1 CSC A fast periodic visual phenomenon is recorded by a normal video camera 25 fps by randomly opening and closing the shutter at high speed 2000 Hz The phenomenon is accurately reconstructed from the captured frames at the high speed shutter rate 2000 fps Instead of direct observation of beat frequencies we exploit a computational camera approach based on different sampling sequences The key idea is to measure appropriate linear combinations of the periodic signal and then decode the signal by exploiting the sparsity of the signal in Fourier domain We observe that by coding during the exposure duration of a low frame rate e g 25 fps video camera we can take appropriate projections of the signal needed to reconstruct a high frame rate e g 2000 fps video During each frame we strobe and capture a coded projection of the dynamic event and store the integrated frame After capturing several frames we computationally recover the signal independently at each pixel by exploiting the Fourier sparsity of periodic signals Our method of coded exposure for sampling periodic signals is termed coded strobing and we call our camera the coded strobing camera CSC Figure 1 illustrates the functioning of CSC A Contributions Computationally reconstruct N 10000 frames Capture M 125 frames in 5s In case of periodic phenomenon strobing is commonly used to achieve aliasing and generate lower beat frequencies While strobing performs effectively when the scene consists of a single frequency with a narrow sideband it is difficult to visualize multiple or a wider band of frequencies simultaneously Time Video camera w frame rate fs 25fps We show that sub Nyquist sampling of periodic visual signals is possible and that such signals can be captured and recovered using a coded strobing computational camera We develop a sparsity exploiting reconstruction algorithm and expose connections to Compressive Sensing We show that the primary benefit of our approach over traditional strobing is increased light throughput and IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2 ability to tackle multiple frequencies simultaneously postcapture on super resolution for reconstruction based algorithms have been explored in 1 22 B Benefits and limitations The main constraint for recording a high speed event is light throughput We overcome this constraint for periodic signals via sufficient exposure duration in each frame and extended observation window multiple frames For welllit non periodic events high speed cameras are ideal For a static snapshot a short exposure photo or single frame of the high speed camera is sufficient In both cases light throughput is limited but unavoidable Periodic signals can also be captured with high speed camera But one will need a well lit scene or must illuminate it with unrealistic bright lights For example if we use a 2000 fps camera for vocal cord analysis instead of strobing using a laryngoscope we would need a significantly brighter illumination source and this creates the risk of burn injuries to the throat A safer option would be 25