Super-ResolutionMotivationApproach1.1 Registration (angle)1.2 Registration (shift)PowerPoint Presentation2.2 Projection onto High-res gridSlide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Super-ResolutionDeepesh JainEE 392J – Digital Video ProcessingStanford UniversityWinter 2003-2004Motivation•Create High Resolution Video from a low-resolution one•Create High Resolution Image(s) from a video or collection of low-res images. Applications:–Action Packed Sports Images (Basketball dunk, Gymnastics, etc) –Astronomy–Medical Imaging•This project – Create a high-res image from bunch of low-res ones (constraints: global motion – shift & rotation)Approach•Image Registration – Motion Estimation•Projection onto High-Res grid–Nonuniform Interpolation –Frequency Domain –Iterative Back Projection (IBP)–POCS (Projection onto convex sets)RegistrationProjectionHigh Res GridLow-res Images Registration (sub-pixel grid)•Rotation Calculation•Correlate 1st LR image with all LR images at all anglesOR•Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle1.1 Registration (angle)Anglei = max index(correlation(I1(θ), Ii (θ)))Energy at angle Ii(θ)LR image 1Energy at angle I2(θ)LR image 2i = 2,3,..,N (number of LR images)1.2 Registration (shift)•Shift Calculated using Frequency Domain MethodΔs  [Δx Δy]Tu  [fx fy]Fi (uT) = ej2πuΔsF1(uT)Δs = angle( Fi (uT) / F1(uT) )2πu•Used only 6% lower u (high freq could be aliased)•Used least square to calculate Δs•Input  Down-sampled aliased images•Goal I Correct the low-freq aliased data•Goal II  Predict the lost high freq values2.1 Frequency Domainπ-πOriginal High-Resπ-πDown-sampledπ/2-π/2 πAliased (fix it)Lost (find it)Up-sampledπ-πDesired High-ResI (known pixel positions) = Known Values I_fft = fft2(I)I_fft(higher Freq) = 0I= ifft2 (I_fft)2.2 Projection onto High-res gridPapoulis-Gerchberg Algorithm (special case of POCS)•Correct the low-freq values. Assumes high-freq part to be zero.•Projection onto 2 convex sets•Known pixel values•Known Cut-off freq in the HR image•Algorithm:Papoulis – Gerchberg AlgorithmTaj Mahal – Low-res image IInitial SetupReconstructed image from known pixelsFFT(Reconstructed image)Papoulis – Gerchberg AlgorithmImage at iteration 0FFTI(high freq) =0Known PixelValuesImage after 1st iterationPapoulis – Gerchberg AlgorithmImage at iteration 1FFTI(high freq) =0Known PixelValuesImage after 10 iterationsPapoulis – Gerchberg AlgorithmTaj Mahal – Low-res image 1After 50 iterationsSR Reconstructed imageBilinear Interpolation Bicubic InterpolationResults (Synthetic Images)•Constructed 4 low-res images by shifting and down-sampling 1 high-res image.•Applied SR algorithm & compared it with bicubic interpolationResults (Real images)•Took 4 snaps using a high-res digital camera•Cropped the same part of each image •Applied SR algorithm & compared it with bicubic interpolationResults (Real Images - I)Original Low-res images(Courtesy: Patrick Vandewalle)Results (Real Images - I)Bicubic InterpolationResults (Real Images - I)Super-resolutionResults (Real Images - II)Low-Res Image I Low-Res Image II•Didn’t WORK !!!•Motion was not restricted to shifts & rotation•Images had affine mapping.•Rule I – Need Correct RegistrationResults (Synthetic Image - I)Original High-ResDown-sampledResults (Synthetic Image - I)Bicubic InterpolationResults (Synthetic Image - I)Super-ResolutionResults (Synthetic Image - II)OriginalBicubic SRWhy didn’t SR work???•Low-res images were created by forcing shifts at critical velocities•Rule II  If low-res images are at critical velocities, can’t create good HR imageResults (Synthetic Image - III)OriginalBicubic SRWhy did SR work so well???•Low-res images were created by forcing shifts at non-critical velocities•Rule III  If low-res images have all the info about high-res then HR image can be perfectly constructedFuture Work•Superresolution with multiple motions between frames  create high res video•Predict the high-res frequency components using wavelet methodsPredictPredictPredictAcknowledgements•Prof John Apostolopoulos•Prof Susie Wee•Patrick Vandewalle•Q & A ???•Comments