1Image-Based Rendering and Modelingl Image-based rendering (IBR): A scene is represented as a collection of imagesl 3D model-based rendering (MBR): A scene is represented by a 3D model plus texture mapsl Differencesu Many scene details need not be explicitly modeled in IBRu IBR simplifies model acquisition processu IBR processing speed independent of scene complexityu 3D models (MBR) are more space efficient than storing many images (IBR)u MBR uses conventional graphics “pipeline,” whereas IBR uses pixel reprojectionu IBR can sometimes use uncalibrated images, MBR cannotIBR Approaches for View Synthesisl Non-physically based image mappingu Image morphingl Geometrically-correct pixel reprojectionu Image transfer methods, e.g., in photogrammetryl Mosaicsu Combine two or more images into a single large image or higher resolution imagel Interpolation from dense image samplesu Direct representation of plenoptic function2Image Metamorphosis (Morphing)l Goal: Synthesize a sequence of images that smoothly and realistically transforms objects in source image A into objects in destination image Bl Method 1: 3D Volume Morphingu Create 3D model of each objectu Transform one 3D object into anotheru Render synthesized 3D objectu Hard/expensive to accurately model real 3D objectsu Expensive to accurately render surfaces such as skin, feathers, fur3Image Morphingl Method 2: Image Cross-Dissolvingu Pixel-by-pixel color interpolationu Each pixel p at time t ∈ [0, 1] is computed by combining a fraction of each pixel’s color at the same coordinates in images A and B:p = (1 - t) pA+ t pBu Easy, but looks artificial, non-physicalpApBpt1-tImage Morphingl Method 3: Mesh-based image morphingu G. Wolberg, Digital Image Warping, 1990u Warp between corresponding grid points in source and destinationimagesu Interpolate between grid points, e.g., linearly using three closest grid pointsu Fast, but hard to control so as to avoid unwanted distortions4Image Warpingl Goal: Rearrange pixels in an image. I.e., map pixels in source image A to new coordinates in destination image Bl Applicationsu Geometric Correction (e.g., due to lens pincushion or barrel distortion)u Texture mappingu View synthesisu Mosaicsl Aka geometric transformation, geometric correction, image distortionl Some simple mappings: 2D translation, rotation, scale, affine, projectiveImage Warpingimage plane in front image plane belowblack areawhere no pixelmaps to5Homographiesl Perspective projection of a planeu Lots of names for this:u homography, texture-map, colineation, planar projective mapu Modeled as a 2D warp using homogeneous coordinatessx' * * * xsy' * * * ys * * * 1          =               H pp′′′′To apply a homography H• Compute p′′′′ = Hp (regular matrix multiply)• Convert p′′′′ from homogeneous to image coordinates– divide by s (third) coordinateExamples of 2D TransformationsOriginal RigidAffineProjective6Mapping TechniquesDestinationImage Bxyl Define transformation as eitheru Forward: x = X(u, v), y = Y(u, v)u Backward: u = U(x, y), v = V(x, y)SourceImage AuvMapping Techniquesl Forward, point-basedu Apply forward mapping X, Y at point (u,v) to obtain real-valued point (x,y)u Assign (u,v)’s gray level to pixel closest to (x,y)u Problem: “measles,” i.e., “holes” (pixel in destination image that is not assigned a gray level) and “folds” (pixel in destination image is assigned multiple gray levels)u Example: Rotation, since preserving length cannot preserve number of pixelsAB7Mapping Techniquesl Forward, square-pixel basedu Consider pixel at (u,v) as a unit square in source image. Map square to a quadrilateral in destination imageu Assign (u,v)’s gray level to pixels that the quadrilateral overlapsu Integrate source pixels’ contributions to each output pixel. Destination pixel’s gray level is weighted sum of intersecting source pixels’ gray levels, where weight proportional to coverage of destination pixelu Avoids holes, but not folds, and requires intersection testMapping Techniquesl Backward, point-basedu For each destination pixel at coordinates (x,y), apply backward mapping, U, V, to determine real-valued source coordinates (u,v)u Interpolate gray level at (u,v) from neighboring pixels, and copy gray level to (x,y) u Interpolation may cause artifacts such as aliasing, blockiness, and false contoursu Avoids holes and folds problemsu Method of choice8Backward Mappingl For x = xmin to xmaxfor y = ymin to ymaxu = U(x, y)v = V(x, y)B[x, y] = A[u, v]l But (u, v) may not be at a pixel in Al (u, v) may be out of A’s domainl If U and/or V are discontinuous, A may not be connected!l Digital transformations in general don’t commutePixel Interpolationl Nearest-neighbor (0-order) interpolationu g(x, y) = gray level at nearest pixel (i.e., round (x, y) to nearest integers)u May introduce artifacts if image contains fine detaill Bilinear (1st-order) interpolationu Given the 4 nearest neighbors, g(0, 0), g(0, 1), g(1, 0), g(1, 1), of a desired point g(x, y), compute gray level at g(x, y):u Interpolate linearly between g(0,0) and g(1,0) to obtain g(x,0)u Interpolate linearly between g(0,1) and g(1,1) to obtain g(x,1)u Interpolate linearly between g(x,0) and g(x,1) to obtain g(x,y)u Combining all three interpolation steps into one we get:u g(x,y) = (1-x)(1-y) g(0,0) + (1-x)y g(0,1) + x(1-y) g(1,0) + xy g(1,1)l Bicubic spline interpolation,1,0≤≤yx9Bilinear Interpolationl A simple method for resampling imagesExample of Backward Mappingl Goal: Define a transformation that performs a scale change,which expands size of image by 2, i.e., U(x) = x/2l A = 0 … 0 2 2 2 0 … 0l 0-order interpolation, I.e., u = x/2B = 0 … 0 2 2 2 2 2 2 0 … 0l Bilinear interpolation, I.e., u = x/2 and average 2 nearest pixels if u is not at a pixelB = 0 … 0 1 2 2 2 2 2 1 0 … 010Image Morphingl Method 4: Feature-based image morphingu T. Beier and S. Neely, Proc. SIGGRAPH ‘92u Distort color and shape⇒ image warping + cross-dissolvingu Warping transformation partially defined by user interactively specifying corresponding pairs of line segment features in the source and destination images; only a sparse set is required (but carefully chosen)u Compute dense pixel correspondences, defining continuous mapping function, based on weighted