Unformatted text preview:

6.801/866Vision for GraphicsImage-based renderingTaxonomyTaxonomyTaxonomyTaxonomyModels from stereoCMU’s 3-D RoomMulti-view stereo for VR“Virtualized Reality”ModelsVisual HullsSmoothed Visual HullSmoothed Visual HullSmoothed Visual Hull ResultSmoothed Visual Hull ResultSmoothed Visual Hull ResultImage-based Visual HullsIBVH VideoModel-based SFMFaçadePhotogrammetry (Model-based SFM)View-dependent textureModel-based stereo for surface detailFaçadeFaçadeFaçade MovieCalibration/model free IBR?TaxonomyAffine image transferAugmented realityModel recoveryTaxonomyThe Plenoptic functionThe Plenoptic functionLightfieldsLightfieldsAlternate representationsLight-field renderingExample lightfieldsExample lightfieldsExample lightfieldsUnstructured LumigraphUnstructured LumigraphUnstructured Lumigraph RenderingTaxonomyEndgameRecapThanks!6.801/866Image-Based RenderingT. DarrellVision for GraphicsSo far: stereo, motion, tracking, model-based recognition, most focusing on recovering 3-D models with accurate shape…One of the main applications of vision is making new pictures!• Do we need detailed models?• Do we need Euclidean 3-D shape?• Are dense range images useful?Image-based renderingSynthesize new views from a set of pictures.Taxonomy1. Build a 3-D model; re-render from new viewpoint– Multi-view stereo; “Virtualized Reality”– Visual Hulls– Model-based stereo2. Establish correspondences; use view transfer– Affine view synthesis3. Model sets of light rays– Lightfields, Lumigraphs, …Taxonomy1. Build a 3-D model; re-render from new viewpoint– Multi-view stereo; “Virtualized Reality”– Visual Hulls– Model-based stereo2. Establish correspondences; use view transfer– Affine view synthesis3. Model sets of light rays– Lightfields, Lumigraphs, …Taxonomy1. Build a 3-D model; re-render from new viewpoint– Multi-view stereo; “Virtualized Reality”– Visual Hulls– Model-based stereo2. Establish correspondences; use view transfer– Affine view synthesis3. Model sets of light rays– Lightfields, Lumigraphs, …Taxonomy1. Build a 3-D model; re-render from new viewpoint– Multi-view stereo; “Virtualized Reality”– Visual Hulls– Model-based stereo2. Establish correspondences; use view transfer– Affine view synthesis3. Model sets of light rays– Lightfields, Lumigraphs, …Models from stereoCMU’s 3-D Room49 camera 3-D room:[Kanade et al. 1998]Multi-view stereo for VRCompute dense range image from 3-6 nearby cameras:Merge into global mesh.Texture and render new views….[Kanade et al. 1998]“Virtualized Reality”[Kanade et al. 1998]Models• Virtualized reality– very accurate– many correspondences– many cameras• What can you do with a few cameras, and just silhouettes?Visual HullsVisual Hull [Laurentini, 91]: the minimal object that produces the given silhouettes- 3-D model contains the true object- visual cone intersection- texture mapped for a desired viewpoint[Matusik]Smoothed Visual HullFit surface spline to mesh; relax model according to smoothness assumption. [Sullivan and Ponce]Smoothed Visual HullSmoothed Visual Hull ResultSmoothed Visual Hull ResultSmoothed Visual Hull ResultImage-based Visual HullsVisual Hull can be computed in from images with pixels, without computing any explicit 3-D geometery (Matusik et al, 2001)Exploit view-dependent texture mapping (more later…)()2KnOKnn×IBVH VideoModel-based SFM• Assume parametric shape model– boxes–prisms– solids of revolution– unknown height, width, etc…– constraints between unknowns• Given marked features, fit model to image using (relatively simple) non-linear search.Façade Visually compelling model from just a few photographs!Three steps:• Photogrammetry (Model-based SFM)• View dependent Texture Mapping• Model-based StereopsisPhotogrammetry (Model-based SFM)Line features recovered model model overlay recovered textureView-dependent textureModel-based stereo for surface detailFaçadeFaçadeFaçade MovieCalibration/model free IBR?• Cameras are hard to calibrate…desirable to have IBR methods that work without external/scene knowledge• Recover affine structure from motion• Use to insert virtual objects that follow camera motion…Taxonomy1. Build a 3-D model; re-render from new viewpoint– Multi-view stereo; “Virtualized Reality”– Visual Hulls– Model-based stereo2. Establish correspondences; use view transfer– Affine view synthesis3. Model sets of light rays– Lightfields, Lumigraphs, …Affine image transferUse affine model …Given P0-P3, andWith appropriate choice of 4 bases we can express projected location of points as: 1. Given m>=2 images of p0-p3and p solve using least-squares for x,y,z2. Use x,y,z and positions of p0-p3in new view to find p in new view.Augmented realityFind cameras with black squares; add virtual object to scene with correct camera motion.Model recovery• View transfer good for many special effects and augmented reality applications.• For model recovery, dense correspondence is needed!• But correspondence is hard! … (and/or models are approximate)• What can we do without correspondence?• Model visible rays, not shape….Taxonomy1. Build a 3-D model; re-render from new viewpoint– Multi-view stereo; “Virtualized Reality”– Visual Hulls– Model-based stereo2. Establish correspondences; use view transfer– Affine view synthesis3. Model sets of light rays– Lightfields, Lumigraphs, …The Plenoptic function• IBR Æ recover geometric and photometric models from photographs, bypass the modeling process.• Plenoptic function: images that can be seen!• What parameterizes visible rays?– Camera position– Viewing angle– Wavelength–Time(In a non-dispersive medium…)The Plenoptic function• Adelson and Bergen’s Plenoptic function•7DÆ 5D Æ 4D Æ 2D7D:5D:4D: 2D:),,,,,,( tccczyxλϕθ),,,,(ϕθzyxccc),(ϕθ),,,(2211yxyx[Shum and He]Lightfields• Approximate Plenoptic function for fixed camera location, time, …• Reparametrize rays based on planar intersection• A “light slab”:Lightfields• Generally, 2D slices of 4D data set• For a new views compute other 2D slices• Challenges:–Capture– Parameterization– Compression–Rendering• Point / angle• Two points on a sphere• Points on two planes• Original images and camera positions…Alternate representationsLight-field rendering• Compute intersection with (u,v) and(s,t)


View Full Document

MIT 6 801 - Image-Based Rendering

Download Image-Based Rendering
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Image-Based Rendering and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Image-Based Rendering 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?