CSE486, Penn StateRobert CollinsLecture 08: Introduction to StereoReading: T&V Section 7.1CSE486, Penn StateRobert CollinsStereo VisionInferring depth from images taken at the same time by two or more cameras.CSE486, Penn StateRobert CollinsBasic Perspective ProjectionXXYYZZffOOp = p = (x,y,f)xxyyZYfyZXfxO.Camps, PSUZP =P =(X,Y,Z)yScene PointImage PointPerspective Projection EqnsYXxCSE486, Penn StateRobert CollinsBasic Perspective ProjectionXXYYZZffOOp = p = (x,y,f)xxyyZYfyZXfxO.Camps, PSUXZP =P =(X,Y,Z)xyScene PointImage PointPerspective Projection EqnsYXxZfCSE486, Penn StateRobert CollinsBasic Perspective ProjectionXXYYZZffOOp = p = (x,y,f)xxyyZYfyZXfxO.Camps, PSUXZP =P =(X,Y,Z)xyScene PointImage PointPerspective Projection EqnsYXxZfZYyfCSE486, Penn StateRobert CollinsWhy Stereo Vision?XXYYZZffOOp = p = (x,y,f)xxyyZYfyZXfxO.Camps, PSUkZkYfkZkXfFundamental Ambiguity:Fundamental Ambiguity:Any point on the ray OP has image p Any point on the ray OP has image p XYZP =P =(X,Y,Z)xyCSE486, Penn StateRobert CollinsWhy Stereo Vision?OOLLPPppOORRA second camera can resolve the ambiguity,A second camera can resolve the ambiguity,enabling measurement of depth via triangulation. enabling measurement of depth via triangulation.CSE486, Penn StateRobert CollinsWhy Stereo Vision?~63mmYour two eyes form a stereo systemThe right and left eyes see the worldfrom slightly shifted vantage points.CSE486, Penn StateRobert CollinsKey Concepts for Today• Parallax• Anaglyphs• Random Dot Stereograms• Mathematics of Simple StereoCSE486, Penn StateRobert CollinsDo-it-Yourself Parallax DemoShow: •Points at different depths displace differently•Nearby points displace more than far onesCSE486, Penn StateRobert CollinsA Hitchhiker’s Guide to Parallax Parallax = apparent motion of scene features located at different distancesNearby guardrailMore distant treeVery distant mountain peakLarge displacementSmaller displacementVery small displacementCSE486, Penn StateRobert CollinsGeneral Idea of StereoInfer distance to scene points by measuring parallax.Large displacementSmaller displacementVery small displacementCloseFarMidrangeINFERCSE486, Penn StateRobert CollinsAnaglyphsPut red filter over left eyeAnaglyphs are a way of encoding parallax in a single picture. Two slightly different perspectives of the same subject are superimposed on each other in contrasting colors, producing a three-dimensional effect when viewed through two correspondingly colored filtersCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsHow Anaglyphs WorkClose right eye, then close left. What do you observe?Red filter selectively passes red color, andsimilarly for cyan filter and cyan color.CSE486, Penn StateRobert CollinsMaking an AnaglyphTake a greyscale stereo pair. Copy the left image to the red channel of a new image(the anaglyph image)Copy the right image to the green and blue channelsof the anaglyph image (note: green+blue = cyan)Now when you view with red-cyan glasses, the lefteye sees only the left image, and the right eye seesonly the right image. The brain fuses to form 3D.CSE486, Penn StateRobert CollinsStereo PyschophysicsHow does stereo depth perception work?In particular, at what “level” in the visual systemdoes it occur at?An early debate: do we infer depth from higher-levelinformation like perspective and contours, or doesit occur at a much lower level?"The basis of this three-dimensional perception was hotly debated between Wheatstone and fellow physicist Sir David Brewster. (Though it may seem odd for physicists to concern themselves with the physiology of optics, this was felt to be a natural extension of the study of the physics of optics.) Brewster opined that perspective was the source of the apprehension of an object's shape. Wheatstone insisted that the images in the each eye had identifiable landmarks that were combined to assign depth to the landmarks.”--Ralph M. Siegel Choices: The Science of Bela JuleszCSE486, Penn StateRobert CollinsHigher-level Depth CuesPerspective (vanishing points)CSE486, Penn StateRobert CollinsHigher-level Depth CuesSimilar sized objectsappear smaller at a distance (this is alsorelated to perspective)CSE486, Penn StateRobert CollinsHigher-level Depth CuesOccluded contours(perceptual completion)CSE486, Penn StateRobert CollinsStereo PyschophysicsObviously perspective and contours are important,(particularly for monocular depth perception), but are they necessary for binocular stereo depth perception?Bela Julesz answered this question in 1960 with hisexperiments with random dot stereograms.“In 1960, Bela's experiment with what eventually became known as Juleszrandom dot stereograms unambiguously demonstrated that stereoscopic depth could be computed in the absence of any identifiable objects, in the absence of any perspective, in the absence of any cues available to either eye alone.” --Ralph M. Siegel Choices: The Science of Bela JuleszCSE486, Penn StateRobert CollinsJulesz Random-Dot ExperimentGenerate a random dot pattern using a computere.g. im = roicolor(rand(300,300), 0.5, 1);By definition, this is just “noise”, so there areobviously no monocular depth cues here.CSE486, Penn StateRobert CollinsJulesz Random-Dot ExperimentClip out a square region and shift it to the leftCSE486, Penn StateRobert CollinsJulesz Random-Dot ExperimentClip out a square region and shift it to the leftFill in the “hole” left behind with more random dots.CSE486, Penn StateRobert CollinsJulesz Random-Dot ExperimentNow view as a stereo pair.Julesz used a special viewer, but we willdisplay as an anaglyph (get your glasses!)Original dot image Dot image with shifted squareCSE486, Penn StateRobert CollinsCSE486, Penn StateRobert CollinsMake Your Own%make an image with random dotsim = roicolor(rand(300,300),.5,1);%second image starts as a copy of thatim2 = im;%shift a square of pixels to the rightim2(100:200,110:210) = im(100:200,100:200);%fill in the "hole" with more random dotsim2(100:200,100:110) = roicolor(rand(101,11),.5,1);%encode image2 in red channel of a color imageana = 255*im2;%encode image1 in blue and green channelsana(:,:,2) = 255*im;ana(:,:,3) = 255*im;%take a look (remember to wear your red/cyan glasses!)image(uint8(ana))Try this: what