1Teleimmersion SystemDr. Gregorij KurilloContentsn Introductionn Teleimmersion @ UCBn Current problemsn Multi-stereo camera systemn Networkingn Renderingn Conclusion2Teleimmersionn Tele-Immersion is aimed to enable users in geographically distributed sites to collaborate in real time in a shared simulated environment as if they were in the same physical roomn Combines computer vision, graphics and network communications3D sceneInteractive 3D displayat remote locationInternet 23D sceneInteractive 3D displayat remote locationInternet 23D sceneInteractive 3D displayat remote locationInternet 2Bidirectional 3D data transmissionWhat is Teleimmersion?n Tele-immersion connects remote places through a shared virtual environment3site 1site 2site 3Meeting in Virtual SpaceApplicationsn Collaborative work¤ 3D CAD design¤ Ergonomics¤ Entertainment (games)¤ Remote learning and training¤ Coordination of activities (dancing, rehabilitation)n Measurement tool¤ 3D motion capture of body segments¤ Medicine & rehabilitation4Teleimmersion @ UCBn CITRIS lab in Hearst Memorial Mining Buildingn 360 degree stereo capturingn Full-body 3D reconstructionn Real-time data recordingn Real-time renderingn Collaboration through internet II with UIUCTeleimmersion @ UCB5TI-Systemn 48 cameras – 12 stereo clustersn 8 IR pattern projectorsn 4 microphones and 4 speakersn 13 PC computers (2 or 4 CPUs, Windows XP)n 2 projectors for passive stereo projection (using circular polarization)n Real time rendering3D ReconstructionImage capture (640x480)Resize (320x240)RectificationBackgroundsubtractionEdge extractionReconstruction(Triangulation)Region-basedcorrelation15~20 ms15~20 ms30 msTotal: 130~180 ms6Edge extraction & RB CorrelationImage capture & ResizeReconstruction – Depth MapBackground subtractionReal World, Real ProblemsMissing dataPoor blending of pixelsHoles & noiseCalibration7Headaches…n Cameras producing a lot of noise (i.e. outliers)n Poor reconstruction (e.g. holes, flickering) n Small range of capturingn Some cameras do not work properlyn Delays in data transferCamerasn 48 Dragonfly CCD Cameras (Point Grey)¤ 12 color and 36 black & white cameras¤ Firewire connection, IEEE 1394¤ 640x480 resolution, 15 FPS¤ Synchronization through hardware triggering8CCD Sensorn 1/3” Sony CCD with square pixelsn 60dB signal/noise ration CCD noise has three major components*:¤ photon noise¤ read noise¤ fixed pattern noisen 100 dark images recorded with delay of 20msn Stability of background noise examined on 100 images recorded with delay of 1s*G. Kamberova, R. Bajcsy, Sensor errors and the uncertainties in stereo reconstruction, in: K.W. Bowyer, P.J. Phillips, eds., Empirical Evaluation Techniques in Computer Vision, pp. 96-116. IEEE Computer Society Press, 1998.0 20 40 60 80 1005.815.825.835.845.855.865.875.885.895.9Mean intensity of a set of dark images (Clust #1, Cam #1)framesIntensity [0..255]0 100 200 300 400 5005.85.815.825.835.845.855.865.875.885.895.9RowsAvg. intensity [0..255]Average intensity of 100 dark images (Clust #1, Cam #1)9Noise Analysis (Dark Images) Camera 1 Camera 2 Camera 3 Cluster min max mean std min max mean std min max mean std 1 4 8 5.8753 0.0392 4 8 5.8830 0.0409 4 16 5.8852 0.0395 2 0 3 0.0003 0.0017 0 1 0.0000 0.0000 0 6 0.0001 0.0013 3 0 4 0.0046 0.0069 0 1 0.0001 0.0012 0 4 0.0004 0.0020 4 0 1 0.0003 0.0016 0 1 0.0000 0.0005 0 3 0.0004 0.0023 5 0 1 0.0037 0.0061 0 3 0.0033 0.0058 0 3 0.0109 0.0104 6 0 3 0.0000 0.0002 0 5 0.0000 0.0001 0 1 0.0000 0.0000 7 0 1 0.0011 0.0032 0 2 0.0034 0.0058 0 1 0.0028 0.0053 8 0 1 0.0000 0.0004 0 1 0.0001 0.0013 0 1 0.0002 0.0007 9 0 3 0.0022 0.0048 0 1 0.0019 0.0043 0 5 0.0006 0.0026 10 0 2 0.0027 0.0051 0 1 0.0032 0.0056 0 1 0.0050 0.0070 11 0 1 0.0000 0.0005 0 1 0.0004 0.0019 0 1 0.0003 0.0016 12 0 2 0.0130 0.0112 0 2 0.0133 0.0123 0 2 0.0120 0.0108 min ... minimal intensity value in the average of 100 dark imagesmax ... maximal intensity value in the average of 100 dark imagesmean ... average intensity of 100 dark imagesstd ... average per-pixel standard deviation of intensity of 100 dark imagesStereo Clustersn 12 camera clustersn Trinocular stereo system + color camera on topn Problem of point correspondencen Optimization of viewing volumeo1o2o0Px1x2x010Rotation of the side cameras for 0 and 3 degreeso1o2o0Px1x2x0Rotation of top camera for 0 and 2 degrees11New Stereo Clustern Optimized position of the camerasn More robust structuren Protection of camera circuit boardsOptimization of Viewing Volumen Configuration of stereo clustersn Volume coverage analysisn Camera configuration analysisn Re-arrange the cameras to increase coveragen Solve existing problems of the ti-systemn Wide coverage of volume (e.g. for dancing)12Volume Coverage Simulator13Camera SimulatorProblems with the old configuration14Camera ArrangementBefore…After…Lensesn 6mm lens with the volume coverage of about 45Hºx34Vºn 4mm lens with the volume coverage of about 64Hºx49Vº15n Resized volume = decreased resolution of captured objectsn Use of wide angle lenses for lower body parts6mm lens 4mm lensIlluminationn Photometric properties of objects:¤ Lambertian (e.g. matte surfaces)¤ Diffuse (diffuser)¤ Reflective (e.g. mirror)¤ Specular (e.g. metal objects)n Specular light creates problem with 3D reconstruction (correspondence of points!)n Diffusive light needed to prevent shadows16Illuminationn Camera hardware parameters:¤ Lens aperturen Camera software parameters:¤ Exposure¤ Brightness¤ Shutter¤ Gainn LightsToo dark…Too bright…- Lens aperture- Shutter- Gain17Lightingn Diffused lightsn Daylight illuminationn brightline SeriesONE 1.1n Eight 55W 5600K lamps¤ 2 ceiling fixtures¤ 4 top-down fixtures¤ 2 floor fixturesWhat Should I Wear???n Texture helps correlationn Use of IR patterns to increase texturen Colors that work well:¤ Skin colors¤ Soft colors (e.g. yellow, pink, off-white)n Black or dark colors tend to blend with the background + less features extracted = poor 3D reconstruction18Camera Calibrationn Tsai* camera calibration methodn Pinhole perspective projection modeln Two step calibration:¤ Intrinsic parameters (grid collection)¤ Extrinsic parameters (LED point collection)n Nonlinear optimizationn Camera Calibration Toolbox for Matlab® (ETHZ)* R.Y. Tsai,An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision. Proceedings of IEEE Conference on Computer Vision and
View Full Document
Unlocking...