01/26/10 15-494 Cognitive Robotics 1The Vision Pipeline andColor Image Segmentation15-494 Cognitive RoboticsDavid S. Touretzky &Ethan Tira-ThompsonCarnegie MellonSpring 201001/26/10 15-494 Cognitive Robotics 2Why Don't Computers See Very Well?Approx. 1/3 of the human brain is devoted to vision!Felleman and Van Essen's Flat Map of the Macaque BrainDJ Felleman and DC Van Essen (1991), Cerebral Cortex 1:1-47.01/26/10 15-494 Cognitive Robotics 3The Macaque “Vision Pipeline”as of December 1990DJ Felleman and DC Van Essen (1991), Cerebral Cortex 1:1-47.RGC = retinal ganglion cellsHC = hippocampus;ER = entorhinal cortex:high level brain areas01/26/10 15-494 Cognitive Robotics 4Why Is Vision Hard?●Segmentation: where are the boundaries of objects?●Need to recover 3-D shapes from 2-D images:–Shape from shading–Shape from texture●Need to fill in occluded elements – what aren't we seeing?●Importance of domain knowledge:–Experience shapes our perceptual abilities–Faces are very special; there are “face cells” in IT (inferotemporal cortex)–Reading is also special; learning to read fluently alters the brain01/26/10 15-494 Cognitive Robotics 5The Segmentation Problem01/26/10 15-494 Cognitive Robotics 6Shape From ShadingImages from: www.cs.ucla.edu/~eprados/01/26/10 15-494 Cognitive Robotics 7Occlusion●How many rectangles can you find?●What shapes are present in the image?01/26/10 15-494 Cognitive Robotics 8Occlusion●How many rectangles can you find?●What shapes are present in the image?None! (Or two.)01/26/10 15-494 Cognitive Robotics 9Vision is Hard!How Can a Poor Robot Cope?●Use color to segment images.●Discard shading and texture cues.●Planar world assumption (can be relaxed later).●Domain knowledge for occlusion(blue/orange occludes pink.)From colors to objects: green = floor pink = board blue, orange = game pieces01/26/10 15-494 Cognitive Robotics 10What is “Color” ?●Humans have 3 types of color receptors (cones).●Dogs have 2: they'rered/green colorblind.●Cats have 3, but sparse:weak trichromants.●Birds have 4 or 5 types.●Birds and honeybees can see ultraviolet;honeybees can't seered.●Rats lack color vision.Image from: http://www.normankoren.com/Human_spectral_sensitivity_small.jpg01/26/10 15-494 Cognitive Robotics 11The Human Retina is Most Responsive to Green LightImages from http://www.cse.lehigh.edu/%7Espletzer/cse398_Spring05/lec002_CMVision.pdfThat's why green laser pointers look brighter than red ones of the same power.01/26/10 15-494 Cognitive Robotics 12Color and Computers●Video cameras don't see color the same way the human eye does:–Different spectral sensitivity curves.●Colors that look different to you may look the same to a computer that sees through a camera, and vice versa.●Computer monitors try to synthesize colors by blending just three frequencies: red(), green(), and blue().●No computer monitor can produce the full range of color sensations of which humans are capable.01/26/10 15-494 Cognitive Robotics 13RGB Color SpaceImage from http://www.photo.net/learn/optics/edscott/vis00020.htm01/26/10 15-494 Cognitive Robotics 14Edge of Fully Saturated HuesImage from http://www.photo.net/learn/optics/edscott/vis00020.htmMove from one corner to the next by increasing or decreasing one of the three RGB components.Example: moving... From green to yellow: [0,255,0]  [255,255,0]From yellow to red: [255,255,0]  [255,0,0]From red to magenta: [255,0,0]  [255,0,255]Saturation in RGB space = max(r,g,b) - min(r,g,b)01/26/10 15-494 Cognitive Robotics 15Saturation in ImageszerosaturationmaximumsaturationImage source: Wikipedia“Color Saturation”01/26/10 15-494 Cognitive Robotics 16YUV / YCbCr Color Space●Y = intensity●U/Cb = “blueness”(green vs. blue)●V/Cr = “redness”(green vs. red)Image from http://www.andrew.cmu.edu/course/15-491/lectures/Vision_I.pdf01/26/10 15-494 Cognitive Robotics 17YUV Color CubeImages from http://commons.wikimedia.org/wiki/Image:Cubo_YUV_con_las_capas_de_color.png01/26/10 15-494 Cognitive Robotics 18Converting RGB to YUV(assuming 8 bits per channel)[YUV]=1256⋅[65.738 129.057 25.064−37.945 −74.494 112.439112.439 −94.154 −18.285][RGB][16128128]01/26/10 15-494 Cognitive Robotics 19HSV Color Space●H = hue●S = saturation●V = value (intensity)Image from http://www.wordiq.com/definition/Image:HSV_cone.jpg01/26/10 15-494 Cognitive Robotics 20Many Cameras Use YUVWhat the robot seesWhat is displayedfor humansSegmented image01/26/10 15-494 Cognitive Robotics 21Color Classification 1●Define a set of color classes: “pink”, “orange”, etc.●Each class is assigned some region of color space.●Simplest case: use rectangles.isOrange[i] = imR[i] >= orangeMinR && imR[i] <= orangeMaxR && imG[i] >= orangeMinG && imR[i] <= orangeMaxG && imB[i] >= orangeMinB && imR[i] <= orangeMaxB;●Drawbacks: (1) the “real” regions aren't rectangular, so errors result; (2) lots of colors = slow processing.01/26/10 15-494 Cognitive Robotics 22Color Classification 2●We can have arbitrary-shaped color regions by creating a lookup table.●For each (R,G,B) value, store the color class (integer).●Problem: 24 bit color = 16 million entries = 16 MB.●Could use fewer bits, but that would reduce accuracy.01/26/10 15-494 Cognitive Robotics 23Color Classification 3●J. Bruce, T. Balch, and M. Veloso, IROS 2000:●Table lookup with bit-wise AND function can handle 32 color classes at once.int Ytable[256], Utable[256], Vtable[256];ColorClasses[i] = Ytable[imY[i]] & Utable[imU[i]] & Vtable[imV[i]];01/26/10 15-494 Cognitive Robotics 24Bruce et al. (continued)●We assigned a bit to each color:1000 = “pink”0100 = “orange”0010 = “blue”0001 = “green”●Suppose the “pink” and “orange” classes both include some colors with a Y value of 214:Ytable[214] = 0x1100●Suppose all four classes include a U value of 56:Utable[56] = 0x1111●If “orange” and “green” both include V values of 118:Vtable[118] = 0x0101●Color classes of (214,56,118) are: 0x0100 = orange01/26/10 15-494 Cognitive Robotics 25Color Classification 4: CMVision●CMVision is a vision package developed by Jim Bruce, Tucker Balch, and Manuela Veloso at Carnegie Mellon. Used for many robotics projects.●Current implementation operates in YUV space. Uses a reduced-resolution lookup table so it's not limited to rectangular decision boundaries.–4 bits for Y,