02/07/08 15-494 Cognitive Robotics 1Ullman's Visual Routines,and Tekkotsu Sketches15-494 Cognitive RoboticsDavid S. Touretzky &Ethan Tira-ThompsonCarnegie MellonSpring 200802/07/08 15-494 Cognitive Robotics 2Parsing the Visual World●How does intermediate level vision work?–How do we parse a scene?●Is the x inside or outside the closed curve?02/07/08 15-494 Cognitive Robotics 3Ullman: Visual Routines●Fixed set of composable operators.●Wired into our brains.●Operate on “base representations”, produce “incremental representations”.●Can also operate on incremental representations.●Examples:–shift of processing focus–indexing (odd-man-out)–boundary tracing–marking–bounded activation (coloring)02/07/08 15-494 Cognitive Robotics 4Base Representations●Derived automatically; no decisions to make.●Derivation is fully parallel.–Multiple parallel streams in the visual hierarchy.●Describe local image properties such as color, orientation, texture, depth, motion.●Marr's “primal sketch” and “2 ½-D Sketch”02/07/08 15-494 Cognitive Robotics 5Primal SketchImages from http://www.cs.ucla.edu/~cguo/primal_sketch.htm02/07/08 15-494 Cognitive Robotics 6Incremental Representations●Constructed by visual routines.●Describe relationships between objects in the scene.●Construction may be inherently sequential:–tracing and scanning take time–the output of one visual routine may be input to another–pipelining may speed things up●Can't compute everything; too many combinations.●The choice of which operations to apply will depend on the task being performed.02/07/08 15-494 Cognitive Robotics 7Dual-Coding Representation●Paivio's “dual-coding theory”:People use both iconic and symbolic mental representations.They can convert between them when necessary, but at a cost of increased processing time.●Tekkotsu implements this idea:●What would Ullman say? Visual routines mostly operate on sketches, but not exclusively.02/07/08 15-494 Cognitive Robotics 8Sketches in Tekkotsu●A sketch is a 2-D iconic (pixel) representation.●Templated class:–Sketch<uchar> unsigned char: can hold a color index–Sketch<bool> true if a property holds at image loc.–Sketch<uint> unsigned int: pixel index; distance; area–Sketch<usint> unsigned short int–Sketch<float>●Sketches are smart pointers.●Sketches live in a SketchSpace: fixed width and height.●A built-in sketch space: camSkS.02/07/08 15-494 Cognitive Robotics 9Making New Sketches●We can use a macro to create new sketches: NEW_SKETCH(name, type, value)●The name will be used as a variable name.●The type should be one of bool, uchar, uint, etc. NEW_SKETCH(camFrame, uchar, sketchFromSeg())02/07/08 15-494 Cognitive Robotics 10VisualRoutinesBehavior●Subclass of BehaviorBase●Provides several SketchSpace / ShapeSpace pairs.●Allows you to view the SketchSpace remotely, using the SketchGUI tool.●Let's try a sample image:02/07/08 15-494 Cognitive Robotics 11First Visual Routines Example#include "DualCoding/DualCoding.h"using namespace DualCoding;class DstBehavior : public VisualRoutinesBehavior { public: DstBehavior() : VisualRoutinesBehavior("DstBehavior") {} void DoStart() { VisualRoutinesBehavior::DoStart(); NEW_SKETCH(camFrame, uchar, sketchFromSeg()); NEW_SKETCH(orange_stuff, bool, visops::colormask(camFrame,“orange”)); NEW_SKETCH(o_edge, bool, visops::edge(orange_stuff)); NEW_SKETCH(o_skel, bool, visops::skel(orange_stuff)); NEW_SKETCH(o_neighbs, uchar, visops::neighborSum(orange_stuff)); }};color name defined inthe .col file02/07/08 15-494 Cognitive Robotics 12Color-Segmented Image02/07/08 15-494 Cognitive Robotics 13visops::colormask(“orange”)02/07/08 15-494 Cognitive Robotics 14visops::edge(orange_stuff)02/07/08 15-494 Cognitive Robotics 15visops::skel(orange_stuff)02/07/08 15-494 Cognitive Robotics 16visops::neighborSum(orange_stuff)02/07/08 15-494 Cognitive Robotics 17Second Example●Find the largest blue region in the image:02/07/08 15-494 Cognitive Robotics 18Second Example void DoStart() { VisualRoutinesBehavior::DoStart(); NEW_SKETCH(camFrame, uchar, sketchFromSeg()); NEW_SKETCH(blue_stuff, bool, visops::colormask(camFrame,“blue”)); NEW_SKETCH(b_cc, uint, visops::labelcc(blue_stuff)); NEW_SKETCH(b_area, uint, visops::areacc(b_cc)); NEW_SKETCH(b_max, bool, b_area == b_area->max()); }};02/07/08 15-494 Cognitive Robotics 19camFrame02/07/08 15-494 Cognitive Robotics 20visops::colormask02/07/08 15-494 Cognitive Robotics 21visops::labelccComponents labeled starting from 1in upper left; max label in lower right.02/07/08 15-494 Cognitive Robotics 22visops::areacc02/07/08 15-494 Cognitive Robotics 23b_area == b_area->max()02/07/08 15-494 Cognitive Robotics 24Third Example●Find the orange region closest to the largest blue one; ignore any orange noise (blobs smaller than 10 pixels).02/07/08 15-494 Cognitive Robotics 25Third ExampleNEW_SKETCH(b_dist, uint, visops::edist(b_max));NEW_SKETCH(orange_stuff, bool, visops::colormask(camFrame,"orange"));NEW_SKETCH(o_cc, uint, visops::labelcc(orange_stuff));NEW_SKETCH(o_area, uint, visops::areacc(o_cc));NEW_SKETCH(o_blobs, bool, o_area > 10);NEW_SKETCH(bo_dist, uint, b_dist*o_blobs);int const min_index = bo_dist->findMinPlus();int const min_label = o_cc[min_index];NEW_SKETCH(bo_win, bool, o_cc == min_label);NEW_SKETCH(camY, uchar, sketchFromRawY());02/07/08 15-494 Cognitive Robotics 26visops::edist(b_max)02/07/08 15-494 Cognitive Robotics 27o_area > 10NEW_SKETCH(o_blobs, bool, o_area > 10);02/07/08 15-494 Cognitive Robotics 28bo_distNEW_SKETCH(bo_dist, uint, b_dist*o_blobs);02/07/08 15-494 Cognitive Robotics 29bo_winNEW_SKETCH(bo_win, bool, o_cc == min_label);02/07/08 15-494 Cognitive Robotics 30Sketch Properties●Every sketch has a color, and a colormap.●Sketch<bool> is rendered in that color.●Sketch properties are inherited from the first argument of any visual routine or sketch operator.●Example: NEW_SKETCH(result, bool, blue_thing > pink_thing);The result will have color blue.●Colormaps: segMap, grayMap, jetMap, jetMapScaled02/07/08 15-494 Cognitive Robotics 31Sketch Constructor #1●Specify a sketch space and a name: Sketch<bool> foo(camSkS, “foo”); foo = false; for ( int i=50; i<90; i++ ) foo(i,i) = true; foo->V();02/07/08 15-494 Cognitive Robotics 32Sketch Constructor #2●Specify a name and a parent sketch to inherit from. Sketch<uchar> bar(“bar”, foo);