Ullman's Visual Routines, and Tekkotsu SketchesParsing the Visual WorldUllman: Visual RoutinesBase RepresentationsPrimal SketchIncremental RepresentationsDual-Coding RepresentationSketches in TekkotsuMaking New SketchesVisualRoutinesBehaviorFirst Visual Routines ExampleColor-Segmented Imagevisops::colormask(camFrame,“orange”)‏visops::edge(orange_stuff)‏visops::skel(orange_stuff)‏visops::neighborSum(orange_stuff)‏Second ExampleSlide 18camFramevisops::colormaskvisops::labelccvisops::areaccb_area == b_area->max()‏Third ExampleSlide 25visops::edist(b_max)‏o_area > 10bo_distbo_winSketch PropertiesSketch Constructor #1Sketch Constructor #2Result of Second Constructor: Sketch barNEW_SKETCH MacroSlide 35Do Tekkotsu's Representations Fit Ullman's Theory?Triesman's Visual Search Expt.Slide 38Slide 39What Do Human Limitations Tell Us About Cognition?Science vs. Engineering14/01/19 15-494 Cognitive Robotics 1Ullman's Visual Routines,and Tekkotsu Sketches15-494 Cognitive RoboticsDavid S. Touretzky &Ethan Tira-ThompsonCarnegie MellonSpring 200814/01/19 15-494 Cognitive Robotics 2Parsing the Visual WorldHow does intermediate level vision work?How do we parse a scene?Is the x inside or outside the closed curve?14/01/19 15-494 Cognitive Robotics 3Ullman: Visual RoutinesFixed set of composable operators.Wired into our brains.Operate on “base representations”, produce “incremental representations”.Can also operate on incremental representations.Examples:shift of processing focusindexing (odd-man-out)boundary tracingmarkingbounded activation (coloring)14/01/19 15-494 Cognitive Robotics 4Base RepresentationsDerived automatically; no decisions to make.Derivation is fully parallel.Multiple parallel streams in the visual hierarchy.Describe local image properties such as color, orientation, texture, depth, motion.Marr's “primal sketch” and “2 ½-D Sketch”14/01/19 15-494 Cognitive Robotics 5Primal SketchImages from http://www.cs.ucla.edu/~cguo/primal_sketch.htm14/01/19 15-494 Cognitive Robotics 6Incremental RepresentationsConstructed by visual routines.Describe relationships between objects in the scene.Construction may be inherently sequential:tracing and scanning take timethe output of one visual routine may be input to anotherpipelining may speed things upCan't compute everything; too many combinations.The choice of which operations to apply will depend on the task being performed.14/01/19 15-494 Cognitive Robotics 7Dual-Coding RepresentationPaivio's “dual-coding theory”:People use both iconic and symbolic mental representations.They can convert between them when necessary, but at a cost of increased processing time.Tekkotsu implements this idea:What would Ullman say? Visual routines mostly operate on sketches, but not exclusively.14/01/19 15-494 Cognitive Robotics 8Sketches in TekkotsuA sketch is a 2-D iconic (pixel) representation.Templated class:Sketch<uchar>unsigned char: can hold a color indexSketch<bool> true if a property holds at image loc.Sketch<uint> unsigned int: pixel index; distance; areaSketch<usint> unsigned short intSketch<float>Sketches are smart pointers.Sketches live in a SketchSpace: fixed width and height.A built-in sketch space: camSkS.14/01/19 15-494 Cognitive Robotics 9Making New SketchesWe can use a macro to create new sketches: NEW_SKETCH(name, type, value)The name will be used as a variable name.The type should be one of bool, uchar, uint, etc. NEW_SKETCH(camFrame, uchar, sketchFromSeg())14/01/19 15-494 Cognitive Robotics 10VisualRoutinesBehaviorSubclass of BehaviorBaseProvides several SketchSpace / ShapeSpace pairs.Allows you to view the SketchSpace remotely, using the SketchGUI tool.Let's try a sample image:14/01/19 15-494 Cognitive Robotics 11First Visual Routines Example#include "DualCoding/DualCoding.h"using namespace DualCoding;class DstBehavior : public VisualRoutinesBehavior { public: DstBehavior() : VisualRoutinesBehavior("DstBehavior") {} void DoStart() { VisualRoutinesBehavior::DoStart(); NEW_SKETCH(camFrame, uchar, sketchFromSeg()); NEW_SKETCH(orange_stuff, bool, visops::colormask(camFrame,“orange”)); NEW_SKETCH(o_edge, bool, visops::edge(orange_stuff)); NEW_SKETCH(o_skel, bool, visops::skel(orange_stuff)); NEW_SKETCH(o_neighbs, uchar, visops::neighborSum(orange_stuff)); }};color name defined inthe .col file14/01/19 15-494 Cognitive Robotics 12Color-Segmented Image14/01/19 15-494 Cognitive Robotics 13visops::colormask(camFrame,“orange”)14/01/19 15-494 Cognitive Robotics 14visops::edge(orange_stuff)14/01/19 15-494 Cognitive Robotics 15visops::skel(orange_stuff)14/01/19 15-494 Cognitive Robotics 16visops::neighborSum(orange_stuff)14/01/19 15-494 Cognitive Robotics 17Second ExampleFind the largest blue region in the image:14/01/19 15-494 Cognitive Robotics 18Second Example void DoStart() { VisualRoutinesBehavior::DoStart(); NEW_SKETCH(camFrame, uchar, sketchFromSeg()); NEW_SKETCH(blue_stuff, bool, visops::colormask(camFrame,“blue”)); NEW_SKETCH(b_cc, uint, visops::labelcc(blue_stuff)); NEW_SKETCH(b_area, uint, visops::areacc(b_cc)); NEW_SKETCH(b_max, bool, b_area == b_area->max()); }};14/01/19 15-494 Cognitive Robotics 19camFrame14/01/19 15-494 Cognitive Robotics 20visops::colormask14/01/19 15-494 Cognitive Robotics 21visops::labelccComponents labeled starting from 1in upper left; max label in lower right.14/01/19 15-494 Cognitive Robotics 22visops::areacc14/01/19 15-494 Cognitive Robotics 23b_area == b_area->max()14/01/19 15-494 Cognitive Robotics 24Third ExampleFind the orange region closest to the largest blue one; ignore any orange noise (blobs smaller than 10 pixels).14/01/19 15-494 Cognitive Robotics 25Third ExampleNEW_SKETCH(b_dist, uint, visops::edist(b_max));NEW_SKETCH(orange_stuff, bool, visops::colormask(camFrame,"orange"));NEW_SKETCH(o_cc, uint, visops::labelcc(orange_stuff));NEW_SKETCH(o_area, uint, visops::areacc(o_cc));NEW_SKETCH(o_blobs, bool, o_area > 10);NEW_SKETCH(bo_dist, uint, b_dist*o_blobs);int const min_index = bo_dist->findMinPlus();int const min_label = o_cc[min_index];NEW_SKETCH(bo_win, bool, o_cc == min_label);NEW_SKETCH(camY, uchar, sketchFromRawY());14/01/19 15-494 Cognitive Robotics 26visops::edist(b_max)14/01/19 15-494 Cognitive Robotics 27o_area > 10NEW_SKETCH(o_blobs, bool, o_area > 10);14/01/19