CSE 591: Visual AnalyticsLecture 4: Visual Importance and SaliencyKlaus MuellerComputer Science DepartmentStony Brook UniversityAttentionThe cognitive process of selectively concentrating on one thing while ignoring other things•detecting features in visual clutter (CAPTCHA, next slide)• detecting coherent speech in noisy environments (cocktail party effect)• ignore features while concentrating on others (recall gorilla)• can also have divided attention (example: cell phone + driving)• heavily studied in psychology and neuroscience• closely tied to perceptionAttention theory is important for visualization as well•in contrast to computer vision, WE design/create the scene • this design guides the attention of the viewer• guidance determined by visualization goalsTherefore it is important to understand mechanism of attentionCAPTCHACAPTCHA: noisy and vastly distorted patterns that are difficult to recognize by machinesCAPTCHABut computer vision algorithms have become more sophisticated at CAPTCHA character recognition•the latest approach is object recognitionVisual Recognition and AttentionTwo opposing theories:•Gestalt• Feature integrationGestalt theory•top-down approach• proposes that the operational principle of the brain is holistic, parallel, and analog, with self-organizing tendencies• important in user interface design (button grouping, etc)Feature integration theory• bottom-up approach• primary visual features are processed and represented with separate feature maps• these are later integrated in a saliency map that can be accessed in order to direct attention to the most conspicuous areasGestalt Theory: Confirming ExamplesEmergenceReificationMulti-StabilityInvarianceGestalt Theory: Opposing ExamplesSelective-Encoding:•involving one to distinguish what is important in a problem and what is irrelevant (i.e., filtering)Selective-Comparison:•identifying information by finding a connection between acquiredknowledge and experienceSelective-Combination:•identifying a problem through understanding the different components and putting everything together.Feature Integration TheoryOne of the most influential influential psychological models of human visual attention in recent yearsTwo types of visual search mechanismsFeature search •can be performed fast and pre-attentively for targets defined by primitive features (such as color, orientation, intensity, etc)Conjunction search • serial search for targets defined by a conjunction of primitive features• much slower• requires conscious attentionVery promising technique for computer vision to detect partially occluded objects (SIFT)What Does It Mean For Visualization?Feature integration theory:•must exploit this to guide attention• relatively “easy” since it involves mostly local enhancements• notion of saliency is important (recall center-surround mechanism)Gestalt theory:• reminds me of ghosting techniques (mental feature completion)• silhouettes and contours for context objects• many techniques used now in illustrative rendering• recall also optical illusionsBasic Techniques: Contours and Outlinesdepth-map(edges are due to C0discontinuities)normal-map(edges are dueto C1discontinuities)combinedBasic Techniques: Contours and Outlinesdepth-mapnormal-mapcombinedBasic Techniques: Contours and Outlinesmixing outlines with volume renderingrendering interior structures as contoursJ. Fisher, D. BartzBasic Techniques: SilhouettesNot an image-space method•uses dot product V⋅N=0 criterion• V: view vector• N: surface normalFinds curves and creases at higher qualityAllows further processing of these (for example hatching)Must disambiguate occlusionsSuggestive ContoursCurves where the surface bends away from the viewer (as opposed bending towards them)D. DeCarloSuggestive ContoursThose locations at which the surface is almost in contour, from the original viewpoint•where the radial curvature (1/curve radius) is zero (w is the projection of V onto the tangent plane)• where V⋅N is a positive local minimum rather than zero. • correspond to true contours in relatively nearby viewpoints.Suggestive Contourscontours suggestive contours (image space vs. object space method)Suggestive ContoursRequire the computation of the second derivative at high accuracy•use high-quality 2ndderivative (curvature-estimation) filters for volume datasetsG. KindlmannCurvature Stroke LinesSemitransparent iso-intensity surface for radiation treatment planning and a tumor inside. Right: Strokes along the principal curvature are added to convey shape V. InterranteHatchingApplies this illustration style as a function of illumination and others portion of the tonal art mapStipplingStippling is yet another illustration technique•vary the density of points with illumination and/or other attributeImportance-Controlled Rendering / VisualizationFirst, classify the scene:•Focus Objects (FO): objects in the center of interest are emphasized in a particular way• Near Focus Objects (NFO): important objects for the understanding of the functional interrelation or spatial location.• Context Objects (CO): all other objects (rendered e.g., as silhouettes)• Container Objects (CAO): one object that contains all other objects.Render these in a certain order to ensure visual consistencyB. PreimGhostingGhosting: Procedureiso-surface outer structureiso-surface inner structuresemi-transparent outer structureghosted view showing both - outer and inner structureGhostingS. BrucknerContext PreservingI. ViolaFansS. BrucknerFansS. BrucknerThe 4thDimensionTime varying effects are very difficult to perceive in detail•recall the 7+-1 rule?Another look at Daniel Simons’ workA challenging area of research:•visualization of time-varying behavior in a single frame• can use illustration techniquesTime-Varying DataThe goal is to depict the time-varying behavior of the data in a single frame via illustrative techniquestypical illustration metaphors applied in visualization A. JoshiSaliency-Guided Enhancement for VisualizationGaze-directed abstraction of photographs (DeCarlo)Saliency-Guided Enhancement for VisualizationUsing saliency to guide geometric mesh simplification (Lee & Varshney)Saliency-Guided Enhancement for