TutorialScale-Space TheoryforMultiscale Geometric Image AnalysisBart M. ter Haar Romeny, PhDUtrecht University, the [email protected] image analysis has gained firm ground in computer vision, image processingand models of biological vision. The approaches however have been characterised by awide variety of techniques, many of them chosen ad hoc. Scale-space theory, as arelatively new field, has been established as a well founded, general and promisingmultiresolution technique for image structure analysis, both for 2D, 3D and time series.The rather mathematical nature of many of the classical papers in this field has preventedwide acceptance so far. This tutorial will try to bridge that gap by giving acomprehensible and intuitive introduction to this field. We also try, as a mutualinspiration, to relate the computer vision modeling to biological vision modeling. Themathematical rigor is much relaxed for the purpose of giving the broad picture. Inappendix A a number of references are given as a good starting point for further reading.The multiscale nature of thingsIn mathematics objects have no scale. We are familiar with the notion of points, thatreally shrink to zero, lines with zero width. In mathematics are no metrical unitsinvolved, as in physics. Neighborhoods, like necessary in the definition of differentialoperators, are defined as taken into the limit to zero, so we can really speak of localoperators.In physics objects live on a range of scales. We need an instrument to do an observation(our eye, a camera) and it is the range that this instrument can see that we call the scalerange. To expand the range of our eye we have a wide armamentarium of instrumentsavailable, like microscopes and telescopes. The scale range known to humankind spansabout 50 decades, as is beautifully illustrated in the book (and movie) "Powers of Ten"[Morrison 1985]. The range one instrument can see is always necessarily bounded on twosides: the inner scale is the smallest detail seen by the smallest aperture (e.g. one CCDelement of our digital camera, a cone or rod on our retina); the outer scale is the coarsestdetail that can be discriminated, i.e. it is the whole image (field of view).In physics dimensional units are essential: we express any measurement in these units,like meters, seconds, candelas, ampères etc. There is no such thing as a physical 'point'.In mathematics the smallest distance between two points can be considered in the limit tozero, but in physics this reduces to the finite aperture separation distance (samplingdistance). Therefore we may foresee serious problems with notions as differentiation,especially for high order (these problems are known as regularization problems), sub-pixel accuracy etc. As we will see, these problems are just elegantly solved by scale-space theory.In front-end vision the apparatus (starting at the retina) is equipped just to extract multi-scale information. Psychophysically is has been shown that the threshold modulationdepth for seeing blobs of different size is constant (within 5%) over more than twodecades, so the visual system must be equipped with a large range of sampling apertures.There is abundant electrophysiological evidence that the receptive fields (RF’s) in theretina come in a wide range of sizes1 [Hubel ‘62, ‘79a, ‘88a].In any image analysis there is a task: the notion of scale is often an essential part of thedescription of the task: "Do you want to see the leaves or the tree"?Linear Scale-Space Theory - Physics of ObservationTo compute any type of representation from the image data, information must beextracted using certain operators interacting with the data. Basic questions then are:What operators to use? Where to apply them? How should they be adapted to the task?How large should they be? We will derive the kernel from first principles (axioms)below.These operators (or filters, kernels, apertures: different words for the same thing) comeup in many tasks in signal analysis. We show that they are a necessary consequence ofthe physical process of measuring data. In this section we derive from some elementaryaxioms a complete family of such filters. As we will see, these filters come at acontinuous range of sizes. This is the basis of scale-space theory.If we start with taking a close look at the observation process, we run into someelementary questions:• What do we mean with the 'structure of images' [Koenderink 1984]?• What is an image anyway?• How good should a measurement (observation) be?• How accurately can we measure?• How do we incorporate the notion of scale in the mathematics of observation?• What are the best apertures to measure with?• Does the visual system make optimal measurements?Any physical observation is done through an aperture. By necessity this aperture has to befinite (would it be zero no photon would come through). We can modify the aperture 1 It is not so that every receptor in the retina (rod or cone) has its own fiber in the optic nerve to furtherstages. In a human eye there are about 150.106 receptors and 106 optic nerve fibres. Receptive fields formthe elementary ’apertures’ on the retina: they consist of many cones (or rods) in a roughly circular areaprojecting to a single (ganglion) output cell, thus effective integrating the luminance over a finite area.considerably by using instruments, but never make it zero width. This implies that wenever can observe the physical reality in the outside world, but we can come close. Wecan speak of the (for us unobservable) infinite resolution of the outside world.We consider here physical observations by an initial stage measuring device (also calledfront-end) like our retina or a camera, where no knowledge is involved yet, no preferencefor anything, and no nonlinearities of any kind. We call this type of observationuncommitted. Later we will relax this notion, among others by incorporating the notion ofa model or make the process locally adaptive to the image content, but here, in the firststages of observation, we know nothing.This notion will lead to the establishment of linear scale-space theory. It is a naturalrequirement for the first stage, but not for further stages, where extracted information,knowledge of model and/or task comes in etc. We then come into the important realm ofnonlinear scale-space theory, which will be discussed in section 4.Scale-space theory is the theory of