Boundary Ownership by Lifting to 2.1DIdo Leichter and Michael LindenbaumTechnion – Israel Institute of TechnologyComputer Science Department, Technion, Haifa 32000, Israel{idol,mic}@cs.technion.ac.ilAbstractThis paper addresses the “boundary ownership” prob-lem, also known as the figure/ground assignment problem.Estimating boundary ownerships is a key step in perceptualorganization: it allows higher-level processing to be ap-plied on non-accidental shapes corresponding to figural re-gions. Existing methods for estimating the boundary owner-ships for a given set of boundary curves model the probabil-ity distribution function (PDF) of the binary figure/groundrandom variables associated with the curves. Instead ofmodeling this PDF directly, the proposed method uses the2.1D model: it models the PDF of the ordinal depths ofthe image segments enclosed by the curves. After this PDFis maximized, the boundary ownership of a curve is de-termined according to the ordinal depths of the two im-age segments it abuts. This method has two advantages:first, boundary ownership configurations inconsistent withevery depth ordering (and thus very likely to be incorrect)are eliminated from consideration; second, it allows for theintegration of cues related to image segments (not neces-sarily adjacent) in addition to those related to the curves.The proposed method models the PDF as a conditional ran-dom field (CRF) conditioned on cues related to the curves,T-junctions, and image segments. The CRF is formulatedusing learnt non-parametric distributions of the cues. Themethod significantly improves the currently achieved fig-ure/ground assignment accuracy, with 20.7% fewer errorsin the Berkeley Segmentation Dataset.1. IntroductionThe “boundary ownership” problem is the problem ofautomatically deciding which of the two image segmentsabutting a provided curve along an object’s boundary in animage is the ‘figure’ (i.e., “owns the boundary”) and whichis the ‘ground’. Although the image segment correspond-ing to the occluding object is usually the figure, boundaryownership is generally a perceptual characteristic. That is,the image region that is perceived as lying in front of theground regions is considered as ‘figure’ [11], even if thisperception contradicts the 3D layout of the scene. Exam-ples of this may be seen in the images in Fig. 1, taken fromthe Berkeley Segmentation Dataset (BSDS). They were firstsegmented by a human observer, after which two additionalhuman observers attributed each of the boundary curves be-tween the segments to one of its two abutting segments. Al-though the water in images (a) and (b) occludes the land andthe rocks beneath it, both human observers marked the landand the rocks as the “owners” of the shorelines. (This mayexplain the name ‘shoreline’ rather than, say, ‘sealine’.) An-other example is provided in image (c), where both humanobservers marked the ducks as the owners of the contactline between the ducks and the water, although the wateroccludes the duck parts below this contact line. A differenttype of example is the occasional perception of a hole as thefigure in cases where the background seen through the holedoes not appear in the image around the object containingthe hole [11].It has been shown that boundary ownership perception isinfluenced by many factors, including local characteristicsof the boundary shape (e.g., convexity [6] and orientation,termed “lower region” [16]), its global characteristics (e.g.,symmetry, size and orientation [6, 11]), the appearance ofthe region itself (e.g., contrast [11]) and also its surround-ings (e.g., parallelism and surroundedness [11]). The orien-tations of the intensity level sets in the close neighborhoodof the boundary have been shown to bear information aboutwhich of the two boundary sides occludes the continuationof the other. (The orientations of the intensity level setsare measured relative to that of the boundary itself.) Thisinformation may thus be used for estimating the boundaryownership as well [5].All methods for automatic figure/ground assignment inimages measure a subset of the above factors. These mea-surements, which are usually ad hoc, are used to make theboundary ownership decisions. In some works, these mea-surements are performed for each boundary pixel by ana-lyzing the boundary curve segment and the image regionin its neighborhood. The decision for each boundary pixel(a) (b)(c)Figure 1. Cases where the boundary side corresponding to theoccluding object was not classified as ‘figure’ by the human ob-servers. See text for details.is then made independently. For example, such an approachwas taken in [5], where the orientations of the intensity levelsets near the boundary were used for deciding which of theboundary sides occludes the continuation of the other. An-other example is [1], where the convexity, lower region andsize cues were locally measured for boundary points andthen used to independently estimate the figure/ground labelat each point. In one of the methods in [15], the assignmentfor each boundary pixel was also made independently, ac-cording to the likelihood of a set of local shape descriptorsthat was based on a reference set of learnt shapemes [9].These shapemes are prototypical shapes that implicitly en-code cues such as convexity and parallelism.The figure/ground assignment along a boundary curvebetween two junctions is constant. This suggests that bet-ter precision can be obtained by using cues related to thewhole curve segment and making a curve-based, rather thana pixel-based, figure/ground assignment. Such an approachwas taken, for example, in [13] and in [15]. In fact, in [15]the figure/ground assignment to the curves was made by av-eraging the local shapeme on the curve. The latter methodprovided significantly more accurate results than the localmodel and reinforced the advantage of making curve-basedassignments.Curve junctions and the angles between the curves at thejunctions provide additional cues for the figure/ground la-bels of the junction curves. Thus, incorporating this ad-ditional information in the boundary ownership estimatesmight increase the precision further. Since each junctionprovides a cue for the mutual figure/ground labels of all itscurves (rather than on each curve independently), the in-tegration of the junction cues into the boundary ownershipmodel causes the curve assignments to be coupled. Curvejunctions in