CSE486Robert CollinsLecture 27:Skin ColorCSE486Robert CollinsReview: Light TransportSource emits photonsPhotons travel in astraight lineThey hit an object. Some areabsorbed, some bounce offin a new direction.And then some reachan eye/camera andare measured.CSE486Robert CollinsColor of Light SourceSpectral Power Distribution:UVIRVisibleWavelength λAmplitudeRelative amount of lightenergy at each wavelengthCSE486Robert CollinsSpectral albedo for several different leavesSpectral AlbedoRatio of incoming to outgoing radiation at different wavelengths.CSE486Robert CollinsSpectral Radianceto aSpectralIrradianceSpectralRadianceSpectral AlbedoCSE486Robert CollinsHuman Eye: Rods and ConesS cones (blue)M cones (green)L cones (red)rods (overall intensity)CSE486Robert CollinsPutting it all Together = Color3 conesCOLOR!CSE486Robert CollinsDescribing ColorToday we consider a sample material, human skin,and look at two approaches to describe the color of skin in order to find it in images.1) physics-based approach2) learning-based approachCSE486Robert CollinsGoal: Label Skin Pixels in an ImageApplications: Person finding/tracking Gesture recognitionCSE486Robert CollinsThe Physics of Skin ColorAnalytic derivation:Moritz Storring, Hans Andersen and Eric Granum, “Skin ColourDetection under Changing Lighting Conditions,” 7th Symposium onIntelligent Robotics Systems, Coimbra Portugal, July 1999.Experimental measurement:Birgitta Martinkauppi, “Face Colour Under Varying Illumination: Analysis and Applications,” Ph.D. Thesis, Oulu University Press, Oulu Finland, 2002.CSE486Robert CollinsProblem: Color VariationSample from Oulu Physics-Based Face DatabaseApparent color varies due to lighting color and and camera spectral response.CSE486Robert CollinsSkin Reflectance ModelSkin is well-modeled by a dichromatic reflectance model. transparent medium (dermis) pigmentations (hemaglobin, melanin) specular reflection (oil on skin)Dichromatic reflectance modelCSE486Robert CollinsMeasuring Spectral Albedo of SkinCSE486Robert CollinsUnderstanding Skin AlbedoblueredCSE486Robert CollinsUnderstanding Skin AlbedoIncrease in melanin yields darker skin, masking the absorbtion band pattern of the hemaglobin.CSE486Robert CollinsAnalytic ModelGenerate different skin albedos by using observed curve forcaucasian, and calculate the reduction in reflectance due to anincrease in melanin (a substance that has a known absorbtion)I1(λ) ~ s I2(λ) ; λ = wavelengthSimpler approximation:s = scale factorCSE486Robert CollinsIlluminant SPDArtificial light sourcesBlackbody sources(for theoretical calculations)CSE486Robert CollinsCamera Spectral ResponseSONY DXC-755P 3CCD(manufacturer can supply this)CSE486Robert CollinsSkin Color Locus : Analytic Computation“Normalized Color”recall simple scalingof SPD curvesCSE486Robert CollinsSkin Color Locus : Experimental MeasureFairly good agreement!CSE486Robert CollinsSkin Locus ExamplesHistograms of skin color for different lightingconditions. Red: high values, blue: low values.CSE486Robert CollinsTighter BoundsIf you know the camera and light source, you canderive much tighter analytic bounds on skin color.CSE486Robert CollinsExampleSame individual under different lighting conditions.Subject 1:CaucasianSubject 2:Asian IndianCSE486Robert CollinsSample ApplicationFace tracking under varying illumination conditionsCSE486Robert CollinsJones and Rehg, 2002“Statistical Color Models with Application to Skin Detection”, M. J. Jones and J. M. Rehg,Int. J. of Computer Vision, 46(1):81-96, Jan 2002General Idea:• Drop the physics. Learn from examples instead.• Learn distributions of skin and nonskin color• Nonparametric distributions: color histograms• Bayesian classification of skin pixelsCSE486Robert CollinsLearning from ExamplesP(rgb | skin) = number of times rgb seen for a skin pixel total number of skin pixels seenP(rgb | not skin) = number of times rgb seen for a non-skin pixel total number of non-skin pixels seenThese statistics stored in two 32x32x32 RGB histogramsRGBRGBSkin histogram Non-Skin histogramFirst, have some poor grad student hand label thousands of imagesCSE486Robert CollinsLearned DistributionsSkin colorNon-Skin colorP(rgb | skin)P(rgb | not skin)CSE486Robert CollinsLikelihood RatioLabel a pixel skin ifP(rgb | skin)P(rgb | not skin)> Θ Θ = (cost of false positive) P( seeing not skin)(cost of false negative) P( seeing skin)0 <= Θ <= 1CSE486Robert CollinsSample Pixel ClassificationsΘ = .4CSE486Robert CollinsSample Application: HCIHaiying Guan, Matthew Turk, UCSBCSE486Robert CollinsSample Application: HCIHaiying Guan, Matthew Turk, UCSBCSE486Robert CollinsSample Use: Adult Image Classification• Percentage of pixels detected as skin.• Average probability of the skin pixels.• Size in pixels of the largest connected component of skin.• Number of connected components of skin.• Percentage of colors with no entries in the skin and non-skin histogramsBased on Five Features:Jones and RehgCSE486Robert CollinsAdult Image ClassificationCSE486Robert CollinsCombining Color and TextCSE486Robert CollinsAdult Image ClassificationOther related work:M.M. Fleck, D.A. Forsyth and C. Bregler, “FindingNaked People,” Proc. European Conf. on ComputerVision, Springer-Verlag, 1996. p. 593-602James Wang, Jia Li, Gio Wiederhold and OscarFirschein, “System for Screen ObjectionableImages” Computer Communications, Vol 21(15),pp.1355-1369, Elsevier, 1998.CSE486Robert CollinsBack to Jones and Rehg ModelA compact description is provided by converting thehistogram-based model into a Gaussian Mixture model.CSE486Robert CollinsJones and Rehg Mixture ModelCSE486Robert CollinsJones and Rehg Mixture ModelCSE486Robert CollinsHomework: Due Friday Dec 7•Download jrmogskin.m from the course web site•Try it on your own images!CSE486Robert CollinsWhat to Hand InA short report, in Angel: 1) one example where it works wonderfully well 2) one example showing false positives (things that are not skin, but that are labeled as skin). 3) one example showing false negatives (a patch of skin that is not labeled), along with an educated guess about why it was missed.CSE486Robert CollinsExamples Working WellCSE486Robert CollinsExample of False PositivesCSE486Robert CollinsExamplesExample ofFalse NegativesExplanation:paint on skinchanges thespectral albedoCSE486Robert CollinsImportant ConstraintNo X-rated images!!!! Keep