CSE486, Penn StateRobert CollinsLecture 24Video Change DetectionCSE486, Penn StateRobert CollinsBasics of VideoFrames come in 30 times per second. This is not much time to processeach image. Real-time algorithms therefore tend to be very simple.One of the main features of video imagery is the temporal consistencyfrom frame to frame. Not much changes during 1/30 of a second!Real-TimeOfflinecamera digitizer PCpre-stored imagesCSE486, Penn StateRobert CollinsDetecting Moving ObjectsAssumption: objects that move are important (e.g. people and vehicles)Basic approach: maintain a model of the static background. Compare the current frame with the background to locate moving foreground objects.CompareBackgroundmaintenanceCurrentframeChanges(objects)BackgroundmodelCSE486, Penn StateRobert CollinsSimple Background Subtraction• Background model is a static image (assumed to have no objects present).• Pixels are labeled as object (1) or not object (0) based on thresholding the absolute intensity difference between current frame and background.B = I(0);…loop time t I(t) = next frame; diff = abs[B – I(t)]; M(t) = threshold(diff,λ); …endM(t)absI(t)BλTCSE486, Penn StateRobert CollinsBackground Subtraction ResultsmovieCSE486, Penn StateRobert CollinsBG ObservationsBackground subtraction does a reasonable job of extractingthe shape of an object, provided the object intensity/color issufficiently different from the background.CSE486, Penn StateRobert CollinsBG ObservationsObjects that enter the scene and stop continue tobe detected, making it difficult to detect new objectsthat pass in front of them. If part of the assumed static background startsmoving, both the object and its negative ghost(the revealed background) are detectedCSE486, Penn StateRobert CollinsBG ObservationsBackground subtraction is sensitive to changingillumination and unimportant movement of thebackground (for example, trees blowing in thewind, reflections of sunlight off of cars or water).Background subtraction cannot handle movementof the camera.CSE486, Penn StateRobert CollinsSimple Frame Differencing• Background model is replaced with the previous image.M(t)absB(0) = I(0);…loop time t I(t) = next frame; diff = abs[B(t-1) – I(t)]; M(t) = threshold(diff,λ); … B(t) = I(t);enddelayI(t)B(t-1)λTCSE486, Penn StateRobert CollinsFrame Differencing ResultsmovieCSE486, Penn StateRobert CollinsFD ObservationsFrame differencing is very quick to adapt to changes in lighting or camera motion.Objects that stop are no longer detected. Objects thatstart up do not leave behind ghosts.However, frame differencing only detects the leading and trailing edge of a uniformly colored object. As a result very few pixels on the object are labeled, and it is very hardto detect an object moving towards or away from the camera.CSE486, Penn StateRobert CollinsDifferencing and Temporal ScaleDefine D(N) = || I(t) - I(t+N) ||Note what happens when we adjust the temporal scale (frame rate) at which we perform two-frame differencing …I(t) D(-1) D(-3) D(-5) D(-9) D(-15)more complete object silhouette, but two copies (one where object used to be, one where it is now).CSE486, Penn StateRobert CollinsThree-Frame DifferencingANDD(-15)D(+15)The previous observation is the motivation behind three-frame differencingwhere object was, and where it is nowwhere object is now, and where it will bewhere object is now!CSE486, Penn StateRobert CollinsThree-Frame Differencing51525354555651Choice of good frame-rate for three-frame differencing depends on the size and speed of the objectThis worked wellfor the person# framesskippedCSE486, Penn StateRobert CollinsAdaptive Background Subtraction• Current image is “blended” into the background model with parameter α• α = 0 yields simple background subtraction, α = 1 yields frame differencingM(t)absB(0) = I(0);…loop time t I(t) = next frame; diff = abs[B(t-1) – I(t)]; M(t) = threshold(diff,λ); … B(t) = α I(t)+(1–α)B(t-1);endI(t)B(t-1)λTα I(t) + (1–α)B(t-1)B(t)delayαCSE486, Penn StateRobert CollinsAdaptive BG Subtraction ResultsmovieCSE486, Penn StateRobert CollinsAdaptive BG ObservationsAdaptive background subtraction is more responsiveto changes in illumination and camera motion.Fast small moving objects are well segmented, but they leave behind short “trails” of pixels.Objects that stop, and ghosts left behind by objectsthat start, gradually fade into the background.The centers of large slow moving objects start tofade into the background too! This can be “fixed”by decreasing the blend parameter A, but then ittakes longer for stopped/ghost objects to disappear.CSE486, Penn StateRobert CollinsPersistent Frame Differencing• Motion images are combined with a linear decay term• also known as motion history images (Davis and Bobick)H(t)B(0) = I(0);H(0) = 0;loop time t I(t) = next frame; diff = abs[B(t-1) – I(t)]; M(t) = threshold(diff,λ); tmp = max[H(t-1)-γ,0)]; H(t) = max[255*M(t),tmp)]; … B(t) = I(t);endM(t)absdelayI(t)B(t-1)λTmaxmaxX2550γH(t-1)CSE486, Penn StateRobert CollinsPersistant FD ResultsmovieCSE486, Penn StateRobert CollinsPersistant FD ObservationsPersistant frame differencing is also responsiveto changes in illumination and camera motion,and stopped objects / ghosts also fade away.Objects leave behind gradually fading trails ofpixels. The gradient of this trail indicates theapparent direction of object motion in the image. Although the centers of uniformly colored objectsare still not detected, the leading and trailing edgesare make wider by the linear decay, so that perceptually (to a person) it is easier to see the whole object.CSE486, Penn StateRobert CollinsComparisonsBG subtract Frame diffAdaptive BG subtract Persistent Frame diffCSE486, Penn StateRobert CollinsComparisonsBG subtract Frame diffAdaptive BG subtract Persistent Frame diffCSE486, Penn StateRobert CollinsComparisonsBG subtract Frame diffAdaptive BG subtract Persistent Frame diffCSE486, Penn StateRobert CollinsComparisonsBG subtract Frame diffAdaptive BG subtract Persistent Frame diffCSE486, Penn StateRobert CollinsComparisonsBG subtract Frame diffAdaptive BG subtract Persistent Frame diffCSE486, Penn StateRobert CollinsComparisonsBG subtract Frame diffAdaptive BG subtract Persistent Frame diffCSE486, Penn StateRobert CollinsVariants of Basic Background SubtractionThere are lots, and more papers every day.CSE486, Penn StateRobert