IEEE Transactions on Robotics and Automation: Special Issue on Multi-Robot Systems Paper:H2001-122MRS 1 Abstract— The objective of the CyberScout project is todevelop an autonomous surveillance and reconnaissance systemusing a network of all-terrain vehicles. In this paper, we focus ontwo facets of this system: 1) vision for surveillance and 2)autonomous navigation and dynamic path planning.In the area of vision-based surveillance, we have developedrobust, efficient algorithms to detect, classify, and track movingobjects of interest (person, people, or vehicle) with a staticcamera. Adaptation through feedback from the classifier andtracker allow the detector to use grayscale imagery, butperform as well as prior color-based detectors. We haveextended the detector using scene mosaicing to detect and indexmoving objects when the camera is panning or tilting. Theclassification algorithm performs well (less than 8% error rate forall classes) with coarse inputs (20x20-pixel binary image chips),has unparalleled rejection capabilities (rejects 72% of spuriousdetections), and can flag novel moving objects. The trackingalgorithm achieves highly accurate (96%) frame-to-framecorrespondence for multiple moving objects in cluttered scenes bydetermining the discriminant relevance of object features.We have also developed a novel mission coordinationarchitecture, CPAD (Checkpoint/Priority/Action Database),which performs path planning via checkpoint and dynamicpriority assignment, using statistical estimates of theenvironment’s motion structure. The motion structure is used tomake both preplanning and reactive behaviors more efficient byapplying global context. This approach is more computationallyefficient than centralized approaches and exploits robotcooperation in dynamic environments better than decoupledapproaches.Index Terms— Multiple autonomous vehicles, dynamic pathplanning, visual surveillance, reconnaissance, motion detectionwith image mosaics, object classification, moving objectcorrespondence.I. INTRODUCTIONCamera-based surveillance has long been used for securityand observation purposes. Surveillance cameras are typicallyfixed at known positions and cover a circumscribed areadefined by the fields of view of the cameras. Although somerecent vision work has addressed autonomous surveillance[1][2][3][4], in most cases humans perform the sensoryprocessing, either in real time, or by reviewing footage.Manuscript received March 27, 2001. The research described in this paperwas carried out by the Institute for Complex Engineered Systems, CarnegieMellon University, under a contract with the Defense Advanced ResearchProjects Agency.All authors are with the Institute for Complex Engineered Systems,Carnegie Mellon University. (address all correspondence to: {mahesh |jmd}@andrew.cmu.edu).Likewise, humans have performed reconnaissance, or scouting,for centuries in military and other applications in order toinspect terrain and identify and classify activities in theenvironment. In the CyberScout project, we combine thesensory capabilities of surveillance with the mobility ofreconnaissance by mounting cameras on mobile roboticplatforms. The resulting groups of collaboratingreconnaissance and surveillance robots pose interestingchallenges in vision-based surveillance algorithms and missionplanning.This paper primarily describes the surveillance-algorithmand mission-sensitive path-planning components ofCyberScout. Section II reviews previous work in autonomoussurveillance with mobile robots. Section III gives a briefoverview of the CyberScout system. Fig. 1 shows thearchitecture of the surveillance system within eachCyberScout. A camera on a CyberScout captures a frame andsends it to the motion detection (detector) algorithm, whichsegments moving objects from the scene. A detected object issent to the classifier, which determines the object type andsends the aggregated information to the correspondencealgorithm, which temporally tracks the object. Thecorrespondence information is then sent back to the classifier,which adjusts its classification decision based on a sequence ofindividually classified and segmented images of the sameobject. The classifier in turn gives its classifications to thedetector, which uses the information in its adaptation process.Sections IV, V, and VI respectively describe the motiondetection, classification and correspondence algorithms. Thedescriptions of the surveillance algorithms include key resultsand evaluations of the techniques. In order to accomplish thesurveillance mission, the CyberScout has to be able to navigateto a specified geographic location and also be able to adjust itsposition based on feedback from the surveillance algorithms.Section VII describes a novel path-planning system (CPAD)for navigation of multiple CyberScouts in a dynamicenvironment, and Section VIII gives conclusions.Distributed Surveillance and ReconnaissanceUsing Multiple Autonomous ATVs: CyberScoutMahesh Saptharishi, C. Spence Oliver, Christopher P. Diehl, Kiran S. Bhat, John M. Dolan, AshiteyTrebi-Ollennu and Pradeep K. KhoslaCameraDetectorClassifierCorrespondenceAgentsTrackingPan/TiltEvent AnalysisUser Interface,and PlanningFrameDetectionsFalse AlarmsDetections,ClassificationsCorrespondencesCorrespondences, Classifications,Object behaviors and actionsMosaic IndexMotion CommandMotion vectorsCameraDetectorClassifierCorrespondenceAgentsTrackingPan/TiltEvent AnalysisUser Interface,and PlanningFrameDetectionsFalse AlarmsDetections,ClassificationsCorrespondencesCorrespondences, Classifications,Object behaviors and actionsMosaic IndexMotion CommandMotion vectorsFig. 1 Surveillance System ArchitectureIEEE Transactions on Robotics and Automation: Special Issue on Multi-Robot Systems Paper:H2001-122MRS 2II. BACKGROUNDIn the past decade there has been considerable effort indeveloping autonomous robotic vehicles for random patrols,barrier assessment, intruder detection, building or terrainmapping, explosives neutralization, and reconnaissance andsurveillance. Mobile robotic platforms with the abovecapabilities improve the ability to counter threats, limit risks topersonnel, and reduce manpower requirements in hazardousenvironments. The ultimate goal of the CyberScout Project atCarnegie Mellon University’s Institute for ComplexEngineered Systems is to develop mobile robotic technologiesthat extend the sphere of awareness and mobility of anindividual or group performing such