Real Time Detection of Groceries ProposalTess WinlockDepartment of Computer Science and EngineeringUniversity of California San DiegoLa Jolla, CA [email protected] problem of detecting objects in a scene with limitedprior data is becoming a more well explored area of com-puter vision. The GroZi project aims at allowing a blinduser to independently navigate a grocery store and collectthe items on their grocery list. Using a shoulder mountedcamera the project aims to detect objects and direct the userto the object. There are several major problems presentedin trying to detect objects in such scenes: occlusion, mo-tion blur, focus, and pose to name a few. This problem hasbeen explored previously by [3], where they explored de-tecting objects in a grocery scene using images collectedfrom ’ideal’ situations. This projects aims to continue thiswork, applying detection algorithms for real time recogni-tion of products from a shopping list. I intend on taking4 promising algorithms and comparing their performanceacross a large number of items and video to determine ifthere is one algorithm which performs best, or if a combina-tion of approaches need to be undertaken. After comparingthe performance of various detection algorithms individu-ally on various scenes, I plan on applying the classifiers inparallel to try to detect which of the desired objects exist ina given image. This will rely heavily on which algorithmswill have been found to be best for this problem.1. QualificationsTess Winlock is a second year masters student, and iscurrently working on the GroZi project. I have previouslycompleted two graduate level classes in machine learning,and a course in computer vision. In my undergrad I workedon a project to build a cheap device to assist those withcerebral palsy communicate. I have also completed 2 in-ternships at Google working on a team that uses machinelearning to detect payment fraud on Google Checkout andGoogle AdWords. Further, Though unrelated during my un-dergraduate career I studied human navigation in a virtualreality environment. Overall this has given me a good back-ground in machine learning and working with the disabled.2. Project OutlineThe final goal of this project is to be able to accuratelydetect the location of one object from a shopping list in animage provided from a video. The algorithms that I plan onexploring are:• SURF - I currently use SURF [1] to match keypointsin the training image to those in the scene, and then tryto find a homography for planar objects.• Boosted HAAR-like Features - Using the Viola Jonesmethod [5] explored by Merler’s and Galleguillos onthis dataset, I intend on expanding their approach bynot only training the entire object, but also on detect-ing sub features (like the brand label) once the locationof the object has been detected, and then applying ge-ometric constraints to these.• Ferns - Ozuysal [4] has used ferns to supposedlyquickly recognize keypoints in a scene, it seems to bean interesting approach that may compete with SURF.• One Way Descriptors [2] - Another real time approachwhich attempts to match patches taken from keypointsin the input image to those in the scene, shows promiseon being quick and providing a homography upon de-tection.I intend on using training data from the GroZi-120 dataset tocompare the performance of all these algorithms, and thencompare their performance across the products. The criteriafor the final algorithm will be heavily driven by these find-ings, as well as the speed of detection. For our final productwe will need to detect multiple objects at once, this couldrequire several classifiers to run in parallel, which meansperformance is a non-trivial concern. Application of theseclassifiers in parallel will rely heavily on the results of thefirst milestone, there may not be a single algorithm whichperforms best under all situations. Thus we may need torun a classifier capable of detecting an object’s presence,and then running the more exhaustive detector. Once thishas been built, the final test will be run on the completeGroZi-120 dataset with randomly generated shopping listsof 10 items.2.1. Milestones1. Week 2: Application of all algorithms - I am currentlyworking on this section of the project, so I hope to beable to apply all algorithms to the data by the end ofthe second week of the quarter.2. Week 5: Comparison of performance of 3 different al-gorithms - Once I have the various approaches com-plete, I intend on testing their performance on the com-plete GroZi-120 dataset. I may also expand this on tovideos that show the products from angles other thanfronto parallel3. Week 8: Development of Parallel Algorithm - Oncethe algorithm(s) have been selected I can try to detectone of many objects in a scene.4. Week 10: Test of Complete application - If/when theparallel approach to application has been determined Iwill run the application on the GroZi-120 dataset witha shopping list of 10 items.3. Questions• Which algorithm shows the best performance at recog-nizing items?• Will one algorithm work for all items or is there somecriterion for using different approaches for differentitems (say for planar objects vs. cylindrical objects)• What is should be used to run these algorithms in par-allel?• How to pass to the detection data to tracker?4. SoftwareI am currently developing the detection algorithms inOpenCV, as they have many useful algorithms already im-plemented. OpenCV has implementations of SURF, Cas-cade classifiers using HAAR, and the OneWayDescriptor. Iplan on continuing to use this throughout the duration of theproject.5. DataI will use video shot in the old UCSD sunshine storesas my primary data set, and possibly videos from Von’s ifwe get permission to shoot new videos. The GroZi-120dataset will provide a good base for comparing the differ-ent algorithm’s performance under ideal conditions, but wewill likely need to expand the data to include perspectivesthat are not just fronto parallel. This will require some sig-nificant gathering and labeling of videos with the variousvisible products of interest. To accommodate this I havea simple program, written using OpenCV, that allows formultiple labeling of objects in a video.References[1] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-uprobust features (surf). 110(3):346–359, June 2008. 1[2] S. Hinterstoisser, O. Kutter, N. Navab, P. Fua, and V. Lepetit.Real-time learning of accurate