1CS 559: Machine LearningCS 559: Machine Learning Fundamentals and Applications1stSet of NotesInstructor: Philippos MordohaiWebpage: www.cs.stevens.edu/~mordohaiEmail:Philippos Mordohai@stevens eduE-mail: [email protected]: Lieb 215Pattern Classification, Chapter 1ObjectivesObjectives•Hands-on experience with fundamentalHandson experience with fundamental algorithmsUseful for everyday problems–Useful for everyday problems• Exposure to state of the art in machine learning and pattern recognitionPattern Classification, Chapter 1 2LogisticsLogistics•Office hours: Tuesday 5-6 and by emailOffice hours: Tuesday 56 and by email• Evaluation:4 h k t (20%)–4 homework sets (20%)– Project (25%)–Pop-up quizzes and participation (10%)– Midterm (20%)–Final exam (25%)Pattern Classification, Chapter 1 3ProjectProject•Pick topic around middle of the semesterPick topic around middle of the semester• I will suggest ideas and datasets in next lectureslectures• Deliverables:–Project proposal– Presentation in class– Poster in CS department event– Final report (around 8 pages)Pattern Classification, Chapter 1 4Project ExamplesProject Examples•Face detection using boostingFace detection using boostingPattern Classification, Chapter 1 5Project ExamplesProject Examples•Detecting bots in Quake 2Detecting bots in Quake 2Pattern Classification, Chapter 1 6Project ExamplesProject Examples•Spam filteringSpam filtering• Gender identification from emailsH d ii ii•Handwriting recognition– Also cool demo, but requires hardwarePattern Classification, Chapter 1 7PrerequisitesPrerequisites•Probability theoryProbability theory • Matlab or C/C++ programming–This could be“language of your choice”butThis could be language of your choice, but then you are responsible for debugging etc.•Some linear algebraSome linear algebra– Must not be afraid of eigenvalues• Your grade will be affected by any weaknesses in theseweaknesses in thesePattern Classification, Chapter 1 8Pattern ClassificationA lot of material in these slides was taken fromPattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley & Sons, 2000with the permission of the authors and the publisherWhat is Pattern Recognition?What is Pattern Recognition?•InformallyInformally– Recognize patterns in dataMore formally•More formally– Assign an object or an event to one of the several prespecifiedcategories(a category isseveral pre-specified categories(a category is usually called a class)Pattern Classification, Chapter 1 10Many of these slides are borrowed from Olga Veksler (U. of Western Ontario)Example: Male or Female?Example: Male or Female?Pattern Classification, Chapter 1 11Example: Photograph or Not?Example: Photograph or Not?Pattern Classification, Chapter 1 12Character RecognitionCharacter Recognition• In this case the classes are all characters in the alphabet digits etcin the alphabet, digits etc.Pattern Classification, Chapter 1 13Speech UnderstandingSpeech Understanding• In this case the classes are all phonemesPattern Classification, Chapter 1 14Machine Learning ResearchMachine Learning Research•Speech recognitionSpeech recognition • Natural language processingCii•Computer vision• Medical outcomes analysis• Robot control•Computational biologyComputational biology• Sensor networksPattern Classification, Chapter 1 15Real-life ApplicationsReallife Applications•Loan applicationsLoan applications• Recommendation systems A N tfli–Amazon, Netflix• Targeted advertising– countless examples…Pattern Classification, Chapter 1 16Chapter 1: Introduction to Pattern Recognition (Sections 1 1-16)Recognition (Sections 1.11.6)•Machine Perception•Machine Perception• An ExamplePtt R iti S t•Pattern Recognition Systems• The Design Cycle• Learning and Adaptation• ConclusionPattern Classification, Chapter 1 17Machine PerceptionMachine Perception• Build a machine that can recognize patterns:gp– Speech recognition– Computer Vision: object recognition, face detection– Fingerprint identificationOCR (O ti l Ch t R iti )–OCR (Optical Character Recognition)– DNA sequence identification Pattern Classification, Chapter 1 18An ExampleAn Example• “Sorting incoming Fish on a conveyorSorting incoming Fish on a conveyor according to species using optical sensing”Sea bassSpeciesSalmonPattern Classification, Chapter 1 19TrainingTraining• Set up a camera and take some sample images –Label these images by handLabel these images by hand• Extract features– Length–Lightness– Width– Number and shape of fins– Position of the mouth, etc…–Test whether this set of features is useful for a classifierPattern Classification, Chapter 1 20PreprocessingPreprocessing• Use a segmentation operation to isolate ff ffishes from one another and from the background•Informationfrom a single fish is sent to a•Information from a single fish is sent to a feature extractor whose purpose is to reduce the data by measuring certainquantitiesthe data by measuring certain quantities•The features are passed to a classifierThe features are passed to a classifier Pattern Classification, Chapter 1 21Pattern Classification, Chapter 1 22ClassificationClassification• Select the length of the fish as a possible feature for discriminationfor discriminationPattern Classification, Chapter 1 23Preliminary ResultsPreliminary Results•Lengthis a poor feature alone!Lengthis a poor feature alone!– About 20% misclassification rate at best threshold choicethreshold choiceSl tli htibl f t•Select lightness as a possible feature.Pattern Classification, Chapter 1 24Pattern Classification, Chapter 1 25The Decision BoundaryThe Decision Boundary– Move the decision boundary toward smaller values of lightness in order to minimize thevalues of lightness in order to minimize the cost (reduce the number of misclassifications)Pattern Classification, Chapter 1 26New ClassifierNew Classifier•Adopt the lightness and add the width ofAdopt the lightness and add the width of the fishFishxT= [x1, x2]LightnessWidthPattern Classification, Chapter 1 27New ClassifierNew ClassifierPattern Classification, Chapter 1 28• We may add other features that are not correlated with the ones we alreadycorrelated with the ones we already have• Intuitively, the best decision boundary hldbth hih idshould be the one which provides an optimal performance such as in the