Introduction to Predictive LearningOUTLINE of Set 11.1.1 Subject Matter(cont’d) Many old men are baldConceptual IssuesBeliefs vs True Theories1.1.2 Philosophical ConnectionsChallenge of Predictive LearningBackground: philosophySlide 10Slide 11Slide 121.1.3 Prerequisites and Hwk1Homework 1(cont’d) Homework 1Slide 161.1.4 Expected Outcomes (of this course)Other Related FieldsSlide 191.2.1Handling Uncertainty and RiskGods, Prophets and ShamansHandling Uncertainty and Risk(2)Cultural and Psychological AspectsHuman GeneralizationSlide 251.3.1 Scientific KnowledgeScientific KnowledgeKnowledge Discovery in Digital AgeReality: many questionable studiesScientific Data Mining: Kepler’s LawsKepler’s LawsKepler’s Laws vs. ‘Lady Gaga’ knowledge1.3.2Growth of empirical knowledge1.3.3 Nature of human knowledgeMore on Empirical KnowledgeRepresentation of Empirical KnowledgeExample: Polynomial Curve FittingModeling Assumptions1.4 General Experimental ProcedureHonest Disclosure of Results1Introduction to Predictive LearningElectrical and Computer EngineeringLECTURE SET 1INTRODUCTION and OVERVIEW2OUTLINE of Set 11.1 Overview: what is this course about:- subject matter- philosophical connections- prerequisites and HW1- expected outcomes of this course1.2 Historical Perspective1.3 Motivation for Empirical Knowledge1.4 General Experimental Procedure for Estimating Models from Data31.1.1 Subject MatterUncertainty and Learning•Decision making under uncertainty•Biological learning (adaptation)(examples and discussion)•Induction, Statistics and PhilosophyEx. 1: Many old men are baldEx. 2: Sun rises on the East every day4(cont’d) Many old men are bald•Psychological Induction:- inductive statement based on experience- has certain predictive aspect- no scientific explanation•Statistical View:- the lack of hair = random variable- estimate its distribution (depending on age) from past observations (training sample)•Philosophy of Science Approach:- find scientific theory to explain the lack of hair- explanation itself is not sufficient- true theory needs to make non-trivial predictions5Conceptual Issues•Any theory (or model) has two aspects: 1. explanation of past data (observations) 2. prediction of future (unobserved) data•Achieving both goals perfectly not possible•Important issues to be addressed: - quality of explanation and prediction- is good prediction possible at all ?- if two models explain past data equally well, which one is better?- how to distinguish between true scientific and pseudo-scientific theories?6Beliefs vs True TheoriesMen have lower life expectancy than women•Because they choose to do so•Because they make more money (on average) and experience higher stress managing it•Because they engage in risky activities •Because …..Demarcation problem in philosophy71.1.2 Philosophical Connections•Oxford English dictionary:Induction is the process of inferring a general law or principle from the observations of particular instances.•Clearly related to Predictive Learning.•All science and (most of) human knowledge involves induction•How to form ‘good’ inductive theories?8Challenge of Predictive Learning•Explain the past and predict the future9Background: philosophy William of Ockham: entities should notbe multiplied beyond necessityEpicurus of Samos: If more than one theory is consistent with the observations, keep all theories10Background: philosophyThomas Bayes: How to update/ revise beliefs in light of new evidenceKarl Popper: Every true (inductive) theory prohibits certain events or occurences, i.e. it should be falsifiable11Background: philosophyGeorge W. Bush: I am The Decider12Background: philosophyBill Clinton: I told the Truth131.1.3 Prerequisites and Hwk1•Math: working knowledge of basic Probability + Linear Algebra• Statistical software (of your choice): - MATLAB, also R-project, Mathematica etc.Note: you will be using s/w implementations of learning algorithm (not writing programs)•Writing: no special requirements•Philosophy: no special requirements14Homework 1•Purpose: testing background on probability and computer skills• Goal: estimate pdf of a random variable X•Real Data: X=daily price changes of SP500i.e. where Z(t) = closing price •Typical + Useful Statistics- Histogram (empirical pdf)- mean, standard deviation%100)1()1()()( tZtZtZtX15(cont’d) Homework 1Histogram = estimated pdf (from data)•Example: histograms of 5 and 30 bins to model N(0,1) also mean and standard deviation (estimated from data) 0100200300400500-3 -2 -1 0 1 2 3 4020406080100-3 -2 -1 0 1 2 3 416(cont’d) Homework 1NOTE: histogram ~ empirical pdf, i.e. y-axis scale is in % (frequency). Example: histogram of SP500 daily price changes in 1981:171.1.4 Expected Outcomes (of this course)Scientific/technical: •Learning = generalization, concepts and issues•Math theory: Statistical Learning Theory aka VC-theory•Implications for Philosophy and ApplicationsPhilosophical:•Nature of human knowledge and intelligence•Demarcation principle•Human learningPractical:•Financial engineering•Security•Genomics•Predicting successful marriage, climate modeling etc., etc.What is this course NOT about181818Other Related Fields•The field of Pattern Recognition is concerned with the automatic discovery of regularities in data.•Data Mining is the process of automatically discovering useful information in large data repositories.•This book (on Statistical Learning) is about learning from data.•The field of Machine Learning is concerned with the question of how to construct computer programs that automatically improve with experience.•Artificial Neural Networks perform useful computations through the process of learning. (1) unnecessary fragmentation  confusion(2) all fields estimate useful models from data, i.e. extract knowledge from data (the same as in classical statistics)Real Issues: what is ‘useful’? What is ‘knowledge’?19OUTLINE of Set 11.1 Overview: what is this course about1.2 Historical PerspectiveHandling uncertainty and risk:- probabilistic vs - risk minimization1.3 Motivation for Empirical Knowledge1.4 General Experimental Procedure for Estimating Models from Data201.2.1Handling Uncertainty and Risk•Ancient times•Probability for quantifying uncertainty- degree-of-belief- frequentist (Cardano-1525, Pascale, Fermat)•Newton and causal determinism•Probability theory and statistics (20th