Mini-course on Artificial Neural Networks and Bayesian NetworksSection 1: IntroductionNetworks (1)Example: collaboration networkNetworks (2)Networks (3)OutlineMotivationQuestionsHistory of (modern) ANNs and BNsSection 2: On-line LearningSection 2.1: The PerceptronThe PerceptronPerceptron: binary outputLearning a linearly separable rule from reliable examplesLearning a linearly… (Cont.)Off-line learningOn-line trainingOn-line training - Statistical Physics approachGeneralizationGeneralization (cont.)Geometric argumentOverlap ParametersDerivation for large NCentral Limit TheoremCentral Limit Theorem (Cont.)Generalization ErrorAssumptions about the dataHebbian learning (revisited) Hebb 1949Hebb: on-lineHebb: on-line (Cont.)Slide 32Slide 33Hebb: on-line mean valuesLearning Curve:  dependent of the order parametersSlide 36Slide 37Asymptotic expansion [draw w. matlab]Questions:Modified Hebbian learningPerceptron Rosenblatt 1959PerceptronOn-line dynamics Biehl and Riegler 1994Slide 44Learning Curve - Hebb and PerceptronSection 2.2: On-line by gradient descentIntroductionLinear perceptron and linear regression (1D)Simple case: ‘Linear perceptron’‘Linear perceptron’ (cont.)Adatron: binary output J() = J(-1) +f(…) ST/NMultilayered feed-forward NNMultilayered ff NN (cont.)Teacher-Student scenarioThe error measureOn-line gradient descentAssumptions and definitionsOrder parameters roleGeneralization error: erf function Saad & Solla 1995Permutation SymmetryA simple caseUpdate of the order parametersDifferential EquationsLearning curvesSection 3: Unsupervised learningSlide 66Potential aimsA simple exampleA simple example (cont)Student scenarioPCA: General setting [matlab]Principle Component AnalysisExample: visionExample: vision (cont)First nine eigen facesDimensionality ReductionSection 4: Bayesian NetworksBayesian StatisticsProbability as Degree of BeliefThe Asia problemGraphical modelsDirected acyclic GraphsDirected Graphical Models (2)Classification problemClassification: assigning labels to dataDensity estimatorDirected graph: ‘real world’ exampleGoalPrevious topic-based modelsSlide 90ClassificationTopics Model for Semantic RepresentationThe DRM ParadigmExample: test of false memory effects in the DRM ParadaigmA Rational Analysis of Semantic MemoryA Spatial Representation: Latent Semantic Analysis (Landauer & Dumais, 1997)Triangle Inequality constraint on words with multiple meaningsA generative model for topicsA toy exampleAll probability to topic 1…All probability to topic 2…Application to corpus dataFitting the modelGibbs Sampling & MCMC see Griffiths & Steyvers, 2003 for detailsA selection from 500 topics [P(w|z = j)]Polysemy: words with multiple meanings represented in different topicsPredicting word associationWord Association (norms from Nelson et al. 1998)P( set contains first associate )Explaining variability in false recallOne recall component: inferencePredictions for the “Sleep” listCorrelation between intrusion rates and predictionsOther recall components??? One possibility: two routes add strengthMini-course on Artificial Neural Networks and Bayesian Networks Michal Rosen-ZviMini-course on ANN and BN, The Multidisciplinary Brain Research center, Bar-Ilan University, May 2004 ד״סשת ןליא ־רב תטיסרבינוא חומה רקחל זכרמה תותשרב ברה ימוחת זכורמ סרוקSection 1: Introduction Mini-course on ANN and BN, The Multidisciplinary Brain Research center, Bar-Ilan University, May 2004 ד״סשת ןליא ־רב תטיסרבינוא חומה רקחל זכרמה תותשרב ברה ימוחת זכורמ סרוקNetworks (1)Networks serve as a visual way for displaying relationships: Social networks are examples of ‘flat’ networks where the only information is relation between entitiesMini-course on ANN and BN, The Multidisciplinary Brain Research center, Bar-Ilan University, May 2004 ד״סשת ןליא ־רב תטיסרבינוא חומה רקחל זכרמה תותשרב ברה ימוחת זכורמ סרוקExample: collaboration networkMini-course on ANN and BN, The Multidisciplinary Brain Research center, Bar-Ilan University, May 2004 ד״סשת ןליא ־רב תטיסרבינוא חומה רקחל זכרמה תותשרב ברה ימוחת זכורמ סרוק1. Analyzing Cortical Activity using Hidden Markov ModelsItay Gat, Naftali Tishby, and Moshe Abeles"Network, Computation in Neural Systems", August 1997.2. Cortical Activity Flips Among Quasi Stationary StatesMoshe Abeles, Hagai Bergman, Itay Gat, Isaac Meilijson, Eyal Seidemann, Naftali Tishby, Eilon VaadiaPrepared: Feb 1, 1995, Appeared in the Proceedings of the National Academy of Science (PNAS)3. Rigorous Learning Curve Bounds from Statistical MechanicsDavid Haussler, Michael Kearns, H. Sebastian Seung, and Naftali TishbyPrepared: July 1994. Full version, Machine Learning (1997). 4. H. S. Seung, Haim Sompolinsky, Naftali Tishby: Learning Curves in Large Neural Networks. COLT 1991: 112-1275. Yann LeCun, Ido Kanter, Sara A. Solla: Second Order Properties of Error Surfaces. NIPS 1990: 918-9246. Esther Levin, Naftali Tishby, Sara A. Solla: A Statistical Approach to Learning and Generalization in Layered Neural Networks. COLT 1989: 245-2607. Litvak V, Sompolinsky H, Segev I, and Abeles M (2003) On the Transmission of Rate Code in Long Feedforward Networks with Excitatory-Inhibitory Balance. Journal of Neuroscience, 23(7):3006-30158. Senn, W., Segev, I., and Tsodyks, M. (1998). Reading neural synchrony with depressing synapses. Neural Computation 10: 815-8198. Tsodkys, M., I.Mit'kov, H.Sompolinsky (1993): Pattern of synchrony in inhomogeneous networks of oscillators with pulse interactions. Phys. Rev. Lett., 9. Memory Capacity of Balanced Networks (Yuval Aviel, David Horn and Moshe Abeles)10. The Role of Inhibition in an Associative Memory Model of the Olfactory Bulb. (Ofer Hendin, David Horn and Misha Tsodyks)11 Information Bottleneck for Gaussian Variables Gal Chechik, Amir Globerson, Naftali Tishby and Yair WeissPrepared: June 2003. Submitted to NIPS-2003 [matlab]Networks (2)Artificial Neural Networks represent rules – deterministic relations - between input and outputMini-course on ANN and BN, The Multidisciplinary Brain Research center, Bar-Ilan University, May 2004 ד״סשת ןליא ־רב תטיסרבינוא חומה רקחל זכרמה תותשרב ברה ימוחת זכורמ סרוקNetworks