MMDSS 2007 Data stream management and miningOutlineWhat is a data stream ?What is a data stream ?OutlineApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingApplications of data stream processingOutlineModels for data streamsModels for data streamsModels for data streamsModels for data streamsModels for data streamsModels for data streamsModels for data streamsModels for data streamsModels for data streamsModels for data streamsOutlineDSMS outlineDSMS: definitionDSMS outlineDSMS: data modelDSMS: data modelDSMS outlineDSMS: queriesDSMS: queriesDSMS: queriesDSMS: queriesDSMS: queriesDSMS: queriesDSMS outlineDSMS: STREAMDSMS: STREAMDSMS: STREAMSlide 45DSMS outlineMain architecture of DSMSMain architecture of DSMSMain architecture of DSMSDSMS outlineApproximate answers to queriesApproximate answers to queriesApproximate answers to queriesApproximate answers to queriesDSMS outlineMain existing DSMSMain existing DSMSMain existing DSMSOutlineData stream mining: outlineData stream mining: definitionData stream mining: definitionData stream mining: definitionData stream mining: definitionData stream mining: outlineData stream mining: decision treeData stream mining: decision treeData stream mining: decision treeData stream mining: outlineData stream mining: additive methodsData stream mining: additive methodsData stream mining: outlineData stream mining: clusteringData stream mining: clusteringData stream mining: clusteringData stream mining: clusteringData stream mining: clusteringData stream mining: clusteringData stream mining: clusteringData stream mining: conclusionOutlineSynopses structuresSynopses structures: random samplesSynopses structuresSynopses structures: sketchesSynopses structures: sketchesSynopses structures: sketchesSynopses structures: sketchesSynopses structures: sketchesSynopses structures: sketchesSynopses structures: sketchesSynopses structures: sketchesOutlineConclusionReferences: generalReferences: DSMSReferences: data stream miningData stream management and mining09/10/2007MMDSS 2007Data stream management and miningSeptember 10th, 2007Georges HébrailENST ParisMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 2Outlinez What is a data stream ?z Applications of data stream managementz Models for data streamsz Data stream management systemsz Data stream miningz Synopses structuresz ConclusionMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 3What is a data stream ?……………15235,020,5223,5216/12/2006-17:2915,8235,680,5283,66616/12/2006-17:2823233,740,5025,38816/12/2006-17:2723233,290,4985,37416/12/2006-17:26……………I 1 (A)U 1 (V)Puis. R (kVAR)Puis. A (kW)Timestampz Golab & Oszu (2003): “A data stream is a real-time, continuous, ordered (implicitly by arrival time or explicitly by timestamp) sequence of items. It is impossible to control the order in which items arrive, nor is it feasible to locally store a stream in its entirety.”z Structured records ≠ audio or video dataz Massive volumes of data, records arrive at a high rateMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 4What is a data stream ?z Golab & Oszu (2003): “A data stream is a real-time, continuous, ordered (implicitly by arrival time or explicitly by timestamp) sequence of items. It is impossible to control the order in which items arrive, nor is it feasible to locally store a stream in its entirety.”z Structured records ≠ audio or video dataz Massive volumes of data, records arrive at a high rate………………ftp80K1816.5.5.819.7.1.212345http58K2614.8.7.412.4.3.812344http24K1612.4.0.318.6.7.112343http20K1216.2.3.710.1.0.212342………………ProtocolBytesDurationDestinationSourceTimestampMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 5Outlinez What is a data stream ?z Applications of data stream processingz Models for data streamsz Data stream management systemsz Data stream miningz Synopses structuresz ConclusionMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 6Applications of data stream processingData stream processingÖ Process queries (compute statistics, activate alarms)Ö Apply data mining algorithms• Real-time processing• One-pass processing• Bounded storage (no complete storage of streams)• Possibly consider several streamsMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 7Applications of data stream processingApplicationsÖ Real-time monitoring/supervision of IS (Information Systems) generating large amountsof data• Computer network management• Telecommunication calls analysis (BI)• Internet applications (ebay, google, recommendation systems,click stream analysis)• Monitoring of power plantsÖGeneric software for applications where basic data is streaming data• Finance (fraud detection, stock market information)• Sensor networks (environment, road traffic, weather forecast, electric power consumption)MMDSS’07 – G.Hébrail – Data stream management and mining – Slide 8Applications of data stream processingGeneric tools for processing dataSpecific developmentwithout databasetechnologyApplications with basic streaming dataQuerying and mining‘on the fly’ (scalable)Data warehouses(unscalable)Monitoring, Business Intelligence applicationsData stream processingtechnologyStandard data processing technologyStandard data processingversus data stream processingMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 9Applications of data stream processingLet’s go deeper into some examplesÖ Network managementÖ Stock monitoringÖ Linear road benchmarkMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 10Applications of data stream processingNetwork managementz Supervision of a computer networkÖ Improvement of network configuration (hardware, software, architecture)Ö Measurements made on routers (Cisco Netflow)Network supervision centerMMDSS’07 – G.Hébrail – Data stream management and mining – Slide 11Applications of data stream processingNetwork managementÖ Information about IP sessions going through a routerÖ Huge