Monitoring Streams- A New Class of Data Management ApplicationsTable of contentsIntroduction ScenarioAuto Service DatabaseScenario SummaryComparisonSo…Implementation - triggerImplementation - middlewareImplementation - AuroraSystem model of AuroraSlide 12Aurora Query SemanticsOperators DiscussionQuery modelOptimizationRunTime OperationWhole Structure RevisitedAurora from AboveRuntime Operation Scheduling: Minimizing Per Tuple Processing OverheadPerformanceDisucssionDiscussionThe final ResultGraphical InterfaceGUI for an ExampleCritiqueConclusionExtra on AuroraTHANK YOU!Monitoring Streams- A New Class of Data Management Applications Presented by Qing Cao at CS@UVA2/30Table of contentsIntroductionAurora System ModelAurora Optimization Real-Time Operation DetailsCritiqueConclusionDiscussion throughout the talk3/30Introduction ScenarioRFID taggedComponentsArmed various sensorsRPM, temperature, pressure, oil status, …Pressure SensorBrightness SensorUser ID and Status4/30Auto Service Database4G WirelessNetworkService centerGPSRepair centerHomevisit serviceNotify Instead ofQuery5/30Scenario SummaryData Streams rather than Static DataParadigm shift from HADP to DAHPCan traditional Database be used to ha-ndle this kind of scenarios?According to the authors, NO!6/30Comparison MonitoringApplicationTraditionalDBMSTypical modelData ActiveHuman PassiveData PassiveHuman ActiveManaging History of valuesrequiredVery hard or inefficientApproximate query resultrequired Not supportedReal-time requirementrequired Not supported7/30So…Quote: The primary goal of the Aurora project is to build a single infrastructure that can efficiently and seamlessly meet the requirements of such demanding applications. To this end, we are currently critically rethinking many existing data management and processing issues, as well as developing new proactive data processing concepts and techniques.8/30Implementation - triggerDataStreamOutput???DBMSDataSubmitterMessagingSystemsQuery registerCHALLENGECHALLENGECHALLENGECHALLENGECHALLENGETrigger: they are not scalableData stream: Not in RealTimeUpdate query: millions update in short time burstQuery management: often update new triggers or queries requested by 3rd partyHistory of values: no scalable way to support latest location of the carCHALLENGEOptimization: Is it helpful doing massive optimization during high load?CHALLENGEQoS: can not ensure service for premium customers9/30Implementation - middlewareDataStreamOutput???DBMSDataSubmitterMessagingSystemsQueryregisterqueryQueryProcessorCHALLENGECHALLENGECHALLENGECHALLENGEQoS: can not ensure service for premium customersQuery management: has to use new query languageData stream: sometimes lost or delivered latelyHistory of values: no scalable way to find latest location of the carOptimization: Can not benefit from query optimizationUpdate query: millions update in short time burstCHALLENGECHALLENGEResource usage: are we efficiently using the system?CHALLENGE10/30Implementation - AuroraDataStreamOutputDBMSDataSubmitterMessagingSystemsQueryregisterCHALLENGEqueryQueryProcessorCHALLENGECHALLENGECHALLENGECHALLENGEQoS: can not ensure service for premium customersQuery management: has to use new query languageData stream: sometimes lost or delivered latelyHistory of values: no scalable way to find latest location of the carOptimization: Can not benefit from query optimizationUpdate query: millions update in short time burstCHALLENGEData stream: new stream processing architectureUpdate queries: new stream processing architectureHistory of the values: new stream processing architectureOptimization: run-time optimizationQuery management: intuitive stream algebra and GUIQoS: specified by application administrator &load sheddingCHALLENGEResource usage: are we efficiently using the system?Resource usage: train scheduling & feed back from/to QoS11/30System model of AuroraExternaldata sourceUser applicationOperatorboxesdata flowContinuous& ad hoc queriesHistoricalStorageAuroraSystemQoS specQuery specApplicationadministrator12/30Implementation - AuroraDataStreamOutputBuffer managerStorageManagerPersistent StoreQ1Q2QmQ1Q2QnSchedulerLoadShedderQoSMonitorCatalogBox ProcessorsσμRouterinputsoutputs13/30Aurora Query SemanticsTraditionalStructured Query LanguageDeclarative query on static dataAuroraData flow model for data streamApplication manager will construct queries using GUIStream Query AlgebraQueries are processed by SQuAl operators on the data stream14/30Operators DiscussionSlideTumbleLatchResampleFilterDropMapGroupByMAP+GROUPBY = CASE15/30Query modelb1 b2 b3b4b5 b6b7b8 b9appappQoS specQoS specQoS speccontinuous queryviewad-hoc queryConnectionpointStorage16/30OptimizationDynamic continuous query optimizationInserting projectionsCombining boxesReordering boxesAd hoc query optimization1st stage : replace implementation (Filter/Join)2nd stage : same as continuous query17/30RunTime OperationQoS Data StructureStorage ManagementReal-time SchedulingLoad Shedding18/30Whole Structure RevisitedDataStreamOutputBuffer managerStorageManagerPersistent StoreQ1Q2QmQ1Q2QnSchedulerLoadShedderQoSMonitorCatalogBox ProcessorsσμRouterinputsoutputs19/30 Aurora from Above...AppQoS...AppQoS.........AppQoS20/30Runtime OperationScheduling: Minimizing Per Tuple Processing OverheadTrain Scheduling:AB… xyzA (x)A (y)A (z) B (A (x))B (A (y))B (A (z)) = Scheduler ActionAB… xyzB (A (x))B (A (y))B (A (z))Box Trains:AB… xyzA (z, y, x)B (A (z), A (y), A (x))Tuple Trains:21/30Performance22/30DisucssionSolution approachRethink about everything for the requirementsQuery modelData flow style query specification and QoSOptimizationDynamic runtime optimizationTrain schedulingQoS specification based resource management23/30DiscussionCan it works in a distributive manner?Aurora projectWhat is the final result?After intensive searching of the tens of papers published on this subject, I finally finds what was implemented:24/30The final ResultThe Aurora stream-processing engine. Aurora is currently operational. It consists of some 100K lines of C++ and Java and runs on both Unix- and Linux-based platf-orms.25/30Graphical Interface26/30GUI for an Example27/30CritiqueThe overall approaches lacks in novelty, e.g. stream operators are ad-hoc.The overall result is not impressing. The