Data Quality Data Integration Monica Scannapieco Department of Computer Engineering Universit di Roma La Sapienza Italy Summary of Previous Lesson We have learnt what is data quality in terms of its major dimensions that are however a subset of larger possible sets that can take into account domain specific features some elements on how assessing quality of data in general contexts data quality is critical in modern information systems like CISs 2 In this lesson Specific data quality problems in data integration systems Focus on instance conflicts resolution Some techniques solving dq problems in data integration systems 3 Cooperative Information Systems CISs abstracting DISs Different Approaches Data Integration techniques Agent based methodologies Process Coordination and Servicebased systems 4 Data Integration Data integration is the problem of combining data stored by different sources and providing the user with a unified view of this data Two main approaches to data integration namely Materialized data integration where the unified view of data is materialized for instance in a data warehouse Virtual data integration where the unified view is virtual and data resides only at sources 5 Basics on Data Integration 6 Virtual Data Integration System Answer Q Query Global Schema Sources Three elements a global schema A source schema including schemas of all sources A mapping between the global schema and the source schema 7 Cont Sources are distributed independent and heterogeneous and in general out of the control of the data integration system A mapping holds the relationships between the sources and the unified view of them When the user issues queries the system carries out the task of extracting data from the sources and reassembling them into the answer 8 Approaches to DI Two basic approaches have been proposed to specify the mapping Global as View GAV requires that the global schema is expressed in terms of the data sources Easier query processing Local as View LAV requires that each data source is expressed as a view over the global schema New data sources easier integrated GLAV approach combines the GAV and LAV approaches and is such that which queries over the sources are put in correspondence with queries over the global schema 9 Example Global View movie Title Year Director european Director review Title Critique Sources r1 Title Year Director since 1960 european directors r2 Title Critique since 1990 Query Title and critique of movies in 1998 movie T 1998 D review T R 10 Global as View Example Global as view relations in the global view are views over the sources movie T Y D r1 T Y D european D r1 T Y D review T R r2 T R The query mr T R movie T 1998 D review T R is processed by means of unfolding i e by expanding the atoms according to their definitions until we come up with source relations In this case r1 T 1998 D r2 T R 11 Local as View Example Local as view relations at the sources are defined as views over the global view r1 T Y D movie T Y D european D Y 1960 r2 T R movie T Y D review T R Y 1990 The query mr T R movie T 1998 D review T R is processed by means of an inference mechanism aiming at re express the atoms of the global view in terms of atoms at the sources In this case r1 T 1998 D r2 T R 12 Data Quality Issues 13 Data Replication Copies of the same data are stored by different organizations or within a single organization Problem data inconsistency 14 Data Inconsistency DB B DB A ID Name Lastname Telephone Num SSN B1 Marie Gold 999555 3222 B2 John Smith 222444 Out of Dateness Name Lastname BirthYear SSN ID Marie Gold 2000 3322 A1 Jhn Smith 1974 xxxy A2 Internal Inaccuracy Incompleteness Inconsistency 15 Data Inconsistencies in CISs Why a problem Organizations do not trust each other if providing inconsistent data lack of cooperation Users of the CISs do not trust organizations providing mutually inconsistent data lack of data service requests Uncontrolled spread of low quality data in the system 16 The Anatomy of Conflicts in DISs Schema level conflicts Instance level conflicts 17 Schema level conflicts 1 Longly studied here a summary Heterogeneity conflicts different data models are used e g XML and relational Semantic conflicts relationship btw model level extensions e g the Person class may have different extensions in different sources disjoint partially overlapping completely overlapping including 18 Schema level conflicts 2 Description conflicts concepts described with different attributes e g different formats different attribute types different scaling etc Structural conflicts different design choices within the same model e g Address represented in one source as a class and in another as an attribute 19 Instance level conflicts 1 Our focus in the next slides few research results so far On the basis of the model element granularity attribute conflicts key conflicts also called entity or tuple conflicts relationship conflicts 20 Instance level conflicts 2 Given two sources S1 and S2 let A1k and A2k represent the same attribute of a real world object represented as t1 in S1 and as t2 in S2 Attribute Conflicts An attribute conflict occurs if t1 A1k t2 A2k Key Conflicts Let us suppose that A1k is primary key for t1 and A2k is primary key for t2 A Key conflict occurs iff t1 A1k t2 A2k and t1 A1i t2 A2i for all i k 21 Instance level conflicts 3 Relationship Conflict Let us suppose that A1k is primary key for t1 and A2k is primary key for t2 Also both S1 and S2 are involved in a relationship R with a third source S3 A relationship conflict occurs if given t1 in relationship R with t3k in S3 i e t1 R t3k and given t2 in relation R with t3h in S3 i e t2 R t3h then t3k t3h 22 Instance Level Conflicts 4 EmployeeID Name Surname Salary Email arpa78 John Smith 2000 smith abc it eugi98 Edward Monroe 1500 monroe abc it ghjk09 Anthony Wite 1250 white abc it treg23 Marianne Collins 1150 collins abc it Attribute conflict EmployeeS1 Key conflict EmployeeID Name Surname Salary Email arpa78 John Smith 2600 smith abc it eugi98 Edward Monroe 1500 monroe abc it ghjk09 Anthony White 1250 white abc it dref43 Marianne Collins 1150 collins abc it EmployeeS2 23 Techniques for Instance Conflicts Resolution Key conflicts require the application of object identification techniques Record Matching techniques Attribute and relationship conflict are solved by Query Time techniques we will see two QT techniques 24 1st Challenge Conclusions Record Matching can be perfomed to