Introduction to Data Quality Monica Scannapieco Department of Computer Engineering Universit di Roma La Sapienza Italy Who I am Research Associate Computer Engineering Department Univ Roma La Sapienza Italy Web page http www dis uniroma1 it monscan Email monscan dis uniroma1 it 2 Outline Introduction What is data quality How to assess quality of data Data quality issues in modern ISs CISs Conclusions 3 Motivations In the last decade increasing demand for Integration Analysis Exchange of very large data bases Critical Problem Data quality 4 Cont Current data quality problems cost U S businesses more than 600 billion dollars a year Data Warehouse Institute Between the 30 to 80 of the data analysis task is spent on cleaning and understanding the data Data mining practioners survey E Government initiatives addressing data quality issues European directive 2003 98 CE on the reuse of public data USA Data Quality Act 5 Data Quality Generically defined as fitness for use It is a complex concept resulting from the composition of various characteristics or dimensions Standard set of dimensions not yet defined though the community agrees on a common minimal set 6 Data Quality Multidimensional Concept Completeness Accuracy Jhn vs John Currency Residence Permanent Address outdated vs updated Consistency ZIP Code and City consistent Prefix StreetName Number ZipCode City Via Salaria 113 00198 Roma Prefix StreetName Number ZipCode Salaria City Attribute Completeness Roma Prefix StreetName Number ZipCode City Via Salaria 113 00198 Roma Via Gracchi 74 00193 Roma Prefix StreetName Number ZipCode City Via Salaria 113 00198 Roma Entity Completeness 7 Example ID Title Director Year Remakes LastRemakeYear 1 Casablanca Weir 1942 3 1940 2 Dead Poets Society Curtiz 1989 0 NULL 3 Rman Holiday Wylder 1953 0 NULL 4 Sabrina NULL 1964 0 1985 8 Cont Movies table with the cells with data quality problems that are shadowed At a quick look only the cell corresponding to the title of movie 3 is wrong i e there is a misspelling in the title accuracy error But many other errors are present 9 Cont Swap in the directors of movies 1 and 2 also occurred accuracy error Missing value for the director of movie 4 completeness error For movie 4 a remake was actually made in 1985 therefore the 0 value for the Remakes attribute is outdated currency error For movie 1 the value of LastRemakeYear cannot be lower than Year also for movie 4 the value of LastRemakeYear and Remakes are contradicting consistency errors 10 Accuracy Can be evaluated for disparate granularity levels of a data model ranging from single values to entire databases For single data values accuracy measures the distance between a value v and a value v which is considered correct Two kinds of accuracy can be identified syntactic accuracy and semantic accuracy 11 Syntactic Accuracy Syntactic accuracy is measured by means of comparison functions that evaluate the distance between v and v Edit distance is a simple example of comparison function taking into account the cost of converting a string s to a string s through a sequence of character insertions deletions and replacements The accuracy error of movie 3 on the Title value is a syntactic accuracy error As the correct value for Rman Holidays is Roman Holidays the edit distance between the two values is equal to 1 12 Semantic Accuracy Semantic accuracy captures the cases in which v is a syntactically correct value but it is different from v In the movie example swapping the directors names for movies 1 and 2 results in a semantic accuracy error Director named Weir for movie 1 is syntactically correct but he is not the director of Casablanca 13 Completeness Completeness can be generically defined as the extent to which data are of sufficient breadth depth and scope for the task at hand Wang 1996 Three types of completeness Schema completeness the degree to which entities and attributes are not missing from the schema Column completeness a function of the missing values in a column of a table Population completeness evaluates missing values with respect to a reference population 14 Cont If considering a specific data model more detailed characterization Example relational data model with null values Different possible meanings for null values Different completeness characterizations 15 Cont Null Values Meaning ID Name Surname BirthDate Email 1 John Smith 03 17 1974 smith abc it 2 Edward Monroe 02 03 1967 NULL 3 Anthony White 01 01 1936 NULL 4 Marianne Collins 11 20 1955 NULL Not Existing Existing But Unknown Not Known If Existing Tuple 2 no incompleteness Tuple 3 incompleteness Tuple 4 possible incompleteness 16 Cont Completeness of relational model elements Value completeness captures the presence of null values for some attributes of tuples Tuple completeness characterizes the completeness of a whole tuple with respect to the values of all attributes Attribute completeness measures the number of null values of a specific attribute in a relation Relation completeness captures the presence of null values in the whole relation But if more tuples should have been present in the relation Open World Assumption things are more difficult 17 Cont Completeness of relational model elements relation value attribute tuple 18 Time Related Dimensions Some data are stable in time birth date surnames eye color etc Some data vary in time ages addresses salaries etc Three time related dimensions Currency Timeliness Volatility 19 Cont Currency Currency measures if or the degree to which data are updated In the movies example Remakes of movie 4 is not current because a remake of the movie 4 had been performed but this information did not result in an increased value If a residence address of a person is updated i e it actually corresponds to the address where the person lives then it is current 20 Cont Volatility Volatility measures the frequency according to which data vary in time Stable data such as birth dates have the lowest value in a given metric scale for volatility as they do not vary at all Stock quotes have a high volatility values 21 Cont Timeliness Timeliness measures how current data are relative to a specific task If considering a timetable for university courses it can be current thus containing the most recent data but it can be not timely if it only becomes available after the start of lessons 22 Consistency The consistency dimension captures the violation of semantic rules defined over a set of data items With