Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45A Long Now Perspective Of Digital Data PreservationKurt BollackerThe Long Now FoundationODU21 April 02005Outline●The problem of digital data preservation●The Long Now Foundation●Data mobility as an approach●The Long Server project●Brief intro to file format conversion●The Long Server project planWhy Do We Need To Preserve Data?●Survival of our culture makes our lives better.●Collective memory is a socio-evolutionary advantage.Why Do We Need To Preserve Data?●Survival of our culture makes our lives better.●Collective memory is a socio-evolutionary advantage.Why Do We Need To Preserve Data?Why Care About Digital Data?●Most data is or will be digital in the future.Components Of Digital Data Preservation●Preservation of the bits: The bits themselves survive.●Access to the bits: The bits' values are available.●Understanding of the bits: The bits' meaning is preserved.Why Is Long Term Digital Data Preservation Difficult?●There are no direct precedents from which to learn.●The indirect precedent of traditional data preservation is insufficient.The World Of Traditional Data Preservation Has:●Centralized control of maintenance by skilled professionals.●Long lived, expensive media.●Stable language, processes, and standards.●Rare and regularly patterned access requests.●Limited data quantities.●A high cost to move data.●Dependency on manual curation.In The World Of Digital Data:●Central control is hard, especially when scaling.●Preservationists are often amateur and distributed.●Media are short lived and cheap.●Technology grows/dies/mutates rapidly.●Access needs often change and scale rapidly.●Dataset sizes can grow exponentially.●It's cheap and fast to move data.●Manual curation is often impossible.Under/Unused Resources For Digital Archiving.●Cheap, unreliable, capacious storage media.●Cheap bandwidth.●Cheap and easy processing/transformation. (Amortizable!)●Lots of creative and motivated labor available.A Model of Time For Understanding The ProblemA Time Scale Mismatch●Digital data operates at the fast layers●Traditional data preservation operates at the slow layers.●It's easy to build fast layers upon slow ones.●It's hard to build slow layers upon fast ones.●Perhaps a reconsideration of digital data preservation is needed.Real World Example:The Yucca Mountain Repository●Engineers were asked to design a 10,000 year no/low maintenance repository for nuclear waste.●Simulations of 10,000 years of aging had unknown accuracy.●Only planning for 100-200 years was feasible.●Lesson: We need long term thinking, not long term planning.The Long Now Foundation●Our goal is to foster long term thinking and responsibility. We do this through iconic and didactic long term engineering projects. Our current existing projects include:–The 10,000 year clock–The Rosetta Project (Slow and Fast Paths)–Long Bets–Seminars About Long Term ThinkingA “Long Now” Paradigm For Long Term Digital Data Preservation●Emphasize learning and process over knowledge and product.●Create preservation meta- preservation processes, rather than preservation processes themselves.●Embrace the new resources while not depending on the old ones.●Preserve preservation knowledge first (keep the bootstrap).●Give away ownership of data. Do not hoard it.●Embed all of these into culture!A First Concept: High Data Mobility●This is the notion that data must be able to easily move around in order to survive rather than stay in a single/few safe places.●This requires a shift in thinking of the preservation problem as a mostly “static” problem to a highly “dynamic” one.●Data mobility is more important than reliability of media or expected stability of standards, institutions, or processes.●Since the digital world operates in the fast layers, data must be able to move around at appropriate speed in order to survive.High Data Mobility – Diversity Is Good!●The concept of mobility includes a diversity of redundancy. ●Dimensions of diversity include:–Media instance–Technology type–Location–Curation–File formats/data encodingsHigh Data Mobility: Hardware/Technology●It should be cheap/easy to make many copies quickly. ●Copies should be made to different classes of hardware.●There should be a diversity of copying policies.High Data Mobility: Location/Environment●Copies of data should be made to multiple, diverse locations.●Copies should be owned by many different types of organizations.●Resources for copies should be available from a diversity of management models.High Data Mobility: Curation●Curation should be automated and distributed if possible.●Multiple curation approaches/perspectives should exist.●Example: The Rosetta Project–Collaborative vetting (not centralized)–Iterative peer review and discussion–Open “in process” repositories.●Other Examples: Wikipedia, FlickrHigh Data Mobility: File Formats/Data Encodings●A diversity of formats is good.●Formats should be well and publicly documented.●It should be easy to convert to/from a format without data loss.How Do We Make The Shift?●Long Now engages in long term engineering projects that teach important lessons and inspire others to think in the long term.●In order to promote the idea of data mobility, Long Now is creating a set of digital archiving tools, collectively known as the Long Server system. These tools include:–A P2P system for personal data archiving.–A metadata assistant for distributing the load of creating schemas and labeling data.–A universal file format converter.Long Server Usability Design Principles●It is important to build digital data preservation tools that:–Engage users by starting simple and emphasizing accessibility and ease of use. (Engage users.)–Allow messy user practices and encourage (but not enforce) proper ones. (Teach good knowledge.)–While working towards the ideal solution, adapt to changes in the world. (Teach adaptability.)–Count on always having insufficient knowledge about the future. (Good enough is better than