DDI Version 30DDI版本30

Introduction to DDI 3.0Sanda Ionescu ICPSRCESSDA Expert Seminar,September 2007DDI Version 3.0Radically different.More complex(but certainly doable!)Brings important benefits.Workshop Schedule 14:30 15:10Overview(40)15:10 15:35Structure and Technical Mechanisms(25)15:35 15:45 Break(10)15:45 16:10Study Unit Modules Content(25)16:10 16:30 Variable Markup Example(20)16:30 16:40 Break(10)16:40 17:10 Grouping Modules Content and Examples(30)17:10 17:30Getting Started(20)DDI 3.0OverviewDDI BackgroundDevelopment History1995 A grant-funded project initiated and organized by ICPSR proposes to create a new standard for documenting social science data,to replace OSIRIS tagged codebooks.First drafts used SGML,then converted to Web-friendly XML.2000 DDI Version 1.0 published as a mainly document-and codebook-centric standard.DDI BackgroundDevelopment History2003 DDI Version 2.0 published with extended scope:Aggregate data coverage(based on matrix structure)Additional geographic representation to assist geographic search systems and GIS usersVersions 1.0 through 2.1(latest published)are backwards compatible,and based on the same structure.DDI BackgroundDevelopment HistoryFebruary 2003 Formation of the DDI Alliance,a self-sustaining membership organization whose members have a voice in the development of the DDI specification.http:/www.ddialliance.org/DDI BackgroundDevelopment HistoryVersion 3.0:2004-2006:Planning and Development November 2006:Internal ReviewFebruary 2007:Public ReviewJuly 2007:Candidate Draft Release http:/www.ddialliance.org/ddi3/index.htmlBenefits of using DDI as an XML-based standardInteroperability:Enables seamless exchange and reuse by other systems.Repurposing:Provides a core document from which different types of outputs can be generated.Value-added documentation:Tagging carries“intelligence”in the document by describing content.Enhanced Data Discovery:Increases precision and granularity of searches.Support for Data Analysis:Variables description is accepted as input by online analysis systems.Multiple presentation formats:ASCII text;PDF;HTML;RTF.Preservation-friendly:Non-proprietary format.Why DDI 3.0?DDI 3.0 presents new features in response to:Perceived needs of:-Data users-Data producers-Data archivists/librariansDevelopments in documenting and archiving dataAdvances in XML technologyDDI 3.0 and the Data Life Cycle ModelDDI Versions 1/2 were codebook-centric:Closely followed the structure of traditional print codebooks.Captured data documentation at a single,“frozen”point in time archiving.DDI 3.0 and the Data Life Cycle Model Version 3.0 is Life Cycle oriented:-Designed to cover all stages in the life cycle of a data collection:pre-production production post-production secondary useLife Cycle Coverage in DDI 3.0 Planning for the Study:Proposal/DesignStudy Purpose/OutlineConceptsStudy PopulationAuthor(s)Funding SourcesVersion 3.1Survey/Sample DesignPre-testingLife Cycle Coverage in DDI 3.0Proposal becomes reality Data Collection methodology:sampling,time,etc.Instrument characteristics QuestionnaireData cleaning,weighting,coding,etc.Life Cycle Coverage in DDI 3.0Publishing the data Intellectual content:Variables,Categories,Codes.Physical representation:Data format,Record structure,Statistics.Life Cycle Coverage in DDI 3.0Archiving/(Re)Distributing the data collectionProcessing checksHoldings,availability and access conditionsLife Cycle Coverage in DDI 3.0DDI becomes“visible”to the outside worldDDI Instance:Pulls together all life cycle stagesAcquires its own identity as an objectBecomes a tool for data discovery and analysisLife Cycle Coverage in DDI 3.0Secondary use of data new conceptual frameworkNew DDI Instance:New PurposeNew Logical ProductNew Physical Description of DataDDI 3.0 and the Data Life Cycle Model Advantages of Life Cycle orientation:Allows capture and preservation of metadata generated by different agents at different points in time.Facilitates tracking changes and updates in both data and documentation.DDI 3.0 and the Data Life Cycle Model Advantages of Life Cycle orientation:Enables investigators,data collectors and producers to document their work directly in DDI,thus increasing the metadatas visibility and usability.Benefits data users,who need information from the full data life cycle for optimal discovery,evaluation,interpretation,and re-use of data resources.New/Extended Functionalities in DDI 3.0:QuestionnaireVersions 1/2:-No instrument coverage.-Question text only as part of variable description.-No documentation for question flow/conditions.Version 3.0:-Full description of instrument as a separate entity.-Documents specific use of questions:flow,conditions,loops.-Compatible with Computer Assisted Interviewing software.New/Extended Functionalities in DDI 3.0:Complex DataVersions 1/2:-Inadequate representation of complex/hierarchical data Version 3.0:-Detailed documentation for complex/hierarchical dataLogical structure of recordsRecord Types and RelationshipsRelevant variables:key-link,case identification,record type locatorPhysical layout of records Single“hierarchical”file for all records,multiple rectangular files,relational database,etc.New/Extended Functionalities in DDI 3.0:Aggregate DataVersions 1/2:-Initially designed for microdata only-Aggregate data section added in V 2.1 to support limited representation(Census-type data,delimited files)Version 3.0:-Adds support for tabular,spreadsheet-type,representation of aggregate data-Aggregate data transport option:cell content may be included inline with the data item description New/Extended Functionalities in DDI 3.0:Data TransportVersions 1/2:-NoneVersion 3.0:-In-line inclusion enabled for both aggregate data and microdataNew/Extended Functionalities in DDI 3.0:Longitudinal/Time Series/Cross-national DataComparabilityVersions 1/2:-NoneVersion 3.0:-Grouping structure documents studies related on one or several dimensions(time,geography,language,etc.)as well as their comparabilityNew/Extended Functionalities in DDI 3.0:Increased Multilingual SupportVersions 1/2:-Limited Version 3.0:-Support for multiple language use and translations Geburtsjahr Year of Birth DDI 3.0 Specification:Schema-basedVersions 1/2:-DTD-basedVersion 3.0:-Schema-based:Data typing supports machine actionabilityUse of namespaces supports-Modularity-Extensibility and reuse-Alignment with/use of other standardsDDI 3.0 Specification:Machine-actionableVersions 1/2:-Machine-readableVersion 3.0:-Machine-actionable:1.Data typing:increased use of controlled vocabularies and standard codes2.Larger set of required elementsPredictable content=a more consistentbase for programming DDI 3.0:Modular StructureVersion 1/2:-Single file,hierarchical designVersion 3.0:-Modular design:-Facilitates reuse-Facilitates versioning and maintenance-Supports life cycle model-Allows flexibility in organizing the DDI Instance-Supports grouping and comparing studies-Supports creation of metadata registriesDDI 3.0:Alignment with other metadata standardsVersions 1/2:-MARC,Dublin Core(bibliographic standards)Version 3.0:-MARC,DC,but also-SDMX(Statistical Data and Metadata Exchange)-ISO 11179(Metadata Registries)-FGDC(Digital Geospatial Metadata)-ISO 19115(Geographic Information Metadata)DDI 1/2 or DDI 3.0?DDI 3.0 will not supersede DDI 2.1.Both versions willcoexistcontinue to be maintainedbe used according to specific needs.All DDI 1/2 markup will not have to be migrated to Version 3.0.DDI 3.0Structure and MechanismsDDI 3.0 Modular Structure Building blocks of DDI 3.0:Modules SchemesDDI 3.0 Modular StructureModules:Document different aspects of a study,or group of studies,following the data through their life cycle(Conceptual Components,Data Collection,Logical Product,Physical Instance,etc.)Schemes:Include collections of sibling“objects”that are traditionally components of a variable description:Concepts,Universes,Questions,Variable Labels and Names,Categories,Codes.DDI 3.0 Modular StructureModules:Can live independently(have their own schemas)or connected to one another within a hierarchical structure.Schemes:Can live semi-independently(need a higher-level wrapper as they do not have their own schemas)or in-line within a Study Unit or Group module.DDI 3.0 Modular Structure DDI 3.0 model=a multi-branched hierarchyModule level:DDI InstanceResource PackageGroupStudy UnitSubgroupStudyUnitConceptualComponentsDataCollectionArchiveOrganizationsStudyUnitSubgroup(Sub)groupStudyUnitDDI 3.0 Modular Structure DDI 3.0 model=a multi-branched hierarchyWithin modules:DataCollectionQuestion SchemeProcessingMethodologySamplingTime MethodQuestionItemQuestionItemWeightingCodingDDI 3.0 Modular StructureRelationships are established through:In-line inclusion (Relational order is explicit)Referencing Internal External (Relational order is implicit)DDI 3.0 Structural mechanisms Enable modular design and help actualize its benefits.InheritanceReferencingIdentificationDDI 3.0:InheritanceInheritance is based on the hierarchical structure of the model.In DDI 3.0 a number of elements are reused at different levels of the hierarchy.When the same element is present at multiple levels,lower levels inherit content from the upper levels,and only need to specify differences(=local overrides).DDI 3.0 InheritanceExampleInstance:Coverage:Spatial:50 US states -Study Unit A no Spatial Coverage defined =will be inherited from Instance-Study Unit B Coverage:Spatial:48 coterminous states =supersedes definition in InstanceDDI 3.0:ReferencingDDI 3.0 modular structure is dependent upon creating relationships by reference.Referencing implies bringing up the content of a DDI object within,or in association with,another object,by specifying its Unique Identifier.Identifiers are the key links between DDI objects.DDI 3.0:ReferencingExampleData Collection Module:Question Scheme:Question:ID:“Q1”Text:“How many days in the past week did you watch the national network news on TV?”Conceptual Components Module:Concept Scheme:Concept:ID:“C1”Description:“Exposure to national TV news”Logical Product Module:Variable Scheme:Variable:ID:“V1”Name:V043014 Label:Days past week watch natl news on TV Question Reference:ID:“Q1”Concept Reference:ID:“C1”DDI 3.0:ReferencingExampleDDI 3.0:IdentificationConsistency in building and using identifiers is needed for:Proper functioning of reference systems,enabling a smooth exchange and reuse of existing metadata.Machine-actionability of DDI instances,allowing them to serve as a basis for running programs and processes.DDI 3.0:IdentificationElement types used in the Identification system:All elementsIdentifiableVersionableMaintainableDDI 3.0:IdentificationElement TypesNon-identified elements:Require context,which is provided by containing parents.Example:codes within code schemesAre not reusable.Example:variable and category statisticsDDI 3.0:IdentificationElement TypesIdentifiables Carry their own IDMay be referenced/reusedCannot be versioned or maintained,except as part of a complex parent element(Example:Variable a change implies a new version of the entire scheme).DDI 3.0:IdentificationElement TypesVersionablesCarry their own IDCarry their own Version:content changes are important to note(Example:Concept may be independently versioned within a scheme).DDI 3.0:IdentificationElement TypesMaintainablesAre higher level DDI objectsAre both identifiable and versionableCan also be published and maintained as separate entities(Example:all modules,schemes,comparison maps)DDI 3.0:Identification StructureMaintainable elements:URN and/or ID+Identifying Agency +Versioning Information:Version Version Date Version Responsibility Version RationaleVersionable elements:URN and/or ID+Versioning InformationIdentifiable elements:URN and/or IDDDI 3.0:Identification StructureNon-specified Identification information is inherited from the levels above.Example 1:Inheritance is assumed.Maintainable:Variable Scheme:ID:VarScheme_AIdentifying Agency:ICPSR Version:1.0 Identifiable:Variable:ID:Var_1 Identifying Agency VersionDDI 3.0:Identification StructureNon-specified Identification information is inherited from the levels above.Example 1:Inheritance is assumedMaintainable:Variable Scheme:ID:VarScheme_A Identifying Agency:ICPSR Version:1.0 Identifiable:Variable:ID:V1 Identifying Agency VersionExample 2:Inheritance is applied by defaultMaintainable:Logical Product ID:LogicalProd_Y Identifying Agency:ICPSR Version:1.0 Maintainable:Variable Scheme:ID:VarScheme_A Identifying Agency:Version:DDI 3.0:Identification Structure:IDsUniqueness of Identifiers is necessary for both internal and external referencing:1)All IDs MUST be unique within a maintainable2)All maintainables MUST have unique IDs across an AgencyDDI 3.0:Identification Structure:Creating unique Identifiers A DDI Instance may include multiple maintainables at different hierarchical levels:Instance(maintainable)unique ID within Identifying Agency Study Unit(maintainable)unique ID within Identifying Agency Logical Product(maintainable)unique ID within Identifying Agency Variable Scheme(maintainable)unique ID within Identifying Agency DDI 3.0:Identification Structure:Creating Unique IdentifiersInstance_A(unique at ICPSR)StudyUnit_1 Logical Product_1 VariableScheme_1 Variable_1Instance_B(unique at ICPSR)StudyUnit_1 Logical Product_1 VariableScheme_1 Variable_1Post-markup:Variable ID:Instance_AStudyUnit_1LogicalProduct_1VariableScheme_1Variable_1Instance_BStudyUnit_1LogicalProduct_1VariableScheme_1Variable_1Markup:DDI 3.0:Identification Structure:URNsHave a fixed structure and MUST include object ID,Identifying Agency,and Version.For versionable and identifiable elements,the containing maintainable is specified.Take precedence when both a URN and the Identification sequence are used for the same object.May be constructed post-markup from the Identification sequence.DDI 3.0:Identification:URN StructureExamples:Maintainables:urn:ddi:3.0:StudyUnit:ddialliance.org:StudyUnit_ID:1.0Versionables:urn:ddi:3.0:ConceptScheme:ddialliance.org:ConceptScheme_ID:1.0:Concept:Concept_ID:2.1Identifiables:urn:ddi:3.0:VariableScheme:ddialliance.org:VariableScheme_ID:1.0:Variable:Variable_IDObject nameIdentifyingAgencyObject IDObjectVersionDDI 3.0:ReferencingReference structure:URN,and/or:Referenced objects ID+Identifying Agency+Version +Containing Module ID +Containing Scheme ID DDI 3.0:Reuse of InformationReferencing Mechanisms for REUSE InheritanceReuse of Information:1.Facilitates development of documentation throughout the study life cycle2.Promotes interoperability and standardization across organizations3.Saves markup time and effort4.Reduces the risk of human entry error5.Provides a basic level of implicit comparabilityDDI 3.0 ModulesContent,Markup ExamplesDDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupResource PackageStudy UnitSubgroupStudy UnitSub(Group)ConceptsData Coll.Logical Pr.etcOther“specialized”DDI 3.0 modulesAggregate Data:NCube Logical ProductInline NCube Record LayoutNCube Record LayoutTabular NCube Record LayoutInline Microdata:DatasetUser-specific Markup Templates:DDI ProfileDDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogical ProductPhysical Data ProductPhysical InstanceArchiveOrganizationsConceptual Component Data CollectionLogical ProductArchiveStudy UnitGroupComparativeDDI 3.0 Modules used to mark up a simple studyDDI 3.0 modules for documenting a single,survey-type studyDDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogical ProductPhysical Data ProductPhysical InstanceArchiveOrganizationsConceptual Component Data CollectionLogical ProductArchiveStudy UnitGroupComparativeDDI 3.0 modules for documenting a single,survey-type studyReusableXHTMLInstanceStudy UnitConceptual ComponentData CollectionLogical productPhysical Data ProductPhysical InstanceArchiveOrganizationsDDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogical ProductPhysical Data ProductPhysical InstanceArchiveOrganizationsConceptual Component Data CollectionLogical ProductArchiveStudy UnitGroupComparativeDDI Instance-wrapper for all modules-IdentificationURNIdentification SequenceNameCitation (+optional DC Elements)CoverageTopicalSpatialTemporalGroup(module)repeatableResource Package(module)-repeatableStudy Unit(module)-repeatableOther Material(s)Note(s)Translation InformationCoverage in DDI 3.0Study:American National Election Study(ANES),2004Topical Coverage:Subject:Historical and Contemporary Electoral ProcessesKeyword:Electoral campaigns Political attitudesPolitical participationSpatial Coverage:Description:United StatesTop level:nationLowest level:congressional districtTemporal Coverage:Date:2004DDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogical ProductPhysical Data ProductPhysical InstanceArchiveOrganizationsConceptual Component Data CollectionLogical ProductArchiveStudy UnitGroupComparativeStudy Unit-documents a single“study”-Identification,Other Material(s),Note(s)CitationAbstractUniverse ReferenceFunding InformationPurposeCoverage Analysis UnitEmbargoConceptual Component(module)Data Collection(module)Logical Product(module)Physical Data Product(module)Physical Instance(module)Archive(module)Organizations(module)DDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogical ProductPhysical Data ProductPhysical InstanceArchiveOrganizationsConceptual Component Data CollectionLogical ProductArchiveStudy UnitGroupComparativeConceptual Component-lists concepts and universes-Identification,Other Material(s),NotesCoverageConcept Scheme or Reference to External SchemeVocabulary describes vocabulary usedConceptLabelDescriptionSimilar ConceptDifferenceConcept GroupConcept Reference(nestable)Universe Scheme or Reference to External SchemeUniverseHuman ReadableMachine ReadableSubuniverseSubuniverseDDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogical ProductPhysical Data ProductPhysical InstanceArchiveOrganizationsConceptual Component Data CollectionLogical ProductArchiveStudy UnitGroupComparativeData CollectionIdentification,Other Material(s),Note(s)CoverageMethodologyTime MethodSamplingCollection EventData CollectorData SourceCollection Date(s)Mode of data collectionQuestion Scheme lists actual questionsInstrument documents question flow,conditionsProcessing EventControl and cleaning operationsWeightingData Appraisal InformationCodingDDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogical ProductPhysical Data ProductPhysical InstanceArchiveOrganizationsConceptual Component Data CollectionLogical ProductArchiveStudy UnitGroupComparativeLogical Product-documents intellectual content of data-Identification,Other Material(s),Note(s)CoverageCategory Scheme or Reference to external category schemeCategoryLabelDerivation(if applicable)DefinitionCode Scheme or Reference to external code schemeCategory Scheme ReferenceHierarchy TypeLevel(in the hierarchy)CodeCategory ReferenceValueCode(nestable)Variable Scheme or Reference to external variable schemeLogical ProductVariable Scheme:VariableVariable or Reference to an externally documented variableIdentificationNameLabelDefinitionUniverse ReferenceConcept ReferenceQuestion Reference Embargo ReferenceResponse UnitAnalysis Unit RepresentationImputationDerivationCoding InstructionsValue Representation:TextDate/TimeNumericCodeLogical ProductVariable Scheme:Variable GroupVariable Group:TypeLabel DefinitionUniverse ReferenceConcept ReferenceVariable Reference(lists variables in the group)Variable Group Reference(allows nesting of groups)Variable Group Reference(use for externally documented Variable Group)DDI Version 3.0 Modules-Structural Overview-DDI InstanceStudy UnitGroupConceptual ComponentData CollectionLogi