ExtendedFrameworkforInformationDiscovery

EXTENDED FRAMEWORK for Information DiscoveryxFIND V0.95 /2000-08-15Extended Framework for Information DiscoveryxFIND V0.96Programmers DocumentationIICM, TU-GRAZ, AUSTRIA2001-02-15C. GtlModule Authors: A. Griesmayer, R. Hatheyer,J. Heber, A. Jurak, B. Kngler, W. Lackner, J. Moser, J. WeitzerCopyright (c) 2001 IICM, TU-GRAZ, AUSTRIAPermission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with the Front-Cover Texts being xFIND - extended Framework for INformation Discovery, xFIND V0.95, IICM, TU-GRAZ, AUSTRIA. Authors: C.Guetl, J. Heber, A. Jurak, B. Knoegler, J. Moser, J. Weitzer, and with the Back-Cover Texts being xFIND - extended Framework for INformation Discovery, xFIND V0.97, IICM, TU-GRAZ, AUSTRIA. Authors: C.Guetl, J. Heber, A. Jurak, B. Knoegler, J. Moser, J. Weitzer. The distributed concept of xFIND will guarantee high scalability and allows to manage a huge information supply. The modular system is subdivided into Gatherers, Indexers, Information Brokers as well as Personal Information Brokers. Consequently this is providing high scalability and an enormous reduction of network and server loads. xFIND improves search results using meta data sets, quality rating labels and Web site descriptions. A copy of the license is included in the section entitled GNU Free Documentation License.Acknowledgements:We would like to thank all members of the IICM for their suggestions and help. Parts of the research and implementation of xFIND, described in this work was conducted as part of the IICM Knowledge Discovery Project, supported by the Austrian Federal Ministry of Science and Transport. Many thanks to Prof. Maurer who had the initial idea for this project and to the xFIND development team.Dedications:To all users who are drowning in information but starving for knowledge. History:2000-02-01 Architectural Design Document: xFIND V0.90Authors: C. Guetl, A. Jurak, B. Knoegler, J. Moser2000-04-28 Programmers Documentation R01: xFIND V0.90Changes:terms of usage changed to GNU Free Documentation LicensexQMS Modules xQMS Metadata scheme from J. Weitzer2000-08-15Programmers Documentation: xFIND V0.95Changes:Sections Brokertool, Rating System (parts from xQMS Modules have been adopted) and Broker added by J. Heber2001-02-14 Programmers Documentation: xFIND V0.96Changes:4.8.3.4 XFBroker (BookSite are modified, add ForgotPassword), 4.8.4 The CGI UserInterface (CGI files are described)modified by W. Lackner5.1 xFIND Gatherer Admin ToolTable Of Contents1.Introduction71.1.Purpose71.2.Scope71.3.Definitions, Acronyms and Abbreviations81.4.References92.General Description112.1.Product Perspective112.2.User Characteristics122.3.General Constraints122.4.Assumptions and Dependencies132.5.Operational Environment133.System Overview143.1.Main Overview143.1.1The Gatherer163.1.2The Indexer173.1.3The Knowledge Broker183.1.4The Rating System193.2.External Interface Definition193.2.1User Interface Definition193.2.2Interfaces to other Systems204.System Design214.1.Design Method214.2.Decomposition Description214.3.Common Modules214.3.1The Logging Module214.3.2The Config Reader224.3.3The Allow-Deny-Filter234.3.4The argument converter234.4.The Communication Layer244.4.1Module Overview244.4.2Overview of the Package Structure274.4.3Description of the Classes284.5.Query Communication Format (QCF)314.5.1Query-Communication-Format Overview314.5.2Query-Communication-Format Keys and Values324.6.The Gatherer324.6.1Module overview324.6.2Overview of the package structure354.6.3Interface to Indexer364.6.4XFGathererMaster374.6.5XFGatherer384.6.6XFGatherObject404.6.7XFGatherLinks414.6.8XFGatherDirectory424.6.9XFObjectFactory and XFObjectCache424.6.10XFRetriever424.6.11XFConverter444.6.12XFAVTable454.6.13XFDatabase454.6.14XFAuthorizer454.6.15XFRetrievedObject and XFRetrievedObjectCache454.6.16XFGathererStartup and XFGathererRemoteStartup464.6.17XFImageConverter and XFImgObj464.7.Indexer464.7.1Module Overview474.7.2Overview of the Package Structure504.7.3Description of the Classes514.8.Brokertool574.8.1Module Overview574.8.2Overview of the Package Structure584.8.3Description of the Classes584.8.4The CGI UserInterface634.9.Rating System684.9.1xQMS Quality Metadata Scheme684.9.2Overview of the package structure774.9.3Description of the Classes784.10.Broker814.10.1Module Overview814.10.2Overview of the package structure824.10.3Description of the Classes835.Addons895.1.xFIND Gatherer Admin Tool895.2.System requirements915.3.Installation915.4.UserInterface Overview925.5.Areaconfiguration-mode925.5.1Available functions935.5.2Parameters955.6.AllowDenyconfiguration-mode975.6.1Available functions975.6.2Filters995.6.3Examples for allowdeny-rules995.6.4Creating new rulesets with the help of predifined templates1005.7.URL Window1006.Appendix1006.1.The Query Communication Format (QCF)1007.GNU Free Documentation License148List of TablesTable 1: Definitions, Acronyms and Abbreviations8Table 2 - 34 attributes of xQMS70Table 3 - Kind of resource71Table 4 - Specifying the type73Table 5 - Characterisation of the language74Table 6 - Age of intended audience75Table 7 - Knowledge of intended audience75Table 8 - Specifying the authority in science76Table 9 - Range for the width of information77Table 10 - Range of depth of information77Table 11 Popupmenu area configuration93Table 12 Popupmenu allowdeny configuration97Table 13 Filters allowdeny99Table 14 Examples: regular expressions for allowdeny configuration99List of FiguresFigure 3.1.1 An overview of the xFIND system15Figure 3.1.2 xFIND Core System and the interaction to other external systems16Figure 4.4.1 Communication flow-chart.26Figure 4.6.1 Overview of the gatherers modules33Figure 4.6.2 Communication with the gatherer37Figure 4.6.3 Communication between classes during gathering of a single object41Figure 4.7.1 Collaboration of the storage parts.48Figure 4.7.2 Collaboration of the retrieval parts.50Figure 4.8.1 - Collaboration of the Brokertool modules.57Figure 4.8.2 - xFIND CGI scripts and their dependences64Figure 4.10.1 - Collaboration of the Broker modules.82Figure 5.4.1 Mainscreen XFExplorer92Figure 5.5.1 Area configuration popupmenu93Figure 5.5.2 Area configuration edit dialog94Figure 5.5.3 Area configuration delete dialog95Figure 5.6.1 Allowdeny edit dialog98Figure 5.6.2 Allowdeny delete dialog98Figure 5.6.3 Allowdeny editdialog; filter combobox open99Figure 5.7.1 URL Window1001. Introduction1.1. PurposeThis document describes the system architecture by decomposing the system into components and modules. Each module of the communication diagram can then be implemented by a group of classes and methods.The programmers documentation describes the architecture of xFIND and makes the system more readable and understandable for the team members and (technical contact persons of) customers, and the quality assurance representative (review).1.2. ScopeThis document is used for the xFIND project. If modules have interfaces to other systems, this must be referenced at concerned chapters (an overview has to be given at 3.1 Main Overview as well as the concerned module in 4 System Design.1.3. Definitions, Acronyms and AbbreviationsDescription of the definitions used. List of the acronyms and abbreviations used.ItemDescriptionACMAssociation for Computing MachineryCGICommon Gateway InterfaceCOOChief Operating OfficerCPUCentral Processing UnitCTOChief Technical OfficerDCDublin Core Meta Data SetDDCDewey Decimal Classification codeEOBEnd Of BlockEODEnd Of DataFTPFile Transfer ProtocolHIKSHierarchical Interactive Knowledge SystemHTTPHypertext Transfer ProtocolIDIdentifierIICMInstitute for Information Processing and Computer Supported new MediaI/OInput / OutputJVMJava Virtual MachineLMTLast Modification TimeLOMLearning Object MetadataMD5Message Digest 5MIMEMultipurpose Internet Mail ExtensionsQA Quality AssuranceQCFQuery Communication FormatSOIFSummary Object Interchange FormatSQLStructured Query LanguageSTDOUTStandard OutputTCPTransmission Control ProtocolTCP/IPTransmission Control Protocol over Internet ProtocolTTLTime To LiveUIUser InterfaceURUser RequirementURDUser Requirement DocumentURIUniversal Resource IdentifierURLUniform Resource LocatorWebThe World Wide WebWPWork PlanxFINDextended Framework for INformation DiscoveryTable 1: Definitions, Acronyms and Abbreviations1.4. References1Gtl, C.; Andrews, K.; Maurer, H.: Future Information Harvesting and Processing on the Web (1998), Presented at Conference European Telematics: advancing the information society, Barcelona, 4-7 February 1998, andhttp:/www2.iicm.edu/cguetl/papers/fihap2Dietinger, T.; Gtl, C.; Maurer, H.; Pivec, M.; Schmaranz, K.: Intelligent Knowledge Gathering and Management as New Ways of an Improved Learning Process (1998), Presented at Conference Webnet 98, Orlando, 4. - 12. Nov. 1998 and Proceedings of WebNet 98 - World Conference of the WWW, AACE, Internet and Intranet, Charlottesville, USA, pp. 244 249 andhttp:/www2.iicm.edu/cguetl/papers/webnet98 3Dietinger, T.; Gtl, C.; Kngler, B.; Neussl, D.; Schmaranz, K.: Dynamic Background Libraries - New Developments In Distance Education Using HIKS (Hierarchical Interactive Knowledge System) (1999), Published in J.UCS, Vol.5 / No.1 / 2- 10 andhttp:/www2.iicm.edu/cguetl/papers/dynamicbacklib 4Gtl, C.; Schmaranz, K.: DigLib 2000 - A Working Prototype for the Next Generation of Digital Libraries (1999), Proceedings 3rd International Austrian-Israeli Technion Symposium cum Industrial Forum Technology for Peace - Science for Mankind, p 123-131 (1999) andhttp:/www2.iicm.edu/cguetl/papers/diglib2000 5Gtl, C:, Jurak, A.; Moser, J.; Neussl, D.; Pivec, M.: Knowledge Transfer and Knowledge Discovery - New Improvements for the Corporate Decision (1999), Presented at Conference Webnet 99, Hawaii, 24. - 30. Oct. 1999 and Proceedings of WebNet 99 - World Conference of the WWW, AACE, Internet and Intranet, Charlottesville, USA, pp. 1280 1281 andhttp:/www2.iicm.edu/cguetl/papers/knowledgdiscovery6Gtl, C.; Maurer, H.; Pivec, M.: Learning on Demand using xFIND: An Improved Way for Ongoing and Lifelong Learning as a Smart Module for the GENTLE Learning Environment (1999), Presented at ICCE99 and Proc. ICCE99, Chiba, Japan, IOS Press, Amsterdam, Vol. I (1999), 592-599 andhttp:/www2.iicm.edu/cguetl/papers/learningondemand 7Bowman, C. M.; Danzig, P. B; Hardy, D. R.; Manber, U.; Schwartz, M. F.; Wessels, D. P.: Harvest: A Scalable, Customisable Discovery and Access System. Technical Report CU-CS-732-94, Department of Computer Science, University of Colorado, Boulder, (1994).8Huang, K.; Lee, Y. W.; Waang, R. Y.:Quality Information and Knowledge (1999), Prentice Hall, New Jersey, 1999.9IEEE: LOM Learning Object Metadata, IEEE Learning Technology Standards Committee, Learning Objects Metadata Working Group,http:/ltsc.ieee.org/wg12/ 10Metadaten-Elemente des Dublin Core, Max-Planck-Institut fr Bildungsforschung, Berlin, 18. Dezember 1996http:/www.mpib-berlin.mpg.de/DOK/metatagd.htm2. General Description2.1. Product PerspectiveThis is a short description of the intentions for developing the product. Today, since the development of simple-to-use access tools, the Internet represents the worlds largest and fastest growing unstructured knowledge and information database. However, exactly this very characteristic makes the discovery of relevant information difficult and prevents efficient use of the World Wide Web. In recent years search engines have been developed to give users the possibility to search and find certain information on the Web. All these search services gather independently information from more or less large parts of the Internet. Present day methods of information collection and the ever-growing quantity of documents and active search engines will increasingly affect future internet applications because of repeated net- and server-load caused by information gathering. xFIND is a scalable and distributed system. It implements new ways such as local gathering and hierarchical search services. Furthermore, current concepts of local search engines and concepts of providing already gathered information for external, hierarchically higher levels of search services are covered. xFIND is designed as a portable and scalable information discovering system which enables providing information contents as well as meta information for internal system use as well as for external search services. An index system with a ranking and relevance system and automatic discriminator terms extraction from the documents will be created. In addition, smart visualisation interfaces will support users needs. The xFIND system can be used in different work scenarios distributed search system dynamic background library intranet information distribution system decision support system news and change tracking system for online information integrated into a work environment advanced search facilities for the Hyperwave Information Server advanced search facilities for the GENTLE System dynamic background library for the GENTLE System advanced search facilities using as stand alone search engine smart search system for portal server applications support certain user groups to achieve certain goals standard users to get information about specified topics learners to get additional information from a dynamic background library trainers to detect new pieces of information in specified subjects libraries to get relevant information human subject experts to detect new pieces of information in specified subjects adaptable for other market opportunities high modularity platform independence can be used in the field of electronic publishing and libraries, the GENTLE system and the Hyperwave information server2.2. User CharacteristicsxFIND user roles may be divided into Users Experts Content providers Administrators2.3. General ConstraintsDescription of aspects which have to be considered technical restrictions (SW and HW) platform independence Software implementation in Java Java API for external system use Interfaces for the integration of external systems organisational restrictions (security, confidentiality, company policy.) reducing of network and server loads information pre-processing and information replication distributed administration trustworthy information handling (digital signatures)2.4. Assumptions and DependenciesxFIND will support the Hyperwave Information Server, the GENTLE System and the DINO system. 2.5. Operational EnvironmentThe xFIND basic system is implemented in Java, additional modules like the data base module may be platform dependent. Basically xFIND is designed for UNIX (Linux, Sun Solaris, Alpha) and Win NT. Please note that the platform availability depends on additional modules and their implementation.3. System OverviewThe system overview describes the components of the xFIND system.3.1. Main OverviewThis overview describes the system in a very abstract form (considering only the level of components). In addition a graphic describes the relationship of the components.The MotivationThe rapid increasing number of accessible and indexable Web documents as well as the large amount of dynamically generated information must lead to new concepts for search services. Nowadays a large number of search engines is gathering any information available. So the same documents are indexed several times. Consequently this causes irksome server and network load. Mankinds knowledge is available on the Internet but the sources are scattered all over the world. To make this knowledge available means must be found to provide the user with accurate and high quality search results.The xFIND systemA future oriented information system must meet the requirements for distributed and multimedia information sources. xFIND provides a scalable, distributed information system, which offers much more than a common search engine. Former experiences and research work lead to the combination of a core information system and some external modules like rating systems, annotation systems and archiving systems. xFIND will bridge the gap between information suppliers and information inquirers. In addition the system supports DC and LOM meta data formats and allows to manage quality meta data for Web servers, Web areas and single documents.The Concept of xFINDxFIND is one possible approach to a future-oriented information discovery system, which can support the corporate decision process and the learning process as well as the interaction and collaboration between users. Useful information can be found in a wide range of information sources inside and outside organisational units. The design basics will sufficient the model of a knowledge management platform as discussed at Huang et. al. 8 This model points out 4 aspects: Making knowledge visible which concerns questions where we can find relevant information and who knows what. Building Knowledge Infrastructure in such ways enables access to external and internal knowledge sources. Building Knowledge Intensity handles the management of knowledge processing and necessary networks communication.The fourth point Developing a Knowledge Culture concerns sharing and exchanging knowledge and establishing its trustworthiness. The later point is influenced by organisational processes and can be supported by proper tools. The concept of xFIND covers the ideas of the discussed model described above. Further experiences from a first prototype called HIKS 3 and tests within the GENTLE learning environment 6 have also influenced the concept and the design of xFIND and its implementation. xFIND offers features to handle the information lifecycle and fits into the concept of a knowledge management environment. Because of its platform independence Java is used as the implementation language. In order to achieve the functionality described above xFIND is split into three main parts: the Gatherer, the Indexer and the Broker (see Fig 1).Figure 3.1.1 An overview of the xFIND systemFigure 3.1.2 xFIND Core System and the interaction to other external systemsFigure 2 visualises the conceptual idea of interacting the xFIND Core System with external systems, like external search services and quality rating systems. The quality aspects are one of the important items of the xFIND concept. The system will manage object data as well as quality meta data. The main parts are described as follows.3.1.1 The GathererThe Gatherer performs the task of visiting servers and gathering information from various sources as well as pre-processing the document data. It identifies data like title, keywords, type, language, and creation or modification time. In case of HTML files also links, images and other embedded objects like Java