Overview of TIOindex implementations The DAG GIDS and

Overview of TIO-index implementations The DAG, GIDS and Desire TIO/LDAP index servers Henny Bekker LDAP/TIO implementations -2 -

Overview of TIO-index implementations Agenda • General overview of LDAP/TIO-indexes – What are TIO indexes What– is The a TIOgeneric index blamodel bla bla bl ablbbabbnsjdsa bla • Some specific implementations – The generic Desire TIO index server – The Ericsson DAG server – The GIDS server • Open Issues – The scope and communication between LDAP/TIO index servers • Exchanging TIO’s – Local access policy • Access restrictions • Security requirements • Senario’s LDAP/TIO implementations -3 -

General overview of LDAP/TIO-indexes Tagged Index Object’s A TIO consists of: • Meta information such as What is mime a TIOheader indexdefining bla the bla object bla bl ablbbabbnsjdsa bla – A – – An object type identifier that uniquely identifies the subtree and scope One or more URI's that will form the base of the created referrals The security options and credentials such as a PGP or S/MIME key The update type indicating the type of TIO (e. g. full or incremental) • The payload – The tokenization types headers (e. g. Full, Token, RFC 822 etc) • Indicating which information is ‘tokenizated’ and which delimiters to use – The TAG list • Containing multiple consecutive tags which might be grouped using a dash. LDAP/TIO implementations -4 -

General overview of LDAP/TIO-indexes Content-Type: application/index. obj. tagged; dsi="1. 3. 6. 1. 4. 1. 5062. 1. 99. 1. 114"; base-uri= "weetmuts. surfnet. nl: 389/o=SURFnet, c=NL" Content-Length: 6219 version: x-tagged-index-1 updatetype: total thisupdate: 950688539 BEGIN IO-Schema sn: FULL cn: FULL. o: TOKEN END IO-Schema BEGIN Index-Info sn: 22/Arends -6/Bezemer -4/Bos -8/Neggers. -2 -3, 5 -9, 11, 14 -15, 18 -19/+31 302 305 -12/030 -2305327 o: 1/SURFnet END Index-Info What is a TIO index bla bla bl ablbbabbnsjdsa bla LDAP/TIO implementations -5 -

General overview of LDAP/TIO-indexes Tagged Index Object’s (cont. ) What is it used for: • Provide pointers to servers which most likely contains the requested information What is a TIO index bla bla bl ablbbabbnsjdsa bla – The number of false hits is depending on the choice of attribute tokenization types – Performing phrase searches is depending on the tokenization of the fields • Features a full or incremental update (which uses potentially less bandwidth) LDAP/TIO implementations -6 -

General overview of LDAP/TIO-indexes The generic model • A TIO interface – For importing, deleting and in some cases exporting TIO’s from the What isindex a TIO index bla bla bl ablbbabbnsjdsa bla – Implementing authentication control • A TIO searchable index – For searching the index on referrals to other information services – Accessible through the TIO query interface • The LDAP query interface – Containing a LDAP gateway to the query interface of the TIO index – Can act as an LDAPv 2 chaining server or as an LDAPv 3 referral server LDAP/TIO implementations -7 -

Some specific implementations The generic Desire TIO index server • Sponsored by the European Community and build by SURFnet & DFN in cooperation with Dante. • What. The is a server TIO indexconsists bla blaof: bla bl ablbbabbnsjdsa bla – The TIO index server • Using the My. SQL database engine for storing and searching the TIO’s • Containing a TIO push/pull interface and a database for storing TIO’s. • An HTTP frontend for direct access to the TIO index server by the NPS. – A Native Protocol Server (NPS) for access using the LDAP protocol • For connecting clients using specific communication protocols such as LDAPv 2, LDAPv 3 or Who. IS++. • For connecting directory servers with a specific communication protocols such as LDAPv 2, LDAPv 3 or Who. IS++. – An Ldap. Crawler for gathering and converting LDIF files to TIO’s • Currently no encryption of TIO’s implemented • Currently only support for LDAPv 2. (no characterset conversion problem) LDAP/TIO implementations -8 -

Some specific implementations The generic Desire TIO index server (cont. ) LDAP/TIO implementations -9 -

Some specific implementations The Desire LDAP/TIO index server (cont. ) LDAP/TIO implementations -10 -

Some specific implementations The Desire LDAP/TIO index server (cont. ) • Unfortunately we don’t have yet any performance figures – The package is on the brink of being completed server will bla be bla faster What–is a. Presumably TIO index blathe bla GIDS bla blindex ablbbabbnsjdsa • The generic My. SQL engine is slow compared to a dedicated TIO database. • Current implementation – Available on Linux and (hopefully) on Digital Unix – The source code and executable for Linux of • The Ldap. Crawler with an integrated LDIF 2 TIO converter • The TIO index (using My. SQL v 3. 23. 6) • The LDAP NPS implemented using the Open-LDAP v 1. 2. 10 with an API to the TIO index LDAP/TIO implementations -11 -

Some specific implementations The Ericsson DAG server • Offspring of the TISDAG project – Aimed to provide a solution for an uniform telephone directory What is acontaining TIO index blanumbers bla blawithout bl ablbbabbnsjdsa bladatabase bla a centralized • The server consists of: – The DAG (Directory Access Gateway) index server • Implemented using the Times. Ten “In-Memory” database engine for storing and searching the TIO’s. – One or more CAP (Client Access Point) modules • For connecting clients using specific communication protocols such as LDAPv 2, LDAPv 3 or Who. IS++. – One or more SAP (Server Access Point) modules • For connecting directory servers with a specific communication protocols such as LDAPv 2, LDAPv 3 or Who. IS++. LDAP/TIO implementations -12 -

Some specific implementations The Ericsson DAG server (cont. ) LDAP/TIO implementations -13 -

Some specific implementations The Ericsson DAG server (cont. ) • Unfortunately we aren’t allowed to present exact figures – The next version is said to be much faster is a TIO index • What. Performance figuresbla bla bl ablbbabbnsjdsa bla – Response times • Use LDAPv 3 referral requests to measure the response time of the referral server without doing chaining or following referrals. • The mean response time related to the number of parallel search queries. (measured with a large number of queries) – Number of queries/second (or minute? ? ) – The number of parallel requests • Related to the response time – Maximum number of entries in the TIO index • Bounded by the memory size and the algorithm used to search the index LDAP/TIO implementations -14 -

Some specific implementations The GIDS server • Offspring of the TISDAG project – Second implementation of the TISDAG TIO index server is a server TIO indexconsists bla blaof: bla bl ablbbabbnsjdsa bla • What. The – An index server • Using a dedicated database engine for storing and searching the TIO’s. • Is using a dedicated communication protocol (analogous with LDAP) to communicate with the CAP and SAP modules. – One or more CAP (Client Access Point) modules • For connecting clients using specific communication protocols such as LDAPv 2, LDAPv 3 and HTTP – One or more SAP (Server Access Point) modules • For connecting directory servers with a specific communication protocols such as LDAPv 2 and LDAPv 3 – An Ldap. Crawler for gathering and converting LDIF files to TIO’s • With support for LDAPv 2 and LDAPv 3 and character-set conversion LDAP/TIO implementations -15 -

Some specific implementations The GIDS server (cont. ) • Performance figures – Response times • Measured with (an LDAPv 3 What is a TIO index bla bla. LDAPv 3 bla bl abl requests ablbbabbnsjdsa bla bind, sending the query, receiving the message, doing an unbind operation) • With one sequence of LDAPv 3 requests, a mean time of approximately 23 msec. per LDAPv 3 request. • With 10 simultaneously LDAPv 3 requests approximately 150 msec per LDAPv 3 request. – Maximum number of queries/second • Approximately 65 LDAPv 3 queries/second • With LDAPv 2 the number will be lower because the server has to do chaining. – Maximum number of entries in the TIO index • Bounded by the memory size and the algorithm used to search the index • Current demo implementation (CH, DE, NL, NO & SE) 120 K tokens of 450 different data sets. (which consumes about 35 -Mbyte of memory). LDAP/TIO implementations -16 -

Open issues The scope and communication between TIO index servers • Scope – Centralized versus distributed LDAP/TIO engines is a TIO index bla bla bl ablbbabbnsjdsa • What. Location of the TIO/LDAP-index serverbla – Located close by (in network terms) the end users to minimize the RTT – Located close by the referred LDAP servers to minimize the RTT related to LDAPv 2 chaining • Exchanging TIO’s – Global TIO collection versus distributed collections on country level • Distributed to country level • Knowledge base or ‘where to find what’? ? – Encrypted transport via HTTP – Push or pull ? ? LDAP/TIO implementations -17 -

Open issues Local access policy • Security requirements – Personal data is subjected to privacy legislation What–is a. For TIOpublic index bla other blasecurity bl ablbbabbnsjdsa blamight be imposed data requirements – No unauthorized access to local directory servers • Only accessible by local inhabitants and peer countries – All applications able to access the index should be known – Only a limited number of referrals might be returned – No ‘access denied’ messages • Don’t show entries which are not accessible • Access restrictions – Restrict access to the TIO/LDAP-index server – Restrict access to the LDAP servers containing the information • Chaining versus LDAPv 3 referral • HTTP access control versus LDAP access control • Access via HTTP proxies versus LDAP proxies LDAP/TIO implementations -18 -

Open issues Senario’s • Create trusted relation between country level TIO servers – Only peers will communicate with each other • Besides the local LDAP clients – A peer will enforce their own local access rules • The TIO index server should only be accessible by known clients – The LDAP query will be chained to the remote peers • The TIO objects of the peer country should deliver referrals which will point to a known access point e. g. an LDAP proxy or the FLDSA • An LDAP search requests from a known LDAP client must be chained to the known access point. – The number of known access points should be limited – The TIO objects cannot be duplicated between the peers LDAP/TIO implementations -19 -

Open issues Senario’s (cont. ) LDAP/TIO implementations -20 -

Discussion. . LDAP/TIO implementations -21 -