RETS Software Development q RETS the standard q

RETS Software Development q RETS the standard q RETS transactions – Identify issues, concerns and recommendations q Sample session workflow File: 05_RETS_Software_Development. PPT 1

RETS Software Development Topic Objectives RETS Software Development q Improve understanding of the RETS architecture q Build familiarity with the RETS document q Enable more productive RETS software development File: 05_RETS_Software_Development. PPT 2

RETS Principles - 1 RETS Software Development q Stateless communications protocol q Carry state using a cookie: – Basically a token that points to a persistent store on a server somewhere – RETS defines an optional cookie for this purpose q Recommendations: – Return all cookies received – Client applications using browsers will need to accept cookies File: 05_RETS_Software_Development. PPT 3

RETS Principles - 2 RETS Software Development q Uses Http-based transport q Uses DMQL - an open query language q Supports XML and site-specific output data formats q Supports site-specific metadata to describe the local MLS system q Recommendation: – Effort to support metadata produces interoperability benefits File: 05_RETS_Software_Development. PPT 4

RETS Document RETS Software Development q Organization follows a typical real estate industry workflow: – Initiate single session – Multiple data requests – Terminate session q References standards produced by other standards organizations RETS document can be challenging to read File: 05_RETS_Software_Development. PPT 5

Simple Collaboration Diagram RETS Software Development File: 05_RETS_Software_Development. PPT 6

Login Transaction Making the Initial Connection RETS Software Development q Authentication – Response with authorized resources q Types: – Basic – Digest using MD 5 or SHA 1 – SSL q Recommendations: – Implement Digest using MD 5 – SHA 1 if you can File: 05_RETS_Software_Development. PPT 7

Login Transaction Authentication RETS Software Development q Most common authentication type is digest using MD 5 – Somewhat tricky to implement algorithm – Easiest to borrow the implementation from one of the open source projects q When in doubt: – Ask the RETS-DEV list – Ask your software vendor File: 05_RETS_Software_Development. PPT 8

Login Transaction Activity Diagram RETS Software Development > < File: 05_RETS_Software_Development. PPT 9

Login Transaction Digest Authentication RETS Software Development q Message Digest algorithms are one-way functions q Easy to perform the calculation q Difficult to reverse the calculation q Both sides calculate the same message digest q Transmit only the digest q Compare your calculated digest with the received digest File: 05_RETS_Software_Development. PPT 10

Login Transaction User. Agent RETS Software Development q Application name accessing the MLS server – All client requests MUST include this field – This is a standard HTTP header field as defined in RFC 2616 q MLS server should validate application name presented against list of allowable applications maintained on a table q Application name and userid is a useful key to a table that tracks MLS server usage File: 05_RETS_Software_Development. PPT 11

Login Transaction Gotchas RETS Software Development q Using operating system libraries for MD 5 can cause authentication failure q Proxies and tunnels can be a problem q Re-authenticate is part of the RFC but few vendors enforce it q Case may matter File: 05_RETS_Software_Development. PPT 12

Sample Login Challenge Response Header 1 RETS Software Development HTTP/1. 1 401 Unauthorized xxx Date: Mon, 02 Aug 2004 05: 23: 54 GMT Server: Web. Logic XMLX Module 8. 1 SP 1 Fri Jun 20 23: 06: 40 PDT 2003 271009 with WWW-Authenticate: Digest realm="users@ftc 2. com", nonce="313035", opaque="6 e" File: 05_RETS_Software_Development. PPT 13

Sample Login Challenge Response Header 1 cont. RETS Software Development Content-Length: 0 Content-Type: text/html RETS-Version: RETS/1. 0 Cache-Control: private File: 05_RETS_Software_Development. PPT 14

Sample Login Nonce RETS Software Development nonce= "313035393731353435373630342058 dd 631265 e 3 360724 c 45 d 15 f 23 aec 7 a" File: 05_RETS_Software_Development. PPT 15

Sample Login Opaque RETS Software Development opaque="6 e 6 f 742075736564" File: 05_RETS_Software_Development. PPT 16

Sample Login Challenge Response Header 2 RETS Software Development HTTP/1. 1 200 OK Date: Mon, 02 Aug 2004 05: 31: 03 GMT Server: Web. Logic XMLX Module 8. 1 SP 1 Fri Jun 20 23: 06: 40 PDT 2003 271009 with Content-Type: text/plain File: 05_RETS_Software_Development. PPT 17

Sample Login Challenge Response Header 2 cont. RETS Software Development Set-Cookie: RETS-Session-ID=20000439936 RETS-Version: RETS/1. 0 Transfer-Encoding: Chunked Cache-Control: private 0230 File: 05_RETS_Software_Development. PPT 18

Sample Login Challenge Response Body 2 RETS Software Development <RETS Reply. Code="0" Reply. Text="Success"> Member. Name=Mark Crawford User=632344, NULL, 632344 Broker=National, 1 Metadata. Version=1. 2. 0 Min. Metadata. Version=1. 1. 7 Office. List=National; 1 Timeout. Seconds=900 File: 05_RETS_Software_Development. PPT 19

Sample Login Challenge Response Body 2 cont. RETS Software Development Search=http: //rets. ftc 2. com: 6103/search Get. Object=http: //rets. ftc 2. com: 6103/get object Login=http: //rets. ftc 2. com: 6103/login Get. Metadata=http: //rets. ftc 2. com: 6103/getmetadata Logout=http: //rets. ftc 2. com: 6103/logout </RETS> 0000 File: 05_RETS_Software_Development. PPT 20

Login Transaction Capability URL List RETS Software Development q Response from the Login transaction includes a list of known resources on the server – Each URL points to a RETS transaction type q Capability URL list may be different based on the user and their authorization level q Required that the server return a list containing at least Search q The URL references are not permanent locations – the standard allows the location to change File: 05_RETS_Software_Development. PPT 21

Action-URL and Get Transaction RETS Software Development q Included in the Capability URL List is an Action-URL q Points to a human readable message – Most useful if changed regularly; perhaps daily q Immediately following successful login, may need to do a Get on the Action-URL – Get Transaction is a file retrieval from the server File: 05_RETS_Software_Development. PPT 22

Metadata RETS Software Development q Description of the structure of the data on a specific server q Unique to each site – There will be common elements q Most powerful feature of the RETS specification Perhaps the most challenging part of the RETS specification File: 05_RETS_Software_Development. PPT 23

Get. Metadata Transaction Overview RETS Software Development q Transaction to retrieve system metadata q Can select a specific level, the level and descendants or a specific metadata ID q Formats supported are COMPACT and Standard. XML q Metadata id value of 0 – Request is for all types at the same level – breadth q Metadata id value of * – Request is for all types at the same level and all child types - depth File: 05_RETS_Software_Development. PPT 24

Metadata Format ID RETS Software Development q Hierarchy of types of metadata – See Figure 11. 1 on page 11 -2 of the RETS specification q When referencing hierarchy as ID – Parent: Child is the form – Example: Type: METADATA-UPDATE_TYPE ID: Property: RES: Add File: 05_RETS_Software_Development. PPT 25

Metadata Format Versioning - 1 RETS Software Development q Hierarchy of metadata is versioned q Changes to lower level of the hierarchy propagate upward – Example: Changed METADATA-TABLE attribute will propagate up through CLASS, RESOURCE and SYSTEM File: 05_RETS_Software_Development. PPT 26

Metadata Format Versioning - 2 RETS Software Development q Caching of metadata is permitted q Use the login information to determine metadata changes q Recommendation: – If possible, get all the metadata on a change instead of attempting to get a specific type File: 05_RETS_Software_Development. PPT 27

Metadata Format Version 1. 5 changes RETS Software Development q Added the Foreign Keys metadata – Provides relationships between the offered resources of the form parent: child q Clarified a number of issues with Metadata Format q Recommendation: – Use 1. 5 as a reference even when implementing 1. 0. 1 compliant software File: 05_RETS_Software_Development. PPT 28

Application Responses to Metadata Changes RETS Software Development q Ignore metadata – Change breaks the application, intervention required to fix q Static metadata – Change is invisible, applications keep working, intervention to required view change q Dynamic metadata – Change is visible to applications without intervention File: 05_RETS_Software_Development. PPT 29

Search Transaction Overview RETS Software Development q Most important RETS transaction q Can use either http GET method or http POST method – Example – GET method request http: //rets. server. com: 6103/search? Class=RES q Issue: GET method is limited in the character length of the request q Recommendation: POST method File: 05_RETS_Software_Development. PPT 30

Search Transaction Client Diagram RETS Software Development File: 05_RETS_Software_Development. PPT 31

Search Transaction Server Diagram RETS Software Development File: 05_RETS_Software_Development. PPT 32

Sample Search Request Header RETS Software Development HTTP/1. 1 200 OK Date: Fri, 31 Jul 2004 23: 22: 43 GMT Server: Web. Logic XMLX Module 8. 1 SP 1 Fri Jun 20 23: 06: 40 PDT 2003 271009 with Content-Type: text/xml File: 05_RETS_Software_Development. PPT 33

Sample Search Request Header cont. RETS Software Development Set-Cookie: RETS-Session-ID=20000439934 Set-Cookie: RETS-Request-ID="42" RETS-Version: RETS/1. 0 Transfer-Encoding: Chunked Cache-Control: private File: 05_RETS_Software_Development. PPT 34

Sample Search Response Body RETS Software Development 0 fe 8 <? xml version="1. 0" ? > <!DOCTYPE RETS SYSTEM "http: //www. ftc 2. com/rets/dtd/RETS 20010812. dtd"> <RETS Reply. Code="0" Reply. Text="Success"> <COUNT Records="2507" /> File: 05_RETS_Software_Development. PPT 35

Sample Search Response Body cont. RETS Software Development <REData> <Properties> <Residential. Property> <Building. Description> <Stories>1</Stories> <Property. Condition>Shows Well</Property. Condition> RETS Software Development 36 File: 05_RETS_Software_Development. PPT

Search Transaction Search. Type Required RETS Software Development q Search. Type: The resource id to search on – Example – Search. Type=Property File: 05_RETS_Software_Development. PPT 37

Search Transaction Class Required RETS Software Development q The class of data within the resource to search against – Example – Class=RES q Issues: – Some of the data may be in system specific Class names rather than the Class name that represents an XML Standard Name File: 05_RETS_Software_Development. PPT 38

Search Transaction Query. Type Required RETS Software Development q Query. Type: The query language – Only two types: DMQL and DMQL 2 – Example – Query. Type=DMQL 2 q Issues: – Mainly stylistic difference, DMQL 2 supports quoted text – Not suitable for highly complex searches – Not possible to perform cross Class searches Example – NOT Residential and Lot Land File: 05_RETS_Software_Development. PPT 39

Search Transaction Query Required RETS Software Development q Query: – Key – value pairs – Uses field name based on metadata or on standard names – Example Query=(List. Price=200000 -300000), (Status=|A) q Issues: – Lookup type queries can execute for a long time File: 05_RETS_Software_Development. PPT 40

Search Transaction Optional Arguments RETS Software Development q Optional argument are just that q A vendor MAY choose to implement none, one or more q Software is RETS compliant without any of the optional arguments implemented q Issue: – Mismatch between client & server feature sets q Recommendation: – Do NOT assume, verify the feature set File: 05_RETS_Software_Development. PPT 41

Search Transaction Count RETS Software Development q Default is no count q Behavior outside of given three values not defined File: 05_RETS_Software_Development. PPT 42

Search Transaction Format RETS Software Development q Return records in XML or column header/row values format – XML is more verbose, but contains structure information – Some issues with site specific XML – clarification in RETS needed q Default is Standard-XML q Issue: – Site specific XML is not defined in the current standard File: 05_RETS_Software_Development. PPT 43

Search Transaction Format Sidebar: XML - 1 RETS Software Development q XML is a meta-language that defines a markup language q RETS is an example of an XML application q An application consists of a DTD and documents that obey the rules embodied in the DTD q The XML DTD is a document that describes the structure and elements of an XML document File: 05_RETS_Software_Development. PPT 44

Search Transaction Format Sidebar: XML - 2 RETS Software Development q The XML document takes data and applies the DTD q RETS will eventually move from a DTD based vocabulary to a Schema-based vocabulary File: 05_RETS_Software_Development. PPT 45

Search Transaction Format Sidebar: Compact Formats RETS Software Development q Provides a delimited column description q Provides delimited data in the same position as the column description q Field delimiter is chosen by the server q Gotcha: – Delimiter tag is a two character representation of the ASCII character – Delimiter value may collide with data values in earlier implementations File: 05_RETS_Software_Development. PPT 46

Search Transaction Limit RETS Software Development q Limits the number of records returned: – The NONE well known name requests a suspension of limit – Behavior is implementation dependent – Requesting a Limit should make the query faster – May still search for all records but only transmit records to a Limit File: 05_RETS_Software_Development. PPT 47

Search Transaction Offset RETS Software Development q Return records starting from the specified point in the result set: – One method to handle very large data pulls – Given that the underlying transport mechanism is not guaranteed, very large results can fail with transport errors – Given that the data volumes can be very large, out of memory errors can occur File: 05_RETS_Software_Development. PPT 48

Search Transaction Standard. Names RETS Software Development q Indicates that the query uses the standard names or the system names: – Servers are not required to implement standard names to meet RETS compliance criteria – Standard Names are intended to be common across all RETS systems Refining the list of Standard Names is an ongoing process File: 05_RETS_Software_Development. PPT 49

Search Transaction Restricted Indicator RETS Software Development q Certain fields may have restricted visibility: – Client may request fields that are restricted be replaced by a specific character string – Example - compensation amount field – Client may request that this amount be masked by ### – Server default is to return a null value File: 05_RETS_Software_Development. PPT 50

Search Transaction Select RETS Software Development q Provides the ability to create a subset of the fields accessible to the client q Default is to return all accessible fields q Gotcha: Select field names are not specified as either the system names or the standard names – Ambiguity in RETS standard – Working assumption is that the server will understand both File: 05_RETS_Software_Development. PPT 51

Search Transaction DMQL RETS Software Development q Defines the search criteria for the search result set q Based on a BNF grammar to provide a neutral representation q RETS Server parses the DMQL and creates a server specific query q Trade off between domain specific query forms and more complex query grammars like SQL-99 q Issues around the lexical parsing of the grammar File: 05_RETS_Software_Development. PPT 52

Update Transaction Overview RETS Software Development q Allows addition of records to the RETS system q Supports Auto-population of fields q References only by System Name, not Standard Name q Validation based on rules described in the metadata q Single record per update – no bulk update File: 05_RETS_Software_Development. PPT 53

Update Transaction Validation RETS Software Development q Basic validation on data types q Additional validation for single lookup, multilookup and range q Additional validation for Boolean expression q Additional validation using external validation q Certain validation steps may be server only File: 05_RETS_Software_Development. PPT 54

Update Transaction Error Block RETS Software Development q Validation tests that fail result in an error block that provides some information to assist in correction q One error block per failed validation test – Means a single field may trigger multiple errors File: 05_RETS_Software_Development. PPT 55

Update Transaction Gotchas and Recommendations RETS Software Development q Gotcha: – RETS 1. 7 d 1 adds a third state to the validate-flag “ 2” meaning “validate the record but do not save it” – Be aware of this coming specification change q Recommendation: – Client-side validation is fast fail – use it File: 05_RETS_Software_Development. PPT 56

Get. Object Transaction Overview RETS Software Development q Provides a way for users to get objects defined in the metadata q Main use is to return pictures although there are other media types q Returns the object in the appropriate mime type format - this may be a binary format q Recommendation: – Ordering of the object ID changed in 1. 5 D 2 – Implement using the most recent RETS version File: 05_RETS_Software_Development. PPT 57

Change. Password Transaction Overview RETS Software Development q Allows a user to change their password q Some ambiguity in the 1. 5 and earlier standard q Cryptographically weak q Recommendations: – Be very careful when implementing – Use the 1. 7 standard addition as described in RCP 041 File: 05_RETS_Software_Development. PPT 58

Logout Transaction Overview RETS Software Development q Courtesy transaction q Communicates that a user has finished the session q A mechanism for the server to: – dispose of any local resources – communicate session and billing information q Recommendation: – Clients should log out – Servers should be aware that clients may never log out File: 05_RETS_Software_Development. PPT 59

Extensions to the RETS Standard RETS Software Development q RETS allows vendors and sites to extend the standard to meet the specific needs of a site q Additional Capability List URLs have no suggested form q Additional parameters to an existing transaction and additional metadata have should have prefix ‘X’ – Example: Add sort direction to Search: XSort. Dir=Asc q Almost everything else is covered by metadata File: 05_RETS_Software_Development. PPT 60

Role of Parser Generators RETS Software Development q Used to produce parsers for: – DMQL – Validation Expressions q DMQL Java code will be used by RETS server to generate SQL from DMQL q Validation expression Java code will be used by RETS server to execute the validation expression File: 05_RETS_Software_Development. PPT 61

Benefits for using a Parser Generator RETS Software Development q Improves productivity for software developers – Not easy to write a good parser – Relatively easy to write the grammar input file for the parser generator q Avoids having to develop a parser – Difficult, demanding, time-consuming q Makes use of available parser generators – Examples: javacc, ANTLR, lexx q Produces more robust Java source code File: 05_RETS_Software_Development. PPT 62

Generating Java Source Code from a BNF Grammar File RETS Software Development BNF Grammar File (produced by developer) BNF grammar for DMQL - specified in RETS specification - see example on next two slides Grammar file syntax - specified in parser generator specification - see example on next two slides Javacc Parser Generator Java Source code File: 05_RETS_Software_Development. PPT 63

BNF Grammar File Fragment DMQL Javacc Input File - 1 RETS Software Development PARSER_BEGIN(DMQL) // The DMQL Javacc file which is used to generate the primary class for parsing DMQL queries public class DMQL {<Your Java code here>} PARSER_END(DMQL) // Parse the DMQL statement. void Input() : {< Java attributes>} {<Javacc expressions & your Java code>} // Match the DMQL Query at the highest level. void Query() : {< Java attributes>} File: 05_RETS_Software_Development. PPT 64

BNF Grammar File Fragment DMQL Javacc Input File - 2 RETS Software Development { Search() (<EOF> | <TERMINATOR>)}. . . BNF and Javacode Productions and additional Java code for processing tokens for parsing DMQL based on the RETS DMQL Grammar. . . // Terminals (Tokens) TOKEN : /* The DMQL Tokens (See RETS Specification) */ { < PLUS: "+" > | < SINGLE_CHAR: "? " > | < MULTI_CHAR: "*" >. | < OR: "|" > } File: 05_RETS_Software_Development. PPT 65

Application of Parser Output RETS Software Development Input rules for update (produced by MLS) Input DMQL (produced by client software) Java Validation Expression Code to evaluate Rules Java code to process DMQL SQL Parser Generator Output Validation Outcome File: 05_RETS_Software_Development. PPT 66

Resources for Parser Generators RETS Software Development q ANTLR, ANother Tool for Language Recognition – http: //www. antlr. org q Java Compilertm – Java. CCtm – https: //javacc. dev. java. net q Lexers and Parser Generators – http: //catalog. compilertools. net/lexparse. html File: 05_RETS_Software_Development. PPT 67

Topic Summary RETS Software Development q RETS provides a rich set of features for implementing a service or client q Careful design will result in a successful project q Work with the vendor or customer to ensure that the correct features are built q Read the RETS document and watch forward and reverse references q Use RETS 1. 5 as a supplement to RETS 1. 0. 1 – it includes many clarifications File: 05_RETS_Software_Development. PPT 68

Software Quality RETS Software Development File: 05_RETS_Software_Development. PPT 69

Top 10 signs Technology Has Taken Over Your Life RETS Software Development 10. You need to fill out a form that must be typewritten, but you can't because there isn't one typewriter in your house - only computers with laser printers. 9. You would rather get more dots per inch than miles per gallon. 8. You rotate your screen savers more frequently than your automobile tires. 7. You think of the gadgets in your office as "friends, " but you forget to send your father a birthday card. 6. You know Bill Gates' e-mail address, but you have to look up your own social security number. 5. On vacation, you are reading a computer manual and turning the pages faster than everyone else who is reading John Grisham novels. 4. The thought that a CD could refer to finance or music never enters your mind. 3. Your letterhead lists a phone number, a fax number, e-mail addresses for two online services, and your web site address but no street address. 2. You have never sat through an entire movie without having at least one device on your body beep or buzz. 1. You sign Christmas cards by putting : -) next to your signature. File: 05_RETS_Software_Development. PPT 70

Typical Session Workflow Login RETS Software Development Update Metadata ? Get Metadata Data about Offices, Agents, Listings Search Get Objects ? Data about available server data Get Object Pictures, Video Documents Logout File: 05_RETS_Software_Development. PPT 71