Innovative Routines International IRI Inc The Co SORT

  • Slides: 18
Download presentation
Innovative Routines International (IRI), Inc. The Co. SORT Company Simple Shared Metadata

Innovative Routines International (IRI), Inc. The Co. SORT Company Simple Shared Metadata

About Us Innovative Routines International (IRI), Inc. n Founded 1978, New York n Released

About Us Innovative Routines International (IRI), Inc. n Founded 1978, New York n Released world’s 1 st commercial sorts for: CP/M, DOS, UNIX, Windows, Linux n 30+ Int’l Agencies, Profitable sinception n Core products: Co. SORT, FACT, Row. Gen n Fast Sorts, ETL, Reorg, Reports, Test Data n Co. SORT recommended by all H/W OEMs n Embedding by/with top ISVs: Acucobol, AMS, Cincom, Experian, IVIS, Kalido, Micro Focus, Sabre, Soliton, Sun, Valdero, Vi. PS, etc. Co. SORT/ IRI, Inc. 2194 Highway A 1 A, Suite 303 Melbourne, FL 32937 -4932 USA www. cosort. com 1 -800 -333 -SORT

Optimizing Data Population & Presentation IRI software stages, reports and synthesizes large data volumes

Optimizing Data Population & Presentation IRI software stages, reports and synthesizes large data volumes outside the database. FACT, Co. SORT, and Row. Gen leverage the speed and simplicity of sequential (flat) file processing. “The Customer Intelligence Lifecycle”

Perform and Speed Oracle Reorgs and ETL ! E T Co. SORT’s FAst extra.

Perform and Speed Oracle Reorgs and ETL ! E T Co. SORT’s FAst extra. CT (FACT) for Oracle Co. SORT’s Sort Control Language (Sort. CL) L SQL*Loader

E TL Co. SORT’s FAst extra. CT (FACT) for Oracle Faster Oracle Unloads •

E TL Co. SORT’s FAst extra. CT (FACT) for Oracle Faster Oracle Unloads • Rapidly Dumps Table Contents (Extract) – Parallel hints, SQL syntax, flat file results! • Creates Co. SORT/Sort. CL Metadata (Transform) – Select, join, sort, aggregate, calculate, convert, and re-format • Creates SQL*Loader Metadata (Load) – Pre-sorted flat files can re-populate Oracle up to 90% faster • Combine all 3 into 1 Fast E-T-L Pass! Ask for demo

Fast data integration & staging (Transformation) all-in-one I/O pass … E T L Co.

Fast data integration & staging (Transformation) all-in-one I/O pass … E T L Co. SORT Sort Control Language (Sort. CL) … through many, large, differentlyformatted inputs: ü Select via record filters or conditional include/omit ü Sort / Merge any number of keys in any position, 2 GB+/min ü Join SQL match syntax 1 -1, many-1, inner & outer ü Convert translate input field data types to new types ü Aggregate min, max, average, sum, count (sub and grand) ü Calculate across rows to perform math (+ sci functions) ü Re-map change field positions, sizes, and values ü Report to highly-formatted, multi-level output targets ü User Exits for custom input, compare and output criteria

IRI’s FACT extracts Oracle tables, and creates metadata for [simultaneous] Co. SORT/Sort. CL transform

IRI’s FACT extracts Oracle tables, and creates metadata for [simultaneous] Co. SORT/Sort. CL transform and [pre-sorted] SQL*Loader bulk load operations. All you do is create the FACT and Sort. CL E and T config. files, and watch it fly! See demo.

Oracle 9 i Table: “Orders” ORDER ----11040 11041 11042 11043 11044 11045 11046 11047

Oracle 9 i Table: “Orders” ORDER ----11040 11041 11042 11043 11044 11045 11046 11047 11048 11049 11050 CUSTOMER -----------------Great Lakes Food Market Chop-suey Chinese Comeio Mineiro Spealitedu monde Wolski Zajazd Bottom-Dollar Markets Die Wandernde Kuh Eastern Connection Bottom-Dollar Markets Gourmet Lanchonetes Folk och fa. B EMPLOYEE --------Peacock, Margaret Leverling, Janet Fuller, Andrew Buchanan, Steven Peacock, Margaret Suyama, Michael Callahan, Laura King, Robert Leverling, Janet Callahan, Laura SHIP_DATE FREIGHT ---------18. 84 28 -APR-98 48. 22 01 -MAY-98 29. 99 29 -APR-98 8. 8 01 -MAY-98 8. 72 70. 58 24 -APR-98 71. 64 01 -MAY-98 46. 62 30 -APR-98 24. 12 04 -MAY-98 8. 34 05 -MAY-98 59. 41 ORDER ----11051 11052 11053 11054 11055 11056 11057 11058 11059 11060 11061 CUSTOMER -----------------La maison d. Asie Hanari Carnes Piccolo und mehr Cactus Comidas para llevar HILARIO-Abastos Eastern Connection North/South Blauer See Delikatessen Ricardo Adocicados Franchi S. p. A. Great Lakes Food Market EMPLOYEE --------King, Robert Leverling, Janet Fuller, Andrew Callahan, Laura King, Robert Callahan, Laura Leverling, Janet Dodsworth, Anne Fuller, Andrew Peacock, Margaret SHIP_DATE FREIGHT ---------2. 79 01 -MAY-98 67. 26 29 -APR-98 53. 05. 33 05 -MAY-98 120. 92 01 -MAY-98 278. 96 01 -MAY-98 4. 13 31. 14 85. 8 04 -MAY-98 10. 98 14. 01

FAst extra. CT (FACT). ini File # "orders. ini" # FACT initialization file for

FAst extra. CT (FACT). ini File # "orders. ini" # FACT initialization file for ETL example # gets run with fact emp. ini DATABASE=ORACLE INSTANCE=test USERID=scott PASSWORD=tiger QUERY=SELECT * FROM orders OUTFILE=stdout # extracted records pipe out to Sort. CL; metadata in stdout. ddf VARIABLE DELIM=, FRAMEFIELD=EMPLOYEE # Specifies the name of the column to be framed FRAMECHAR=" # Framed column to be enclosed in double quotes FETCHSIZE=auto LOADTABLE=orders_sorted LOADTYPE=TRUNCATE # sorted records from table ‘orders’ will insert into new table DATAEXT=. dat CTLEXT=. ctl DDFEXT=. ddf REPORTEXT=. log DATEFORMAT=YYYY/MM/DD

Co. SORT (Sort. CL) File Metadata [cosort@demo example]$ cat stdout. ddf /FILE=stdin /FIELD=(ORDER_ID, POS=1,

Co. SORT (Sort. CL) File Metadata [cosort@demo example]$ cat stdout. ddf /FILE=stdin /FIELD=(ORDER_ID, POS=1, SEP=', ') /FIELD=(CUSTOMER, POS=2, SEP=', ') /FIELD=(EMPLOYEE, POS=3, SEP=', ', FRAME='"') /FIELD=(SHIP_DATE, POS=4, SEP=', ') /FIELD=(FREIGHT, POS=5, SEP=', ', NUMERIC) Notes: 1) FACT created this metadata during the Oracle table unload 2) stdin will be the piped input into Co. SORT’s Sort. CL program 3) /FIELD statements define each fixed/floating data column 4) Sort. CL uses field names for transforms and output remapping 5) Centralized metadata can be (re)used in Sort. CL & Row. Gen apps 6) Meta. Integration tool users can also build. ddf’s automatically!

Co. SORT (Sort. CL) Application [cosort@demo example]$ cat orders_sorted. scl /INFILE=stdin /SPEC=stdout. ddf #

Co. SORT (Sort. CL) Application [cosort@demo example]$ cat orders_sorted. scl /INFILE=stdin /SPEC=stdout. ddf # FACT-created metadata description /CONDITION=(change_emp_cust, TEST=(EMPLOYEE OR CUSTOMER)) /SORT # Pre-sorted loads are faster! /KEY=CUSTOMER # 2 -key sort, ascending /KEY=SHIP_DATE /OUTFILE=stdout. dat # Streaming output to SQL*Loader /OUTFILE=orders. csv # Outfile 2: Table replication (sorted) /OUTFILE=orders. sum # Outfile 3: Freight total report /SPEC=order 2. scl # Metadata for Sort. CL aggregation app Notes: 1) All Sort. CL scripts flow this way: Input Process Output 2) stdin is the piped input from FACT’s table unload 3) /SPEC references /FIELD and other Sort. CL metadata syntax 4) Sort. CL can map >1 inputs & formats to >1 outputs & formats

Single-Pass Oracle ETL Operation [cosort@demo example]$ vi orders_ETL. sh rm -r stdout. dat mkfifo

Single-Pass Oracle ETL Operation [cosort@demo example]$ vi orders_ETL. sh rm -r stdout. dat mkfifo stdout. dat fact orders. ini info_file fact orders. ini | sortcl /spec=orders_sorted. scl & sqlldr scott/tiger control=stdout. ctl DIRECT=TRUE~ Notes: 1) rm -r stdout. dat clears any same-named prior FIFO buffer 2) mkfifo stdout. dat creates a new FIFO buffer for stdout. dat 3) fact info_file orders. ini creates both metadata and log files 4) fact orders. ini extracts and reformats table data (see. ini file) 5) | sortcl drives FACT-unloaded data into Co. SORT (Sort. CL) 6) & sqlldr is named pipe from Co. SORT (Sort. CL) to SQL*Loader

ET L Co. SORT’s Row. Gen Data Synthesizer New Product! Build Your Own Custom

ET L Co. SORT’s Row. Gen Data Synthesizer New Product! Build Your Own Custom Files ü Prototype Applications – Create data and file formats your projects need ü Share File Formats – Provide accurate layouts without real data ü Specify Value Ranges – Use selection and set files for realistic data ü Simulate DB Ops – Quickly test table loading and query scenarios ü Benchmark Testing – Gen big files for hardware and software Po. Cs

Sort. CL & Row. Gen: Same Metadata! /INFILE=orders_data # source (Sort. CL) / placeholder

Sort. CL & Row. Gen: Same Metadata! /INFILE=orders_data # source (Sort. CL) / placeholder (Row. Gen) # layouts for: input data (Sort. CL) / generated data (Row. Gen) /FIELD=(ORDER_ID, set=order_id. set, POS=1, SIZE=5. 0, SEP='|', NUMERIC) /FIELD=(CUSTOMER, set=names. set, POS=2, SEP='|') /FIELD=(SHIP_DATE, SET=dates. set, POS=3, SIZE=10, SEP='|', ISO_TIMESTAMP) /FIELD=(ORDER_VALUE, SET=values. set, POS=4, SIZE=6. 2, SEP='|', NUMERIC) # set file values for Row. Gen (ignored by Sort. CL) /INCOLLECT=252 # 252 records processed (Sort. CL) / generated (Row. Gen) /SORT /KEY=CUSTOMER /KEY=SHIP_DATE # 2 -key sort, ascending /OUTFILE=customer_orders # names the output target # this layout (1 of 3) derives subtotals /DATA={14}" " /DATA=CUSTOMER /FIELD=(SUM_ORDER_VALUE, POS=46, SIZE=9. 2, CURRENCY) /FIELD=(CT_ORDER, POS=58, SIZE=2) /DATA="n" /DATA={59}"-" /DATA="n" /SUM SUM_ORDER_VALUE FROM ORDER_VALUE BREAK CUSTOMER /COUNT CT_ORDER BREAK CUSTOMER

Transform & Synthesis MD (Con’t. ) /OUTFILE=customer_orders # same file name (format overlay) #

Transform & Synthesis MD (Con’t. ) /OUTFILE=customer_orders # same file name (format overlay) # this layout (2 of 3) derives grand totals /FIELD=(SUM_ORDER_VALUE, POS=45, SIZE=10. 2, CURRENCY) /FIELD=(CT_ORDER, POS=57, SIZE=3) /SUM SUM_ORDER_VALUE FROM ORDER_VALUE /COUNT CT_ORDER /OUTFILE=customer_orders # same file name (format overlay) # this layout (3 of 3) contains detail records /HEADREC=“Ship Date Customer Order ID Value Ct. n------------------------------n" /FIELD=(SHIP_DATE, POS=1, SIZE=10, ISO_TIMESTAMP) /FIELD=(ORDER_ID, POS=36) /FIELD=(ORDER_VALUE, POS=47, SIZE=8. 2, CURRENCY)

Synthesize, Simulate, Share Need Sample Data? Row. Gen Can Build You: • Simultaneous, multiple

Synthesize, Simulate, Share Need Sample Data? Row. Gen Can Build You: • Simultaneous, multiple target files in 1 or more formats, featuring: – Random field values in >100 data types • ASCII upper/lowercase, EBCDIC, COBOL, IP Addresses, Timestamps, etc. – Popular file formats • CSV, Line / Record / Variable Sequential, MF ISAM, ELF, Unisys VBF, etc. – Randomly-selected data from real “set” files, incl. DB extracts – – • Conditional and duplicate filters + custom output mapping (data segmentation) Detail records for aggregation, with simultaneous check file(s) created Sorting, aggregation, and cross-field expressions (vertical and horizontal calcs) Numeric fields in precisely-defined ranges and layouts Multiple record layouts in one or more simultaneous targets – Using same metadata as/for Sort. CL transforms and reports! • New target data in the same file format (via random seed values) • Mixed valid and invalid records (to test app filtering) • Realistic, structured, fully-custom report formats

Innovative Routines International (IRI), Inc. The Co. SORT Company cosor t. com 1 -800

Innovative Routines International (IRI), Inc. The Co. SORT Company cosor t. com 1 -800 -333 -SORT