ICU Build Installation Issues and Beyond Yves Arrouye

ICU Build / Installation Issues and Beyond Yves Arrouye <yves@realnames. com> Steven R. Loomis <srl@jtcsv. com> Cupertino, CA, USA / September 11, 2000 1 First ICU Developer. Workshop : Build/Installation. . .

Topics • • • Configuring and building ICU Integrating ICU into your product Porting ICU to a new platform Migrating an existing product to ICU Adding new data to ICU Packaging application data Cupertino, CA, USA / September 11, 2000 2 First ICU Developer. Workshop : Build/Installation. . .

Build Process – Directory Structure • icu – data – mostly text source files for data – source • • common - core unicode functionality config – build configuration files data – data files used during the build extra - ustdio (file handling, used by tools) i 18 n – higher level resource-based localization samples test – cintltst (C), intltest (C++) • tools – for processing and packaging data files Cupertino, CA, USA / September 11, 2000 3 First ICU Developer. Workshop : Build/Installation. . .

File Name Conventions • Library – Win 32: icuuc. dll, icui 18 n. dll, icudata. dll – Unix: libicu-uc. so, libicu-i 18 n. so, libicudata. so • (Could be. a, . sl, . x depending on platform) • Include file – Public API’s under “unicode/” directories – Example: search path icu/source/common #include “unicode/ucnv. h” Cupertino, CA, USA / September 11, 2000 4 First ICU Developer. Workshop : Build/Installation. . .

Dependencies • Required – ANSI C & C++ compiler • NO: templates, exceptions, RTTI, STL – Working math library – stdio (or wrapper, especially for tools) • Nice to have – – Dynamic loading Memory mapping Threads IEEE 754 numerics Cupertino, CA, USA / September 11, 2000 5 First ICU Developer. Workshop : Build/Installation. . .

Unix – “configure” script • GNU Autoconf system – – • Detects platform capabilities Generates Makefiles Install prefix: --prefix=/usr/local – – – /usr/local/libicu-uc. so /usr/local/share/icu (data) /usr/local/include/unicode /usr/local/sbin (tools) /usr/local/bin (samples) Cupertino, CA, USA / September 11, 2000 6 First ICU Developer. Workshop : Build/Installation. . .

Unix – configure options • Things to turn off or on (use --help) Threads, hpuxcma, Mapped, Tests, Extras, Samples • Out of source build (--srcdir) – Share one read-only source base with multiple targets • mkdir /build/hpux 11 -hppa-gcc/ • cd /build/hpux 11 -hppa-gcc/ • /icu/source/configure –srcdir=/icu/source • Example call: sh. /configure --enable-threads=no --prefix=/opt Cupertino, CA, USA / September 11, 2000 7 First ICU Developer. Workshop : Build/Installation. . .

Unix – Building gmake all code, data, tests, tools gmake check run all tests gmake install optional: install to $(prefix) (Use just ‘make’ if your make is GNU. . ) Cupertino, CA, USA / September 11, 2000 8 First ICU Developer. Workshop : Build/Installation. . .

Unix – configure variables • Shell variables CC=/usr/opt/bin/cc CXX=/usr/opt/bin/CC (C++) – CC and CXX can need full paths CPPFLAGS= -Dmydefine=3 -I/path (both C++ and C compiler) LDFLAGS=-L/usr/local/lib –lotherlib FORCE_LIBS=-lm –lpthread –lstdc++ PATH=… • sh CC=/usr/opt/bin/cc. /configure … Cupertino, CA, USA / September 11, 2000 9 First ICU Developer. Workshop : Build/Installation. . .

Unix – run. Configure. ICU • For a specific set of platforms – Pre-selects compiler and options for reference platforms and then some. – Use instead of ‘configure’ sh run. Configure. ICU AIX 4. 3 xl. C … • Supported platforms: AIX 4. 1. 5 xl. C SOLARISCC SOLARISGCC Linux. Red. Hat 5. 2 Linux. Red. Hat 6. x HP/UX 10. 2 CC HP/UX 10. 2 ACC HP/UX 11 ACC AIX 4. 3 xl. C SOL 2. 7 CC PTX SOLARISX 86 AIX 4. 3 VA Cupertino, CA, USA / September 11, 2000 10 First ICU Developer. Workshop : Build/Installation. . .

Win 32 • Use ‘allinone’ project • ICU_DATA – sets target location for icudata. dll (use a trailing backslash ) – (Default: same as other DLL locations) • Target: Debug vs. Release • Makedata project builds and updates ICU data • Hit F 7! Cupertino, CA, USA / September 11, 2000 11 First ICU Developer. Workshop : Build/Installation. . .

Running the Tests • Set ICU_DATA (trailing slash or backslash) and set TZ=PST 8 PDT. . (or “gmake check”) • Intltest – shows errors after each level – – Intltest LIST list possible tests Intltest format Intltest rbbi/LIST Intltest –v format • Cintltest –l - list possible tests – Cintltest /tsvconv – Cintltest –v /tsformat Cupertino, CA, USA / September 11, 2000 12 First ICU Developer. Workshop : Build/Installation. . .

Integrating ICU – Compiling • UNIX: can include …/share/icu/Makefile. inc or, -I$(prefix)/include • Win 32: add icuinclude to search path Cupertino, CA, USA / September 11, 2000 13 First ICU Developer. Workshop : Build/Installation. . .

Integrating ICU – Linking • May not need to link in “i 18 n” • Unix: use $(ICULIBS) or –licu-uc –licu-i 18 n • Win 32: iculibDebugicuuc. lib, iculibDebugicui 18 n. lib (or Release) Cupertino, CA, USA / September 11, 2000 14 First ICU Developer. Workshop : Build/Installation. . .

Integrating ICU – Data • u_set. Data. Directory() or ICU_DATA – Set these before any ICU calls! • Modes: – DLL - copy libicudata. so or icudata. dll with other libs – Common – set a path to dir containing icudata. dat – Linked – Link app with data dll, call u_set. Common. Data() before any ICU calls are made. (see udata. h ) Cupertino, CA, USA / September 11, 2000 15 First ICU Developer. Workshop : Build/Installation. . .

Port ICU to new platform • Keep us in the loop! – (You will be glad, the next time you don’t have to back-port) • Bring up incrementally, prioritize (if needed) • If you fix a general bug, don’t #ifdef it • Test for features, not platforms (except putil. c) – #if U_HAVE_XXX instead of #if MYPLATFORM • Read and respect the ICU coding guidelines Cupertino, CA, USA / September 11, 2000 16 First ICU Developer. Workshop : Build/Installation. . .

Port – Sane order • Common – umachine. h platform include. h • Not needed under autoconf • Base types (int 32_t, . . ) – cmemory. h, cstring. h, putil. c, umutex. c • Platform utilities: strcpy() through default time zone. • putil. c has POSIX routines for many of these tasks – udata. c – dynamic loading, memory map. . • I 18 n • samples/date – test linking an executable! • Toolutils Cupertino, CA, USA / September 11, 2000 17 First ICU Developer. Workshop : Build/Installation. . .

Port – Sane order, cont’d • Tools – – – Can skip pkgdata/genccode/gencmn if need be Makeconv & gencnval – converter files Ustdio – UFILE* library, used in. . Genrb – Resource Bundle compiler Ctestfw – used in cintltest • Test – Intltest – C++ [tsmthred. cpp: requires threads] – Cintltest – C Cupertino, CA, USA / September 11, 2000 18 First ICU Developer. Workshop : Build/Installation. . .

Port – Sane order, #3 • Other tools (gentz, gencnval, . . ) • Data tools – Determine data loading options (dll, memory map, . . ) – Go back and update udata. c if needed – Pkgdata/genccode/gencmn • Gotchas: – Sometimes C++ linkage is needed for icudata. – Exporting symbols (see U_EXPORT macros) – Tests sensitive to limits of floating point capability Cupertino, CA, USA / September 11, 2000 19 First ICU Developer. Workshop : Build/Installation. . .

Port – keeping autoconf happy • autoconf 2. 13 depends on GNU m 4 – Only edit “. in” files [if possible!] • Add compiler choice in run. Configure. ICU • Find a ‘close’ mh- file in source/config, copy to mh-yourplatform • Platform detection: – Aclocal. m 4, config. guess, config. sub • Custom tweaks in configure. in – Search for pthreads, headers, math lib – rpath separator • Test other platforms! Cupertino, CA, USA / September 11, 2000 20 First ICU Developer. Workshop : Build/Installation. . .

Migrating to ICU - Context • What does the “international contexts” of your application consist of? – Locale • System/Default Locale – for OS interface • Locale of an end user • Locale of external services – Codepage • System/Default • File format or communications protocol • Internal processing – Timezone Cupertino, CA, USA / September 11, 2000 21 First ICU Developer. Workshop : Build/Installation. . .

Migrating – Binding • To what are these contexts bound? – User (may be the same as system settings) – Connection, thread, transaction (server) • What type of international data does my application need? – Localized messages – Collation sequences – New converters • See ICU Design Principles & Services Guide – Converters, Formatters, etc. are per-thread Cupertino, CA, USA / September 11, 2000 22 First ICU Developer. Workshop : Build/Installation. . .

Migrating – Data Path • Either widen the data path… – Unicode everywhere – Direct manipulation easier • …Or build wrappers – Knowledge of Unicode only at certain levels – Pass data as opaque ‘blobs’ of text elsewhere Cupertino, CA, USA / September 11, 2000 23 First ICU Developer. Workshop : Build/Installation. . .

Migrating – Input/Output • Application data input and output – Conversion APIs to obtain data in Unicode, output in legacy codepages – ICU locale suport to manipulate dates, times, currencies – ICU collation support for all comparison, break iterators instead of scanning for whitespace to separate words and sentences. Cupertino, CA, USA / September 11, 2000 24 First ICU Developer. Workshop : Build/Installation. . .

Migrating - Messages • Use Resource Bundles to externalize locale-depentant data such as messages. • Use Message. Format, ustdio • Translate the messages. Cupertino, CA, USA / September 11, 2000 25 First ICU Developer. Workshop : Build/Installation. . .

Adding Data to ICU • Locale language _ country _ variant – Determine where data belongs in hierarchy • Language – Don’t set specific currency (format with ¤ OK) – Base translations: country/language names, days of weeks, months • Country – Currency codes – Formatting variations • Variant (e. g. Euro) – Add to: data/index. txt, tools/genrbfiles. mk Reslocal. mk for temporary additions – Use Locale Explorer for examples Cupertino, CA, USA / September 11, 2000 26 First ICU Developer. Workshop : Build/Installation. . .

Add – Converter Alias • Alias – convrtrs. txt ibm-367 us-ascii {MIME} ANSI_X 3. 4 -1968 {IANA} ascii # Table file: ibm-367. ucm – Standard name follows alias (above: us-ascii is the MIME name) – Algorithmic types listed in ucnv_bld. c – Locale argument: ISO_2022, locale=jp • Rebuild alias table with gencnval – (“gmake all” in tools, or rebuild “makedata”project) Cupertino, CA, USA / September 11, 2000 27 First ICU Developer. Workshop : Build/Installation. . .

Add – Converter Table • Format: Modified IBM CDRA • Choose a type – SBCS – single byte <U 0127> <UFF 01> x. B 1 x 20 # comment |0 # fallback |1 – DBCS – double byte <U 2537> x 43x. EC # |0 – MBCS / EBCDIC_STATEFUL • Mixed single and double byte. See documentation • The LAST entry overrides if there is a conflict Cupertino, CA, USA / September 11, 2000 28 First ICU Developer. Workshop : Build/Installation. . .

Add – Converter Table to Build • Add. ucm file to tools/makeconv/ucmfiles. mk ucmlocal. mk for temporary additions Cupertino, CA, USA / September 11, 2000 29 First ICU Developer. Workshop : Build/Installation. . .

Packaging Data What is a package? • 1. 6: 160+ converters, 170+ bundles = ~7. 6 Mb • udata: unified API to create, locate, and manage data • Raw form of data (usable in this way) – – Single directory (no subdirs) Individual files (. res, . dat, . cnv, . . ) Each file has versioning, endianness, etc. . Created with unewdata. h (toolutils library) Cupertino, CA, USA / September 11, 2000 30 First ICU Developer. Workshop : Build/Installation. . .

Pkgdata – Modes • Common (icudata. dat) – Single file, with internal table of contents – Memory mapped – Portable among platforms which share endianness and base charset (ASCII vs EBCDIC) • Shared Library (icudata. dll) – Data converted into static byte arrays – One symbol per library: table of contents – Can be used with u_set. Common. Data() for compile time linkage – DLL search path finds your data! Cupertino, CA, USA / September 11, 2000 31 First ICU Developer. Workshop : Build/Installation. . .

Pkgdata – List File • Manifest for package creation – Root. res ja_jp. res c: otherfilesotherdata. dat • Relative to –s source directory Cupertino, CA, USA / September 11, 2000 32 First ICU Developer. Workshop : Build/Installation. . .

Pkgdata – Options • -m mode (DLL or Common) • -O buildoptions (Unix: path to Makefile. inc, Win 32: path to ICU dir) • -p shortname (e. g. ‘icudata’ or ‘myappdata’) • -I installdir • Examples: pkgdata –m dll –p xyz –O /opt/icu/Makefile. inc xyz. lst pkgdata –m dll –p xyz –O D: E: Workicu xyz. lst (Win 32 Debug) Cupertino, CA, USA / September 11, 2000 33 First ICU Developer. Workshop : Build/Installation. . .

Q&A Cupertino, CA, USA / September 11, 2000 34 First ICU Developer. Workshop : Build/Installation. . .