ICU Data Loading, Packaging, Installation, Update, Customization

Summary of discussions 2005-aug..sep

• Meta data should cover the properties of IDs and items, including hidden items, available  IDs that don't have files/items, and dependencies.
• "Normal" items/IDs should not need to be listed manually. That is, the meta data input files that are created manually should not need entries for stored items that correspond to available IDs.
• Any meta data that is reasonably discoverable by build-time tools should be discovered by them. In particular, dependencies between items that can be discovered by looking into them.
• This is to keep explicit meta data at a minimum, to make it easy, and to keep it from getting out of sync with actual relationships.
• Not listing "normal" items means that explicit meta data in many cases can be empty.

Design meeting 2005-oct-12 (Markus Scherer, George Rhoten, Raghuram Viswanadha, Steven R Loomis, Vladimir Weinstein)

Development so far: The icupkg tool can now automatically discover dependencies between .cnv conversion tables (from delta-only to base tables) and between .res resource bundles (from aliases).

The dependencies between a resource bundle and its parent (according to its locale fallback) will be handled by the following:

• The .res file format will get a new flag for whether a .res file participates in locale fallback. Most do, except for ones like transliterator bundles, zoneinfo.res, and similar.
• The genrb tool will accept new syntax only for the top-level item, like be:table(nofallback) {...}.
• The ICU runtime libraries will treat a nofallback bundle as if ures_openDirect() had been called to open it. ures_openDirect() will still behave slightly better for such files, because if the bundle is missing altogether, it does not attempt to open a root bundle, unlike ures_open(). If the flag is missing completely, it is treated as using fallbacks.
• The icupkg and similar tools will assume a dependency on the parent bundle if the flag exists and is set to nofallback.

Alternatively, we could have a single per-tree flag in the meta data for whether all bundles participate in locale fallback. However, this would mean that users would be forced to use data trees (which we think are rarely used and poorly understood) to avoid mixing locale bundles and non-locale bundles on top level.

Consider moving locale bundles to a locales/ data tree (or non-locale bundles to a misc/ tree) for better separation between data items of different types and usage.

icupkg needs to support tree names in .res aliases: /ICUDATA-coll/...

With the above, icupkg will handle the majority of item dependencies. We decided that for now we will not handle other dependencies via meta data. For issues and details for future work on this see Jitterbug 4877 RFE: ICU data packaging meta data: support explicit item dependencies on package level

Decisions for meta data work for ICU 3.6:

• Use the converter alias table data formats (text & binary) for meta data.
• Use hidden tags (tags that won't be listed by the conversion API as available "standards") for implicit items ("normal" items/IDs that were added by icupkg rather than included in the explicit meta data), nofile IDs (IDs that are listed as available but don't have items/files), and hidden items (items/files that are loadable but not listed as available IDs). Use the existing syntax for aliases.
• Use some text file syntax to indicate whether name comparison is precise/binary (like strcmp(), for meta data for locale IDs) vs. loose (for case/space/dash-insensitive converter aliases). Store an enum value for this in the binary form.
• Store the number of hidden tags in the binary file if implicit/nofile/hidden tags are used. (Currently, there are two hidden tags hardcoded.)
• Use the old data format version (in the binary) unless the new features are used.
• New runtime code for available IDs skipping hidden items and providing a sorted list.
• May need to add filtering for available IDs.
• icupkg adds implicit items to manual/explicit meta data. It probably needs to first remove them when opening the meta data, then do all of the requested item updates (add/remove), then re-add implicit items that are in the resulting package.
• Use meta data aliases instead of .res bundles with just %%ALIAS entries.
• Explicit meta data is compiled for ICU4C into a binary file, and used and updated by icupkg. The result (not the manual/explicit input) contains the complete list of available and loadable items for runtime use.
• icupkg needs to be able to output dependency data for possible UI configurator/packaging tools.

ICU4J:

• Create a code (a tool?) for handling manual/explicit meta data in its text form and adding implicit items for runtime use.
• Create a pure Java version of icupkg. There, require "b" type data instead of using swapper code.
• Be able to use ICU4C's icupkg tool, with text output of the meta data files.
• Create Java code for writing .res files, for XLIFF2ICUConverter and LDML2ICUConverter so that the genrb tool from ICU4C need not be used as well. Don't include a .txt bundle syntax parser, just write the binary data from the XLIFF/LDML tools' internal data structures.

Examples for meta data usage:

• iw->he, currently a %%ALIAS-only bundle, becomes just an alias in meta data
• ja__TRADITIONAL listed as {hidden}
• fr listed implicitly
• coll/nl, currently empty to just document it as available (with standard UCA collation), listed as {nofile}

Markus Scherer, 20050824..26

Problem

Several ways to build/load/ship ICU and ICU-application data.

Hard to change data: Updating a .dat package means extracting all items and a manifest file via decmn, modifying the files and the manifest in the file system, swapping new items to the common type, repackaging with gencmn, and maybe swapping again to a new type. (All of this can be done with icupkg [below] with a single command-line invocation.) We also use index files for getAvailable() functions, and one would have to manually update these index files, at least for ICU4C.

Or, starting from a data-source download (now available only through CVS, not as .zip/.tgz) and modifying files, then packaging. Some of the index files are generated by the build.

Not a problem

This is about packaging and indexing. It is not about changing the format of .res files or any other ICU data files. It is also not about how ICU4J data is packaged in general (putting individual data files into the .jar).

However, as a result of the discussion, index files (at least in ICU4C) may be switched from being .res files to using the existing data format of the converter alias table. Such index files would probably use only a subset of that format's capabilities.

New icupkg tool

There is a new command-line tool that can create .dat files, swap them, add/remove/extract and list items, change copyright/comment, input package and item files in any platform type, output .dat package and extracted files in desired platform type. It does not yet handle indexes in any way. It could replace gencmn, decmn, and icuswap completely.

The icupkg tool is not finished. It needs testing/debugging, and if we decide to use automatically updated index files, such code will need to be written and integrated. Its package-handling code (mostly a C++ class) has been moved (aug26) to the toolutil library so that we can easily reuse this code in other codes, via Java JNI, etc. icupkg is part of CVS since 20050826.

Proposal

• Main design point and recommendation to users: Use .dat package files.
  • Portable (within l, b, e platforms).
  • Can be taken apart & updated.
  • Can be used without file system access if application provides pointer.
  • Built without (sometimes expensive or not installed) platform build tools (assembler, compiler, linker).
  • Needs only a single copyright statement per package.
  • Runtime: Needs either a hardcoded path, or a provided path, or a provided pointer.
• Use index files inside .dat packages for getAvailable() functions, for .cnv and .res items.
• Use cnvalias.icu file format for index files.
  • Use aliases for iw=he and for validSubLocales etc. (We currently use empty resource bundles instead.)
  • Single format for indexes (rather than two or more) reduces code for merging & updating indexes.
  • This format needed anyway for conversion aliases.
• Provide tool(s) for updating/modifying .dat package files and automatically updating and/or merging included indexes.
• Keep DLL packages as an option. (Its one advantage is not requiring configuration.)

Questions

• Keep single files as an option? (Used [e.g. by Apple] for adding/overriding packaged items. Harder to update index and/or enumerate items.)
• Change default loading from files-then-package to package-only? (We have a new function for controlling this in ICU4C 3.4.)
• Should we have the same index/enumeration method in ICU4C and ICU4J? In particular, if we choose index files, should we have the very same index files and index file formats in ICU4C and ICU4J?
• The new icupkg tool subsumes the functionality of the gencmn, decmn, and icuswap tools. Should we remove the older tools because they are not necessary any more, or should we keep them for a kind of tool/script backward compatibility? Or keep them for one release (as "@obsolete") and then remove them?

Alternatives

• DLLs: Platform-specific (not portable), cannot be taken apart or modified, need linker and (on most platforms) assembler or C compiler for building, two different ToC formats (embedded .dat vs. pointers from single .obj's). Found via system loader without configuration.
• Single files: Large number of files, require file system access, need copyright per file.
• Enumerate items rather than using and updating indexes: Needs custom code for file systems, easy for .dat packages and DLL packages, needs additional method (indexes?!) for aliases. In ICU4J in WebSphere, we don't know yet how to enumerate data items, and ICU4J 3.4 uses indexes again.
• Use .res files for .res indexes: Need additional code for merging & updating such indexes.

Scenarios

What do we want to achieve? Thoughts:

• Add a locale data file, or a localization resource bundle (to application data), or a conversion table.
  • At build time using the ICU source download.
  • After an application and ICU installation have been done on a production machine.
• Change the Swedish time separator character.
• Add/remove items to customize the data file for distribution. Especially now that the ICU4C source download contains the .dat package instead of the source files. (Without the new icupkg tool, customizing users effectively have to download the sources via CVS.)
• Create locale data for a new locale that's not in CLDR. (Requires multiple related CLDR files [root, parent, supplemental], some ICU-supplementary files, LDML2ICUConverter, etc.)

Not everything here is easy, nor handled nearly completely with the above proposal.

Clarification before the discussion

(Emails 20050824)

Doug asked

Here's my thoughts from an ICU4J perspective.

My main concern is to keep it easy for Java users to build and customize ICU4J with a minimum of tooling. Currently, all you need is Ant, JDK 1.4, and the ICU4J source jar. ICU4J uses locale-specific .res files (in the main) so that removing particular resources is 'easy' (disregarding problems that arise from removing some resources that other resources alias, or messing up inheritance). Adding resources to our resource tree is not really possible, but people can build their own resource trees using standard Java resources. I suspect the main demand is to reduce footprint by removing unused resources.

Requiring an ICU4C tool and accompanying ICU4C installation is a significant addition to the toolset. I'd like to keep the common case where ICU4J users just want to trim data in 'chunks'-- e.g. remove a bunch of locales, or all data related to an unused service-- as simple as it is currently.

Changes to internal structure of the data is simply a porting issue-- ICU4J needs to know how to read the .res files. If ICU4J retains the per-locale .res file format ICU4J is pretty much untouched, it already builds its indexes at build time by looking at what resources are present during the build. They are simply a list of strings naming the locales.

More significant customization would be done (I presume) by more sophisticated users who would be willing to install ICU4C and learn (at least) to use the tool. If ICU4J retains the per-locale res file model, then ideally these users could download the ICU .dat files and then use the tool to create the per-locale .res files (I don't know the tool's capabilities) without having to know how to build ICU and run other tools to do this. One-stop shopping is a plus here.

A Java tool that provided a GUI, and used ICU4JNI or called the ICU4C tool directly, would be a plus.

Markus replied

Doug, your concern is perfectly placed. Let me clarify.

The proposal does not change at all how ICU4J stores and loads .res files. If someone works from a .jar file or from its extracted contents, then nothing on that level will change. Changing the internal structure of .res files is not on the table.

The part of the discussion that affects ICU4J is the question of whether to use index files for getAvailable() and for aliases/equivalents/validSubLocales, or whether instead to enumerate the relevant items. We should have the same solution (index vs. enumeration) for both ICU4C and ICU4J.

The real change here would be that the index file would come from ICU4C (and an ICU4C tool) instead of from the ICU4J build process.

In addition, if we decide to go with index files, we need to decide whether to keep them in resource bundle format, or whether to use the data format of the conversion alias table. (The latter's source text form is actually simpler to edit than resource bundle sources.)

For generating new .res files, ICU4J already relies on ICU4C tools. If we add tool support for updating index files, this may or may not mean another ICU4C tool for that.

Another possible impact of the discussion on ICU4J: Ram has written a script that takes the ICU4C data, generates the Unicode properties files, and creates the ICU4J data .jar file. If the input to this script is a .dat package, then updating the .dat package with tools would also be a way to customize ICU4J data. The index files would be updated automatically before the script gets the .dat file.

Yes, the new icupkg tool (and the old decmn tool) can take a .dat file and extract all of the .res and other items in it as separate files. (Much like unzip.)