PROLOGIA-I18N
User-manual
SourceForge.net Logo
( back to Prologia I18N Homepage )

I - Prologia-I18N overview

Imagine you have been written an app for some years. Imagine that at the beginning it was only a simple app but it has grown to several thousands lines of code. Imagine that at the start it was only focused on the English market so didn't bother with internationalization & hard-coded every string in a given language. Now that the app is mature and that foreign customers are interested, how can you do to easily localize and translate this app ?

Prologia-I18N is the solution and is intended to help you localize Java (up to 1.4) source code.

Here's a summary of the process:


scheme


Black Arrow    : mandatory steps

Red Arrow    : not mandatory but may happen throughout the I18N process life

1 - Analyzer

The analyzer analyses java source files contained in a directory. If it finds string concatenations with variables (such as "hi " + sName + " ! How are you doing ?") it will choose to use a messageFormat. For a simple string with no concatenation or no variable, it will only replace the string by a function call and a key.

If you want to know more about message format, please see the Internationalization trail in the Java Tutorial here, and more specifically this section.

It is important to understand that the Analyzer does not actually modify the files. It just analyses them (hence the name!) and stores the results in AaA file.

Back to index

2 - User choice

The GUI allows you to have a look at all the different strings found in the sources. For each string or for messageFormat, you can choose:

File -> open…

You choose an AaA file (the one you want to display the content) & the directory where the original Java sources are stored (in order to display the context of the string & being able to decide accordingly).

File -> Save

Saves the last AaA file opened.

File -> Save as…

            Allow you to save the last AaA file opened with the last modifications done under a different name.

File -> Update from…

            This is one important feature. If you start the I18N process again against non-internationalized sources, you will have to make again all your choices… In order to bypass this boring task you can update the new AaA (the one just coming from the analyzer) with an old one where you had saved all your choices. All the Strings from the old AaA having "USER_CHOICE" or "UPDATED_FROM_AAA" as their "Reason" field will replace the default ones in the new AaA.

Tips:

1- If you want to select all packages (ie all files), select one package (ie one file) and press ctrl+A.

2 - If you ever clic on a cell in the 3rd column, the "reason" field of the string will be changed to "USER_choice" (even if you didn't actually even modified the old value). Hence, it will be used to update the future AaA created if you use the "update" function and this file. There is no mean to change the "reason" field so be warned as it may have long-term consequences.

3 - AaA key and AaA : each AaA file is associated with a AaAKey files. If your AaA file is named "MyAaA10-10-2004", your key file will be named "MyAaA10-10-2004Key". This

AaAkey file is generated automatically and is very important. So if you were to move your AaA file to another directory, be sure to move the associated key file too.

Back to index

3 - Internationalizer

            This is the one that will do the effective work. It takes the original sources and the choices made in AaA as input and produces internationalized java source code.

Note that the internationalizer takes a resourceBundle too as input in which it will store the keys and the default translation. A resourceBundle file (even if empty) must exist for the internationalizer to fill it in.

Back to index

4 - Reusing externationalized sources

You can add new files, modify the files, add code etc to the already I18Nized files and then apply the process again. But the 4' step is then mandatory in order to get back the keys corresponding to already internationalized strings in the code.

Back to index

II - Running the software

The first time you use the internationalization process, you should use the main GUI to familiarize yourself with the process and its output. Once familiarize, you may want to automate the tasks using batch files. We provide command line options for this.

1 - Main GUI:

GUI

The following command launches the main GUI, allowing you to use steps 1, 2 and 3 of the first diagram:

java -cp antlr.jar -jar I18N.jar

Note: ensure that antlr.jar is located in the same directory than I18N.jar

The following command-line enables to have default file specified 

java -cp antlr.jar -jar I18N.jar "sourceCodeDir" "TargetSourceCodeDir" "AaAFile" "ResourceBundle" ("listOfFiles")*

Note:

For large projects, the programs may be short on memory. It is recommended that you add the following Virtual Machine arguments:

-Xms256m -Xmx256m -XX:MaxPermSize=256m

You can enable assertions too, and report us any failed assertion. It is useful for debugging the program. Add the following argument:

-ea

exemple:

java -ea -Xms256m -Xmx256m -XX:MaxPermSize=256m -cp antlr.jar -jar I18N.jar "C:\bamboo\I18N\Code source Bamboo actuel" "C:\bamboo\I18N\Cible" "C:\bamboo\I18N\aaa" "C:\bamboo\I18N\Cible"

Back to index

2 - Batch files

Besides using the main GUI, you can decide to launch the 3 steps separately. This enables you to write scripts automating the work. Here is how to do it and the command line parameters used.

a/ Analyzer:

This command will launch the analyzer's GUI allowing you to graphically choose the directory to analyze & the AaA file:

java -cp antlr.jar -jar I18N.jar -Analyzer

The following command-line bypasses the GUI and sets default files for automatic analysis:

java -cp antlr.jar -jar I18N.jar  -Analyzer "sourceCodeDir" "AaAFile"

Example:

java -ea -Xms256m -Xmx256m -XX:MaxPermSize=256m -cp antlr.jar -jar I18N.jar -Analyzer "C:\Documents and Settings\Francois\Bureau\test" "C:\Documents and Settings\Francois\Bureau\TestCible\AaA"

b/ Updating AaA:

This command will update a AaA file (the target) with user-choices made in another AaA File (the source):

java -ea -Xms256m -Xmx256m -XX:MaxPermSize=256m -jar I18N.jar -UpdateAAA  "targetAaAFile" "sourceAaAFile"

c/ Internationalizer:

This command will launch the internationalizer GUI allowing you to graphically choose the files & directory :

java -jar I18N.jar -internationaliser

The following command-line enables to have default file specified for automatic internationalization::

java -cp antlr.jar -jar I18N.jar  -internationaliser "sourceCodeDir" "TargetSourceCodeDir" "AaAFile" "RessourceBundleDirectory"

Example:

java -jar I18N.jar - internationaliser "C:\Documents and Settings\Francois\Bureau\Bamboo 4.10.14 Sources modifiee 1 fichier" "C:\Documents and Settings\Francois\Bureau\Bamboo 4.10.14 Modif 1 fichier Target" "C:\Documents and Settings\Francois\Bureau\Bamboo 4.10.14 Modif 1 fichier Target\AaA" "C:\Documents and Settings\Francois\Bureau\Bamboo 4.10.14 Modif 1 fichier Target"

d/ Example

In order to sum things up, here is a MS-Windows batch file: it runs the 3 steps and quit if a fatal error is encountered.
All you have to do is to replace the folders locations in the file your own paths (where there is "TODO" written).
Batch_NO_GUI.gif
Use the "save as" feature of your browser ti save this file or click to open it in text-mode.

Back to index

III - The I18N Process in practice

1 - Import the BooI18N class to your un-internationalized code

First of all, your code should contain the BooI18N class. It contains static variables mandatory for compiling the internationalized code. Once internationalized, the strings are replaced by a call to a resourceBundle, which is defined as static in BooI18N. The BooI18N should be provided with the source or the executables of Prologia-I18N.

You may want to change the package defined at the beginning of BooI18N.java file to adapt it to where you want the file to be in your project.

BooI18N.java

The BooI18N class loads the language setting contained in the file language.properties. So if you want your app to be in French, just write "language=fr" if you want it to be in English write "language=en" (assuming that you have the right localized resourceBundles).

language.properties

Tip: At run-time, before loading any translation, the resourceBundle has to be loaded first. Hence ensure that it is loaded before the initialization of translated static strings, otherwise it would result as an error.

Back to index

2 - Analyzer

Analyze your code with the analyzer to produce an AaA (always associated with a AaAKey file) file. You may want to check section "Analyzer configuration file" to learn about the config file and how to have custom function names or string patterns to be automatically detected by the analyzer as not to be translated.

Back to index

3 - User choices, GUI, update

UserChoice_GUI_small2.gif

1- Package selection (if any)
2- File selection
    tip: you can multi-select packages, files and strings using ctrl and/or shift keys. Only the strings contained in selected packages/files are shown in '3'.
    If you multi-select packages, you can't choose which files are displayed: they are all displayed.
3- Source-code strings
4- Box is checked if the string is already internationalized (the string is then a key from a resourceBundle)
5- Action wanted for this string
6- String Information panel (state, action wanted and why, line number, key(s)...)
7- Source code preview. Allow you to see where the selected string comes from.
8- Internationalization preview. Allow to see the code generated by the Internationalization of the selected string.


Open the analyzer-generated AaA file, and change the values according to your preferences and your knowledge of the code.

If you started the process from an un-internationalized version of the code and wish to re-use the choices you had made during a previous internationalization, do not forget to update the current AaA file with the old one.

Do not forget to save the AaA file, and you're done!

Back to index

4 - Internationalizer

It takes the original source code and the AaA file (the one which comes from the analysis performed on the same original source code! Otherwise the result may be unpredictable…). If parts of your original source code are already internationalized, the corresponding resourceBundle should be provided as input too. Nevertheless, a BambooLanguage.properties MUST be given as input. If you are internationalizing from scratch, just create an empty file named BambooLanguage.properties.

It outputs the internationalized source code and a modified resourceBundle (containing the keys corresponding to the newly translated strings).

5 - use the internationalized sources

a/ Compiling:

According to 1, you have already imported the BooI18N java class in your code. If not, do it! There, the code (your original java source code modified by the internationalizer) code should compile correctly.

b/ Run-time:

In order to run it, the internationalizer-generated resourceBundle (bambooLanguage.properties) has to be located in a directory called "Internationalisation". This directory itself located at the root of the java project.

If you get errors running the internationalized program, please have a look at "V - misc problems and solutions + programmer's guide" where known issues are listed.

Back to index

IV - I18N of large existing projects: different philosophies

This section is designed to help those who already have an existing (large) project and want to internationalize it.

1 - I18N From scratch with AaA update

This solution allows you & your devs to keep on working as you were used to on your un-localized sources. Each time you want to produce an internationalized version of the sources:

/!\ Keep & backup your old AaA file in order to be able to update new AaA !

Pro:

Con:

2 - One-time Internationalization

            Internationalize one time your app, keep on working with the internationalized sources and do not hard-code any un-localized string any more. Then you only need to use Prologia-I18N one & only one time.

Pro:

less error prone on internationalization problems as coders cope directly with internationalized code

Cons:

- the output code (internationalized) is harder to read, harder to understand.

Back to index

3 - Alternative method

            This method combines the two first. You internationalize your sources and use the new sources as the starting point for your future coding, but you still use hard-coded un-localized strings

Pro:

            - no need to keep old AaA files any more (for updating user choices)

Con:

            - source code  contains 2 kinds of text (internationalized & non- internationalized ones)

Back to index

V - Misc Problems and solutions / programmer's guide

1 - Change your programming habits

a/ Example 1

If you are used to base your coding on string comparison, and even worst on some parts of a string variable, please, change your habits. Here is an example:

Imagine you have a menu in you app that, according to the context, enables you to remove clients or supplier. A string contains the words that will be shown in a menu. If somewhere else in your program you want to know the context, you may be tempted to do this way:

// here, sMenuDelete should contain "delete customer"
// or "delete client"
if (sMenuDelete.endswith("customer") )
     ;// do customer deletion action
else
     ;// do client deletion actions

The problem is that the translator will have to translate 3 strings: "delete customer", "delete client" and "customer". In English, that is not a problem, but who knows that the word customer will be the last word of "delete customer" once translated in another language? The program wouldn't work anymore.

What we advise you is to avoid making your code rely on string comparison. Programm logic should never rely on string content but on separated variable, which may influence directly the strings content.

// boolDeleteCustomer is a Boolean (either true or false)
if ( boolDeleteCustomer )
     ;// do customer deletion action
else
     ;// do client deletion actions

b/ Example 2:

Generally speaking, do your best to avoid string concatenation. For example, you may be used to do multiple tests and construct a string piece by piece depending on the result of each test.

String s = "";
If ( isAGuy )
            s += "Hi Mr";
else
     s += "Hi Ms";
s += " " + username;
if ( isATown )
     s += " welcome to my town";
else
     s += " welcome to my village";

This kind of code will give a hard time to the translator: he will have to translate many little pieces of string without knowing the context. So we recommend coding this way:

If  (isAGuy && isATown)
     s = "Hi Mr " + username + " welcome to my town";
else if (isAGuy && ! isATown)
s = "Hi Mr " + username + " welcome to my village";
else if (!isAGuy && isATown)
s = "Hi Ms " + username + " welcome to my town";
else
            s = "Hi Ms " + username + " welcome to my village";

Back to index

2 - Common exceptions caused by I18N

a/ nullPointerExceptions (Ternary operators and null checks)

Imagine the following line of java code:

(var != null) ? dothis("you won: " + var) : dothat();

If this code is to be internationalized using a message format, the following line will be included before the test "var != null" :

MessageFormat formatter000 = new MessageFormat("");
formatter000.setLocale(BooI18N.BOO_LOCALE);
Object[] messageArguments000 = {var+"" }; // /!\ problem /!\
formatter000.applyPattern(BooI18N.BOO_RESSOURCE_BUNDLE.getString("key"));
(var != null) ? dothis(formatter000.format(messageArguments000)) : dothat();

At run time, it will produce an error if var is null. Such case can't be automatically detected by Prologia-I18N: it is the programmer's task to be aware of the problem and to produce internationalizable code, compatible with Prologia-I18N.

There are 3 solutions here:

Back to index

b/ ExceptionInInitializer error:

This exception usually happens when you try to load a translation (BooI18N.BOO_RESOURCE_BUNDLE.getString("XXX") ) before the resourceBundle is even loaded (at run-time). You have to ensure that the resourceBundle is loaded before the call.



VI - Advanced I18N

Here are discussed advanced Prologia-I18N topics.

1 - Markups

We made it possible for programmers to specify, using specific comments in the java code, if they want the strings to be internationalized or not, and if they want messageFormat to be used or not.

a/ Grammar

S is the grammar axiom.

Non-terminal symbols are in bold italic.

Terminal symbols are underlined.

 

S à     //INT  optionsDeLigne         |           INTBloc

INTBloc à    //INT_BEGIN options \n code \n //INT_END

code à une ou plusieurs lignes de code Java

(all lines not containing « //INT_END »)

optionsDeLigne à    key=KeyValue, options        

|           options

options à      int=BoolValue, options

|           msg="MsgValue", options

|           msgFormat=BoolValue, options

|           Ø

KeyValue à alphanuméric character string (no spaces, but underscore possible): [a-z][A-Z][0-9][_]

BoolValue àtrue | false

MsgValue à KeyValue MsgValue | espace MsgValue

Notice: MsgValue & KeyValue are identical except that MsgValue can contain spaces.

b/ Example

//INT_BEGIN int=1, msgFormat=1, msg="I18N forced & messageFormat will be used"
if (idAgent == -1)
            return new BooReturn("L'agent " + trigram + " n'existe pas !!!", false);
if (idPri == -1)
            return new BooReturn("Le privilege " + pa + " n'existe pas !!!", false);
//INT_END
BooReturn br = new BooReturn("Valeur de retour:" + value); //INT int=1, msgFormat=0, key="returnValue"

Here, we tell the analyzer that we don't want to use a messageFormat & that the key used should be named "returnValue".

Warning: it is impossible for the analyzer to distinguish between I18N Markups and usual comments that contain the letters "INT". So be warned that a malformed I18N Markup will not be taken into account.

According to how your Analyzer.properties is configured, a warning message can be displayed in the console for each malformed markup. But there are chances that many of the warning messages will be issued by usual comments. For example:

//INT_BEGIN int=1, msggggggFormat=1
will issue a warning because of the way msgFormat is written.
// let's try to write INT in the comment
will issue a warning to  because it contains "INT".

Back to index

2 - Update only a few files (new!)

This option is useful if you have a large project but have modified only a few files since the last internationalization. It will prevent you from analyzing again the whole source directory.

For the time being, there is no command line for automating this process. You can only specify default files on the command line (cf II - Running the software / 1 - main GUI).

Once you have added the files (which MUST be in the same directory that the sources files) press the "AaA update" button. The following steps will happen:

Back to index

VII - Analyser config file

Analyzer.properties

In the file Analyzer.properties, you can set some settings.

1- Debug setting

The settings beginning with "DEBUG_" enable you to have a look at the internal of Prologia-I18N. They have the value 0 (disable) or 1 (enable).

# Shows ANTLR generated tree
DEBUG_ShowTree = 0
# Shows I18NStrings reconstructed by Analyzer. Those strings are read in
# the AaA file once it has been completely written.
DEBUG_PrintStrings = 1
# Display a warning message if a malformed markup is found (because
# we're unable to distinct a malformed markup from a normal comment
# countaining "INT")
DEBUG_ShowMarkupWarnings = 1

Back to index

2 - Keywords

The following keys define specific keywords

forbiddenFunctionCall: function names whose string parameters will not be translated.

SQLKeywords: if a string ever contains one of these keywords, it will not be translated and the reason for that will be set to "SQL".

OtherKeywords: if a string ever contains one of these keywords, it will not be translated and the reason for that will be set to "Other".

How to specify the keywords ?

The keywords should be separated by comma. Each space will be taken into account so be careful with spaces. For clarity's sake, we put all keywords on one line each but we must use a backslash ("\") character at the end of the previous line in order to forget blank spaces or tabulations preceding the keyword.

Examples :

forbiddenFunctionCall = System.out.println,\
                                 System.err.println,\
                                   getLong,\
                                   getInt,\
                                   getShort,\
                                   getString,\
                                   getDate,\
                                   WriteMessageInLogFile,\
                                   Font, java.awt.Font,\
                                   setString,\
                                   setName,\
                                   setActionCommand,\
                                   FileReader,FileWriter,\
                                   Superviser.getProperty,\
                                   .exec,\
                                    newFile,\
                                   ImageIcon,\
                                    getImage,\
                                   messageLog SQLKeywords = SELECT,\
                        FROM,\
                        WHERE ,\
                        UPDATE ,\
                        DELETE ,\
                        UPDATE ,\
                        HAVING ,\
                        AND ,\
                        ORDER BY ,\
                         VALUES ,\
                        VALUES(,\
                        INSERT INTO,\
                        IS NULL,\
                        IS NOT NULL,\
                        SET ,\
                        OR ,\
                        \ IN OtherKeywords = Throwable,\
                        caught,\
                        SQL error,\
                        .gif,\
                        .png,\
                        .rtf,\
                        ==>\

Back to index

3 - Other

alreadyI18N: used by the analyzer to recognize strings already internationalized.

TODO by Prologia-I18N developers: modify it in order to have only the resourceBundle name which should be used by both the analyzer and the internationalizer.

# Pattern indicating already externalized strings

already18NString = BOO_RESSOURCE_BUNDLE.getString

do_not_translate_there: location of  not-to-be translated parts of the program. Use of these setting is not recommended. Instead you should use a markup in order to advise the analyzer not to translate this string.

Nevertheless, if you want to play with this setting,  you must enter a series of filenames and the corresponding line number as follow:

do_not_translate_there = Superviser.java,68,\
                             Superviser.java,95,\
                             fileName.java,15,\
                             fileName2.java, 30

Back to index

VIII - ResourceBundles Maintenance

Here, RB will be used indifferently for "ResourceBundle" or for "property file" (backing up a resouceBundle).

1 - General

RBs contain the keys used in the app that match translated text in the original source code. The defaultResourceBundle (named bambooLanguage.properties) is created automatically by Prologia-I18N. The keys there match the original strings written by your developers.

For each language you want to support, you have to create a localized resourceBundle where the same keys will be associated with the matching translations. At run-time, your programs loads language.properties and sets the local according to what is written in that file. Then it loads the resourceBundle corresponding to that local (bambooLanguage_en.properties for example) and if it needs a key, which has not yet been translated, it uses the one contained in the default resourceBundle.

More info about resourceBundle can be found here: http://java.sun.com/docs/books/tutorial/i18n/resbundle/propfile.html

Back to index

2 - Editing ResourceBundles

a/ ResourceBundles structure

Keys are grouped together by where they are used. A comment, before each group of key, lists where (in which files) the keys in the group are used.

Example:

#package1/classe1
keyUsedOnlyInPackage1_classe1_1=texte1
keyUsedOnlyInPackage 1_classe1_2=texte2
#package1/classe2
keyUsedOnlyInPackage 1_classe2_1=texte3
keyUsedOnlyInPackage 1_classe2_2=texte4
#package1/classe1, package1/classe2
commonKeyInPackage1_classe1ANDpackage1_classe2_1=texte5
commonKeyInPackage1_classe1ANDpackage1_classe2_2=texte6
…

Back to index

b/ Modify ResourceBundle with "RBEditor" and Attesoro

RBEditor is a front-end to Attesoro (http://ostermiller.org/attesoro) which is a free open-source translation editor for Java programs. Our front-end enables you to choose which packages you want to translate. Then you can edit your translations package by package and the internationalization of your app will be done step by step, some parts totally translated while some others are waiting for translation. Then you should avoid the problem arising when you translate keys in alphabetical order and on the same screen you find translated and non-translated strings.

In order to run the RBEditor, simply double-click on the icon RBEditor.jar. Or on a command-line, execute "java -jar RBEditor.jar". In order for Attesoro to be launched, the file attesoro_1_5.jar should be in the same directory as RBEditor.jar.

Once the front-end is launched, open the default resourceBundle. The front-end will display a list containing all the files containing strings.

Front-end

Choose the files that are of interest and click "extract & edit" button. The corresponding keys and translations will be extracted in temporary files for each resourceBundle (including bambooLanguage_en.properties, bambooLanguage_it.properties, bambooLanguage_fr.properties, and so on if they exist). Attesoro will then be launched and will help you translate those strings and let you know which strings are not translated in which language.

Once you're done with the translations, save (the temporary files will be saved) and exit from Attesoro. The front-end will issue a Warning-box asking if you would like to "Save the changes made with Attesoro ?". Answer "OK" and the modifications done by Attesoro in the temp files will be inserted in your resourceBundles for all languages edited.

Concerning the Attesoro software please read its documentation here: http://ostermiller.org/attesoro/Using_Attesoro.pdf

Known bug: in Attesoro, you can choose to delete a key, but if you do so, the key won't be deleted in your resourceBundles. If you want to delete a key and Attesoro is not opened for editing, just delete the key in all the resourceBundles in which it is referenced. If Attesoro is opened, delete the key both in Attesoro and in the resourceBundles.

c/ Modify ResourceBundles "by hand"

Adding a key

You want to add a key corresponding to a test. If this key does already exist:

If the key does not already exist, simply add it to the right group of key or create a group of key (with the java filename in commentary).

Deleting a key:

If the key is in a group containing only one java class, just delete it. Else you have to change the group of the key and create (or move the key to, if already existing) a group where the java class are the same except that it does not contain the java class you want the key to be removed from.

Back to index

IX - Known bugs & feature request


see sourceforge's Prologia-I18N page for more (I hope not so !) and up-to-date bugs:
https://sourceforge.net/tracker/?atid=675023&group_id=116498&func=browse

and for feature requests: http://sourceforge.net/tracker/?atid=675026&group_id=116498&func=browse

1 - Analyzer

.java file should not end with a comment line. A carriage return (a blank line) must be inserted after any comment at the end of .java files.
status: not important, the grammar would have to be changed so no correction planned

2 - Internationalyzer

When internationalizing files (this happens if a string in the file is marked as "TO_TRANSLATE"), the internationalizer adds 3 impirts to the file:
import java.util.*;
import java.text.*;
import bambooAL.internationalization.*;

even if those import are already in the source-code, they will be imported again...
Worst the third one may not be valid for your project... We'll have to correct it.
status: urgent !
TODO: check for the import and only add needed imports.



( back to Prologia I18N Homepage )