     [file version.texi not included]

GNU Libidn
**********

This manual is last updated UPDATED for version VERSION of GNU Libidn.

Copyright © 2002-2016 Simon Josefsson.

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

Introduction
************

GNU Libidn is a fully documented implementation of the Stringprep, Punycode and IDNA specifications.  Libidn's purpose is to encode and decode internationalized domain name strings.  There are native C, C# and Java libraries.

The C library contains a generic Stringprep implementation.  Profiles for Nameprep, iSCSI, SASL, XMPP and Kerberos V5 are included. Punycode and ASCII Compatible Encoding (ACE) via IDNA are supported. A mechanism to define Top-Level Domain (TLD) specific validation tables, and to compare strings against those tables, is included. Default tables for some TLDs are also included.

The Stringprep API consists of two main functions, one for converting data from the system's native representation into UTF-8, and one function to perform the Stringprep processing.  Adding a new Stringprep profile for your application within the API is straightforward.  The Punycode API consists of one encoding function and one decoding function.  The IDNA API consists of the ToASCII and ToUnicode functions, as well as an high-level interface for converting entire domain names to and from the ACE encoded form.  The TLD API consists of one set of functions to extract the TLD name from a domain string, one set of functions to locate the proper TLD table to use based on the TLD name, and core functions to validate a string against a TLD table, and some utility wrappers to perform all the steps in one call.

The library is used by, e.g., GNU SASL and Shishi to process user names and passwords.  Libidn can be built into GNU Libc to enable a new system-wide getaddrinfo flag for IDN processing.

Libidn is developed for the GNU/Linux system, but runs on over 20 Unix platforms (including Solaris, IRIX, AIX, and Tru64) and Windows.  The library is written in C and (parts of) the API is also accessible from C++, Emacs Lisp, Python and Java.  A native Java and C# port is included.

Also included is a command line tool, several self tests, code examples, and more.

Getting Started
===============

This manual documents the library programming interface.  All functions and data types provided by the library are explained. Included are also examples, and documentation for the command line tool idn that provide a quick interface to the library.  The Emacs Lisp bindings for the library is also discussed.

The reader is assumed to possess basic familiarity with internationalization concepts and network programming in C or C++.

This manual can be used in several ways.  If read from the beginning to the end, it gives a good introduction into the library and how it can be used in an application.  Forward references are included where necessary.  Later on, the manual can be used as a reference manual to get just the information needed about any particular interface of the library.  Experienced programmers might want to start looking at the examples at the end of the manual (see Examples), and then only read up those parts of the interface which are unclear.

Features
========

This library might have a couple of advantages over other libraries doing a similar job.

It's Free Software
     Anybody can use, modify, and redistribute it under the terms of a free software license.

It's thread-safe
     No global state is kept in the library.  All functions are re-entrant.

It's portable
     The code is intended to be written in pure ANSI C89.  It has been tested on many Unix like operating systems, and Windows.

It's modularized
     The library is composed of several modules, and the only interaction between modules is through each modules' public API.  If you only need one piece of functionality, it is possible to take the files you need and incorporate them into your own project.

It's not bloated
     The design of the library is based on the smallest API necessary to implement the basic functionality.  It has been carefully extended with a small number of high-level wrappers to make it comfortable to use the library.  However, it does not implement additional functionality just for the sake of completeness.

It's documented
     Sadly, not all software comes with documentation these days.  This one does.

Library Overview
================

The following illustration show the components that make up Libidn, and how your application relates to the library.  In the illustration, various components are shown as boxes.  You see the generic StringPrep component, the various StringPrep profiles including Nameprep, the Punycode component, the IDNA component, and the TLD component.  The arrows indicate aggregation, e.g., IDNA uses Punycode and Nameprep, and in turn Nameprep uses the generic StringPrep interface.  The interfaces to all components are available for applications, no component within the library is hidden from the application.

Supported Platforms
===================

Libidn has at some point in time been tested on the following platforms.  Build reports for each platforms and Libidn version is available at http://autobuild.josefsson.org/libidn/.

 1. Debian GNU/Linux 3.0 (Woody)

     GCC 2.95.4 and GNU Make. This is the main development platform. alphaev67-unknown-linux-gnu, alphaev6-unknown-linux-gnu, arm-unknown-linux-gnu, armv4l-unknown-linux-gnu, hppa-unknown-linux-gnu, hppa64-unknown-linux-gnu, i686-pc-linux-gnu, ia64-unknown-linux-gnu, m68k-unknown-linux-gnu, mips-unknown-linux-gnu, mipsel-unknown-linux-gnu, powerpc-unknown-linux-gnu, s390-ibm-linux-gnu, sparc-unknown-linux-gnu, sparc64-unknown-linux-gnu.

 2. Debian GNU/Linux 2.1

     GCC 2.95.1 and GNU Make. armv4l-unknown-linux-gnu.

 3. Tru64 UNIX

     Tru64 UNIX C compiler and Tru64 Make. alphaev67-dec-osf5.1, alphaev68-dec-osf5.1.

 4. SuSE Linux 7.1

     GCC 2.96 and GNU Make. alphaev6-unknown-linux-gnu, alphaev67-unknown-linux-gnu.

 5. SuSE Linux 7.2a

     GCC 3.0 and GNU Make. ia64-unknown-linux-gnu.

 6. SuSE Linux

     GCC 3.2.2 and GNU Make.  x86_64-unknown-linux-gnu (AMD64 Opteron ``Melody'').

 7. SuSE Enterprise Server 9 on IBM OpenPower 720

     GCC 3.3.3 and GNU Make.  powerpc64-unknown-linux-gnu.

 8. RedHat Linux 7.2

     GCC 2.96 and GNU Make. alphaev6-unknown-linux-gnu, alphaev67-unknown-linux-gnu, ia64-unknown-linux-gnu.

 9. RedHat Linux 8.0

     GCC 3.2 and GNU Make. i686-pc-linux-gnu.

 10. RedHat Advanced Server 2.1

     GCC 2.96 and GNU Make. i686-pc-linux-gnu.

 11. Slackware Linux 8.0.01

     GCC 2.95.3 and GNU Make. i686-pc-linux-gnu.

 12. Mandrake Linux 9.0

     GCC 3.2 and GNU Make. i686-pc-linux-gnu.

 13. IRIX 6.5

     MIPS C compiler, IRIX Make. mips-sgi-irix6.5.

 14. AIX 4.3.2

     IBM C for AIX compiler, AIX Make.  rs6000-ibm-aix4.3.2.0.

 15. Microsoft Windows 2000 (Cygwin)

     GCC 3.2, GNU make. i686-pc-cygwin.

 16. HP-UX 11

     HP-UX C compiler and HP Make. ia64-hp-hpux11.22, hppa2.0w-hp-hpux11.11.

 17. SUN Solaris 2.7

     GCC 3.0.4 and GNU Make. sparc-sun-solaris2.7.

 18. SUN Solaris 2.8

     Sun WorkShop Compiler C 6.0 and SUN Make. sparc-sun-solaris2.8.

 19. SUN Solaris 2.9

     Sun Forte Developer 7 C compiler and GNU Make. sparc-sun-solaris2.9.

 20. NetBSD 1.6

     GCC 2.95.3 and GNU Make. alpha-unknown-netbsd1.6, i386-unknown-netbsdelf1.6.

 21. OpenBSD 3.1 and 3.2

     GCC 2.95.3 and GNU Make. alpha-unknown-openbsd3.1, i386-unknown-openbsd3.1.

 22. FreeBSD 4.7 and 4.8

     GCC 2.95.4 and GNU Make. alpha-unknown-freebsd4.7, alpha-unknown-freebsd4.8, i386-unknown-freebsd4.7, i386-unknown-freebsd4.8.

 23. MacOS X 10.2 Server Edition

     GCC 3.1 and GNU Make. powerpc-apple-darwin6.5.

 24. MacOS X 10.4 ``Tiger'' with Xcode 2.0

     GCC 4.0 and GNU Make. powerpc-apple-darwin8.0.

 25. Cross compiled to uClinux/uClibc on Motorola Coldfire

     GCC 3.4 and GNU Make m68k-uclinux-elf.

 26. Cross compiled to ARM using Glibc

     GCC 2.95 and GNU Make arm-linux.

 27. Cross compiled to Mingw32.

     GCC 3.4.4 and GNU Make i586-mingw32msvc.

 28. OS/2

     GCC.

If you use Libidn on, or port Libidn to, a new platform please report it to the author.

Getting help
============

A mailing list where users of Libidn may help each other exists, and you can reach it by sending e-mail to help-libidn@gnu.org. Archives of the mailing list discussions, and an interface to manage subscriptions, is available through the World Wide Web at http://lists.gnu.org/mailman/listinfo/help-libidn.

Commercial Support
==================

Commercial support is available for users of GNU Libidn.  The kind of support that can be purchased may include:

  * Implement new features. Such as country code specific profiling to support a restricted subset of Unicode.

  * Port Libidn to new platforms. This could include porting Libidn to an embedded platforms that may need memory or size optimization.

  * Integrating IDN support in your existing project.

  * System design of components related to IDN.

If you are interested, please write to:

Simon Josefsson Datakonsult AB
Hagagatan 24
113 47 Stockholm
Sweden

E-mail: simon@josefsson.org

If your company provides support related to GNU Libidn and would like to be mentioned here, contact the author (see Bug Reports).

Downloading and Installing
==========================

The package can be downloaded from several places, including:

ftp://alpha.gnu.org/pub/gnu/libidn/

The latest version is stored in a file, e.g., libidn-VERSION.tar.gz where the VERSION value is the highest version number in the directory.

The package is then extracted, configured and built like many other packages that use Autoconf.  For detailed information on configuring and building it, refer to the INSTALL file that is part of the distribution archive.

Here is an example terminal session that download, configure, build and install the package.  You will need a few basic tools, such as sh, make and cc.

     $ wget -q ftp://alpha.gnu.org/pub/gnu/libidn/libidn-VERSION.tar.gz
     $ tar xfz libidn-VERSION.tar.gz
     $ cd libidn-VERSION/
     $ ./configure
     ...
     $ make
     ...
     $ make install
     ...

After that Libidn should be properly installed and ready for use.

A few configure options may be relevant, summarized in the table.

--enable-java
     Build the Java port into a *.JAR file.  See Java API, for more information.

--disable-tld
     Disable the TLD module.  This would typically only be useful if you are building on a memory restricted platforms.  See TLD Functions, for more information.

--enable-csharp[=IMPL]
     Build the C# port into a *.DLL file.  See C# API, for more information.  Here, IMPL is pnet or mono, indicating whether the PNET cscc compiler or the Mono mcs compiler should be used, respectively.

--disable-valgrind-tests
     Disable running the self-checks under Valgrind (http://valgrind.org/).  Normally Valgrind does not cause problems and can detect some severe memory errors.  If you are getting errors from Valgrind that are caused by the compiler or libc (possibly as a result of special optimization flags), you may use this option to disable the use of Valgrind.

For the complete list, refer to the output from configure --help.

Installing under Windows
------------------------

There are two ways to build Libidn on Windows: via MinGW or via Visual Studio.

With MinGW, you can build a Libidn DLL and use it from other applications.  After installing MinGW (http://mingw.org/) follow the generic installation instructions (see Downloading and Installing).  The DLL is installed by default.

For information on how to use the DLL in other applications, see: http://www.mingw.org/mingwfaq.shtml#faq-msvcdll.

You can build Libidn as a native Visual Studio C++ project.  This allows you to build the code for other platforms that VS supports, such as Windows Mobile.  You need Visual Studio 2005 or later.

First download and unpack the archive as described in the generic installation instructions (see Downloading and Installing).  Don't run ./configure.  Instead, start Visual Studio and open the project file windows/libidn.sln inside the Libidn directory.  You should be able to build the project using Build Project.

Output libraries will be written into the windows/lib (or windows/lib/debug for Debug versions) folder.

When working with Windows you may want to look into the special memory handling functions that may be needed (see Memory handling under Windows).

Bug Reports
===========

If you think you have found a bug in Libidn, please investigate it and report it.

  * Please make sure that the bug is really in Libidn, and preferably also check that it hasn't already been fixed in the latest version.

  * You have to send us a test case that makes it possible for us to reproduce the bug.

  * You also have to explain what is wrong; if you get a crash, or if the results printed are not good and in that case, in what way. Make sure that the bug report includes all information you would need to fix this kind of bug for someone else.

Please make an effort to produce a self-contained report, with something definite that can be tested or debugged.  Vague queries or piecemeal messages are difficult to act on and don't help the development effort.

If your bug report is good, we will do our best to help you to get a corrected version of the software; if the bug report is poor, we won't do anything about it (apart from asking you to send better bug reports).

If you think something in this manual is unclear, or downright incorrect, or if the language needs to be improved, please also send a note.

Send your bug report to:

bug-libidn@gnu.org

Contributing
============

If you want to submit a patch for inclusion -- from solve a typo you discovered, up to adding support for a new feature -- you should submit it as a bug report (see Bug Reports).  There are some things that you can do to increase the chances for it to be included in the official package.

Unless your patch is very small (say, under 10 lines) we require that you assign the copyright of your work to the Free Software Foundation. This is to protect the freedom of the project.  If you have not already signed papers, we will send you the necessary information when you submit your contribution.

For contributions that doesn't consist of actual programming code, the only guidelines are common sense.  Use it.

For code contributions, a number of style guides will help you:

  * Coding Style. Follow the GNU Standards document (see GNU Coding Standards (standards)).

     If you normally code using another coding standard, there is no problem, but you should use indent to reformat the code (see GNU Indent (indent)) before submitting your work.

  * Use the unified diff format diff -u.

  * Return errors. No reason whatsoever should abort the execution of the library.  Even memory allocation errors, e.g. when malloc return NULL, should work although result in an error code.

  * Design with thread safety in mind. Don't use global variables and the like.

  * Avoid using the C math library. It causes problems for embedded implementations, and in most situations it is very easy to avoid using it.

  * Document your functions. Use comments before each function headers, that, if properly formatted, are extracted into GTK-DOC web pages.  Don't forget to update the Texinfo manual as well.

  * Supply a ChangeLog and NEWS entries, where appropriate.

Preparation
***********

To use `Libidn', you have to perform some changes to your sources and the build system.  The necessary changes are small and explained in the following sections.  At the end of this chapter, it is described how the library is initialized, and how the requirements of the library are verified.

A faster way to find out how to adapt your application for use with `Libidn' may be to look at the examples at the end of this manual (see Examples).

Header
======

The library contains a few independent parts, and each part export the interfaces (data types and functions) in a header file.  You must include the appropriate header files in all programs using the library, either directly or through some other header file, like this:

     #include <stringprep.h>

The header files and the functions they define are categorized as follows:

stringprep.h

     The low-level stringprep API entry point.  For IDN applications, this is usually invoked via IDNA. Some applications, specifically non-IDN ones, may want to prepare strings directly though, and should include this header file.

     The name space of the stringprep part of Libidn is stringprep* for function names, Stringprep* for data types and STRINGPREP_* for other symbols.  In addition, _stringprep* is reserved for internal use and should never be used by applications.

punycode.h

     The entry point to Punycode encoding and decoding functions.  Normally punycode is used via the idna.h interface, but some application may want to perform raw punycode operations.

     The name space of the punycode part of Libidn is punycode_* for function names, Punycode* for data types and PUNYCODE_* for other symbols.  In addition, _punycode* is reserved for internal use and should never be used by applications.

idna.h

     The entry point to the IDNA functions.  This is the normal entry point for applications that need IDN functionality.

     The name space of the IDNA part of Libidn is idna_* for function names, Idna* for data types and IDNA_* for other symbols.  In addition, _idna* is reserved for internal use and should never be used by applications.

tld.h

     The entry point to the TLD functions.  Normal applications are not expected to need this functionality, but it is present for applications that are used by TLDs to validate customer input.

     The name space of the TLD part of Libidn is tld_* for function names, Tld_* for data types and TLD_* for other symbols. In addition, _tld* is reserved for internal use and should never be used by applications.

pr29.h

     The entry point to the PR29 functions.  These functions are used to detect ``problem sequences'' (see PR29 Functions), mostly for use in security critical applications.

     The name space of the PR29 part of Libidn is pr29_* for function names, Pr29_* for data types and PR29_* for other symbols.  In addition, _pr29* is reserved for internal use and should never be used by applications.

idn-free.h

     The entry point to the Windows memory de-allocation function (see Memory handling under Windows).  It contains only one function idn_free.

All header files defined and use the symbol IDNAPI to decorate the API functions.

Initialization
==============

Libidn is stateless and does not need any initialization.

Version Check
=============

It is often desirable to check that the version of `Libidn' used is indeed one which fits all requirements.  Even with binary compatibility new features may have been introduced but due to problem with the dynamic linker an old version is actually used.  So you may want to check that the version is okay right after program startup.

     [file stringprep_check_version.texi not included]

The normal way to use the function is to put something similar to the following first in your main:

       if (!stringprep_check_version (STRINGPREP_VERSION))
         {
           printf ("stringprep_check_version() failed:\n"
                   "Header file incompatible with shared library.\n");
           exit(EXIT_FAILURE);
         }

Building the source
===================

If you want to compile a source file including e.g. the `idna.h' header file, you must make sure that the compiler can find it in the directory hierarchy.  This is accomplished by adding the path to the directory in which the header file is located to the compilers include file search path (via the -I option).

However, the path to the include file is determined at the time the source is configured.  To solve this problem, `Libidn' uses the external package pkg-config that knows the path to the include file and other configuration options.  The options that need to be added to the compiler invocation at compile time are output by the --cflags option to pkg-config libidn.  The following example shows how it can be used at the command line:

     gcc -c foo.c `pkg-config libidn --cflags`

Adding the output of pkg-config libidn --cflags to the compilers command line will ensure that the compiler can find e.g. the idna.h header file.

A similar problem occurs when linking the program with the library. Again, the compiler has to find the library files.  For this to work, the path to the library files has to be added to the library search path (via the -L option).  For this, the option --libs to pkg-config libidn can be used.  For convenience, this option also outputs all other options that are required to link the program with the `libidn' library.  The example shows how to link foo.o with the `libidn' library to a program foo.

     gcc -o foo foo.o `pkg-config libidn --libs`

Of course you can also combine both examples to a single command by specifying both options to pkg-config:

     gcc -o foo foo.c `pkg-config libidn --cflags --libs`

Autoconf tests
==============

If your project uses Autoconf (see GNU Autoconf (autoconf)) to check for installed libraries, you might find the following snippet illustrative.  It add a new configure parameter --with-libidn, and check for idna.h and -lidn (possibly below the directory specified as the optional argument to --with-libidn), and define the CPP symbol LIBIDN if the library is found.  The default behaviour is to search for the library and enable the functionality (that is, define the symbol) when the library is found, but if you wish to make the default behaviour of your package be that Libidn is not used (even if it is installed on the system), change libidn=yes to libidn=no on the third line.

     AC_ARG_WITH(libidn, AC_HELP_STRING([--with-libidn=[DIR]],
                                     [Support IDN (needs GNU Libidn)]),
       libidn=$withval, libidn=yes)
     if test "$libidn" != "no"; then
       if test "$libidn" != "yes"; then
         LDFLAGS="${LDFLAGS} -L$libidn/lib"
         CPPFLAGS="${CPPFLAGS} -I$libidn/include"
       fi
       AC_CHECK_HEADER(idna.h,
         AC_CHECK_LIB(idn, stringprep_check_version,
           [libidn=yes LIBS="${LIBS} -lidn"], libidn=no),
         libidn=no)
     fi
     if test "$libidn" != "no" ; then
       AC_DEFINE(LIBIDN, 1, [Define to 1 if you want IDN support.])
     else
       AC_MSG_WARN([Libidn not found])
     fi
     AC_MSG_CHECKING([if Libidn should be used])
     AC_MSG_RESULT($libidn)

If you require that your users have installed pkg-config (which I cannot recommend generally), the above can be done more easily as follows.

     AC_ARG_WITH(libidn, AC_HELP_STRING([--with-libidn=[DIR]],
                                     [Support IDN (needs GNU Libidn)]),
       libidn=$withval, libidn=yes)
     if test "$libidn" != "no" ; then
       PKG_CHECK_MODULES(LIBIDN, libidn >= 0.0.0, [libidn=yes], [libidn=no])
       if test "$libidn" != "yes" ; then
         libidn=no
         AC_MSG_WARN([Libidn not found])
       else
         libidn=yes
         AC_DEFINE(LIBIDN, 1, [Define to 1 if you want Libidn.])
       fi
     fi
     AC_MSG_CHECKING([if Libidn should be used])
     AC_MSG_RESULT($libidn)

Memory handling under Windows
=============================

Several functions in the library allocates memory.  The memory is expected to be de-allocated using the free function.  Under Windows, it is sometimes necessary to de-allocate memory in the same module that allocated a memory region.  The reason is that different modules use separate heap memory regions.  To solve this problem we provide a function to de-allocate memory inside the library.

Note that we do not recommend using this interface generally if you do not care about Windows portability.

Header file idn-free.h
======================

To use the function explained in this chapter, you need to include the file idn-free.h using:

     #include <idn-free.h>

Memory de-allocation function
=============================

     [file idn_free.texi not included]

Utility Functions
*****************

The rest of this library makes extensive use of Unicode characters. In order to interface this library with the outside world, your application may need to make various Unicode transformations.

Header file stringprep.h
========================

To use the functions explained in this chapter, you need to include the file stringprep.h using:

     #include <stringprep.h>

Unicode Encoding Transformation
===============================

     [file stringprep_unichar_to_utf8.texi not included]
     [file stringprep_utf8_to_unichar.texi not included]
     [file stringprep_ucs4_to_utf8.texi not included]
     [file stringprep_utf8_to_ucs4.texi not included]

Unicode Normalization
=====================

     [file stringprep_ucs4_nfkc_normalize.texi not included]
     [file stringprep_utf8_nfkc_normalize.texi not included]

Character Set Conversion
========================

     [file stringprep_locale_charset.texi not included]
     [file stringprep_convert.texi not included]
     [file stringprep_locale_to_utf8.texi not included]
     [file stringprep_utf8_to_locale.texi not included]

Stringprep Functions
********************

Stringprep describes a framework for preparing Unicode text strings in order to increase the likelihood that string input and string comparison work in ways that make sense for typical users throughout the world. The stringprep protocol is useful for protocol identifier values, company and personal names, internationalized domain names, and other text strings.

Header file stringprep.h
========================

To use the functions explained in this chapter, you need to include the file stringprep.h using:

     #include <stringprep.h>

Defining A Stringprep Profile
=============================

Further types and structures are defined for applications that want to specify their own stringprep profile.  As these are fairly obscure, and by necessity tied to the implementation, we do not document them here.  Look into the stringprep.h header file, and the profiles.c source code for the details.

Control Flags
=============

 -- Stringprep flags: Stringprep_profile_flags STRINGPREP_NO_NFKC
     Disable the NFKC normalization, as well as selecting the non-NFKC case folding tables.  Usually the profile specifies BIDI and NFKC settings, and applications should not override it unless in special situations.

 -- Stringprep flags: Stringprep_profile_flags STRINGPREP_NO_BIDI
     Disable the BIDI step.  Usually the profile specifies BIDI and NFKC settings, and applications should not override it unless in special situations.

 -- Stringprep flags: Stringprep_profile_flags STRINGPREP_NO_UNASSIGNED
     Make the library return with an error if string contains unassigned characters according to profile.

Core Functions
==============

     [file stringprep_4i.texi not included]
     [file stringprep_4zi.texi not included]
     [file stringprep.texi not included]
     [file stringprep_profile.texi not included]

Error Handling
==============

     [file stringprep_strerror.texi not included]

Stringprep Profile Macros
=========================

 -- Function: int stringprep_nameprep_no_unassigned (char * IN, int MAXLEN)

     IN: input/ouput array with string to prepare.

     MAXLEN: maximum length of input/output array.

     Prepare the input UTF-8 string according to the nameprep profile.  The AllowUnassigned flag is false, use stringprep_nameprep for true AllowUnassigned.  Returns 0 iff successful, or an error code.

 -- Function: int stringprep_iscsi (char * IN, int MAXLEN)

     IN: input/ouput array with string to prepare.

     MAXLEN: maximum length of input/output array.

     Prepare the input UTF-8 string according to the draft iSCSI stringprep profile.  Returns 0 iff successful, or an error code.

 -- Function: int stringprep_plain (char * IN, int MAXLEN)

     IN: input/ouput array with string to prepare.

     MAXLEN: maximum length of input/output array.

     Prepare the input UTF-8 string according to the draft SASL ANONYMOUS profile.  Returns 0 iff successful, or an error code.

 -- Function: int stringprep_xmpp_nodeprep (char * IN, int MAXLEN)

     IN: input/ouput array with string to prepare.

     MAXLEN: maximum length of input/output array.

     Prepare the input UTF-8 string according to the draft XMPP node identifier profile.  Returns 0 iff successful, or an error code.

 -- Function: int stringprep_xmpp_resourceprep (char * IN, int MAXLEN)

     IN: input/ouput array with string to prepare.

     MAXLEN: maximum length of input/output array.

     Prepare the input UTF-8 string according to the draft XMPP resource identifier profile.  Returns 0 iff successful, or an error code.

Punycode Functions
******************

Punycode is a simple and efficient transfer encoding syntax designed for use with Internationalized Domain Names in Applications. It uniquely and reversibly transforms a Unicode string into an ASCII string. ASCII characters in the Unicode string are represented literally, and non-ASCII characters are represented by ASCII characters that are allowed in host name labels (letters, digits, and hyphens). A general algorithm called Bootstring allows a string of basic code points to uniquely represent any string of code points drawn from a larger set. Punycode is an instance of Bootstring that uses particular parameter values, appropriate for IDNA.

Header file punycode.h
======================

To use the functions explained in this chapter, you need to include the file punycode.h using:

     #include <punycode.h>

Unicode Code Point Data Type
============================

The punycode function uses a special type to denote Unicode code points.  It is guaranteed to always be a 32 bit unsigned integer.

 -- Punycode Unicode code point: uint32_t punycode_uint
     A unsigned integer that hold Unicode code points.

Core Functions
==============

Note that the current implementation will fail if the input_length exceed 4294967295 (the size of punycode_uint).  This restriction may be removed in the future. Meanwhile applications are encouraged to not depend on this problem, and use sizeof to initialize input_length and output_length.

The functions provided are the following two entry points:

     [file punycode_encode.texi not included]
     [file punycode_decode.texi not included]

Error Handling
==============

     [file punycode_strerror.texi not included]

IDNA Functions
**************

Until now, there has been no standard method for domain names to use characters outside the ASCII repertoire. The IDNA document defines internationalized domain names (IDNs) and a mechanism called IDNA for handling them in a standard fashion. IDNs use characters drawn from a large repertoire (Unicode), but IDNA allows the non-ASCII characters to be represented using only the ASCII characters already allowed in so-called host names today. This backward-compatible representation is required in existing protocols like DNS, so that IDNs can be introduced with no changes to the existing infrastructure. IDNA is only meant for processing domain names, not free text.

Header file idna.h
==================

To use the functions explained in this chapter, you need to include the file idna.h using:

     #include <idna.h>

Control Flags
=============

The IDNA flags parameter can take on the following values, or a bit-wise inclusive or of any subset of the parameters:

 -- Return code: Idna_flags IDNA_ALLOW_UNASSIGNED
     Allow unassigned Unicode code points.

 -- Return code: Idna_flags IDNA_USE_STD3_ASCII_RULES
     Check output to make sure it is a STD3 conforming host name.

Prefix String
=============

 -- Macro: #define IDNA_ACE_PREFIX
     String with the official IDNA prefix, xn--.

Core Functions
==============

The idea behind the IDNA function names are as follows: the idna_to_ascii_4i and idna_to_unicode_44i functions are the core IDNA primitives.  The 4 indicate that the function takes UCS-4 strings (i.e., Unicode code points encoded in a 32-bit unsigned integer type) of the specified length.  The i indicate that the data is written ``inline'' into the buffer.  This means the caller is responsible for allocating (and de-allocating) the string, and providing the library with the allocated length of the string. The output length is written in the output length variable.  The remaining functions all contain the z indicator, which means the strings are zero terminated.  All output strings are allocated by the library, and must be de-allocated by the caller.  The 4 indicator again means that the string is UCS-4, the 8 means the strings are UTF-8 and the l indicator means the strings are encoded in the encoding used by the current locale.

The functions provided are the following entry points:

     [file idna_to_ascii_4i.texi not included]
     [file idna_to_unicode_44i.texi not included]

Simplified ToASCII Interface
============================

     [file idna_to_ascii_4z.texi not included]
     [file idna_to_ascii_8z.texi not included]
     [file idna_to_ascii_lz.texi not included]

Simplified ToUnicode Interface
==============================

     [file idna_to_unicode_4z4z.texi not included]
     [file idna_to_unicode_8z4z.texi not included]
     [file idna_to_unicode_8z8z.texi not included]
     [file idna_to_unicode_8zlz.texi not included]
     [file idna_to_unicode_lzlz.texi not included]

Error Handling
==============

     [file idna_strerror.texi not included]

TLD Functions
*************

Organizations that manage some Top Level Domains (TLDs) have published tables with characters they accept within the domain.  The reason may be to reduce complexity that come from using the full Unicode range, and to protect themselves from future (backwards incompatible) changes in the IDN or Unicode specifications.  Libidn implement an infrastructure for defining and checking strings against such tables. Libidn also ship some tables from TLDs that we have managed to get permission to use them from.  Because these tables are even less static than Unicode or StringPrep tables, it is likely that they will be updated from time to time (even in backwards incompatible ways). The Libidn interface provide a ``version'' field for each TLD table, which can be compared for equality to guarantee the same operation over time.

From a design point of view, you can regard the TLD tables for IDN as the ``localization'' step that come after the ``internationalization'' step provided by the IETF standards.

The TLD functionality rely on up-to-date tables.  The latest version of Libidn aim to provide these, but tables with unclear copying conditions, or generally experimental tables, are not included.  Some such tables can be found at https://github.com/gnuthor/tldchk.

Header file tld.h
=================

To use the functions explained in this chapter, you need to include the file tld.h using:

     #include <tld.h>

Core Functions
==============

     [file tld_check_4t.texi not included]
     [file tld_check_4tz.texi not included]

Utility Functions
=================

     [file tld_get_4.texi not included]
     [file tld_get_4z.texi not included]
     [file tld_get_z.texi not included]
     [file tld_get_table.texi not included]
     [file tld_default_table.texi not included]

High-Level Wrapper Functions
============================

     [file tld_check_4.texi not included]
     [file tld_check_4z.texi not included]
     [file tld_check_8z.texi not included]
     [file tld_check_lz.texi not included]

Error Handling
==============

     [file tld_strerror.texi not included]

PR29 Functions
**************

A deficiency in the specification of Unicode Normalization Forms has been found.  The consequence is that some strings can be normalized into different strings by different implementations.  In other words, two different implementations may return different output for the same input (because the interpretation of the specification is ambiguous). Further, an implementation invoked again on the one of the output strings may return a different string (because one of the interpretation of the ambiguous specification make normalization non-idempotent).  Fortunately, only a select few character sequence exhibit this problem, and none of them are expected to occur in natural languages (due to different linguistic uses of the involved characters).

A full discussion of the problem may be found at:

http://www.unicode.org/review/pr-29.html

The PR29 functions below allow you to detect the problem sequence.  So when would you want to use these functions?  For most applications, such as those using Nameprep for IDN, this is likely only to be an interoperability problem.  Thus, you may not want to care about it, as the character sequences will rarely occur naturally.  However, if you are using a profile, such as SASLPrep, to process authentication tokens; authorization tokens; or passwords, there is a real danger that attackers may try to use the peculiarities in these strings to attack parts of your system.  As only a small number of strings, and no naturally occurring strings, exhibit this problem, the conservative approach of rejecting the strings is recommended.  If this approach is not used, you should instead verify that all parts of your system, that process the tokens and passwords, use a NFKC implementation that produce the same output for the same input.

Technically inclined readers may be interested in knowing more about the implementation aspects of the PR29 flaw. See PR29 discussion.

Header file pr29.h
==================

To use the functions explained in this chapter, you need to include the file pr29.h using:

     #include <pr29.h>

Core Functions
==============

     [file pr29_4.texi not included]

Utility Functions
=================

     [file pr29_4z.texi not included]
     [file pr29_8z.texi not included]

Error Handling
==============

     [file pr29_strerror.texi not included]

Examples
********

This chapter contains example code which illustrate how `Libidn' can be used when writing your own application.

Example 1
=========

This example demonstrates how the stringprep functions are used.

     [file example.c not included]

Example 2
=========

This example demonstrates how the punycode functions are used.

     [file example2.c not included]

Example 3
=========

This example demonstrates how the library is used to convert internationalized domain names into ASCII compatible names.

     [file example3.c not included]

Example 4
=========

This example demonstrates how the library is used to convert ASCII compatible names to internationalized domain names.

     [file example4.c not included]

Example 5
=========

This example demonstrates how the library is used to check a string for invalid characters within a specific TLD.

     [file example5.c not included]

Invoking idn
************

Name
====

GNU Libidn (idn) -- Internationalized Domain Names command line tool

Description
===========

idn allows internationalized string preparation (stringprep), encoding and decoding of punycode data, and IDNA ToASCII/ToUnicode operations to be performed on the command line.

If strings are specified on the command line, they are used as input and the computed output is printed to standard output stdout. If no strings are specified on the command line, the program read data, line by line, from the standard input stdin, and print the computed output to standard output.  What processing is performed (e.g., ToASCII, or Punycode encode) is indicated by options.  If any errors are encountered, the execution of the applications is aborted.

All strings are expected to be encoded in the preferred charset used by your locale.  Use --debug to find out what this charset is. You can override the charset used by setting environment variable CHARSET.

To process a string that starts with -, for example -foo, use -- to signal the end of parameters, as in idn --quiet -a -- -foo.

Options
=======

idn recognizes these commands:

  -h, --help               Print help and exit

  -V, --version            Print version and exit

  -s, --stringprep         Prepare string according to nameprep profile

  -d, --punycode-decode    Decode Punycode

  -e, --punycode-encode    Encode Punycode

  -a, --idna-to-ascii      Convert to ACE according to IDNA (default mode)

  -u, --idna-to-unicode    Convert from ACE according to IDNA

      --allow-unassigned   Toggle IDNA AllowUnassigned flag (default off)

      --usestd3asciirules  Toggle IDNA UseSTD3ASCIIRules flag (default off)

      --no-tld             Don't check string for TLD specific rules
                             Only for --idna-to-ascii and --idna-to-unicode

  -n, --nfkc               Normalize string according to Unicode v3.2 NFKC

  -p, --profile=STRING     Use specified stringprep profile instead
                             Valid stringprep profiles: `Nameprep',
                             `iSCSI', `Nodeprep', `Resourceprep',
                             `trace', `SASLprep'

      --debug              Print debugging information

      --quiet              Silent operation

Environment Variables
=====================

The CHARSET environment variable can be used to override what character set to be used for decoding incoming data (i.e., on the command line or on the standard input stream), and to encode data to the standard output.  If your system is set up correctly, however, the application will guess which character set is used automatically. Example usage:

     $ CHARSET=ISO-8859-1 idn --punycode-encode
     ...

Examples
========

Standard usage, reading input from standard input:

     jas@latte:~$ idn
     libidn 0.3.5
     Copyright 2002, 2003 Simon Josefsson.
     GNU Libidn comes with NO WARRANTY, to the extent permitted by law.
     You may redistribute copies of GNU Libidn under the terms of
     the GNU Lesser General Public License.  For more information
     about these matters, see the file named COPYING.LIB.
     Type each input string on a line by itself, terminated by a newline character.
     räksmörgås.se
     xn--rksmrgs-5wao1o.se
     jas@latte:~$

Reading input from command line, and disable printing copyright and license information:

     jas@latte:~$ idn --quiet räksmörgås.se blåbærgrød.no
     xn--rksmrgs-5wao1o.se
     xn--blbrgrd-fxak7p.no
     jas@latte:~$

Accessing a specific StringPrep profile directly:

     jas@latte:~$ idn --quiet --profile=SASLprep --stringprep teßtª
     teßta
     jas@latte:~$

Troubleshooting
===============

Getting character data encoded right, and making sure Libidn use the same encoding, can be difficult.  The reason for this is that most systems encode character data in more than one character encoding, i.e., using UTF-8 together with ISO-8859-1 or ISO-2022-JP.  This problem is likely to continue to exist until only one character encoding come out as the evolutionary winner, or (more likely, at least to some extents) forever.

The first step to troubleshooting character encoding problems with Libidn is to use the --debug parameter to find out which character set encoding idn believe your locale uses.

     jas@latte:~$ idn --debug --quiet ""
     system locale uses charset `UTF-8'.

     jas@latte:~$

If it prints ANSI_X3.4-1968 (i.e., US-ASCII), this indicate you have not configured your locale properly.  To configure the locale, you can, for example, use LANG=sv_SE.UTF-8; export LANG at a /bin/sh prompt, to set up your locale for a Swedish environment using UTF-8 as the encoding.

Sometimes idn appear to be unable to translate from your system locale into UTF-8 (which is used internally), and you get an error like the following:

     jas@latte:~$ idn --quiet foo
     idn: could not convert from ISO-8859-1 to UTF-8.
     jas@latte:~$

The simplest explanation is that you haven't installed the iconv conversion tools.  You can find it as a standalone library in GNU Libiconv (http://www.gnu.org/software/libiconv/).  On many GNU/Linux systems, this library is part of the system, but you may have to install additional packages (e.g., glibc-locale for Debian) to be able to use it.

Another explanation is that the error is correct and you are feeding idn invalid data.  This can happen inadvertently if you are not careful with the character set encoding you use.  For example, if your shell run in a ISO-8859-1 environment, and you invoke idn with the CHARSET environment variable as follows, you will feed it ISO-8859-1 characters but force it to believe they are UTF-8.  Naturally this will lead to an error, unless the byte sequences happen to be valid UTF-8.  Note that even if you don't get an error, the output may be incorrect in this situation, because ISO-8859-1 and UTF-8 does not in general encode the same characters as the same byte sequences.

     jas@latte:~$ idn --quiet --debug ""
     system locale uses charset `ISO-8859-1'.

     jas@latte:~$ CHARSET=UTF-8 idn --quiet --debug räksmörgås
     system locale uses charset `UTF-8'.
     input[0] = U+0072
     input[1] = U+4af3
     input[2] = U+006d
     input[3] = U+1b29e5
     input[4] = U+0073
     output[0] = U+0078
     output[1] = U+006e
     output[2] = U+002d
     output[3] = U+002d
     output[4] = U+0072
     output[5] = U+006d
     output[6] = U+0073
     output[7] = U+002d
     output[8] = U+0068
     output[9] = U+0069
     output[10] = U+0036
     output[11] = U+0064
     output[12] = U+0035
     output[13] = U+0039
     output[14] = U+0037
     output[15] = U+0035
     output[16] = U+0035
     output[17] = U+0032
     output[18] = U+0061
     xn--rms-hi6d597552a
     jas@latte:~$

The sense moral here is to forget about CHARSET (configure your locales properly instead) unless you know what you are doing, and if you want to use it, do it carefully, after verifying with --debug that you get the desired results.

Emacs API
*********

Included in Libidn are punycode.el and idna.el that provides an Emacs Lisp API to (a limited set of) the Libidn API.  This section describes the API.  Currently the IDNA API always set the UseSTD3ASCIIRules flag and clear the AllowUnassigned flag, in the future there may be functionality to specify these flags via the API.

Punycode Emacs API
==================

 -- Variable: punycode-program
     Name of the GNU Libidn idn application.  The default is idn.  This variable can be customized.

 -- Variable: punycode-environment
     List of environment variable definitions prepended to process-environment.  The default is ("CHARSET=UTF-8"). This variable can be customized.

 -- Variable: punycode-encode-parameters
     List of parameters passed to PUNYCODE-PROGRAM to invoke punycode encoding mode.  The default is ("--quiet" "--punycode-encode"). This variable can be customized.

 -- Variable: punycode-decode-parameters
     Parameters passed to PUNYCODE-PROGRAM to invoke punycode decoding mode.  The default is ("--quiet" "--punycode-decode"). This variable can be customized.

 -- Function: punycode-encode string
     Returns a Punycode encoding of the STRING, after converting the input into UTF-8.

 -- Function: punycode-decode string
     Returns a possibly multibyte string which is the decoding of the STRING which is a punycode encoded string.

IDNA Emacs API
==============

 -- Variable: idna-program
     Name of the GNU Libidn idn application.  The default is idn.  This variable can be customized.

 -- Variable: idna-environment
     List of environment variable definitions prepended to process-environment.  The default is ("CHARSET=UTF-8"). This variable can be customized.

 -- Variable: idna-to-ascii-parameters
     List of parameters passed to IDNA-PROGRAM to invoke IDNA ToASCII mode.  The default is ("--quiet" "--idna-to-ascii" "--usestd3asciirules").  This variable can be customized.

 -- Variable: idna-to-unicode-parameters
     Parameters passed IDNA-PROGRAM to invoke IDNA ToUnicode mode. The default is ("--quiet" "--idna-to-unicode" "--usestd3asciirules").  This variable can be customized.

 -- Function: idna-to-ascii string
     Returns an ASCII Compatible Encoding (ACE) of the string computed by the IDNA ToASCII operation on the input STRING, after converting the input to UTF-8.

 -- Function: idna-to-unicode string
     Returns a possibly multibyte string which is the output of the IDNA ToUnicode operation computed on the input STRING.

Java API
********

Libidn has been ported to the Java programming language, and as a consequence most of the API is available to native Java applications. This section contain notes on this support, complete documentation is pending.

The Java library, if Libidn has been built with Java support (see Downloading and Installing), will be placed in java/libidn-VERSION.jar.  The source code is below java/ in Maven directory layout, and there is a Maven pom.xml build script as well.  Source code files are in java/src/main/java/gnu/inet/encoding/.

Overview
========

This package provides a Java implementation of the Internationalized Domain Names in Applications (IDNA) standard. It is written entirely in Java and does not require any additional libraries to be set up.

The gnu.inet.encoding.IDNA class offers two public functions, toASCII and toUnicode which can be used as follows:

     gnu.inet.encoding.IDNA.toASCII("blöds.züg");
     gnu.inet.encoding.IDNA.toUnicode("xn--blds-6qa.xn--zg-xka");

Miscellaneous Programs
======================

The java/src/util/java/ directory contains several programs that are related to the Java part of GNU Libidn, but that don't need to be included in the main source tree or the JAR file.

GenerateRFC3454
---------------

This program parses RFC3454 and creates the RFC3454.java program that is required during the StringPrep phase.

The RFC can be found at various locations, for example at http://www.ietf.org/rfc/rfc3454.txt.

Invoke the program as follows:

     $ java GenerateRFC3454
     Creating RFC3454.java... Ok.

GenerateNFKC
------------

The GenerateNFKC program parses the Unicode character database file and generates all the tables required for NFKC. This program requires the two files UnicodeData.txt and CompositionExclusions.txt of version 3.2 of the Unicode files. Note that RFC3454 (Stringprep) defines that Unicode version 3.2 is to be used, not the latest version.

The Unicode data files can be found at http://www.unicode.org/Public/.

Invoke the program as follows:

     $ java GenerateNFKC
     Creating CombiningClass.java... Ok.
     Creating DecompositionKeys.java... Ok.
     Creating DecompositionMappings.java... Ok.
     Creating Composition.java... Ok.

TestIDNA
--------

The TestIDNA program allows to test the IDNA implementation manually or against Simon Josefsson's test vectors.

The test vectors can be found at the Libidn homepage, http://www.gnu.org/software/libidn/.

To test the transformation manually, use:

     $ java -cp .:/usr/share/java/libidn.jar TestIDNA -a <string to test>
     Input: <string to test>
     Output: <toASCII(string to test)>
     $ java -cp .:/usr/share/java/libidn.jar TestIDNA -u <string to test>
     Input: <string to test>
     Output: <toUnicode(string to test)>

To test against draft-josefsson-idn-test-vectors.html, use:

     $ java -cp .:/usr/share/java/libidn/libidn.jar TestIDNA -t
     No errors detected!

TestNFKC
--------

The TestNFKC program allows to test the NFKC implementation manually or against the NormalizationTest.txt file from the Unicode data files.

To test the normalization manually, use:

     $ java -cp .:/usr/share/java/libidn.jar TestNFKC <string to test>
     Input: <string to test>
     Output: <nfkc version of the string to test>

To test against NormalizationTest.txt:

     $ java -cp .:/usr/share/java/libidn.jar TestNFKC
     No errors detected!

Possible Problems
=================

Beware of Bugs: This Java API needs a lot more testing, especially with "exotic" character sets. While it works for me, it may not work for you.

Encoding of your Java sources: If you are using non-ASCII characters in your Java source code, make sure javac compiles your programs with the correct encoding. If necessary specify the encoding using the -encoding parameter.

Java Unicode handling: Java 1.4 only handles 16-bit Unicode code points (i.e. characters in the Basic Multilingual Plane), this implementation therefore ignores all references to so-called Supplementary Characters (U+10000 to U+10FFFF). Starting from Java 1.5, these characters will also be supported by Java, but this will require changes to this library.  See also the next section.

A Note on Java and Unicode
==========================

This library uses Java's built-in 'char' datatype. Up to Java 1.4, this datatype only supports 16-bit Unicode code points, also called the Basic Multilingual Plane. For this reason, this library doesn't work for Supplementary Characters (i.e. characters from U+10000 to U+10FFFF). All references to such characters are silently ignored.

Starting from Java 1.5, also Supplementary Characters will be supported. However, this will require changes in the present version of the library. Java 1.5 is currently in beta status.

For more information refer to the documentation of java.lang.Character in the JDK API.

C# API
******

The Libidn library has been ported to the C# language.  The port reside in the top-level csharp/ directory.  Currently, no further documentation about the implementation or the API is available.  However, the C# port was based on the Java port, and the API is exactly the same as in the Java version.  The help files for the Java API may thus be useful.

Acknowledgements
****************

The punycode implementation was taken from the IETF IDN Punycode specification, by Adam M. Costello.  The TLD code was contributed by Thomas Jacob.  The Java implementation was contributed by Oliver Hitz. The C# implementation was contributed by Alexander Gnauck.  The Unicode tables were provided by Unicode, Inc.  Some functions for dealing with Unicode (see nfkc.c and toutf8.c) were borrowed from GLib, downloaded from http://www.gtk.org/.  The manual borrowed text from Libgcrypt by Werner Koch.

Inspiration for many things that, consciously or not, have gone into this package is due to a number of free software package that the author has been exposed to.  The author wishes to acknowledge the free software community in general, for giving an example on what role software development can play in the modern society.

Several people reported bugs, sent patches or suggested improvements, see the file THANKS in the top-level directory of the source code.

History
*******

The complete history of user visible changes is stored in the file NEWS in the top-level directory of the source code tree.  The complete history of modifications to each file is stored in the file ChangeLog in the same directory.  This section contain a condensed version of that information, in the form of ``milestones'' for the project.

Stringprep implementation.
     Version 0.0.0 released on 2002-11-05.

IDNA and Punycode implementations, part of the GNU project.
     Version 0.1.0 released on 2003-01-05.

Uses official IDNA ACE prefix xn--.
     Version 0.1.7 released on 2003-02-12.

Command line interface.
     Version 0.1.11 released on 2003-02-26.

GNU Libc add-on proposed.
     Version 0.1.12 released on 2003-03-06.

Interoperability testing during IDNConnect.
     Version 0.3.1 released on 2003-10-02.

TLD restriction testing.
     Version 0.4.0 released on 2004-02-28.

GNU Libc add-on integrated.
     Version 0.4.1 released on 2004-03-08.

Native Java implementation.
     Version 0.4.2-0.4.9 released between 2004-03-20 and 2004-06-11.

PR-29 functions for ``problem sequences''.
     Version 0.5.0 released on 2004-06-26.

Many small portability fixes and wider use.
     Version 0.5.1 through 0.5.20, released between 2004-07-09 and 2005-10-23.

Native C# implementation.
     Version 0.6.0 released on 2005-12-03.

Windows support through cross-compilation.
     Version 0.6.1 released on 2006-01-20.

Library declared stable by releasing v1.0.
     Version 1.0 released on 2007-07-31.

PR29 discussion
***************

If you wish to experiment with a modified Unicode NFKC implementation according to the PR29 proposal, you may find the following bug report useful.  However, I have not verified that the suggested modifications are correct.  For reference, I'm including my response to the report as well.

From: Rick McGowan <rick@unicode.org>
Subject: Possible bug and status of PR 29 change(s)
To: bug-libidn@gnu.org
Date: Wed, 27 Oct 2004 14:49:17 -0700

Hello. On behalf of the Unicode Consortium editorial committee, I would
like to find out more information about the PR 29 fixes, if any, and
functions in Libidn. Your implementation was listed in the text of PR29 as
needing investigation, so I am following up on several implementations.

The UTC has accepted the proposed fix to D2 as outlined in PR29, and a new
draft of UAX #15 has been issued.

I have looked at Libidn 0.5.8 (today), and there may still be a possible
bug in NFKC.java and nfkc.c.

------------------------------------------------------

1. In NFKC.java, this line in canonicalOrdering():

      if (i > 0 && (last_cc == 0 || last_cc != cc)) {

should perhaps be changed to:

      if (i > 0 && (last_cc == 0 || last_cc < cc)) {

but I'm not sure of the sense of this comparison.

------------------------------------------------------

2. In nfkc.c, function _g_utf8_normalize_wc() has this code:

	  if (i > 0 &&
	      (last_cc == 0 || last_cc != cc) &&
	      combine (wc_buffer[last_start], wc_buffer[i],
		       &wc_buffer[last_start]))
	    {

This appears to have the same bug as the current Python implementation (in
Python 2.3.4). The code should be checking, as per new rule D2 UAX #15
update, that the next combining character is the same or HIGHER than the
current one. It now checks to see if it's non-zero and not equal.

The above line(s) should perhaps be changed to:

	  if (i > 0 &&
	      (last_cc == 0 || last_cc < cc) &&
	      combine (wc_buffer[last_start], wc_buffer[i],
		       &wc_buffer[last_start]))
	    {

but I'm not sure of the sense of the comparison (< or > or <=?) here.

In the text of PR29, I will be marking Libidn as "needs change" and adding
the version number that I checked. If any further change is made, please
let me know the release version, and I'll update again.

Regards,
	Rick McGowan

From: Simon Josefsson <jas@extundo.com>
Subject: Re: Possible bug and status of PR 29 change(s)
To: Rick McGowan <rick@unicode.org>
Cc: bug-libidn@gnu.org
Date: Thu, 28 Oct 2004 09:47:47 +0200

Rick McGowan <rick@unicode.org> writes:

> Hello. On behalf of the Unicode Consortium editorial committee, I would
> like to find out more information about the PR 29 fixes, if any, and
> functions in Libidn. Your implementation was listed in the text of PR29 as
> needing investigation, so I am following up on several implementations.
>
> The UTC has accepted the proposed fix to D2 as outlined in PR29, and a new
> draft of UAX #15 has been issued.
>
> I have looked at Libidn 0.5.8 (today), and there may still be a possible
> bug in NFKC.java and nfkc.c.

Hello Rick.

I believe the current behavior is intentional.  Libidn do not aim to
implement latest-and-greatest NFKC, it aim to implement the NFKC
functionality required for StringPrep and IDN.  As you may know,
StringPrep/IDN reference Unicode 3.2.0, and explicitly says any later
changes (which I consider PR29 as) do not apply.

In fact, I believe that would I incorporate the changes suggested in
PR29, I would in fact be violating the IDN specifications.

Thanks for looking into the code and finding the place where the
change could be made.  I'll see if I can mention this in the manual
somewhere, for technically interested readers.

Regards,
Simon

On Label Separators
*******************

Some strings contains characters whose NFKC normalized form contain the ASCII dot (0x2E, ``.'').  Examples of these characters are U+2024 (ONE DOT LEADER) and U+248C (DIGIT FIVE FULL STOP).  The strings have the interesting property that their IDNA ToASCII output will contain embedded dots.  For example:

     ToASCII (hi U+248C com) = hi5.com
     ToASCII (räksmörgås U+2024 com) = xn--rksmrgs.com-l8as9u

This demonstrate the two general cases: The first where the ASCII dot is part of an output that do not begin with the IDN prefix xn--.  The second example illustrate when the dot is part of IDN prefixed with xn--.

The input strings are, from the DNS point of view, a single label. The IDNA algorithm translate one label at a time.  Thus, the output is expected to be only one label.  What is important here is to make sure the DNS resolver receives the correct query.  The DNS protocol does not use the dot to delimit labels on the wire, rather it uses length-value pairs.  Thus the correct query would be for {7}hi5.com and {22}xn--rksmrgs.com-l8as9u respectively.

Some implementations  (Notably Microsoft's Internet Explorer and Mozilla's Firefox, but not Apple's Safari.) have decided that these inputs strings are potentially confusing for the user.  The string hi U+248C com looks like hi5.com on systems that support Unicode properly.  These implementations do not follow RFC 3490.  They yield:

     ToASCII (hi U+248C com) = hi5.com
     ToASCII (räksmörgås U+2024 com) = xn--rksmrgs-5wao1o.com

The DNS query they perform are {3}hi5{3}com and {18}xn--rksmrgs-5wao1o{3}com respectively.  Arguably, this leads to a better user experience, and suggests that the IDNA specification is sub-optimal in this area.

Recommended Workaround
======================

It has been suggested to normalize the entire input string using NFKC before passing it to IDNA ToASCII.  You may use stringprep_utf8_nfkc_normalize or stringprep_ucs4_nfkc_normalize.  This appears to lead to similar behaviour as IE/Firefox, which would avoid the problem, but this needs to be confirmed.  Feel free to discuss the issue with us.

Alternative workarounds are being considered.  Eventually Libidn may implement a new flag to the idna_* functions that implements a recommended way to work around this problem.

Copying Information
*******************

GNU Free Documentation License
==============================

     [file fdl-1.3.texi not included]