The Linux keyboard and console HOWTO: How to make other programs work with non-ASCII chars

12. How to make other programs work with non-ASCII chars

In the bad old days this used to be quite a hassle. Every separate program had to be convinced individually to leave your bits alone. Not that all is easy now, but recently a lot of gnu utilities have learned to react to LC_CTYPE=iso_8859_1 or LC_CTYPE=iso-8859-1. Try this first, and if it doesn't help look at the hints below. Note that in recent versions of libc the routine setlocale() only works if you have installed the locale files (e.g. in /usr/lib/locale).

NOTE! The above was written years ago. Today locale stuff is a bit different. Try the command locale -a to see which locales are available. Then use one of these locale names instead of the iso_8859-1 mentioned above. For example, LC_CTYPE=fr_FR.ISO-8859-1 or LC_CTYPE=fr_FR@euro.

NOTE! Some of the below may still be true. Most of it is outdated. (Please report on what is incorrect today, so that it can be deleted.)

First of all, the 8-th bit should survive the kernel input processing, so make sure to have stty cs8 -istrip -parenb set.

A. For emacs the details strongly depend on the version. The information below is for version 19.34. Put lines


        (set-input-mode nil nil 1)
        (standard-display-european t)
        (require 'iso-syntax)

into your $HOME/.emacs. The first line (to be precise: the final 1) tells emacs not to discard the 8-th bit from input characters. The second line tells emacs not to display non-ASCII characters as octal escapes. The third line specifies the syntactic properties and case conversion table for the Latin-1 character set These last two lines are superfluous if you have something like LC_CTYPE=ISO-8859-1 in your environment. (The variable may also be LC_ALL or even LANG. The value may be anything with a substring `88591' or `8859-1' or `8859_1'.)

This is a good start. On a terminal that cannot display non-ASCII ISO 8859-1 symbols, the command


        (load-library "iso-ascii")

will cause accented characters to be displayed comme {,c}a. If your keymap does not make it easy to produce non-ASCII characters, then


        (load-library "iso-transl")

will make the 2-character sequence Ctrl-X 8 a compose character, so that the 4-character sequence Ctrl-X 8 , c produces c-cedilla. Very inconvenient.

The command


        (iso-accents-mode)

will toggle ISO-8859-1 accent mode, in which the six characters ', `, ", ^, ~, / are dead keys modifying the following symbol. Special combinations: ~c gives a c with cedilla, ~d gives an Icelandic eth, ~t gives an Icelandic thorn, "s gives German sharp s, /a gives a with ring, /e gives an a-e ligature, ~< and ~> give guillemots, ~! gives an inverted exclamation mark, ~? gives an inverted question mark, and '' gives an acute accent. This is the default mapping of accents. The variable iso-languages is a list of pairs (language name, accent mapping), and a non-default mapping can be selected using


        (iso-accents-customize LANGUAGE)

Here LANGUAGE can be one of "portuguese", "irish", "french", "latin-2", "latin-1".

Since the Linux default compose character is Ctrl-. it might be convenient to use that everywhere. Try


        (load-library "iso-insert.el")
        (define-key global-map [?\C-.] 8859-1-map)

The latter line will not work under xterm, if you use emacs -nw, but in that case you can put


        XTerm*VT100.Translations:       #override\n\
              Ctrl <KeyPress> . : string("\0308")

in your .Xresources.)

B. For less, put LESSCHARSET=latin1 in the environment. This is also what you need if you see \255 or <AD> in man output: some versions of less will render the soft hyphen (octal 0255, hex 0xAD) this way when not given permission to output Latin-1.

C. For ls, give the option -N. (Probably you want to make an alias.)

D. For bash (version 1.13.*), put


        set meta-flag on
        set convert-meta off
        set output-meta on

into your $HOME/.inputrc.

E. For tcsh, use


        setenv LANG     en_US
        setenv LC_CTYPE iso_8859_1

If you have nls on your system, then the corresponding routines are used. Otherwise tcsh will assume iso_8859_1, regardless of the values given to LANG and LC_CTYPE. See the section NATIVE LANGUAGE SYSTEM in tcsh(1). (The Danish HOWTO says: setenv LC_CTYPE ISO-8859-1; stty pass8)

F. For flex, give the option -8 if the parser it generates must be able to handle 8-bit input. (Of course it must.)

G. For elm, set displaycharset to ISO-8859-1. (Danish HOWTO: LANG=C and LC_CTYPE=ISO-8859-1)

H. For programs using curses (such as lynx) David Sibley reports: The regular curses package uses the high-order bit for reverse video mode (see flag _STANDOUT defined in /usr/include/curses.h). However, ncurses seems to be 8-bit clean and does display iso-latin-8859-1 correctly.

I. For programs using groff (such as man), make sure to use -Tlatin1 instead of -Tascii. Old versions of the program man also use col, and the next point also applies.

K. For rlogin, use option -8.

L. For joe, metalab.unc.edu:/pub/Linux/apps/editors/joe-1.0.8-linux.tar.gz is said to work after editing the configuration file. Someone else said: joe: Put the -asis option in /isr/lib/joerc in the first column.

M. For LaTeX: \documentstyle[isolatin]{article}. For LaTeX2e: \documentclass{article}\usepackage{isolatin} where isolatin.sty is available from ftp.vlsivie.tuwien.ac.at/pub/8bit.

A nice discussion on the topic of ISO-8859-1 and how to manage 8-bit characters is contained in the file grasp.insa-lyon.fr:/pub/faq/fr/accents (in French). Another fine discussion (in English) can be found in rtfm.mit.edu:pub/usenet-by-group/comp.answers/internationalization/iso-8859-1-charset.

If you need to fix a program that behaves badly with 8-bit characters, one thing to keep in mind is that if you have a signed char type then characters may be negative, and using them as an array index will fail. Several programs can be fixed by judiciously adding (unsigned char) casts.

Next Previous Contents