preconv(1) — Linux manual page

Name | Synopsis | Description | Options | See Also | COLOPHON

preconv(1)                 General Commands Manual                preconv(1)

Name         top

       preconv - prepare files for typesetting with GNU roff

Synopsis         top

       preconv [-dr] [-D default-encoding] [-e encoding] [file ...]

       preconv -h
       preconv --help

       preconv -v
       preconv --version

Description         top

       preconv reads each file, converts its encoded characters to a form
       groff(1) can interpet, and sends the result to the standard output
       stream.  Currently, this means that code points in the range 0–127
       (in US-ASCII, ISO 8859, or Unicode) remain as-is and the remainder
       are converted to the groff “special character” form “\[uXXXX]”, where
       XXXX is a hexadecimal number of four to six digits corresponding to a
       Unicode code point.  By default, preconv also inserts a roff .lf
       request at the beginning of each file, identifying it for the benefit
       of later processing (including diagnostic messages); the -r option
       suppresses this behavior.

       In typical usage scenarios, preconv need not be run directly; instead
       it should be invoked with the -k or -K options of groff.

       preconv tries to find the input encoding with the following
       algorithm, stopping at the first success.

       1.     If the input encoding has been explicitly specified with
              option -e, use it.

       2.     Check whether the input starts with a Unicode Byte Order Mark.
              If so, determine the encoding as UTF-8, UTF-16, or UTF-32

       3.     If the input stream is seekable, check the first and second
              input lines for a recognized GNU Emacs file-local variable
              identifying the character encoding, here referred to as the
              “coding tag” for brevity.  If found, use it.

       4.     If the input stream is seekable, and if the uchardet library
              is available on the system, use it to try to infer the
              encoding of the file.

       5.     If the -D option specifies an encoding, use it.

       6.     Use the encoding specified by the current locale (LC_CTYPE),
              unless the locale is “C”, “POSIX”, or empty, in which case
              assume Latin-1 (ISO 8859-1).

       Note that the coding tag and uchardet methods in the above procedure
       rely upon a seekable input stream; when preconv reads from a pipe,
       the stream is not seekable, and these detection methods are skipped.
       If character encoding detection of your input files is unreliable,
       arrange for one of the other methods to succeed by using preconv's -D
       or -e options, or by configuring your locale appropriately.
       Furthermore, groff supports a GROFF_ENCODING environment variable
       which is equivalent to its option -k.

   Coding tags
       Text editors that support more than a single character encoding need
       tags within the input files to mark the file's encoding.  While it is
       possible to guess the right input encoding with the help of
       heuristics that are reliable for a preponderance of natural language
       texts, they are not absolutely reliable.  Heuristics can fail on
       inputs that are too short or don't represent a natural language.

       Consequently, preconv supports the coding tag convention (with some
       restrictions) used by GNU Emacs.  These are indicated in specially-
       marked regions of an input file designated for “file-local

       preconv interprets the following syntax if it occurs in a roff
       comment in the first or second line of the input file.  Both “\"” and
       “\#” comment forms are recognized, but the control (or non-breaking
       control) character must be the default and must begin the line.
       Similarly, the escape character must be the default.
              -*- [...;] coding: encoding; [...] -*-

       The only variable preconv interprets is “coding”, which can take the
       values listed below.

       The following list comprises all MIME “charset” parameter values rec‐
       ognized, case-insensitively, by preconv.
              big5, cp1047, euc-jp, euc-kr, gb2312, iso-8859-1, iso-8859-2,
              iso-8859-5, iso-8859-7, iso-8859-9, iso-8859-13, iso-8859-15,
              koi8-r, us-ascii, utf-8, utf-16, utf-16be, utf-16le

       In addition, the following list of other coding tags is recognized,
       each of which is mapped to an appropriate value from the list above.
              ascii, chinese-big5, chinese-euc, chinese-iso-8bit, cn-big5,
              cn-gb, cn-gb-2312, cp878, csascii, csisolatin1,
              cyrillic-iso-8bit, cyrillic-koi8, euc-china, euc-cn,
              euc-japan, euc-japan-1990, euc-korea, greek-iso-8bit,
              iso-10646/utf8, iso-10646/utf-8, iso-latin-1, iso-latin-2,
              iso-latin-5, iso-latin-7, iso-latin-9, japanese-euc,
              japanese-iso-8bit, jis8, koi8, korean-euc, korean-iso-8bit,
              latin-0, latin1, latin-1, latin-2, latin-5, latin-7, latin-9,
              mule-utf-8, mule-utf-16, mule-utf-16be, mule-utf-16-be,
              mule-utf-16be-with-signature, mule-utf-16le, mule-utf-16-le,
              mule-utf-16le-with-signature, utf8, utf-16-be,
              utf-16-be-with-signature, utf-16be-with-signature, utf-16-le,
              utf-16-le-with-signature, utf-16le-with-signature

       Trailing “-dos”, “-unix”, and “-mac” suffixes on coding tags (which
       indicate the end-of-line convention used in the file) are disregarded
       for the purpose of comparison with the above tags.

   iconv support
       preconv itself only supports three encodings: Latin-1, code page
       1047, and UTF-8.  If iconv support is configured at compile time and
       available at run time, all other encodings are passed to iconv
       library functions.  The command “preconv -v” discloses whether iconv
       support is configured.

       The use of iconv means that characters in the input that encode
       invalid code points for that encoding may be dropped from the output
       stream or mapped to the Unicode replacement character (U+FFFD).  Com‐
       pare the following examples using the input “café” (note the “e” with
       an acute accent), which due to its short length challenges inference
       of the encoding used.
              printf 'caf\351\n' | LC_ALL=en_US.UTF-8 preconv
              printf 'caf\351\n' | preconv -e us-ascii
              printf 'caf\351\n' | preconv -e latin-1
       The fate of the accented “e” differs in each case.  In the first,
       uchardet fails to detect an encoding (though the library on your sys‐
       tem may behave differently) and preconv falls back to the locale set‐
       tings, where octal 351 starts an incomplete UTF-8 sequence and
       results in the Unicode replacement character.  In the second, it is
       not a representable character in the declared input encoding of US-
       ASCII and is discarded by iconv.  In the last, it is correctly
       detected and mapped.

Options         top

       -h and --help display a usage message, while -v and --version show
       version information; all exit afterward.

       -d     Emit debugging messages to the standard error stream.

       -D default-encoding
              Report default-encoding if all detection methods fail.

       -e encoding
              Override detection procedure and assume encoding.  This
              corresponds to groff's “-K encoding” option.

       -r     Write files “raw”; do not add .lf requests.

See Also         top

       groff(1), iconv(3), locale(7)

COLOPHON         top

       This page is part of the groff (GNU troff) project.  Information
       about the project can be found at 
       ⟨⟩.  If you have a bug report for
       this manual page, see ⟨⟩.  This
       page was obtained from the project's upstream Git repository
       ⟨⟩ on 2020-07-14.  (At that
       time, the date of the most recent commit that was found in the repos‐
       itory was 2020-07-12.)  If you discover any rendering problems in
       this HTML version of the page, or you believe there is a better or
       more up-to-date source for the page, or you have corrections or
       improvements to the information in this COLOPHON (which is not part
       of the original manual page), send a mail to

groff            27 May 2020                      preconv(1)

Pages that refer to this page: groff(1)