lkml.org 
[lkml]   [2007]   [Jan]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: OT: character encodings
    Am 07.01.2007 18:06 schrieb Russell King:
    >
    > $ git log | head -n 1000 | tail -n 200 > o
    > $ file -i o
    > o: text/plain; charset=us-ascii
    > $ git log | head -n 1000 | tail -n 300 > o
    > $ file -i o
    > o: text/plain; charset=us-ascii
    > $ git log | head -n 1000 | tail -n 400 > o
    > $ file -i o
    > o: text/plain; charset=utf-8
    >
    > (and you know what charset the file is thought to have with all 1000
    > lines in it.)

    What the "file" command thinks is hardly relevant here. "file" just
    attempts to guess what the contents of a file might be, by applying
    a simple set of heuristics. Your results only highlight the actual
    problem: "git" is apparently unable to handle character sets properly
    and instead produces a mix of encodings as output.

    > All on a system with LANG set to en_GB (iow ISO-8859-1).

    For software with proper multilingual support, that should have been
    enough to make sure that all its output would be in iso-8859-1, too.
    Obviously "git" doesn't fall into that category.

    >> Yes. When you stored it on disk, the character set information was lost.
    >
    > The same thing actually happens when I look at it via:
    >
    > $ git log | head -n 1000 | less

    The loss has happened long before you run that command, when the
    data was committed into "git".

    > So, I think you'll find that the contents of git _is_ an ad-hoc collection
    > of character sets which people happen to have in use on their machines.

    Exactly.

    >> A mixed charset environment was _already_ a pain in the butt, because
    >> almost nobody got labelling right. It's wrong to blame that on UTF-8.
    >
    > I'm not talking about a mixed charset environment. I'm talking about
    > non-UTF-8 single charset environments being broken by programs which
    > universally think the universe is UTF-8 only.

    The problem is not programs thinking the universe is UTF-8 only; it's
    people mixing different charsets, in conjunction with programs not
    caring about charsets at all.

    Specifically, your non-UTF-8 single charset environment was not broken
    by git thinking everything was UTF-8, but to the contrary by some data
    in the git repository actually being UTF-8 and git *not* thinking that.

    And that problem is, I repeat, much older than UTF-8.

    HTH
    Tilman

    --
    Tilman Schmidt E-Mail: tilman@imap.cc
    Bonn, Germany
    Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
    Ungeöffnet mindestens haltbar bis: (siehe Rückseite)

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

    \
     
     \ /
      Last update: 2007-01-07 20:29    [W:5.454 / U:0.060 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site