Messages in this thread | | | Date | Sun, 07 Jan 2007 20:29:46 +0100 | From | Tilman Schmidt <> | Subject | Re: OT: character encodings |
| |
Am 07.01.2007 18:06 schrieb Russell King: > > $ git log | head -n 1000 | tail -n 200 > o > $ file -i o > o: text/plain; charset=us-ascii > $ git log | head -n 1000 | tail -n 300 > o > $ file -i o > o: text/plain; charset=us-ascii > $ git log | head -n 1000 | tail -n 400 > o > $ file -i o > o: text/plain; charset=utf-8 > > (and you know what charset the file is thought to have with all 1000 > lines in it.)
What the "file" command thinks is hardly relevant here. "file" just attempts to guess what the contents of a file might be, by applying a simple set of heuristics. Your results only highlight the actual problem: "git" is apparently unable to handle character sets properly and instead produces a mix of encodings as output.
> All on a system with LANG set to en_GB (iow ISO-8859-1).
For software with proper multilingual support, that should have been enough to make sure that all its output would be in iso-8859-1, too. Obviously "git" doesn't fall into that category.
>> Yes. When you stored it on disk, the character set information was lost. > > The same thing actually happens when I look at it via: > > $ git log | head -n 1000 | less
The loss has happened long before you run that command, when the data was committed into "git".
> So, I think you'll find that the contents of git _is_ an ad-hoc collection > of character sets which people happen to have in use on their machines.
Exactly.
>> A mixed charset environment was _already_ a pain in the butt, because >> almost nobody got labelling right. It's wrong to blame that on UTF-8. > > I'm not talking about a mixed charset environment. I'm talking about > non-UTF-8 single charset environments being broken by programs which > universally think the universe is UTF-8 only.
The problem is not programs thinking the universe is UTF-8 only; it's people mixing different charsets, in conjunction with programs not caring about charsets at all.
Specifically, your non-UTF-8 single charset environment was not broken by git thinking everything was UTF-8, but to the contrary by some data in the git repository actually being UTF-8 and git *not* thinking that.
And that problem is, I repeat, much older than UTF-8.
HTH Tilman
-- Tilman Schmidt E-Mail: tilman@imap.cc Bonn, Germany Diese Nachricht besteht zu 100% aus wiederverwerteten Bits. Ungeöffnet mindestens haltbar bis: (siehe Rückseite)
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |