<beerman>
its rather strange really.. and I had the same problem on 3.6, i am not too sure about the locale, but might as well be.. I am pretty clueless at this point, I didn't try to run it through gdb or anything yet
<jaeger>
You had pkgadd lose files installing go on 3.6?
<beerman>
pkgmk
<beerman>
the package itself just didn't contain stuff, I didn't debug it on 3.6 that way though, I am actually just now building an updated go package with gnu tar to bump grafana etc
<jaeger>
Are we talking about minor changes like a few files here and there or modes.... or are we talking everything from /usr/lib/go/* missing?
<beerman>
the latter
<jaeger>
I've only seen the latter in 3.7
<beerman>
i practically never got the same footprint twice it seemed, sometimes I got a working pkg including the symlink, sometimes not
<beerman>
and I didn't fiddle with my pkgmk on 3.6, although I might still have a patch from braewoods in there which I might have forgotten about
<jaeger>
On 3.6 it always has minor footprint differences with fakeroot, consistent without fakeroot... but 3.7 is far more broken
<jaeger>
I've never seen the symlink missing OR the /usr/lib/go/* problem in 3.6
<beerman>
first try on 3.6 and the pkg works out of the box
<beerman>
using gnu tar..
<beerman>
i dunno :D thats the problem I was talking about all the time and didn't find a solution to
<jaeger>
weird... and frustrating
<beerman>
indeed
<beerman>
but at least now I can guarantee myself to get a working pkg so thats great.
<beerman>
I guess this needs to be discussed upstream libarchive
<jaeger>
Yeah, I'd just really like to know the root cause
<beerman>
using dot instead of asterisk doesn't do any magic either, everytime I used bsdtar it just failed horribly
<beerman>
and yes, me too. Would be great to keep bsdtar around here, best solution to get it working that I can see right now is to refactor pkgmk to allow overriding how the tarball is created, like unpack_source()
<jaeger>
for even more weirdness, a package built with LANG=C before the bsdtar calls in pkgmk has 12100 files (expected 13056, original failure was 3280)
<beerman>
headache material
<jaeger>
indeed. And speaking of heacahe, back to work for now
<beerman>
heh, good luck
<jaeger>
er, headache
<jaeger>
thanks
<stenur>
has anyone tried to change C.UTF-8 with en_US.UTF-8 in /usr/bin/pkgmk then?
<jaeger>
yes. I don't remember which results that gave but it didn't help
<jaeger>
(have tried a LOT of different things)
<stenur>
hmm, has anyone ever tried to build libarchive _without_ iconv?
<jaeger>
OK, I couldn't let it go and have just stumbled upon one possible solution that works... and now I really have to get back to work. Will detail later.
<stenur>
since CRUX now will ship with C.UTF-8 available all the time (will it), that conversion step could simply be changed
<stenur>
and MAC uses a different UTF-8 character set (decomposed normalized or what)
<stenur>
heck libarchive/archive_string.c is a mess with all these special cases; iconv is a really bad thing, i always said that
<stenur>
uuuuuhhhh :)
<stenur>
wrong channel sorry
farkuhar has quit [Read error: Connection reset by peer]
farkuhar has joined #crux-devel
<jaeger>
OK, got another few minutes during lunch. Here's the change I'm now testing: http://ix.io/3TX1
<jaeger>
a quick test with just go seems to be happy with that change. Currently I'm rebuilding all the ISO packages with that change to see if anything else is affected
<beerman>
you added --format=gnutar?
<jaeger>
yes
<beerman>
neat, cool find
<stenur>
how about --format=pax then?
<jaeger>
If it doesn't introduce any unexpected weirdness I'd be fine with making the change for 3.7
<jaeger>
stenur: haven't tested yet (I do plan to) but some online stuff indicates that's the default (and the cause of this issue)
<stenur>
Oh. :)
<stenur>
(I am reading your link right now)
<jaeger>
I would have to check the bsdtar source to know for sure but haven't done that yet
<stenur>
I just switched to PAX format for my mailer and the TZDB did not too far ago.
<jaeger>
For most situations it's probably fine (tm), I have only seen this issue with go so far
<jaeger>
But if the fix is simple and doesn't break anything else, yay
<stenur>
it is not true what he says regarding "the os will convert internally".
<stenur>
Well if the encodings do not match you have borked filenames. On the other hand pkgmk sets C.UTF-8 now, anyhow. So if C.UTF-8 will always be installed on the system (and it can only in 3.7 because only ever since glibc supports that local _really_), then the effect should be somehow identical, no?
<jaeger>
In theory, yeah... but 3.7 is where the biggest problem is happening
<stenur>
dunno how that can happen, and noone said something about "XY can't be converted from UTF-8 to current locale"
<jaeger>
In 3.6 we'd see some noise about pathnames failing to be converted but in 3.7 pkgadd is actually skipping files during the install
<stenur>
And C.UTF-8 as created by newest glibc _is_ installed?
<jaeger>
indeed
<stenur>
Then the locale is borked no? UTF-8 -> UTF-8 should be possible.
<jaeger>
I don't know, locales are voodoo to me
<jaeger>
I see no indication from a non-expert perspective that the locales are misconfigured.
<stenur>
This is all insane. QT had a filesystem-locale in addition, twenty years ago. Some network systems use windows-1252, but ok samba or what will convert on the fly to the charset it should convert to, whether that is the user's setting, i have no idea
<stenur>
C.UTF-8 was broken. Sigh. I think some experts were talking on how it should behave, this is not standardized yet. (I think _i_ have opened a standards issue on that, but i could be mistaken.)
<jaeger>
That makes me wonder if another test is warranted
<jaeger>
Instead of setting the format, maybe directly invoking bsdtar with "LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 bsdtar" as well as the same but with C.UTF-8 and see what happens
<jaeger>
When I have time I'll test that as well
<stenur>
(LANG is not inspected when LC_ALL is set, unless anything is borked.)
<jaeger>
That was my understanding as well but I noticed at least one place in docker where they set both
<stenur>
Didn't you say you were testing that case already? Anyhow this problem cannot be helped out unless people are enforced to use an UTF-8 locale when using CRUX. No matter what.
<jaeger>
I tested one of those cases, not both and compared
<stenur>
If i would use de_DE.ISO-8859-1 and CRUX 3.7 pkgadd gives me C.UTF-8 the name of the file will be stored in the wrong encoding on the disk.
<stenur>
The _only_ real possibility would be to forcefully convert all filenames to 7-bit before packing, then with LC_ALL=C
<stenur>
Otherwise end-user boxes _must_ comply with XY.UTF-8
<stenur>
Storing data "binary" is an option, but an UTF-8 filename on a swedish non-UTF-8 box looks grazy.
<jaeger>
Is grazy a german thing?
<stenur>
Megalomania hits everybody
<jaeger>
As for the locale stuff, I don't pretend to know all about it, I just want my system to work right :) Which for me includes go as I use it a lot
<stenur>
I do not understand why "LC_ALL=C.UTF-8 XY" and "LC_ALL=en_US.UTF-8 XY" produce different results regarding iconv(3). This is not ok, but again C.UTF-8 is not truly standardized yet (as far as i know).
<stenur>
Maybe CRUX should use en_US.UTF-8 in /usr/bin/pkgmk then. I see no point in C.UTF-8 if it cannot fully convert the Unicode range, aka UTF-8 -> UTF-8.
<stenur>
So core/glibc/Pkgfile had to generate that instead/in addition.
<stenur>
Anyhow --format=gnutar will not help _if_ the external ball uses --format=pax, as TZDB and my one do, and surely a lot of others. Only by sheer luck -- no real Unicode names involved.
<stenur>
i only track the git repo
SiFuh has quit [Ping timeout: 260 seconds]
SiFuh has joined #crux-devel
braewoods has quit [Quit: WeeChat 2.8]
braewoods has joined #crux-devel
<beerman>
just tried --format=gnutar, seems to work fine here
<stenur>
always had that gut feeling you are a lucky fellow