<RP>
JPEW: I was confusing myself last night. The issue is that there are two different unihash mappings for a given taskhash in the DB. Obviously this should not happen, yet it has and when it does, the behaviour is problematic.
<RP>
JPEW: I dumped this from the database on the live server:
<RP>
JPEW: when just asked for an equivalence, the first is returned but there is now sstate object for it so it rebuilds. When the object is built, it sends the outhash and gets the latter back
<RP>
JPEW: I think the code needs to error if this happens
tre has joined #yocto
m4ho has quit [Read error: Connection reset by peer]
<agherzan>
I got into an interesting (and unexpected issue today). qemu (native recipes) inject host pkgconfig paths trying to detect host's SDL. This has a pretty obvious side effect - configure will use pkgconfig to detect other libraries - e.g. libnfs. I had this library for a while on the host (without even knowing as a dependency to VLC) - all good. Once that host dependency disappeared (doesn't matter why) or got upgraded, qemu started to fail.
<agherzan>
Obviously nothing re-triggered this on the build's side so linking started to fail. I propose to disable it by default to avoid this (libnfs is only in some meta-kodi layer). We could try to make this a bit smarted by making the pkgconfig path injection more specific (only for SDL).
<RP>
agherzan: setting configure options explicitly is definitely a good thing
<RP>
agherzan: if we can limit the sdl pkgconfig search paths that would also be good
<RP>
kanavin: interesting. It is a different way to do it :)
<agherzan>
I don't yet know how to achieve the latter RP . I'll think about it a bit but I'd start with forcing a disable for now just to get us out of this for now.
<RP>
agherzan: that is definitely the right place to start
<kanavin>
RP: I've just kicked off a massive a-full with all of my stuff that's piled up :) will follow that by a meta-oe build as well
<agherzan>
Agreed.
jwillikers has joined #yocto
<RP>
kanavin: I see you're continuing with the native/target vendor changes :/
jwillikers has quit [Remote host closed the connection]
<RP>
The trouble is nobody is paying any attention to the release freeze or the need to fix bugs before we can release
<RP>
I'm getting really really depressed with it :(
jwillikers has joined #yocto
<kanavin>
RP: that's the only way to avoid patching rust assumptions regarding system strings all over the place for the foreseeable future :(
<kanavin>
RP: if you need help, I have time
<agherzan>
RP: I've looks a bit more into this - and I fail to understand this qemu sdl quirk anymore. The native recipes don't have SDL enabled (as per PACKAGECONFIG) but we still enable global host pkgconfig path even if PACKAGECONFIG has it disabled. Was this meant for the cases where something enables sdl in PACKAGECONFIG externally?
<kanavin>
agherzan, I think at some point there was a use case for building qemu-native with the distro SDL
<kanavin>
agherzan, what branch are you on?
<agherzan>
On dunfell but this is on master too kanavin
<kanavin>
agherzan, it adds the host paths after the native path, so if something from the host leaks in, this means qemu has auto-detection for that something, and there should be an explicit setting via PACKAGECONFIG
<kanavin>
is there an option for libnfs, or how is that controlled?
<agherzan>
kanavin: qemu does pkgconfig on libnfs but we don't provide libnfs
<kanavin>
just add a PACKAGECONFIG for it and you'll be fine
<kanavin>
it will pass the same disabling, but then it's a setting instead of just blank prohibition
<agherzan>
But we don't provide the dependency
<agherzan>
I know what PACKAGECONFIG does. It's just that libnfs is not provided by classic layers
<agherzan>
It's only in some kodi one
<agherzan>
So why would I include the knob if there is no provider of it in the classic layers?
<kanavin>
there are plenty of knobs which silently regress because they're off by default, referring to a recipe that hasn't yet been written is ok too in my opinion
<RP>
kanavin: we could just change the values within rust though
<RP>
kanavin: I'm worried about the "change the rest of the system which is fine to match rust" aspect of this
<RP>
agherzan: I think in most cases we end up enabling that and sdl generally ends up coming from the host rather than our own recipes as we don't want to build a native GL stack
<RP>
agherzan: we could make the pkgconfig from the host conditional upon those things being enabled I guess
<kanavin>
RP: I looked into reusing rust's built-in targets, it's not going to happen :( rust upstream wants them 'sacred and immutable' (actual quote from the definitions)
<agherzan>
Right. Now I remember RP . It's though the local configuration (sample for example). That explains it now.
<kanavin>
RP: so we'll have to define and maintain our own targets, and those need to be fully formed regarding the vendor and libc
<agherzan>
through*
<RP>
kanavin: can't we just map our target list into the ones needed for rust?
mckoan is now known as mckoan|away
<kanavin>
RP: I started patching rust all over the place to make it happen, but then thought I might as well fix the issue at its source and make oe-core targets fully formed and consistent
<RP>
we already map things in the openssl build, in the kernel and so on, which is partly why changing our own definitions worries me a lot, particularly if you're telling me that in future we have to then follow exactly what rust does
<RP>
In fact because of that, the answer is definitely no, we're not going this direction
<RP>
We cannot be coupled to something like that
<kanavin>
rust only needs them to be fully formed, and doesn't mind the content - the issue is that a) -gnu suffix is absent for native and target when plain libc is in use; b) vendor is absent for native. I'm only fixing those tw.
<kanavin>
it's not rust specific in any way, I'm just filling in the missing bits
<RP>
kanavin: you're forcing rust policy onto the rest of the system
<RP>
kanavin: can't we just "fully form" the pieces rust needs fully formed for rust where it uses them?
<RP>
we already map things in the openssl build, in the kernel and so on, which is partly why changing the core values for one language worries me a lot
<kanavin>
RP: I replaced 'use TARGET_SYS directly' with 'use TARGET_ARCH, use TARGET_VENDOR if not empty, otherwise use oenative, use TARGET_OS, use -ABIEXTENSION if not empty, otherwise use -gnu' all over the place in rust, and found it ugly :(
vd898 has joined #yocto
<kanavin>
RP: but once I iron out the other issues, I can look into that again
* RP
joins the queue to wait for autobuilder time to try and find out where the sstate corruption is coming from
vd89 has quit [Ping timeout: 256 seconds]
<RP>
although maybe my plan B just yielded results
<RP>
If anyone has a build of target gcc handy, could you run "sha256sum tmp/work/XXX/gcc/11.2.0-r0/temp/depsig.do_populate_lic" ?
<RP>
I'm not interested if the checksum is db208b112ca041e00a5d514fc60d37b538b3a6dd14c78541eb1207a591944d9d but if it isn't, I wouldn't mind a look at the file
<RP>
The background is that something is making these files non-deterministic. I have a theory but I can't quite prove it :(
Ad0 has quit [Ping timeout: 245 seconds]
<RP>
it could be umask issues
<kanavin>
I have one, but it's against my mountain of changes :)
<RP>
kanavin: it shouldn't matter for a populate_lic
<RP>
kanavin: unless you changed gcc ?
<kanavin>
I do not think so ;) Let me see what that prints
artri has joined #yocto
<kanavin>
RP: d2187...
<kanavin>
will send you the file in a sec
<kanavin>
RP: done
<RP>
kanavin: thanks!
<neverpanic>
RP: Would you be interested in 9.3.0-r0 as well, or just 11.2.0?
<RP>
neverpanic: just 11.2.0 thanks
<RP>
kanavin: I managed to paste the wrong checksum above, d2187 is the one I have :/
<kanavin>
RP: :(
<RP>
I'd love to know what the file with checksum afd11c366050bcd75ad763e898e4430e2a60659b26f83fbb22201a60672019fa looks like
<RP>
kanavin: thanks though, the "right" value seems widespread. I just don't know how/where the wrong one came from
Ad0 has joined #yocto
<qschulz>
michaelo: I haven't received patch 3/4 on your latest patchseries for the docs, is that expected?
<JPEW>
RP: Sorry, between conferences and remodeling my kitchen, I've been away a lot
* JPEW
reads the backlog
<JPEW>
RP: My gcc depsig file is db208b112ca041e00a5d514fc60d37b538b3a6dd14c78541eb1207a591944d9d
<rburton>
doing a build here now
<rburton>
mine is d2187ee3a8966db10b34fe0e863482288d9a6185cb8ef58a6c1c6ace87a2f24c /yocto2/ross/build/tmp/work/armv8a-poky-linux/gcc/11.2.0-r0/temp/depsig.do_populate_lic.2962112
<JPEW>
rburton can you cat it for us?
<rburton>
though I did have to -f to get it to re-run, if that would make a difference
<rburton>
the generic file is a copy and it got hardlinked to deploy
<rburton>
guessing the copy codepath just needs to preserve permissions
<JPEW>
Ya, it looks like generic_GPLv3 is hardlinked all over the place for me
xmn has quit [Remote host closed the connection]
yates has quit [Remote host closed the connection]
<RP>
rburton: hardlinks or not shouldn't matter in this context. The group permissions definitely do but that still doesn't get us the afd11 checksum the autobuilder generated
<RP>
JPEW: I don't think depsig should be looking at group permissions outside pseudo context
<RP>
JPEW: but I think there is still a piece of this we're not unravelling and secondly, we need to get better feedback from hashequiv when this happens :/
paulg has joined #yocto
<JPEW>
RP: You could enable the extra reporting in hashserver
<JPEW>
It would store the depsigs in the database
<RP>
JPEW: given the database is already at 88GB, we can't afford that
xmn has joined #yocto
xmn has quit [Client Quit]
<JPEW>
RP: depsig does run in pseudo context
<JPEW>
RP: Ya, it is a lot of extra data
<RP>
JPEW: not for populate_lic
<JPEW>
Ah, we could add a task flag to selectively report it
<RP>
JPEW: I think we could key off some of the existing ones. I would like to understand why the default umask isn't working though
<JPEW>
RP: Sure. The extra data reporting is triggered with the `report_taskdata` variable in `report_unihash`, so it should be easy to make that True on whatever criteria you want
<JPEW>
The more difficult part is you sort of want it retroactively.... hashserver needs a time travel option I guess
<JPEW>
RP: Hmm, what if we compressed the depsig file?
<RP>
JPEW: Perhaps if we had some better way of cleaning up the DB periodically rather than it growing indefinitely
<RP>
JPEW: I don't think we even support access times against the data at present though? (due to the write load it could generate?)
<JPEW>
Right, we have a no-write path in the critical case right now
<RP>
JPEW: I think even compressed, this is too much data for the main server :(
<JPEW>
... but we might be able to figure out if there are things in the db that are irrelevent
<RP>
JPEW: with access time?
<RP>
JPEW: I was wondering if we could have some kind of low priority background "update modified time" thread which doesn't block the queries
<RP>
Ah, the issue is these license files are created outside of bitbake in the checkout
<RP>
which will use the host umask
<RP>
so the correct fix is to mask these bits as they aren't interesting here
<JPEW>
You'd have to be really careful to avoid priority inversion when locking the database
<RP>
JPEW: yes :/
<JPEW>
RP: r.e. umask: But why are they correct when copied then? The correct umask is applying in that case
<RP>
JPEW: Aren't the permissions just being preserved in the copies?
<JPEW>
RP: It doesn't appear so based on rbutons logs
<JPEW>
When it does the copy, the umask is applied
<JPEW>
but when it hardlinks, it is not
<JPEW>
(the bitbake umask)
<RP>
JPEW: right, the copy is applying the umask, hardlinks arne't
<JPEW>
The hardlink must not count as a "new file"
<RP>
JPEW: no, hardlinks share the same permissions on linux
<JPEW>
RP: Oh! I (wrongly) assumed each hardlink was a new directory entry that referenced the same on disk file object, and the perms/owners were in the directory entry.... one or more of those assumption must be wrong
<RP>
JPEW: I think you could do that in principle but it is a security nightmare so more unixes don't do that
<JPEW>
RP: Ya, that makes sense
alessioigor has joined #yocto
alessioigor has quit [Remote host closed the connection]
rob_w has quit [Remote host closed the connection]
<JPEW>
Ya, that looks fine.... do we even care about the owner permissions in that case?
<RP>
JPEW: probably at least executable
sstiller has quit [Quit: Leaving]
<JPEW>
Looking at the sqlite data; it wouldn't mess up the task if the second later entry had the same unihash as the first
<JPEW>
The problem (maybe) is that it is changing the unihash to something new
kiran has joined #yocto
roussinm has joined #yocto
tre has quit [Remote host closed the connection]
Sion has quit [Quit: Konversation terminated!]
<RP>
JPEW: the example I pasted earlier is that issue, yes. You get inconsitent results depending on whether your quiery the taskhash or taskhash+outhash. The bouncing is the issue
<JPEW>
Yep
<RP>
JPEW: if there is an existing mapping, perhaps we should just use that?
<JPEW>
RP: Right, I think so
<RP>
JPEW: and maybe print something client side to say there is non-determinism in that task?
<JPEW>
Need to think about it, but the bouncing definetly seems like an issue
<JPEW>
Ya, we could possible extend the protocol to include that
<JPEW>
What would be the most useful.... the expected outhash maybe?
<JPEW>
If you're reported outhash is not the same we report an on the client side?
<RP>
JPEW: returning the other outhash would be helpful for debugging. I had to hack the database to get it to debug this time
<kanavin>
RP: I dropped the invasive system patches, let's see if the problem is solvable with targeted fixes
<RP>
kanavin: I'm hoping we can do something that isn't too ugly and sorts that out
<RP>
JPEW: we could work around it with the existing code just by querying the server again after reporting
<RP>
kanavin: it is a decent queue :)
<RP>
JPEW: do we need anything further from the current state of things or do I try and fix the sstatesig code for the umask issue and the try and proceed with the release?
<kanavin>
RP: not until I get green a-full and green meta-oe for it :) I'm trying to start those when nothing else is going on, as you're trying to sort a release
<RP>
JPEW: I'm assuming we're going to need a little time to think about exactly how to address this
<JPEW>
RP: Ya we need some time to fix it; we can backport the fix once we figure it out
<JPEW>
I think the taskhash query is correct, but I don't want to break things worse
zpfvo has quit [Remote host closed the connection]
amitk has quit [Quit: leaving]
<JPEW>
RP: Ah! that taskhash divergance problem is *also* a race condition
<JPEW>
RP: When a task reports an outhash, it provides the unihash it's using got the task (e.g. the unihash the server reported when tasks first started parsing
<JPEW>
But, based on the timestamps, the first entry in the database wasn't there when when that happened, so the second builder (correctly) defaulted to the unihash = taskhash
<JPEW>
Then, when it reports that second entry gets added
<JPEW>
I need to run through the proof to make sure it's not an additional race to do the unihash query after the outhash query
<JPEW>
but it's sounding more correct
<vd898>
Can I set a bitbake variable from an environment variable? e.g. IMAGE_VERSION_SUFFIX ?= "-${SOME_JENKINS_BUILD_NUMBER}"
<JPEW>
vd898: I think so, but you need to add the variable to a list to make it "visible" as a bitbake variable
<vd898>
hum I think I saw that in the bitbake source or OE-Core init script...
<JPEW>
vd898: BB_ENV_EXTRAWHITE (it's a shell variable)
<vd898>
yay! I guess BB_ENV_EXTRAWHITE defaults to "BBPATH" ;-)
florian has quit [Quit: Ex-Chat]
<kergoth>
heh, we used to set bbpath and bbfiles in the environment back before bblayers was implemented :)
roussinm has joined #yocto
<RP>
JPEW: ah, right. Should the server not take the unihash but query itself I wonder?
<vd898>
kergoth: do you mean that bitbake will now try to look at the hardcoded path "./conf/bblayers.conf" if BBPATH is empty?
<RP>
JPEW: the race condition makes sense
<JPEW>
Ya, I think it needs to query itself somehow
<RP>
JPEW: can't you just do that in the sql?
vd898 has quit [Quit: Client closed]
<JPEW>
Yes, the part the think though is making sure the place we query also is not a race
vd898 has joined #yocto
florian has joined #yocto
GillesM has joined #yocto
<RP>
JPEW: could we do it all as part of the same sql query?
<JPEW>
RP: No
<JPEW>
Well... maybe. Let me look closer
florian has quit [Ping timeout: 245 seconds]
<JPEW>
RP: No. It has to be atomic with the `INSERT` statement, which can't easily be expressed with SQL (at least not AFAICT)
<JPEW>
The other option is query *after* the row is inserted and update the inserted row if the unihash is incorrect... Not sure if I like that
<RP>
JPEW: you can have INSERT INTO SELECT statements, not sure if sqlite3 supports them though
<JPEW>
It does
<JPEW>
The select statement columns have to exactly match the insert columns (maybe that's easier than I'm making it)
* JPEW
seems to have forgotten a lot of SQL
<RP>
JPEW: I didn't realise about the column constraints. Not sure if that is surmountable or not
<JPEW>
Question though: If the reports in our example were reversed that wouldn't be correct either
<JPEW>
You wouldn't want the report with unihash 3bde230 to be changed to 6259ae8
<JPEW>
Although, that's not possible I guess
<JPEW>
Either there was already a taskhash -> unihash mapping in the database at parse time (which is obviously not the case here)
<RP>
JPEW: I was wondering if something "like insert into tasks_v2 (method, outhash, taskhash, unihash, created) select 'string', 'string2', 'string3', unihash, DATETIME() from tasks_v2" would work
<JPEW>
or there was a matching outhash, in which case we would not be quering the unihash
<RP>
JPEW: right, I think there are only the match, add or race cases
* RP
hasn't used SQL for a while either
<JPEW>
either way, if the order of the reports was revered, there would be no unihash in the db to ask about
<RP>
yes
<JPEW>
Do we even need that entry in the database?
<JPEW>
I guess it allows a taskhash to map to multiple outhashes
GillesM has quit [Quit: Leaving]
<RP>
JPEW: it does flag up an issue but perhaps we simply shouldn't add it. Not sure you can do that race free either though
<JPEW>
Ya, probably not
<RP>
I think adding it map allow the system to "heal" from bad reproducibility too
<JPEW>
RP: Right
<JPEW>
we don't know which of the outhashes is "correct", but we do know they should all be the same unihash
<RP>
JPEW: exactly
frieder has quit [Remote host closed the connection]
jsandman has quit [Quit: Ping timeout (120 seconds)]
florian has joined #yocto
ant__ has joined #yocto
<override>
is it ok to write bbappends for PREFERRED_PROVIDER_virtual/kernel???
jsandman has joined #yocto
<kergoth>
a bbappend operates against a recipe, based on the recipe filename. direct filename match, has nothing to do with providers
<override>
i was trying to ask if virtual/kernel.bbappend works
<override>
from what I understand PREFERRED_PROVIDER works off of recipe filenames too?
<ant__>
you normally set this in machine conf / BSP isn't?
<override>
like the PN or whstevr part of the recipe filename
<ant__>
the BSP know which kernel is preferred
manuel1985 has joined #yocto
creich has quit [Quit: Leaving]
creich has joined #yocto
<override>
let me put it this way
<override>
for preferred_version we suffix it with the variale with the recipe name, what do we suffix the preferred_provider with???
* JPEW
wonders if a schema change is needed in the hash equiv database
<override>
thanks smurray: forgot about how we can put stuff like PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
yocton has quit [Remote host closed the connection]
<jonmason>
RP: it's amazing you haven't pulled all your hair out ;-)
<jonmason>
or is that why you don't turn your video on during zoom calls?
<RP>
jonmason: exactly. You can't see the bottles that way ;-)
<RP>
JPEW: hate to say this but the sstate reuse of my test against the last build is dreadful. I think the equiv racing is hurting things :(
<JPEW>
RP: Ya, that is not suprising
<RP>
JPEW: it was really good on the last one, this one is very poor. I guess it is just luck
<JPEW>
RP: I... think... we can make this a lot better, but it might involve a new database schema
<RP>
JPEW: I think that is ok
<JPEW>
RP: Ok, let me play around with it and see what I get
<RP>
JPEW: thanks. The big question is whether we wait and see how this works out or do we decide to fix post release
<RP>
JPEW: will it break hashequiv protocol from older versions?
<JPEW>
I don't think so
<RP>
ok, that would be nice!
<JPEW>
I think it will just be a server side change in bitbake
<JPEW>
no promises until I'm done though!
<RP>
JPEW: no, fair enough. I'm just trying to get an idea of what to prepare for!
<JPEW>
no plan survives contact with the enemy and all that :)
<RP>
totally, particularly in this terrain
<jonmason>
in the jungles of bitbake, we need an agent orange
Dracos-Carazza has quit [Quit: ZNC 1.8.2 - https://znc.in]
Dracos-Carazza has joined #yocto
prabhakarlad has quit [Quit: Client closed]
zen-coder has quit [Quit: Client closed]
<Tokamak>
hit a quite curious null pointer dereference on bootup. only occurs when passing the kernel arg 'debug'. curious if anyone has seen this before. Also not sure if this is init or kernel dereferencing null? https://pastebin.com/hGyMcmcU
<vd898>
is it preferable for a distro to configure package in its distro conf file with pn-recipe overrides or in recipe bbappends with distro overrides?