manuel has quit [Remote host closed the connection]
manuel has joined #yocto
<LetoThe2nd>
yo dudX
<LetoThe2nd>
jackos888[m]: first - please at least have a short topic in the post here. many do not open links without context.
<LetoThe2nd>
jackos888[m]: second: it just doesn't work like that. recipe data is local, configuration data is global. hence, one recipe cannot affect another one. if you want to set something for the whole build, it needs to go into {local, machine, distro}.conf
alessioigor has joined #yocto
pbergin has joined #yocto
goliath has joined #yocto
manuel1985 has joined #yocto
camus has quit [Ping timeout: 252 seconds]
mckoan|away is now known as mckoan
<mckoan>
good morning
camus has joined #yocto
amitk_ has joined #yocto
<mckoan>
The Yocto Project Summit 2022.11 starts today \o/
<LetoThe2nd>
mckoan: woohoo! I have a demo setup that seems to be working, a 2hr live coding slot and beer in the fridge. what could possibly go wrong?
rfuentess has joined #yocto
nemik has quit [Ping timeout: 260 seconds]
nemik has joined #yocto
camus has quit [Ping timeout: 255 seconds]
nemik has quit [Ping timeout: 264 seconds]
nemik has joined #yocto
d-fens has joined #yocto
<jclsn>
You could drop the beer while working from the fridge to your desk
<jclsn>
s/working/walking/
goliath_ has joined #yocto
goliath has quit [Ping timeout: 260 seconds]
goliath_ is now known as goliath
<d-fens>
hi, searched the docs but couldn't find it: when building in a CD pieline i need to set variables (e.g. ssid) to a specific value to be used in a receipt. 1) how should i pass it in - environment, local.conf ? 2) how can i access the variable in the receipt?
<LetoThe2nd>
d-fens: depends. either environment, then you need to them to BB_ENV_PASSTHROUGH, otherwise they will be filtered out, or injecting into auto.conf or local.conf, depending on the mechanisms you use.
<d-fens>
BB_ENV_PASSTHROUGH helped, thx
zpfvo has joined #yocto
<mcfrisk>
note that BB_ENV_PASSTHROUGH usage is bad design, it leaks the variable everywhere. I think it's better to amend variables into bitbake variables via local.conf
BCMM has joined #yocto
jclsn has quit [Quit: WeeChat 3.7.1]
nemik has quit [Ping timeout: 248 seconds]
nemik has joined #yocto
hiyorijl[m] has quit [Quit: You have been kicked for being idle]
goliath has quit [Quit: SIGSEGV]
jclsn has joined #yocto
nemik has quit [Ping timeout: 265 seconds]
nemik has joined #yocto
olani_ has joined #yocto
urpauollu has joined #yocto
olani_ has quit [Client Quit]
olani_ has joined #yocto
florian has joined #yocto
nemik has quit [Ping timeout: 260 seconds]
nemik has joined #yocto
<d-fens>
mcfrisk: ok i understand, is there a built in method to abort a build when a variable is not set like abort_if_unset(VAR) ?
d-s-e has joined #yocto
<mcfrisk>
nope. but you can set a variable in local.conf so that it's visible in one recipe only and in that recipe fail the build if input isn't what's expected.
<qschulz>
d-fens: I guess you could write some anonymous python that would get the variable and bb.fatal otherwise
<qschulz>
I guess you probably need to set BB_DONT_CACHE also to make sure it's always run and not using cache
<d-fens>
oha, good hint
<qschulz>
RP: re the small cache optimization. I'd really recommend not changing the meaning of ordered parameters in a function, just add the new parameter at the end
<qschulz>
it makes refactoring much harder and unexpected bugs/side effects to happen :/
<qschulz>
wishing y'all a nice YP Summit, looking forward to checkout the recordings, couldn't join this time. Have fun!
tomzy_0 has joined #yocto
goliath has joined #yocto
aleksandar-simon has joined #yocto
aleksandar-simon is now known as aleksandarsimono
aleksandarsimono has quit [Client Quit]
<d-fens>
qschulz: does BB_DONT_CACHE also work in a bbappend file? i have to bump PR to get it reparsed
<qschulz>
d-fens: if the bbappend is parsed, I guess so yes.
mschnelte has joined #yocto
aleksandarsimono has joined #yocto
prabhakarlad8 has quit [Ping timeout: 260 seconds]
onkelpit has joined #yocto
florian_kc has joined #yocto
djs has quit [Quit: Client closed]
Circuitsoft has quit [Quit: Connection closed for inactivity]
<mschnelte>
Hi everyone, I am new to yocto and I am trying to build a package that during build is using python to generate some code. This is done using the Cheetah module. I have added DEPENDS = "python3-native python3-cheetah-native" to the recipe. If i am in the devshell and do a 'which python3' i can see that it is using the host tool. The only way i
<mschnelte>
found was to explicitly call nativepython3. That does not feel correct. It means i need to change my python file from #!/usr/bin/env python3 to #!/usr/bin/env nativepython3. Is there a better way to do that?
<mschnelte>
If i use python3 from the host then the cheetah module is not there (Unless i install it - and i do not want to make assumptions about installed python modules on the host)
<RP>
qschulz: true, although in this case it is effectively internal API
<mschnelte>
yes, i saw that the recipe is in recipes-devtools. Am i not supposed to use this but use another layer instead?
<qschulz>
mschnelte: inherit python3native in your recipe instead of just the DEPENDS on python3-native
d-s-e has quit [Ping timeout: 248 seconds]
<qschulz>
mschnelte: you can take inspiration from other recipes inheriting the python3native class but I think that's pretty much all you need
<mschnelte>
Tnx for the quick help! I will try that out
Tyaku has joined #yocto
<qschulz>
mschnelte: good luck, let us know how it went
<Tyaku>
Hello, We have a problem in our Yocto board. We have a JAVA program that is sometimes killed by oom_reaper, sometimes we also have kernel panic. We see that available memory decrease by the time BUT with ps -aux or top, we don't see the memory of any programs increasing by the time. Do you have any tips to identify the cause of the memory leak ?
GNUmoon has joined #yocto
d-s-e has joined #yocto
<Tyaku>
Note: I think the JAVA program is killed by oom-reaper because in case where the system has low memory, oom-reaper try to kill the software with most memory usage. Our JAVA software use 3Mo of memory (mainly due to JVM) and the memory usage is stable.
<mschnelte>
qschulz: Your info did the trick! Now i "only" need to fix the python script as it is python2 based... Tnx a lot
nemik has quit [Ping timeout: 252 seconds]
nemik has joined #yocto
leon-anavi has joined #yocto
nemik has quit [Ping timeout: 264 seconds]
nemik has joined #yocto
prabhakarlad has joined #yocto
<rburton>
Tyaku: dmesg will tell what process was killed and how much memory it was using when it OOM kills
seninha has quit [Quit: Leaving]
OnkelUlla has quit [Read error: Connection reset by peer]
OnkelUlla has joined #yocto
amelius has joined #yocto
amelius is now known as aduda
denisoft81 has joined #yocto
Qorin has joined #yocto
otavio has joined #yocto
amitk__ has joined #yocto
hcg has quit [Quit: leaving]
amitk_ has quit [Ping timeout: 268 seconds]
denisoft81 has quit [Quit: Leaving]
olani_ has left #yocto [Using Circe, the loveliest of all IRC clients]
behanw has joined #yocto
jtoomey has joined #yocto
d-s-e has quit [Ping timeout: 264 seconds]
alessioigor has quit [Quit: alessioigor]
<JPEW>
RP: Regardless of what I try, I can't seem to get the parsing time below about 2x with the extra data
<JPEW>
I think it's just so much more data to transfer
<JPEW>
For poky + all of meta-openembedded, I get ~10s -> ~20s
<JPEW>
I tried to add a bunch of layers in case the increase isn't linear, but it appears to be approximately linear
<JPEW>
The main thing I've noticed is that the parser processes finish about 2/3s of the way through the parsing and the main process is a "long tail" with the extra data, but with master the worker processes are running the whole parsing time
<JPEW>
e.g. the main process can't keep up with the volume of data over the queue from the workers
<JPEW>
But I don't think there is any way to acclerate that due to the GIL
EarnestSon has joined #yocto
<JaMa>
EarnestSon: hi
<EarnestSon>
Hi~
gho has joined #yocto
xmn has joined #yocto
frieder has joined #yocto
frieder has quit [Remote host closed the connection]
d-s-e has joined #yocto
<RP>
JPEW: with codeparser, we write the cache in each of the parser threads, then the main thread just reads, merges and writes. Not sure if that is a good idea here or not
d-s-e has quit [Ping timeout: 260 seconds]
gho has quit [Ping timeout: 256 seconds]
kscherer has joined #yocto
gho has joined #yocto
sakoman has joined #yocto
<JPEW>
I tried a form of that and it didn't really help much
<JPEW>
I had the parser processes write out the objects to a file instead of back to the pipe, but it didn't make much difference; the slow down (AFAICT) is that the main process still has to wait to read all of the data from the workers
<JPEW>
(and deduplication of data takes some time also, but it's offset by transfering less data)
neil499 has joined #yocto
amitk__ has quit [Ping timeout: 260 seconds]
grjicwvfr has joined #yocto
wulkigztq has joined #yocto
vvcaqmixw has joined #yocto
snyvlxltc has joined #yocto
<RP>
JPEW: fair enough, I'll drop that idea then :)
<RP>
for codeparser at least the dedup is easy
<JPEW>
I even tried doing some crazy stuff with memfds, but ran out of file descriptors :)
<RP>
JPEW: haha, impressive :D
<RP>
JPEW: I think our best option might be to reduce the data, which is where addpylib is going
<qschulz>
had you had a look at multiprocessing.Manager?
<qschulz>
we'd bypass the GIL with it (with both its benefit and downsides), but if everything is handled as a dict in the cache, then dict would guarantee this dedup by design directly from the writer processes
<qschulz>
but multiprocessing is a PITA, I very much didn't like writing any of it
<qschulz>
just throwing ideas, I don't know much of the context or attempts that have been made :)
vvcaqmixw has quit [Quit: Using Circe, the loveliest of all IRC clients]
wulkigztq has quit [Quit: Using Circe, the loveliest of all IRC clients]
grjicwvfr has quit [Quit: Using Circe, the loveliest of all IRC clients]
snyvlxltc has quit [Quit: Using Circe, the loveliest of all IRC clients]
pbergin_ has joined #yocto
pbergin has quit [Ping timeout: 264 seconds]
Haxxa has quit [Remote host closed the connection]
pbergin__ has joined #yocto
GNUmoon has quit [Ping timeout: 255 seconds]
Haxxa has joined #yocto
pbergin__ has quit [Client Quit]
pbergin_ has quit [Ping timeout: 248 seconds]
GNUmoon has joined #yocto
<RP>
qschulz: could be interesting, I suspect the overhead may be too great but might be worth a try
nmehta3999 has joined #yocto
gho1 has joined #yocto
gho has quit [Read error: Connection reset by peer]
gho2 has joined #yocto
d-s-e has joined #yocto
neil499 has quit [Ping timeout: 264 seconds]
gho1 has quit [Ping timeout: 268 seconds]
<JPEW>
qschulz: I'm not sure it would help. I don't think it uses SHM, so it still has to serialize reading the data on the server side? It was on my list to look at though
neil499 has joined #yocto
Haxxa has quit [Quit: Haxxa flies away.]
Haxxa has joined #yocto
nmehta3999 has quit [Ping timeout: 260 seconds]
<JPEW>
Ah, it looks like it does use SHM; might be worth a try
<qschulz>
JPEW: they do say in the docs the objects needs to be pickle-able though
<qschulz>
where did you see the shm part?
<JPEW>
Read the code
<JPEW>
I read the code that is :)
<qschulz>
RTFC :)
Vonter has quit [Ping timeout: 264 seconds]
Vonter has joined #yocto
urpauollu has quit [Remote host closed the connection]
rob_w_ has quit [Quit: Leaving]
alessioigor has joined #yocto
Tyaku_ has quit [Quit: Lost terminal]
alessioigor has quit [Quit: alessioigor]
olani_ has joined #yocto
olani_ has quit [Ping timeout: 265 seconds]
manuel1985 has quit [Ping timeout: 252 seconds]
<vvn>
if distro-b requires distro-a but doesn't append recipe foo, bitbake will reuse the sstate cache and won't recompile foo, right?
goliath has quit [Quit: SIGSEGV]
<mschnelte>
I have configured an sstate_mirror without a hash equivalence server.
<mschnelte>
Now I get the warning that this "will likely mean no sstate will match from the mirror".
<mschnelte>
After having watched the video of Joshua Watt I come to the conclusion that only SOME (instead of NO) sstate will not match, i.e. those for which more than one entry in the Hash server exists.
<mschnelte>
Because in those cases there is no sstate entry because it is matched by a previous entry and the client would not know about this because it does not have access to the Hash Equivalence server. Is this correct?
gho2 has quit [Ping timeout: 260 seconds]
<mschnelte>
Secondly I was wondering how a hash equivalence CI setup would look like and I did not find any documentation on this.
<mschnelte>
I understand that I will need to have one read-only instance of the server for all those clients that will not populate the sstate, alongside with another hash server that allows writes. The second hash server is using the same database as the read-only server and builds using the writable server will also populate the sstate mirror. Is this
<mschnelte>
correct?
<RP>
mschnelte: it is possible some would match, it just becomes very unlikely over time as the hash equivalence causes the hashes to diverge
<RP>
mschnelte: that sounds correct to me
aleksandarsimono has quit [Remote host closed the connection]
<mschnelte>
RP: Thanks!
GNUmoon has quit [Ping timeout: 255 seconds]
d-s-e has quit [Quit: Konversation terminated!]
olani_ has joined #yocto
gho has joined #yocto
protu[m] has quit [Quit: You have been kicked for being idle]
GNUmoon has joined #yocto
rstreif has joined #yocto
tor has quit [Quit: Leaving]
<prabhakarlad>
Hi all, when using the -static flag for compilation with the SDK build by yocto I see this error "cannot find -lc". when I use the same static flag when using the ubuntu cross toolchain I dont see this issue. any pointer on what I could be missing?
rfuentess has quit [Remote host closed the connection]
gho has quit [Ping timeout: 265 seconds]
gsalazar has quit [Ping timeout: 268 seconds]
x2s has joined #yocto
<x2s>
Hi. I'm currently porting a project from rocko to dunfell and in that process switching from X to wayland. Now I think I've understood mostly the differences now, but I wonder about best practices. The old project uses two applications, one that displays an animation for a while and quits, while the real application loads in the background and takes over. This is purely done without a window manager
<x2s>
in X, so it's a bit "magical" how it works.
<x2s>
I think I can do the same with weston, and since both applications were written in Qt it should work mostly like before. But that's a point where I'm looking for suggestions.
<x2s>
Also for debug purposes on rocko there was always a vnc server running so you could connect remotely and click on things. Now with wayland this has become a bit of a problem and that's where I'm looking for an advice. How can I get a remote screen connection to wayland to see what's displayed (in weston)?
Andrew20 has joined #yocto
Kleist has quit [Ping timeout: 260 seconds]
gho has joined #yocto
<JPEW>
RP, qschulz: Synchronization on the object manager looks like it's going to be too slow
nmehta3999 has joined #yocto
gho has quit [Ping timeout: 265 seconds]
gsalazar has joined #yocto
neil499 has quit [Ping timeout: 260 seconds]
gho has joined #yocto
goliath has joined #yocto
<vvn>
Compiling the initramfs image with PREFERRED_PROVIDER_virtual/kernel = "linux-dummy" is likely a bad idea, right?
<RP>
JPEW: I feared that may be the case :(
Andrew205 has joined #yocto
Andrew20 has quit [Ping timeout: 260 seconds]
gsalazar has quit [Ping timeout: 265 seconds]
gsalazar has joined #yocto
gho has quit [Quit: Leaving.]
zpfvo has quit [Remote host closed the connection]
neil499 has joined #yocto
nmehta3999 has quit [Ping timeout: 264 seconds]
mckoan is now known as mckoan|away
<qschulz>
JPEW: are you passing the Manager to threads or processes?
<JPEW>
processes
<qschulz>
mmm so pickling is THAT expensive I guess?
<JPEW>
Pretty sure is the locks that are expensive
<qschulz>
I guess you could use a Queue then, have the main thread read the queue that is filled by all other cache processes separately and then do the dedup and what not to create the final dictionary?
nmehta3999 has joined #yocto
<qschulz>
or is this what you already did? (Manager is just the interface to a proxy object in the end)
florian has quit [Quit: Ex-Chat]
neil499 has quit [Ping timeout: 264 seconds]
Sario has joined #yocto
florian_kc has quit [Ping timeout: 268 seconds]
<JPEW>
Ya, my current patch uses a queue. The workers send a tuple (new_dedup, os.getpid(), data, dedup_data). new_dedup is the new object that were first seen in this object (so each object is only sent once), data is __dict__ minus the keys to be deduplicated, and dedup_data is the __dict__ keys that should be deduplicated, where each one has it's dictionary values replaced with the index of the deduplicated object (previously sent in new_dedup)
<JPEW>
So, bascially every unique deduplicated object is only sent once (per worker)
neil499 has joined #yocto
nmehta3999 has quit [Ping timeout: 256 seconds]
nmehta3999 has joined #yocto
<qschulz>
JPEW: and I guess using a Manager.dict() directly instead of Manager.Queue() wouldn't be that smart because of all the serializing to do between processes during each access?
<qschulz>
in which case you would directly modify the Manager.dict() from the workers
<JPEW>
We have to post process the data, so its not quite a simple as letting the workers update the dict directly
<qschulz>
I guess you'd need a lock if you need to do multiple operations atomically in one process
<qschulz>
ah ok :/
<JPEW>
I mean, we could have them populate _a_ dict, but we'd still need to process it in the main process
<JPEW>
So a queue makes more sense
<qschulz>
yeah I get that :/
<qschulz>
sad it didn't help, glad we don't go with multiprocessing (at least for that)
neil499 has quit [Ping timeout: 268 seconds]
<qschulz>
it's always the beginning of nightmares (I still have PTSD and won't touch my project anymore mostly because of that)
<RP>
JPEW: I'll be quite interested to see if reducing the size of the data helps or not. The trouble is we'll basically have to go ahead and do nearly all the work of converting to find out...
<JPEW>
ya
* RP
thinks it probably is the right thing anyway
<RP>
I was thinking if the inspect on the python funcs ends up too slow, we can cache the result
tor has joined #yocto
EarnestSon has quit [Quit: Client closed]
neil499 has joined #yocto
nmehta3999 has quit [Ping timeout: 264 seconds]
Mark_L has quit [Quit: Mark_L]
nmehta3999 has joined #yocto
neil499 has quit [Ping timeout: 260 seconds]
ecdhe_ has joined #yocto
ecdhe_ has quit [Client Quit]
BCMM has quit [Quit: Konversation terminated!]
neil499 has joined #yocto
nmehta3999 has quit [Ping timeout: 268 seconds]
<JPEW>
Ah, CoreRecipeInfo is used both for the parser task -> main task IPC _and_ to serialize the cache to disk
<vvn>
aleksandarsimono: don't copy/paste on IRC, use services such as pastebin.com or ix.io
<aleksandarsimono>
ok sorry
<vvn>
also the deploy class is for copying files to DEPLOY_DIR, i.e. output binaries, not copy files in the image rootfs
<vvn>
you want to copy the .dtbo in the image rootfs, right?
<aleksandarsimono>
yes
<aleksandarsimono>
image rootfs (/boot/overlays)
Qorin has quit [Quit: Client closed]
vmeson is now known as JackHandey
JackHandey is now known as vmeson
<vvn>
aleksandarsimono: look at the raspberrypi3.conf machine definition (which I presume you are using). It includes rpi-base.inc, which defines RPI_KERNEL_DEVICETREE_OVERLAYS with a bench of .dtbo. Grep that variable in meta-raspberrypi to understand how it's used.
<aleksandarsimono>
Yes, I'm using raspberrypi3.conf machine
<aleksandarsimono>
I will take a look
<aleksandarsimono>
thank you very much
<vvn>
(you likely just have to append "overlays/mcp2515-can2.dtbo" to RPI_KERNEL_DEVICETREE_OVERLAYS if that dtbo is already built by the kernel)
qschulz has quit [Ping timeout: 260 seconds]
<aleksandarsimono>
I'm not sure that it is built
qschulz has joined #yocto
<vvn>
aleksandarsimono: remove your custom recipe and try to add this in your local.conf: RPI_KERNEL_DEVICETREE_OVERLAYS:append = " overlays/mcp2515-can2.dtbo"
<aleksandarsimono>
I will try
fray has joined #yocto
<fray>
I'm trying to diagnose an issue where the system is telling me hashs are changing during the build.. I can't figure out how to dump the before/after hash so I can bitbake-diffsigs.. Any suggestions?
mschnelte has quit [Quit: Client closed]
aleksandarsimono has quit [Ping timeout: 260 seconds]
amitk has quit [Ping timeout: 265 seconds]
<JPEW>
RP: Well, I eeked out a little more: 11.4s -> 19.5s (71%) officially.
onkelpit has quit [Quit: hibternate]
<fray>
figured it out.. two different distros.. one was setting PRSERV_HOST the other was not, which changd the value of PRSERV_ACTIVE during the build!