<Raman>
Can someone tell me why there is a limitation of 20 on number of parallel bb tasks. If I am building on high performance servers, should I do?
<Raman>
BB_NUMBER_THREADS = "64"
<Raman>
PARALLEL_MAKE = "-j 64"
<Raman>
Just want to know it's a general guidance or some known issues exist.
mrpelotaz0 has quit [Quit: Hasta la vista!]
mrpelotazo has joined #yocto
Lihis has quit [Quit: Quitting]
shoragan has joined #yocto
leon-anavi has joined #yocto
amitk has joined #yocto
Kubu_work has joined #yocto
Lihis has joined #yocto
rob_w has joined #yocto
frieder has joined #yocto
tgamblin has quit [Remote host closed the connection]
tgamblin has joined #yocto
mvlad has joined #yocto
Raman has quit [Quit: Client closed]
Xagen has quit [Ping timeout: 268 seconds]
Xagen has joined #yocto
<AdrianF>
Raman: It's a trade of between RAM and CPU cores. For example, with your settings bitbake starts 64 make processes which start 64 g++ processes. If each of the g++ processes allocates 2GB of RAM you need 8TB of RAM. Probably there is a high risk for failed builds due to OOMs.
<philipl_>
JPEW: In a sense that the hash equivalence database may contain "spurious" entries (i.e. entries that may never be referred to again)?
zpfvo has joined #yocto
Raman has joined #yocto
prabhakalad has quit [Quit: Konversation terminated!]
prabhakalad has joined #yocto
Raman has quit [Client Quit]
Jones42 has joined #yocto
florian_kc has joined #yocto
altru has joined #yocto
florian_kc has quit [Ping timeout: 240 seconds]
enok has joined #yocto
enok has quit [Ping timeout: 256 seconds]
enok has joined #yocto
ahussain has joined #yocto
ray-san has joined #yocto
ahussain1 has joined #yocto
ahussain has quit [Ping timeout: 255 seconds]
ehussain has quit [Ping timeout: 272 seconds]
ahussain1 is now known as ehussain
ahussain has joined #yocto
ehussain has quit [Ping timeout: 252 seconds]
ahussain is now known as ehussain
mbulut has joined #yocto
Daaanct12 has joined #yocto
<Jones42>
what's the most idiomatic way to change/overwrite a recipe's task order? i.e. change "addtask T before do_X after do_A" to "addtask T before do_Y after do_A"
Daanct12 has quit [Ping timeout: 268 seconds]
<Jones42>
should I change the varflags of do_X/Y or should I just deltask and add it again? or is there a better way?
<kanavin_>
Jones42, it helps if you explain what the problem is
<Jones42>
sure! I'm using kernel-fitimage.bbclass and I'd like to move "assemble_fitimage before do_install after do_compile" to "before do_deploy"
<Jones42>
(give me a second to find the explanation in my notes...)
<Jones42>
I want do decouple the rootfs generation from the kernel's assemble-fitimage generation, because I need to put the rootfs' dm-verity hash into the fitimage.
<Jones42>
It's a bit complicated to explain: I don't need the fitimage for the kernel do_install. But do_rootfs depends on the kernel's do_package/do_install
<Jones42>
so by moving the assemble fitimage a step "later" (before do_install -> before do_deploy), I can build the rootfs (and my ext4.verity image) without having to assemble_fitimage first.
<Jones42>
so by the time I assemble_fitimage, I have the rootfs.ext4.verity roothash and can put said hash into the fitimage.
<mcfrisk>
Jones42: the "before" and "after" for the tasks needs to be adjusted where they are set, IMO. I'm using meta-security dm-verity-img.bbclass for a /usr partition, fwiw.
xmn has quit [Ping timeout: 264 seconds]
<Jones42>
mcfrisk: hmm... I hoped I could overwrite it somewhere because wanted to avoid messing with any "official" files. But I think that's indeed not possible due to the local scope of each recipe :/
altru has quit [Ping timeout: 250 seconds]
<kanavin_>
Jones42, I don't think it's possible to reorder tasks after the fact, you're probably better off writing a new class, or maybe adding a new task into the correct position
enok has quit [Ping timeout: 256 seconds]
<RP>
Jones42: the constraints do "stack" so you can addtask again and add new constraints. You can't change existing ones though without deleting the task
<Jones42>
kanavin_,RP: thanks! I found the addtask code, but if I can't "bbclassappend" or overwrite the task's varflags from outside the recipe, then the only solution seems to be to write a new class
<Jones42>
wait - I could just bbappend the recipe that inherits the class, right?
<kanavin_>
yes, for sure
<Jones42>
nice, that might be the least ugly solution. :-)
<mcfrisk>
Jones42: what kind of problem are you solving? I had issues using dm-verity-img.bbclass with .wic images and ended up with two image recipe solution: one for dm-verity-img, another for .wic. then connecting output from dm-verity to .wic was a bit ugly but it worked. trying to upstream the bits and pieces at some point when there is a clear path to do so...
<Jones42>
mcfrisk: It's one of those "It should be simple in practice, but how do people actually make it work?"-Problems...
<Jones42>
I'm having a simple fitimage, no initramfs and a dm-verity rootfs. all in a wic image
<Jones42>
I'm using the image_types_verity.bbclass as seen on TV, er, on one of the CLT2024 talks. I wasn't aware of the meta-security dm-verity-img.bbclass. I only found that one later
<Jones42>
My issue is that for some reasons, my do_rootfs has dependencies on my kernel's do_package->do_install->do_assemble_fitimage->do_compile
<mcfrisk>
yes, saw those patches and was wondering what the use case really is, compared to meta-security. The problem for me was the ordering of image processing tasks, which can't be changed so easily. I use uki image with kernel and initrd, signed for uefi secure boot, then /usr on dm-verity partition tied to the kernel command line. Then tpm backed rootfs is created on first boot, with machine specific key.
<Jones42>
So that all explodes when I want to get my u-boot script with the roothash into my fitimage, when the fitimage is already built before my rootfs is dond
<Jones42>
*done
<mcfrisk>
sounds quite similar to my problems with wic image type
<mcfrisk>
also had to generate the roothash and then feed it back to initrd cmdline
<Jones42>
the wic image was no problem for me, since that one was built as very last step. By then I have the fitimage with the roothash as part of the u-boot script and the rootfs.ext4.verity that I can just rawcopy like in that example
<Jones42>
mcfrisk: ah, I was about to ask how you install packages into a seperate partition. rootfs_usr_strip explains that :-)
<Jones42>
I'm in this secure-boot-rabbithole for a few months now, and I'm surprised how complicated those things still are and how much manual scripting still is required
<mcfrisk>
many of these pieces are almost(tm) ready for upstreaming, but some reference builds are needed to make sure they keep working. secure boot should not be this hard...
alessioigor has quit [Remote host closed the connection]
alessioigor has joined #yocto
sakoman has quit [Ping timeout: 264 seconds]
mckoan has quit [Ping timeout: 256 seconds]
mckoan has joined #yocto
sakoman has joined #yocto
Guest348 has joined #yocto
berton has joined #yocto
ehussain has quit [Remote host closed the connection]
ahussain has joined #yocto
ahussain is now known as ehussain
Guest39 has joined #yocto
Daaanct12 has quit [Quit: WeeChat 4.3.3]
PIBA has joined #yocto
goliath has joined #yocto
<PIBA>
has anyone suggestions for good trainings for yocto beginner in english or german ?
<Jones42>
PIBA: german will be tough, but I can share some of my bookmarks
<Guest0>
Hello, I have a more general question: if you have a yocto based os that is build for multiple devices (x86 and aarch64) + custom applications, how can one run the automated tests for those apps for each of the devices?
<rburton>
if you want to run the tests _on target_ then oeqa can do that, although it's not used much. it should still work though... another option would be labgrid or lava.
<Jones42>
or tbot
<Guest0>
do they have anything to do with ptest?
<mcfrisk>
and a lot of board/SoC/variant specific details for automatic flashing, power control, recovery, and then the test framework for tests (or oeqa, possibly exported from bitbake build), and then the framework for managing the whole test farm and infra around it. ptest run as part of oeqa, for example.
<rburton>
Guest0: ptest is a way of packaging up tests inside a package so you can run them on target
<rburton>
so yes if you have a custom app called foo, then the recipe also packaging its own tests would be one way
<Guest0>
thanks for the details, much appreciated!
rber|res has quit [Remote host closed the connection]
rber|res has joined #yocto
florian_kc has joined #yocto
<Guest348>
I have an archive with a pre-built application and I want to have it unpacked and the contents just copied to the image without DNF (or perhaps something else?) trying to guess what the files actually mean.
<Guest348>
Regardless of whether I use bin_package class or copy the files over in do_install, DNF complains, during do_rootfs, that nothing provides some 64bit dependencies, which is probably correct, but it is absolutely out of my power to fix that, since I'm targeting arm and I can't just remove the odd parts of the app and rebuild. I also know for a fact
<Guest348>
that the app works just fine if I just copy the files over.
<Guest348>
Is it possible to inhibit this behavior and just make it copy the files over?
<kanavin_>
Guest348, it helps if you inspect what is in the .spec for the package, and how it gets there. I think there's a way to suppress generating those auto-dependencies, but I need to see what is it specifically.
<kanavin_>
you find the .spec in ${WORKDIR} for the recipe
<rburton>
Guest348: 32-bit arm target?
<Guest348>
It's an i.MX8 so I believe it's 64-bit.
Guest348 has quit [Quit: Client closed]
Guest348 has joined #yocto
<Guest348>
kanavin_ there's a lot of lines like Requires: libc.so.6()(64bit). So basically all of the ones that DNF says "nothing provides" about (libdl.so.2(GLIBC_2.2.5)(64bit) libdl.so.2(GLIBC_2.3.3)(64bit) liblttng-ust.so.0()(64bit) libm.so.6(GLIBC_2.2.5)(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit) librt.so.1(GLIBC_2.2.5)(64bit)) and then some. Now for how
<Guest348>
it got there, my best guess is that it is automatically extracted somehow from the files I want to copy over. Since if I replace the archive with a different one that only contains text files, it copies the files to the image without any issues.
<kanavin_>
Guest348, do_package does this for shared libraries. I don't remember off the top of my head how to suppress this, but there is a way.
<kanavin_>
and it's not dnf-specifc, you would get the same errors with opkg or apt
<Jones42>
uh, so I tried to "deltask/addtask" in a bbappend to change the task order that's set in the bbclass, but it seems as if the bbclass' addtask was executed *after* my bbappend, so my deltask is not working as intended. are there any other ideas or do I really have to create my own bbclass?
Guest348 has quit [Quit: Client closed]
Saur_Home has joined #yocto
frieder has quit [Remote host closed the connection]
alessioigor has joined #yocto
Guest348 has joined #yocto
<rburton>
Guest348: try setting EXCLUDE_FROM_SHLIBS="1" in the recipe
<rburton>
that should stop the shared library depends being extracted at all
<rburton>
you'll be entirely responsible for ensuring depends are installed, obviously
<Guest348>
kanavin_ Great tip. The do_package and do_packagedata info in ${WORKDIR}/pkgdata/runtime and ${WORKDIR}/pkgdata/runtime-reverse contains files with FILERDEPENDS:path_to_a_file: a bunch of space separated packages like libpthread.so.0(GLIBC_2.17)(64bit) and also a bunch of FILERPROVIDES which I also probably don't want. Now I just need to figure out
<Guest348>
the next step. Thanks.
<rburton>
the issue is you've got precompiled libraries that link to specific library versions and they're either not in the sysroot when your recipe builds (ie solve that by DEPENDing on lttng-ust) or the versions don't quite match with what we're shipping.
<Guest348>
rburton setting EXCLUDE_FROM_SHLIBS="1" has no effect. I did a cleanall and ran bitbake again. I'm not sure if it's related, but the ${WORKDIR}/pkgdata/shlibs2 directory was already empty even before adding this setting.
PIBA has quit [Quit: Client closed]
Saur_Home has quit [Quit: Client closed]
Saur_Home has joined #yocto
<Guest348>
kanavin_ I found a way to supress the behavior I added
<Guest348>
python package_do_filedeps() {
<Guest348>
print('SKIPPING')
<Guest348>
}
<Guest348>
to my recipe. Is this something that looks like the right way to do that or did I do something dumb? The image builds and the files are there.
<rburton>
Guest348: setting SKIP_FILEDEPS="1" would do the same thing
<rburton>
(package_do_filedeps() calls oe.package.process_filedeps() and the first line of that is to return early if that is set)
<rburton>
but yes that's the function i meant when i was skimming package.py
<Guest348>
I'll have to try that. The manual says SKIP_FILEDEPS is "…removal of all files from the “Provides” section of an RPM package…" and I clearly want the depends. How can I easily find the source for this that you mention?
<yocton>
RP: sorry about the line wrapped patch :( we recently had to change our git-sendemail config. We'll look a it and resend
<RP>
yocton: np, it happens
<RP>
yocton: I'm happy to have the fixes!
altru has joined #yocto
<rburton>
yocton: if smtp continues to be a pain, b4 send works well
<RP>
v2 is applied, thanks!
<yocton>
rburton: does it integrate with Google Oauth stuff?
<rburton>
yocton: it does something like sending to lore which then forwards
<yocton>
rburton: Oooohhhh sounds lovely
Guest348 has quit [Quit: Client closed]
Guest0 has quit [Quit: Client closed]
altru has quit [Ping timeout: 250 seconds]
jmiehe has joined #yocto
jmiehe has quit [Quit: jmiehe]
beroset has joined #yocto
<beroset>
I've submitted an upstream patch to cracklib that should allow us to eliminate the currently maintained yocto patch for cracklib: https://github.com/cracklib/cracklib/pull/86
<beroset>
The patch uses various htole* functions, but the original 2015 patch in yocto says "We can't use the endian.h, htobe* and be*toh functions because they are not available on older versions of glibc, such as that found in RHEL 5.9."
<beroset>
Is that still a concern?
xmn has joined #yocto
<RP>
beroset: I suspect not, thankfully!
mbulut has quit [Ping timeout: 268 seconds]
<beroset>
I was hoping that was the case! The idea was to fix a problem, not to cause an additional one.
alessioigor has quit [Remote host closed the connection]
alessioigor has joined #yocto
alessioigor has quit [Remote host closed the connection]
<RP>
beroset: thankfully we have moved on since then!
<RP>
beroset: it is nice to reduce the number of patches we're carrying
<beroset>
That was my intent. Is there a list of such patches I could consult? There may be more I can do to upstream them.
<RP>
beroset: you grep for Upstream-Status: on OE-Core to get an idea of the status of them
<beroset>
RP: thanks! I'll try that.
florian has quit [Quit: Ex-Chat]
florian_kc has quit [Ping timeout: 256 seconds]
<RP>
abelloni: the reproducibility issues look like some kind of world build issue with virtual/ PROVIDES recipes
<RP>
abelloni: what I don't understand is what changed :(
zpfvo has quit [Remote host closed the connection]
<khem>
RP: seeing https://snips.sh/f/G-UyWJwYxG quite often these days when cancelling ( Ctrl+c ) a bitbake invocation the sad part is it persists on subsequent runs unless -ccleansstate is run for the failing recipe and it usually runs into next recipe which was also perhaps building something when CTRTL+c was used and list goes on. Practically deleting tmp is the reasonable solution to make progress
<khem>
which causes recipes from devtool workspace to rebuild even nothing changed for them they are not used from sstate for whatever reason
<RP>
khem: always glibc-locale or various recipes?
<khem>
it could be any recipe
<RP>
khem: it sounds like cleanup is being done outside pseudo and the pseudo db isn't updated :(
<khem>
yeah I was suspecting that pseudo is being bypassed somehow, but dont see particularly how will that happen. The build is running inside a container
<RP>
khem: it is more that files will be deleted when pseudo isn't preloaded into memory
<khem>
I have seen this on native system too, so I think container is not in picture here
<RP>
khem: there are two possible improvements: a) when pseudo loads, iterate the DB and delete any entries not present on disk, or b) when it hits this corner case, check if something exists on disk and if not, delete the entry
<RP>
fray: thoughts?
<mckoan>
hello, I'm not an expert at kernel 'audit'. I am trying to enable auditd in my minimal image but I get errors with auditctl. Yocto kirkstone
<khem>
yeah, IMO a robust cleanup would go long way, because I have seen this being reported by few devs as well
<RP>
khem: it would be slow and not necessarily catch all issues as there are genuine ways this can be a real problem
<mckoan>
root@qemuarm:~# auditctl -w /etc/passwd -p wa -k user_identity
<mckoan>
Error sending add rule data request (Invalid argument)
<RP>
khem: perhaps a cleanup command...
<mckoan>
Question: do I need SElinux to use auditd or it should work as is?
ehussain has quit [Quit: ehussain]
rob_w_ has quit [Ping timeout: 240 seconds]
<khem>
RP: yeah even if its a cmd that will help a lot
<khem>
far better than deleting tmp and rebuilding browser from devtool workspace
vthor has quit [Ping timeout: 268 seconds]
rob_w has quit [Quit: Leaving]
<RP>
khem: someone just needs to write it. Feel free to file a bug about needing such a tool
rfuentess has quit [Remote host closed the connection]
vthor has joined #yocto
vthor has quit [Changing host]
vthor has joined #yocto
mckoan is now known as mckoan|away
leon-anavi has quit [Remote host closed the connection]
Saur_Home has quit [Quit: Client closed]
Saur_Home has joined #yocto
florian_kc has joined #yocto
florian_kc has quit [Ping timeout: 255 seconds]
leon-anavi has joined #yocto
leon-anavi has quit [Remote host closed the connection]
florian_kc has joined #yocto
berton has quit [Quit: Connection closed for inactivity]
PaowZ has quit [Ping timeout: 240 seconds]
xmn has quit [Ping timeout: 246 seconds]
Jones42 has quit [Ping timeout: 246 seconds]
RyanEatmon has quit [Remote host closed the connection]
RyanEatmon has joined #yocto
<vvn>
hi there -- in order to provide sstate and downloads mirrors to developers built by the CI, do you guys sync DL_DIR/SSTATE_DIR after a successful CI build, or do you setup the CI to use the mirrors directly as its DL_DIR/SSTATE_DIR?
<RP>
vvn: more the latter
alessioigor has joined #yocto
<vvn>
RP: is the latter robust in case of failing builds? e.g. if you have broken recipes, you'll end up polutting the mirrors?
alessioigor has quit [Remote host closed the connection]
<RP>
vvn: the hashes will likely be different so the pollution generally doesn't matter
<vvn>
ok, it may just result in wasting a bit of space, but nothing worrying I presume
<RP>
vvn: we're not worrying about it
<vvn>
I guess with the former method I can use PREMIRRORS and SSTATE_MIRRORS in my site.conf for both the CI and the developers
rber|res has quit [Read error: Connection reset by peer]
Saur_Home2 has joined #yocto
Saur_Home has quit [Ping timeout: 250 seconds]
Jones42 has joined #yocto
<kanavin_>
vvn, syncing separately means there's a window of time where latest sstate isn't available, and developers will face long local builds
Jones42 has quit [Ping timeout: 255 seconds]
<vvn>
that's a good point, the sync time isn't negligible
<kanavin_>
vvn, the good news is, you can write to DL_DIR and SSTATE_DIR directly, and share that as a r/o http thing
<vvn>
kanavin_: can you have PREMIRRORS/DL_DIR and SSTATE_MIRRORS/SSTATE_DIR pointing to the same content respectively? (wondering if the CI and developer can share the same site configuration)
<vvn>
it's a bit pointless to have PREMIRRORS and SSTATE_MIRRORS defined for the CI, but it would allows to have this configuration statically defined in site.conf for the company
<kanavin_>
vvn, that I don't know
<kanavin_>
but I would say yes
<kanavin_>
beroset, yeah I got notified by email. Is there a version update and a patch we can drop associated with that?
<RP>
khem: I did look through our gdb patches and was tempted to drop them all and see what still breaks
<beroset>
kanavin_: yes to both. The new version of cracklibhasn't been released yet, but is planned to be 2.10. I'm not sure how to point to the patch, but I think it's the only one currently associated with cracklib.
<beroset>
Unrelated to that, I'm moving a project from mickledore to scarthgap. Do I need to delete my tmp-glibc directory before rebuilding? Or is there a smarter way to do such an upgrade?
<kanavin_>
beroset, in theory that isn't necessary. but in practice bitbake will more or less delete all recipe workdirs anyway due to having to rebuild them
<beroset>
kanavin_, thanks!
mbulut has joined #yocto
alessioigor has joined #yocto
Kubu_work has quit [Quit: Leaving.]
alessioigor has quit [Remote host closed the connection]
alessioigor has joined #yocto
mvlad has quit [Remote host closed the connection]
alessioigor has quit [Remote host closed the connection]
tangofoxtrot has quit [Remote host closed the connection]