ArmbianHelper changed the topic of #armbian to: armbian - Linux for ARM development boards | www.armbian.com | Github: github.com/armbian | Commits: #armbian-commits | Developer talk: #armbian-devel | Forum/Twitter feed: #armbian-rss | Logs: -> irc.armbian.com
<Manouchehri> stipa: that makes sense, since the r8169 will never be seen if the PCI bridge isn't working
<Manouchehri> alright 1500ms failed, adding an insane 5000ms as a test
<stipa> what about USB ports?
<stipa> seems like they're on that bus too
<Manouchehri> pretty sure the USB ports aren't on the PCI bus
<stipa> whre would they be?
<stipa> the pcix is the fastest thing that could host usb3 on that chip
<Manouchehri> stipa: the USB controller is on the SoC itself
<Manouchehri> stipa: yeah so even with a 5 second delay, still doesn't work without modprobe -r and modprobe
<stipa> dunno, try my link
<ArmbianHelper> ^ What is the Linux built-in driver load order? - Stack Overflow
<stipa> yeah
<stipa> first time there's an error [ 4.793347] rockchip-pcie: probe of f8000000.pcie failed with error -110
<stipa> that's what you're trying to fix with your timer thing i guess
<stipa> so, idk, pci could depend on something that loads later
<stipa> after 4.7 secs
<ArmbianHelper> ^ gist:98cccfb749992f99fbb32102be2fa735 · GitHub
<stipa> you could maybe rearrange stuff
<stipa> put pci to initialise after boot of armbian
<stipa> for example since nothing depends on it
<stipa> since armbian loads fine
<stipa> if possible
<stipa> if there's some driver to load it's not even possible to initialise pci if the kernel isn't runnign i guess
<stipa> so, you're expecting to load pci driver before armbian starts to boot
<stipa> after the error [ 4.793347] rockchip-pcie: probe of f8000000.pcie failed with error -110 it looks to me like some ubuntu is being booted
<stipa> [ 11.087483] systemd[1]: Created slice Slice /system/modprobe.
<stipa> maybe that has something to do with drivers
<stipa> that's long after 4.7 secs
<Manouchehri> yeah ubuntu jammy is being booted
<stipa> at 4.7 pci was expecting maybe some driver or something
<stipa> from a linux that's not there
<stipa> and gave up on it
<Manouchehri> stipa: this is where the failure is, I don't see how it's expecting another driver. https://github.com/torvalds/linux/blob/master/drivers/pci/controller/pcie-rockchip-host.c#L330-L333
<ArmbianHelper> ^ linux/pcie-rockchip-host.c at master · torvalds/linux · GitHub
<stipa> it looks like it could be on a bootloader level
<Manouchehri> huh
<stipa> uboot
<Manouchehri> if it's at the bootloader level, then why does removing and probing again work?
<stipa> because ubuntu loads after the pci issue
<stipa> the uboot loads it
<Manouchehri> huh?
<stipa> from the sd card
<Manouchehri> the linux kernel has clearly loaded here, no........? https://gist.github.com/Manouchehri/182e865a76fbe04e1c3c0c17befb8236#file-gistfile1-txt-L2
<ArmbianHelper> ^ gist:182e865a76fbe04e1c3c0c17befb8236 · GitHub
<stipa> [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
<stipa> that's not a message linux is showing
<stipa> that's uboot
<Manouchehri> which line is linux starting...?
norwich_ has joined #armbian
<stipa> uboot is similar to bios
<stipa> so, first it detects hardware
<stipa> then it loads
norwich has quit [Ping timeout: 260 seconds]
norwich_ is now known as norwich
<stipa> and tell to linux where hardware can be located and what it found
<Manouchehri> k, but you're saying that my dmesg log is actually uboot and not linux
<stipa> so, it detects hardware and loads linux
<stipa> right, some of the first messages are uboot
<Manouchehri> and which is the first message after uboot is done?
<stipa> is that what you see on terminal during boot?
<Manouchehri> this is 100% from dmesg
<stipa> Manouchehri: i guess it starts to load after the pci error
<stipa> so, you have two problems actually, uboot and linux
<stipa> why uboot failed
<stipa> and linux can start the pcix
<Manouchehri> huh, where do you see uboot failing?
<stipa> that's a guess
<stipa> that's to research a bit more
<Manouchehri> How is that a guess though? Like... I've clearly booted up.
<stipa> i could be completely wrong
<Manouchehri> u-boot doesn't stay running like EFI services, does it....?
<stipa> it failed to start pcie
<stipa> the memory is not on the pcie bus
<Manouchehri> okay but like
<Manouchehri> if the memory wasn't there, why would waiting a bit and then loading the module work?
<Manouchehri> uboot isn't running anymore between the first and my attempt
<stipa> i duuno, maybe the pci depends on some linux driver
<Manouchehri> I do appreciate the help... but are you wildly guessing? 😅
<stipa> whole time, yea
<Manouchehri> oh
<stipa> what are your guesses?
<stipa> the question is if your tweak even reaches the pci hardware
<stipa> is that something in pci chip or?
<stipa> some linux script that never works
<Manouchehri> sorry, I don't really get what you're saying
<stipa> the time you're moving around
<stipa> you're increasing the amount of time
<stipa> can you feel the boot is longer?
<Manouchehri> sure?
<Manouchehri> maybe?
<ArmbianHelper> ^ gist:710a2250b2ab4edab1c25da63a883166 · GitHub
<Manouchehri> I actually managed to get the "PCIe link training gen1 timeout!" error twice
Masterphi has quit [*.net *.split]
pablocastellanos has quit [*.net *.split]
pablocastellanos has joined #armbian
DarkL0rd has quit [Remote host closed the connection]
Unit193 has quit [Read error: Connection reset by peer]
Unit193 has joined #armbian
indy has joined #armbian
lyri has quit [Remote host closed the connection]
DarkL0rd has joined #armbian
DarkL0rd has quit [Ping timeout: 260 seconds]
DarkL0rd has joined #armbian
DarkL0rd has quit [Ping timeout: 260 seconds]
<Armbian-Discord> <Z​anoryt> Armbian-related in that it’s the base for this new SDR based OS. https://twitter.com/airframesio/status/1582980469330653185?s=46&t=0ejpRmqpsi88ExVM658IEA
<ArmbianHelper> ^ Time for a (hopefully) last poll about the OS.Previous great ideas were not Google-able, had existing OS name, or domain was not avail. I think we have it now. @ADSBexchange discord resounded w/ AirwavesOS.What do YOU think?Please RT!#adsb #acars #Avgeeks #SDR #airframes - Airframes (@AirframesIO) October 20, 2022
archetech has joined #armbian
CosmicDJ has quit [Quit: So that's all DJ, the time has come.]
willmore has quit [Remote host closed the connection]
ajfriesen has quit [Ping timeout: 264 seconds]
ajfriesen has joined #armbian
indy has quit [Ping timeout: 246 seconds]
indy has joined #armbian
haritz has quit [Remote host closed the connection]
haritz has joined #armbian
norwich has quit [Ping timeout: 272 seconds]
Unit193 is now known as Montresor
lemonzest has joined #armbian
DarkL0rd has joined #armbian
DarkL0rd has quit [Ping timeout: 255 seconds]
lyri has joined #armbian
willmore has joined #armbian
<Manouchehri> [TheBug]: so I currently believe problem is that the kernel module itself is buggy (race condition is my guess). I’m going to do a hacky workaround by writing a systemd service to reload the module if it fails to load for now.
<ArmbianHelper> ^ PCI: rockchip: Fix timeout in rockchip_pcie_host_init_port() - Patchwork
archetech has quit [Quit: Konversation terminated!]
archetech has joined #armbian
<Manouchehri> After reading the kernel driver code a bit more, I realized the timeout error will appear even if there’s a training error unrelated to timing out.
The_Loko has joined #armbian
ajfriesen6 has joined #armbian
ajfriesen has quit [Ping timeout: 260 seconds]
ajfriesen6 is now known as ajfriesen
LanDi has joined #armbian
norwich has joined #armbian
alekksander has joined #armbian
heartburn has quit [Ping timeout: 240 seconds]
LanDi has quit [Remote host closed the connection]
heartburn has joined #armbian
indy has quit [Ping timeout: 246 seconds]
indy has joined #armbian
alekksander has quit [Quit: Konversation terminated!]
<stipa> Manouchehri: is there any other SBC with the same soc and nic?
<stipa> maybe having a working reference could help
<stipa> at least a working code if not a board
<Manouchehri> stipa: I already solved it. https://github.com/armbian/build/pull/4308
<ArmbianHelper> ^ Add an attempt counter, which helps buggy PCIe links. by Manouchehri · Pull Request #4308 · armbian/build · GitHub
<stipa> right right
<stipa> cool fix, that one is a guess for sure
<IgorPec> Manouchehri great job
<Armbian-Discord> <M​anoftheSea> grats and thanks!
<Manouchehri> IgorPec: I don't get how Rockchip themselves didn't notice and/or fix this loooong before me.
<Armbian-Discord> <I​gorPec> this driver was ported by someone from rockchip legacy kernel
<Armbian-Discord> <c​0rnelius> it doesn't apply to a clean source tree.
<Armbian-Discord> <d​ave> hmm, it applied to 5.19 for me
<Armbian-Discord> <c​0rnelius> after reworking, it will apply to 5.15 thru 6.0. but as it sits it fails.
<Armbian-Discord> <I​gorPec> i am also reworking it 😉
<Armbian-Discord> <d​ave> looks like I'm passing the github runner? https://github.com/armbian/build/actions/runs/3292336624/jobs/5427633950
<ArmbianHelper> ^ Add an attempt counter, which helps buggy PCIe links. · armbian/build@d2e4eaf · GitHub
<Armbian-Discord> <I​gorPec> yes, but armbian has some patche before
<Armbian-Discord> <I​gorPec> and it says [ warn ] * l rk3399-rp64-pcie-retry.patch [ failed ]
<Armbian-Discord> <I​gorPec> CI does not stop on patch warnings
<Armbian-Discord> <c​0rnelius> just letting you know is all. https://paste.debian.net/1257771/
<ArmbianHelper> ^ debian Pastezone
<Armbian-Discord> <d​ave> oh
<Armbian-Discord> <d​ave> you're right
<Armbian-Discord> <d​ave> Are you currently working on it? Just need it for some of my testing 😛
<Armbian-Discord> <I​gorPec> perhaps this https://paste.armbian.com/ilifaroxut.yaml
<ArmbianHelper> ^ hastebin
<stipa> Manouchehri: because they don't care
<Armbian-Discord> <I​gorPec> this is for 5.15.y
<stipa> Manouchehri: they sell socs claiming they work and that's enough for them, fixing bugs is no profit for them
<Armbian-Discord> <I​gorPec> applies for 5.19.y too
<Armbian-Discord> <d​ave> mind pushing that to my branch?
<Armbian-Discord> <d​ave> doesn't seem to apply nicely for me. dave@client:~/build$ git apply meh.patch meh.patch:49: trailing whitespace. error: drivers/pci/controller/pcie-rockchip-host.c: No such file or directory
<Armbian-Discord> <I​gorPec> i will push up any moment, just doing test build on 5.19.y with few additional switches on kernel config
<Armbian-Discord> <d​ave> Thanks!
<Manouchehri> stipa: the thing is, it really doesn't work ;P
<Manouchehri> having an ethernet port often fail on a device that's geared towards being a router is terrible.
<stipa> Manouchehri: yeah, they ignore it and sales go on
<stipa> i know some devs over here do contact companies but they ignore
<Armbian-Discord> <d​ave> So, what do you folks think of having a KERNEL_HARDEN build flag? Basically applying many of the recommendations here. https://github.com/a13xp0p0v/kconfig-hardened-check
<ArmbianHelper> ^ GitHub - a13xp0p0v/kconfig-hardened-check: A tool for checking the security hardening options of the Linux kernel
<Armbian-Discord> <I​gorPec> @dave done
<Armbian-Discord> <d​ave> ty, rebuilding now
<Armbian-Discord> <d​ave> Like this one? https://github.com/rockchip-linux/kernel
<ArmbianHelper> ^ GitHub - rockchip-linux/kernel: BSP kernel source
<Armbian-Discord> <I​gorPec> yeah, don't look into 😉
<Armbian-Discord> <d​ave> basic question for ya! root=UUID=efd35bda-e273-466b-9f9d-affb8db9358b rootwait rootfstype=f2fs splash=verbose console=ttyS2,1500000 consoleblank=0 loglevel=1 ubootpart=b1f26560-01 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u net.ifnames=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 ^ who is setting cgroup_enable=cpuset and cgroup_enable=memory?
<Armbian-Discord> <I​gorPec> probably boot script
<Armbian-Discord> <d​ave> if test "${docker_optimizations}" = "on"; then setenv bootargs "${bootargs} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1"; fi ah you're bang on
<ArmbianHelper> ^ kernel/pcie-rockchip.c at 9ed2be4b9c001ca8006cb4c72928c09927c44f89 · rockchip-linux/kernel · GitHub
<Armbian-Discord> <I​gorPec> workarouds and bugs comes along too 😉
<Armbian-Discord> <c​0rnelius> my nanopc t4 didn't boot after applying that.
<Armbian-Discord> <d​ave> got console logs?
<Armbian-Discord> <c​0rnelius> currently no.
<Armbian-Discord> <c​0rnelius> i'm being lazy.
<stipa> i'll buy you a ftdi converter
<Armbian-Discord> <c​0rnelius> I always just assumed the timeout had to do with there not being a drive in place. it looks and then times out because it can't find an nvme drive.
<Armbian-Discord> <d​ave> you should still always see the PCI controller itself
<Armbian-Discord> <d​ave> root@nanopi-r4s:~# lspci -nn 00:00.0 PCI bridge [0604]: Rockchip Electronics Co., Ltd RK3399 PCI Express Root Port [1d87:0100]
<Armbian-Discord> <c​0rnelius> there is no nvme port on that device though.
<Armbian-Discord> <d​ave> so?
<Armbian-Discord> <d​ave> the controller is it's own thing
<Armbian-Discord> <c​0rnelius> so what happens when it has one is what I am saying
<Armbian-Discord> <d​ave> doesn't matter if anything is connected to it or not
<Armbian-Discord> <d​ave> if it has one, you'll see 1 more PCIe device
<Armbian-Discord> <d​ave> e.g. in my case: root@nanopi-r4s:~# lspci -nn 00:00.0 PCI bridge [0604]: Rockchip Electronics Co., Ltd RK3399 PCI Express Root Port [1d87:0100] 01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
<Armbian-Discord> <c​0rnelius> it always saw the lspci anyway. I think what is happening is the board is locking up because its now being forced to retry.
<Armbian-Discord> <c​0rnelius> find out in a bit.
<Armbian-Discord> <d​ave> to confirm, you saw: 00:00.0 PCI bridge [0604]: Rockchip Electronics Co., Ltd RK3399 PCI Express Root Port [1d87:0100] but still saw PCIe link training gen1 timeout! at the same time?
<Armbian-Discord> <c​0rnelius> yeah
<Armbian-Discord> <c​0rnelius> I'll confirm is a little bit. making a fresh img for it.
<Armbian-Discord> <d​ave> that's... really odd. You shouldn't have a PCI bridge there if the training failed afaik
<Armbian-Discord> <I​gorPec> i am about to test on hw now
<Armbian-Discord> <I​gorPec> R4S
<Armbian-Discord> <d​ave> I'll wait for you before I reboot my R4S.
<Armbian-Discord> <d​ave> already tested on my R4SE just fine 😛
<Armbian-Discord> <I​gorPec> haha
<Armbian-Discord> <I​gorPec> my device is somewhere in rack
<Armbian-Discord> <I​gorPec> if it fails ... have to dig in deep
<Armbian-Discord> <d​ave> but neither device is local to me. My R4SE is in another country, and my R4S is at my office (which I don't feel like driving to).
<Armbian-Discord> <I​gorPec> ok, mine is not far
<Armbian-Discord> <d​ave> sweet
<Armbian-Discord> <I​gorPec> still not up
<Armbian-Discord> <d​ave> my R4SE took a little bit to start
<Armbian-Discord> <d​ave> if you weren't hitting the timeout before, then the patch shouldn't do anything.
<Armbian-Discord> <I​gorPec> i didn't have 2nd nic on this device
<Armbian-Discord> <I​gorPec> no, its not booting
<Armbian-Discord> <d​ave> got serial or logs of some sort?
<Armbian-Discord> <I​gorPec> don't have anything on that device
<Armbian-Discord> <d​ave> pstore?
<Armbian-Discord> <I​gorPec> i think i am done with this for today
<Armbian-Discord> <d​ave> My R4S booted up fine (but not surprising).
The_Loko has quit [Quit: Leaving]
Malditron has joined #armbian
<Armbian-Discord> <d​ave> Vulnerabilities: Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Vulnerable Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Vulnerable Srbds: Not affected Tsx async
<Armbian-Discord> abort: Not affected Is there just no mitigation for Spectre V2 and Spec store bypass...?
archetech has quit [Quit: Konversation terminated!]