bob_x1 has quit [Remote host closed the connection]
bob_x1 has joined ##raspberrypi-internals
dolphinana has joined ##raspberrypi-internals
dolphinana has quit [Ping timeout: 264 seconds]
<clever>
nice!
<clever>
G33KatWork: i also had an idea on how to confirm the watchdog and 2nd core stuff, bootstrap just core0 as before, but then core0 will write to the watchdog registers some more and sev, then core1 can blink an led
<f_>
clever: armstub is based on tf-a..
<f_>
funny. Am reversing a proprietary bl2 based on thay too
<clever>
f_: they havent released the source/changes yet, so its kinda the same
<clever>
previously, every armstub has been under 1kb, and in pure asm
<f_>
so we only have the binary?
<clever>
and they are all on github
<clever>
correct
<clever>
i expect them to release something eventually
<f_>
I could reverse it if they don't
<clever>
let me see where i left it...
<f_>
Fairly familiar with TF-A code
<f_>
and I can compare with upstream
<clever>
ah, not extracted yet
<clever>
one sec
<f_>
Does it log stuff to uart?
<clever>
it has lots of printf strings
<clever>
the filename is armstub8-2712.bin
<clever>
it starts with the byte sequence 00 04 00 14
<f_>
strings armstub8-2712.bin ?
<clever>
i dont have it as its own file yet
<clever>
its embedded within bootmain.elf
<clever>
ah, found it in ghidra, its 28764 bytes long
<f_>
then grep Built
<clever>
Built : 14:26:57, Jun 22 2023
<f_>
indeed tf-a
<clever>
the armstub is at 3fc6_5b3c within bootmain.elf
<f_>
grep BL31
<clever>
yeah, i already decompiled a number of functions, and matched them up
<clever>
this BL31 just needs to setup to catch PSCI calls, and then execute the kernel in EL2
<clever>
the firmware checks for the magic at 0xf0, then fills in the other 2 blanks, before releasing arm reset
<clever>
and this BL31 then acts as some glue, between PSCI and the mailbox stuff
<f_>
haha
<f_>
It's a mess then
<f_>
BL31 is supposed to invoke a *bootloader*, not linux
<f_>
usually..
<clever>
except, the bootloader has already finished its job, before BL31 even started
<f_>
but then...BL33 isn't ran at EL2..
<f_>
It's ran in the non-secure world..
<clever>
and switching things to the standard, would then mean all of the old rpi stuff doesnt work
<f_>
BL33 is the non-trusted bootloader.
<f_>
clever: sure
<clever>
there is a lot of undocumented changes to control registers, that has to align with the device-tree
<f_>
but why is the kernel ran at EL2 in the first place?
<clever>
to allow kvm
<f_>
Looks like a hacky workaround to me.
<f_>
IIRC it's BL32 who's supposed to be ran at el2..
<clever>
a linux without kvm support will just drop itself down to EL1
<clever>
with kvm support, it sets up a proxy, and then drops itself to EL1
<clever>
and that proxy allows linux to make use of EL2 features
<f_>
hmm ok
<clever>
EL2 lacks a TTBR0/TTBR1 split
<clever>
so linux cant run directly in EL2
<clever>
but there is an arm extension that fixes that
<clever>
it lets you stay in EL2, and aliases a bunch of EL1 regs, so linux can act like its in EL1
<clever>
then you dont have to EL1->EL2->EL1 bounc every time you context switch
<clever>
i briefly went over the psci dispatch function, and made note of every thing it supports
<clever>
but havent fully gone over the binary
<clever>
f_: under normal conditions, i would expect BL31 to talk to the dram controller and protect itself, would you be able to find that area of the code?
<f_>
I doubt bl31 is supposed to do that
<f_>
but let's see..
<clever>
i feel like something in EL3 is responsible for that
<clever>
to ensure nothing untrusted can make its way into EL3
<f_>
Could be in MMU init
<f_>
Funnily enough, docs/design/firmware-design.rst says that BL1, BL2, BL31 all enable MMU
<clever>
but my understanding, is that EL1's mmu maps virtual to real physical
<clever>
(when no hypervisor is at play)
<clever>
and EL3 cant change that
<clever>
when the hypervisor is at play, its nested paging tables, where EL1's mmu maps virtual to guest-physical and then EL2 maps guest-physical to real-physical
<clever>
and EL3 is protected, by flagging a region of ram as secure only
<f_>
yup
<f_>
causing a kernel panic when trying to read from that secure region
<clever>
f_: and if there is a standard place to configure that within BL31, we would discover if the 2712 supports that
<Tiersten>
is swd blocked from accessing the secure region?
<clever>
Tiersten: in a truely secure system, yeah, but it tends to be a security flag, and then somebody with the keys could disable it
<Tiersten>
was wondering about the pi 5
<Tiersten>
in its stock config
<clever>
yeah, that is a good question
<clever>
on past models (pi0-pi3), it wasnt locked down
<clever>
so you could jtag secure mode just fine
<f_>
clever: almost done with the website btw
<clever>
nice
<f_>
damn
<f_>
Can't remember which binaries to use for what
<f_>
ok..
<f_>
vc4-stage1 lk.bin as bootcode.bin..
<f_>
and rpiX-test lk.elf...
<f_>
clever: correct me if I'm wrong.
<clever>
vc4-stage1 lk.bin is bootcode.bin
<clever>
vc4-stage2 lk.elf is /boot/lk.elf
<f_>
vc4-stage2?
<f_>
Thought it was rpiX-test..?
<clever>
rpiX-test lk.bin gets baked into vc4-stage2 at compile time
<f_>
ahhhh yes
<f_>
I remember now
<f_>
Thanks
<f_>
tbh vc4-stage2 is confusingly-named..
<f_>
since it runs on the arm cpu, not the vc4 IIRC?
<f_>
They also reverse-engineered Amlogic S905 BL31!
<clever>
nice
<f_>
then someone else REd S905X BL31
<f_>
then someone else REd G12 BL31
<f_>
speaking of amlogic, have a really nice gxl/S905X board looking very much like a pi3
<f_>
(except the header bases are coloured and the pcb is white)
<f_>
That's where I'm reverse-engineering Amlogic S905X BL2 now..previously REd S905 BL2 on a cheap aliexpress set-top box
<f_>
porting U-Boot SPL
<f_>
actually learnt quite a bit about how to u-boot spl and u-boot
<f_>
(could be useful for rpi)
<clever>
under normal conditions, SPL is loaded by the rom, and has to do dram init
<clever>
and then SPL loads u-boot proper
<clever>
but in the rpi, bootcode.bin/vc4-stage1 is acting as the SPL
<f_>
except spl is supposed to be loaded on the arm cpu
<clever>
yeah
<clever>
rpi-open-firmware bootcode.bin is 2 binaries in one
<clever>
the arm payload is in the .rodata section of the bootcode.bin
<clever>
and there is no vc4-stage2
<clever>
but i had stability problems with it
<clever>
changing one thing breaks everything
<clever>
and there isnt much room left within the 128kb limit
<f_>
but on my hacked-up u-boot tree SPL is loaded by the ROM, does dram init, pll init and power init, loads and runs the scp firmware, BL31 and BL33/U-Boot
<clever>
yeah, thats much more "proper", according to the arm docs/specs
<clever>
another way you can think of it, look at the intel ME or the amd PSP
<clever>
thats basically the role the VPU has
<clever>
i need to go check the mail, i'll be back within an hour or 2
dolphinana has joined ##raspberrypi-internals
<f_>
gonna push
<clever>
back
<f_>
I'll push to the website repo. Want to see a screenshow?
<clever>
sure
<f_>
*screenshot
<G33KatWork>
clever: the rp1 rom has read errors btw. I found that if I read 64 bytes in once go via i2c, sometimes random bits flip
<G33KatWork>
I have to do another dump and read in 4 byte chunks
<G33KatWork>
I saw that happening yesterday when staring at register regions and I just found a corrupted opcode where the uppermost bit was flipped to a 1. luckily the upper mirrored half of the rom had the correct instruction
<f_>
the 'List of pages' button opens a 'window' inside the webpage BTW.
<clever>
G33KatWork: oh, that would explain why chunks cant disassembly cleanly!
<G33KatWork>
yes. I am trying to get the second core running, but poking the right watchdog regs and executing sev on the first core doesn't do anything
<G33KatWork>
I'll send you the new binary
jcea has joined ##raspberrypi-internals
<clever>
the bit errors, make me worry about the stability of the bootstrap process, and question how the clocks differ
<clever>
are you clocking the i2c too fast?
<clever>
is the read routine meant for only 4 bytes?
<G33KatWork>
yes, I should benchmark that and dump a section of known memory repeatedly and compare that
<clever>
might the real bootstrap have the same bit flips?
<G33KatWork>
I am not clocking anything, it's all default settings coming from the device tree
<clever>
but that node was disabled, and it may have undergone less testing
<G33KatWork>
yes, 4 bytes seem to work though
<G33KatWork>
as soon as I did that, the register dumps stopped looking wonky
<G33KatWork>
and I hammered that thing when brute forcing the rests
<G33KatWork>
didn't get any false positives due to read errors
<G33KatWork>
sent you the new binary
<G33KatWork>
I also just dumped 32K starting from 0x8000 now
<clever>
got it
<clever>
oh!
<clever>
that might explain why i saw some jumps to the 0x8000 alias
<G33KatWork>
yep
<clever>
maybe that was a flip in the addr, that just happened to land in the alias!
<G33KatWork>
it's always the uppermost bit that flips
<G33KatWork>
ha! core is running
<clever>
ghidra is far more happy now
<clever>
it decompiled the entire .text in one shot with zero errors
<G33KatWork>
ncie
<G33KatWork>
btw, I load my device tree overlays on boot now to disable the whole rp1 and expose the i2c and gpio pin
<G33KatWork>
wifi still works
<clever>
perfect
<clever>
and you have the 2nd core running?
<G33KatWork>
okay, I needed to clear another reset
<G33KatWork>
yes
<clever>
nice
<G33KatWork>
now to figure out which reset that is
<clever>
there is still a single sev in the entire binary, even after correcting for the bitflips
<G33KatWork>
yeah, that's the boot path of the second core
<G33KatWork>
you need to set the other two watchdog regs with the PC and SP
<G33KatWork>
PC xored with 0x4FF83F2D
<clever>
oh right, but i'm looking at the wrong binary
<G33KatWork>
which is 0xB007C0DE ^ 0xfffffff3
<G33KatWork>
then issue an sev
<clever>
aha, i was wondering what that magic was
<clever>
ah, there is what i was looking for!, the rp1.bin (from bootmain.elf) has 2 sev's
<clever>
one in the main entry point
<clever>
and one in a random part of the code
<clever>
kind of feels like a "im done setting all resets" area
<clever>
core0 is done bringing up the system, and now core1 can safely start
<f_>
clever: how does the website look like? does it look good?
<clever>
f_: i think it looks good
<f_>
nice
<f_>
I'll push to the repo tomorrow
<clever>
the bootmain.elf rp1.bin, never touches watchdog registers
<clever>
so i think, you could push 2 payloads, the official rp1.bin and your custom one
<clever>
and set core0 to run rp1.bin, and core1 custom, in the watchdog regs
<clever>
and when rp1.bin has booted up, it will sev and execute your custom code
<clever>
G33KatWork: have you had any luck executing rp1.bin?
<G33KatWork>
haven't tried again
<G33KatWork>
and yes, we should be able to have the original firmware running and just use core 2 by poking the right registers. we still have to issue an sev somehow
<clever>
which is why i checked rp1.bin for sev
<clever>
and it seems to fire one after its done booting
<G33KatWork>
oh, that's cool
<clever>
only problem is, you have to reset the whole rp1, to do that again, with your extra payload
<clever>
and linux is going to be upset
<G33KatWork>
I am going to try and reload the firmware and then attach the whole rp1 again using a device tree overlay
<clever>
i also see an 0x43 at 0x520 into the rom
<clever>
that is the only 0x43 i can find in the operands, and i suspect its the i2c setup code
<G33KatWork>
bit 31 in 0x40014000 is the reset bit for the 2nd core
<G33KatWork>
either that or some pll that supplies it
<G33KatWork>
either way, clearing it makes the core go brrr
<clever>
several functions in the rom write to that reg
<G33KatWork>
yes, it's the first reset register
<G33KatWork>
there are 2 more at offset 0x4 and 0x8
<clever>
the rom seems to never use the one at 8
<G33KatWork>
the ROM must also configure some clocking. and the clocks for both cores are different. the second core blinks slower than the first one
<clever>
interesting
<G33KatWork>
I only found four bits there: usbhost0/1, vbusctrl and VIDEO_OUT_DPI
<clever>
is the 1st core doing anything?
<clever>
wfe or busy loop?
<G33KatWork>
busy spinning in a while(1)
<clever>
that might be bus contention
<G33KatWork>
right, code is in sram
<clever>
core0 keeps winning the fight over the bus
<clever>
spin on wfe, and see if core1 speeds up
<G33KatWork>
you are right
<G33KatWork>
it's bus contention
<G33KatWork>
a wfe makes it go faster
<clever>
it likely has priorities, like the rp2040
<clever>
heres a crazy idea, what about non word aligned reads
<clever>
what happens if you read from 0 to 8, each byte?
<clever>
are registers 0, 1, 2, ... each special?
<G33KatWork>
reading the rom?
<clever>
yeah
<clever>
i see a few switch-case blocks, that handle 0-7
<G33KatWork>
I get zeros back
<clever>
ah dang
<clever>
any clue which peripheral is i2c-slave capable?
<G33KatWork>
I2C5 is the interface they are using
<G33KatWork>
they are also configuring SPI8
<G33KatWork>
in the bootrom
<clever>
ah
<G33KatWork>
the rp1 firmware resets both of them
<clever>
that implies it may be able to boot from spi
<clever>
which is good news, for an x86 rp1 card
<G33KatWork>
and I found that both are used by the ROM because for SPI8 registers were funny until reset
<G33KatWork>
and resetting I2C5 killed comms
<clever>
heh
<clever>
cutting the branch your standing on!
<G33KatWork>
same for all the IO_BANK and PAD peripherals
<clever>
it looks like 0x10002130, is a variable, that says which i2c controller to use
<clever>
and 0x10002000, is an array of i2c controller configs, including the base addr
<clever>
and i2c5, is the 0th entry in that array
<G33KatWork>
maybe they have strapping pins
<clever>
nothing seems to write to that variable
<clever>
maybe dead code?
<clever>
one fo the first things it does, is overwrite the 1st int32 in the array, with 0x43 (i2c slave addr)
<clever>
but this is .data, and could have just been initialized at compile time!
<clever>
this switch-case definitely looks like a state machine
<clever>
let me map out the states....
<G33KatWork>
okay, I just loaded the original firmware
<G33KatWork>
then loaded a dtbo which enables the rp1 node again
<G33KatWork>
and I got a pcie- and ethernet-link :)
<clever>
just gives a chaotic tangle of nodes
<clever>
nice!
<clever>
now try loading 2 binaries at once, official and custom
<clever>
if that works, then we have a way to co-exist, but can only do it once per boot
<clever>
2 different paths in the bootrom state machine, can configure the watchdog, to begin executing at some pc, with a defined stack
<clever>
but its not clear how that pc gets set
<clever>
0x4cc in the rom, if your wondering where i am
<G33KatWork>
oh yeah, that's the monster function I called main
<clever>
oh right, the rp1 has otp!!, and i just found a routine accessin git!
<G33KatWork>
afk, eating...
<clever>
kk
<clever>
oh!!
<clever>
that 2130 variable, a pointer to it, is passed to an otp routine!
<clever>
so, you can configure which i2c controller it boots from, via i2c
<clever>
via otp*
<clever>
dang typos!
<clever>
too many tla's in my head
<clever>
and same for spi as well
<clever>
some of the states in 4cc (main) are i2c based, and some are spi based