15:01 <marex> maz_: hi, do you happen to have a moment for a question about threaded IRQ handling ?

15:06 <marex> so anyway, I'm looking at this https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/irqchip/irq-xilinx-intc.c?h=linux-5.10.y#n163 here ... that runs in a kernel thread

15:06 <marex> I have a xiic attached to that controller (xilinx i2c controller), it requests threaded irq with IRQF_ONESHOT, the irq is level triggered

15:07 <marex> I noticed that every now and then, I miss an interrupt, and I suspect it is because both the xiic and the xilinx intc irq handlers run in a thread, but each in a different thread, and they access action->thread_flags in __irq_wake_thread() and irq_wait_for_interrupt() in kernel/irq/{handle,manage}.c

15:08 <marex> but I might be wrong, the xiic driver is horribly buggy and racy, I started fixing those races and I think I got most of them sorted out , except for this one which I really just cannot nail down

15:09 <marex> (I had to patch the xiic considerably, currently the xiic driver uses local_irq_save()/local_irq_restore() as locking primitive, which isn't working too well on SMP ... so ... that needed fixing)

15:21 <marex> monstr: ^

15:32 <maz_> marex: are you sure that the xilinx intc runs in a thread? or maybe you are forcing all interrupts to be threaded?

15:34 <marex> maz_: ftrace says it runs either in <idle>-0 or kworker/* , so yeah, sometimes

15:35 <marex> maz_: I do have full preemption enabled, so the interrupt handler could be threaded I think ?

15:35 <marex> CONFIG_IRQ_FORCED_THREADING=y this is enabled too

15:36 <maz_> right, so that's the reason why this is threaded.

15:36 <marex> yeah ... but ... shouldn't that still work and not be racy ?

15:37 <marex> btw this is zynqmp, the the xiic-i2c is in FPGA, so is the xilinx-intc

15:37 <maz_> it *should* work.

15:37 <marex> the xiic-i2c is plugged into the intc and intc is plugged into gic

15:37 <maz_> is the intc a secondary irqchip?

15:37 <maz_> right.

15:37 <maz_> i2c->intc->gic.

15:37 <marex> yep

15:38 <marex> btw I really suspect the xiic driver, it is horribly broken, although the xilinx intc would be second on my list of possible issues

15:41 <maz_> marex: the intc_enable_or_unmask() functon is super suspicious in the way it acks interrupts.

15:42 <marex> yes it is, I already dug in that one, but I don't think that is the culprit

15:43 <marex> but I just had a horrid recollection of an issue where GIC got somehow stuck on the zynqmp and didn't distribute interrupts

15:43 <marex> wait a minute

15:44 <marex> if the intc somehow got gic to hang , it could also block IPIs, which would explain why wake_up_process() call doesn't wake the IRQ thread almost imediatelly on the other CPU

15:44 <marex> no ?

15:45 <maz_> everything is possible if you prevent the GIC from EOI-ing an interrupt.

15:46 <marex> maz_: can I detect such a condition ?

15:46 <marex> probably add trace_printk into the gic driver is one way to do that

15:47 <maz_> you can check whether every read of the GIC IAR register results in a writeback to the EOI register with the same value.

15:47 <maz_> if you don't see that, you're in trouble (unless the value is 1023, of course, which indicates a spurious signal).

15:49 <marex> maz_: that would be gic_handle_irq() , do_read_iar() and do_write_eoir() ?

15:50 <maz_> yup, that'd be it.

15:50 <maz_> you probably only want to consider eoimode0.

15:51 <maz_> unless you boot at HYP/EL2?

15:51 <marex> [ 0.041173] CPU: All CPU(s) started at EL2

15:51 <marex> but I dont think it is HYP mode

15:51 <maz_> ah.

15:51 <maz_> so you need to look at eoimode1, and instrument the DIR accesses too.

15:52 <maz_> EL2=HYP.

15:52 <marex> oh

16:09 <marex> maz_: btw shouldnt there be some IPI watchdog in the kernel, which monitors whether the other CPU cores are still operable ?

16:38 <marex> maz_: OK, we can rule out the IPI, I still have this problem even on UP