08:59 <arnd> milkylainen: yes, drivers need to call dma_set_mask() if they support a larger address range

09:01 <arnd> there may also be limitations on the bus, so if you are missing dma-ranges properties, or they define a translation that doesn't contain all of RAM, calling dma_set_mask() may still not result in high DMA

09:01 <milkylainen> arnd: but the default is always 32? for endpoints on pcie? I mean, there is no way of telling if the device can do more, by probing extension registers or something like that?

09:02 <milkylainen> I just know basic stuff unfortunately. :\ I was a bit surprised. Was assuming that the DMA mask would be part of the walking the bus for devices.

09:03 <milkylainen> Sure, the device driver can rearrange that, but that the default would be more close to device capability per default.

09:48 <milkylainen> So many questions. :) Does devices really care? Or are they hoping for iommu single bus cycle (to map to 64-bit) over dual address cycles?

09:50 <sven> there are devices that only support 32bit DMA even if they are on a 64bit capable bus

09:51 <sven> https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt, "DMA addressing capabilities"

09:53 <sven> essentially you can only do 64bit DMA if the bus and the iommu and the device (and maybe something else that's in the way) all support it. dma_set_mask() tells the DMA API that your devices supports that mask.

10:00 <milkylainen> sven: Yes. But I was expecting the device to be able to expose it's DMA capability? To me it looks like the mask information is "We know the device can do this so lets set it..."

10:02 <sven> some devices expose it somewhere in their MMIO

10:18 <milkylainen> sven: ok.

10:19 <milkylainen> Would be nice if the pci standard had exposed it somewhere in status or config regs.

12:35 <robmur01> milkylainen: it's not necessarily one single value though - we already have two separate masks in Linux because some devices have mixed capabilities, and there have also been drivers that adjust one mask between allocations/mappings to cope with different requirements of different buffers/descriptors/etc.

12:38 <robmur01> PCI cares about whether the device sends a 32-bit or a 64-bit address for a transaction, but not how many of those bits are significant beyond that

12:40 <milkylainen> robmur01: Do you know if there are hard requirements for pcie compatible devices to do > 32-bit dma?

12:41 <bjdooks> iirc pcie yes, but pcie<>pci bridges exist

12:43 <milkylainen> robmur01: I was thinking that it ends up with modern devices always expanding the bitmask. I was thinking if the standard would have exposed the device capabilities for dma, something sane could have been done at probing, instead of every device redoing all of the dma setup. I guess most of the dma setups in drivers are very normal, duplicated code?

12:43 <milkylainen> Like this amd gpu I have. Default 32-bit mask unless a driver says otherwise.

12:44 <milkylainen> Maybe there is no better way, because the standard does not expose anything that can be used to do sane dma defaults?

12:44 <milkylainen> It always has to end up in the hands of the driver.

12:45 <milkylainen> Which then always ends up with a potential collision with the reality of the physical machine. :)

12:46 <milkylainen> It ends up colliding with reality either way I guess. Just with a lot of duplicated driver code.

13:06 <robmur01> I believe 32-bit PCIe devices are allowed and do exist (and there are certainly some nominally-64-bit ones whose handling of >32-bit addresses is just broken)

13:07 <robmur01> the only thing you can detect generically is whether a device supports 64-bit MSI addresses or not, but that isn't necessarily indicative of its DMA capability in general

13:11 <robmur01> I can't see that there could be any standard way to describe that, say, a device supports 44-bit DMA addresses for its control ring buffer but 64-bit addresses for data descriptors, because those concepts are defined by that device and its driver

13:29 <biju> we have 34 bit DMA capbale devices in RZ/V2{M,MA} series, currently we use dma_set_mask() for 34-bit dma operation. but unfortunately the drivers are designed for 32 bit dma map/unmap, How do we extend it to 34 bit dma map/unmap()?? currently we use hacks for unmap operation.

13:39 <robmur01> biju: sorry, I don't follow... dma_set_mask() is for the streaming DMA API which *is* dma_{map,unmap}_*() - if you're already setting a 34-bit mask then it will already allow mappings to return up to 34-bit DMA addresses (assuming dma_addr_t is 64-bit)

13:41 <robmur01> if a driver is stuffing a dma_addr_t into a u32 somewhere then the only answer is "don't do that"

13:49 <bjdooks> re-education with cluebat

14:01 <biju> robmur01: It is returning 34 bit mappings. But the driver uses le32_to_cpu(desc->dptr) for dma_unmap that is causing issues. for eg:- https://elixir.bootlin.com/linux/v6.1-rc5/source/drivers/net/ethernet/renesas/ravb_main.c#L219

14:02 <biju> I need to hack BIT(32)| le32_to_cpu(desc->dptr))) to make it work

14:03 <sven> "don't do that" ;)

14:05 <biju> what is the best way to handle both cases. 1 is for 32 bit machines and another for > 32 bits

14:05 <sven> le32_to_cpu sounds like you devices actually only reads 32bit from dptr - so how can it see the extra two bits?

14:05 <sven> *your

14:13 <biju> sven: you mean to support >32 bit just use desc->dptr instead??

14:13 <robmur01> um, yeah, AFAICS the DMA addresses are always truncated by cpu_to_le32() to begin with, so I can't see how it works at all.

14:13 <sven> ^-- that

14:14 <robmur01> if your DMA descriptor can only encode a 32-bit address to pass to the device, then you only support 32-bit DMA

14:16 <biju> But the we need to use a memory beyond 32 bit address range. for eg:- reg = <0x1 0x80000000 0x0 0x80000000>;

14:17 <biju> this is the memory map of the system. with dma_setmask, 34 and hacks in unmap, it works fine.

14:17 <robmur01> well then presumably you must have an IOMMU or some kind of dma-ranges offset in the interconnect

14:19 <robmur01> note that the DMA mask represents the size of address that the device itself can emit, not whatever it might get translated to downstream

14:29 <biju> Not IOMMU , but there is an IP, that has a bank switching register to extend the two higher-order bits of the addresses and output to the ICB (common to the AR and AW channels) to make it 34 bits.

21:43 <ukleinek> arnd: i have a build failure here:

21:43 <ukleinek> /home/uwe/gsrc/linux/arch/arm/mach-s3c/devs.c:32:10: fatal error: linux/platform_data/dma-s3c24xx.h: No such file or directory

21:43 <ukleinek> 32 | #include <linux/platform_data/dma-s3c24xx.h>

21:43 <ukleinek> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

21:43 <ukleinek> I guess that's the fault of cccc46ae362398e43c6b8be38fdb7e39def9e21b

21:44 <ukleinek> vkoul, krzk: ^

21:50 <vkoul> ukleinek: no such failure reported on linux-next, I think ardb has sent patches to all subsystems

22:06 <ukleinek> vkoul: https://lore.kernel.org/linux-arm-kernel/63754698.I%2Ffgy2FgVzs0e3UW%25lkp@intel.com/

22:11 <ukleinek> vkoul: anyhow, patch in your inbox :-)