09:33 <arnd> I did some performance tests using kernel builds with different-sized RAM, different page sizes, and with the folio patch set, as those topics keep coming up: https://docs.google.com/spreadsheets/d/1Y-eeXEHr8Tud2ul4iApn6hLEQoMsl3Bsud-vlW3V2t8/edit#gid=0

09:35 <arnd> The 16KB page kernels are noticeably better in this test case than I had expected, and this is not even on Apple hardware that has additional optimizations for 16KB pages (as used by MacOS and iOS)

09:36 <arnd> One funny bit that I found was that as soon as the machine went deeply into swap, I could never get it to recover from that other than by rebooting

09:38 <arnd> e.g. building kernels with 2GB of RAM and 16 cores works fine without paging, hogging some of the memory using mlock() obviously makes it slower, but after freeing the memory hog it would need more than 2GB to build a kernel without paging

09:50 <milkylainen> I wouldn't have expected a 16k page to make that much difference. Pressure on TLB? Hardware counters? Sounds too large a gap for just TLB pressure?

09:53 <ukleinek> I wonder if Apple also used "Linux kernel builds per hour" as a benchmark before deciding to pick 16k page sizes.

10:04 <arnd> I think Apple went the other way, and their hardware team said they really want large L1 caches, so the OS team made their kernel work better with 16KB pages. There is usually a penalty in L1 access otherwise. With 128KB 8-way set-associative L1 VIPT Dcache and 16KB pages, you get a really nice balance. With 4KB pages you'd need either a PIPT cache or (theoretically) 32 ways.

16:58 <alyssa> and now we get to deal with both! wow!