Rust: reading very large files for the billion row challenge https://video.infosec.exchange/videos/watch/869a3df8-2b78-476c-8771-03ab778b1fb3 Checking out whether memmap can help us read very large files as fast as possible, and wondering how wc manages to be so fast. This is the next bit of the Billion Row Challenge, and probably the closest part to black magic. Read my blog at https://artificialworlds.net/blog Follow me on mastodon: @andybalaam@mastodon.social Fri, 10 Apr 2026 11:24:26 GMT https://validator.w3.org/feed/docs/rss2.html PeerTube - https://video.infosec.exchange Rust: reading very large files for the billion row challenge https://video.infosec.exchange/client/assets/images/icons/icon-1500x1500.png https://video.infosec.exchange/videos/watch/869a3df8-2b78-476c-8771-03ab778b1fb3 All rights reserved, unless otherwise specified in the terms specified at https://video.infosec.exchange/about and potential licenses granted by each content's rightholder. <![CDATA[Rust: reading very large files for the billion row challenge - Andy Balaam]]> https://video.infosec.exchange/w/hC32rdos46nZkXCtYvkGzi;threadId=262847 https://video.infosec.exchange/w/hC32rdos46nZkXCtYvkGzi;threadId=262847 Fri, 10 Apr 2026 08:49:09 GMT @sebsch@chaos.social thanks that sounds quite likely. Maybe something to investigate if I want to squeeze out the max performance.

]]> Andy Balaam <![CDATA[Rust: reading very large files for the billion row challenge - sebsch]]> https://video.infosec.exchange/w/hC32rdos46nZkXCtYvkGzi;threadId=262847 https://video.infosec.exchange/w/hC32rdos46nZkXCtYvkGzi;threadId=262847 Fri, 10 Apr 2026 08:35:10 GMT @andybalaam i did not have a chance looking at your video.

But I think one msin reason for the immense throughput of the GNU tools is, they operating directly on file descriptors.

Rust has nice Linux integrations with https://docs.rs/nix/latest/nix/ if you want to handle the raw fds in a save manner.

]]> sebsch