Hacker GitHub

← All repos

DiskFrame/disk.frame

View on GitHub ↗ ★ 594 R

Fast Disk-Based Parallelized Data Manipulation Framework for Larger-than-RAM Data

3 mentions on Hacker News by 3 people: 0 stories, 3 comments · first 2019-06-27 · latest 2021-02-02

Mentions per year 3 mentions · busiest year 2019 (1)
20082026
AllStoriesComments NewestTopOldest
  1. …the disk rather than in memory [1, 2]. Did you benchmark RAID 0 with less than four disks? [1] R: https://github.com/xiaodaigh/disk.frame [2] Python: https://docs.dask.org/en/latest/dataframe.html
    comment by teruakohatu · 2021-02-02 · in We Test PCIe 4.0 Storage: The AnandTech 2021 SSD Benchmark Suite
  2. I have not tested it myself but now there is a disk.frame ( https://github.com/xiaodaigh/disk.frame ). As far as I know, other options exist too.
    comment by vasili111 · 2020-02-13 · in Does my data fit in RAM?
  3. …a very similar idea recently! I used split to split a large CSV file and then used disk.frame ( https://github.com/xiaodaigh/disk.frame ), which is my package, to read in a large file. Not 25TB though!
    comment by xiaodai · 2019-06-27 · in Using AWK and R to parse 25TB