Hacker GitHub

← All repos

capitalone/DataProfiler

View on GitHub ↗ ★ 1,594 Python

What's in your data? Extract schema, statistics and entities from datasets

29 mentions on Hacker News by 5 people: 11 stories, 18 comments · first 2021-03-26 · latest 2024-02-22

Mentions per year 29 mentions · busiest year 2021 (22)
20082026
AllStoriesComments NewestTopOldest
  1. It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area.…
    comment by lettergram · 2024-02-22 · in LongRoPE: Extending LLM Context Window Beyond 2M Tokens
  2. ▲ 3
    Data Profiler – What's in your data?
    story by PaulHoule · 2023-06-08 · 0 comments
  3. ▲ 1
    Show HN: Dataprofiler v0.8.1 Release
    story by taylorfturner · 2022-10-05 · 0 comments
  4. ▲ 2
    Show HN: Dataprofiler 0.8.0 Release with graph support!
    story by taylorfturner · 2022-09-29 · 0 comments
  5. ▲ 2
    Show HN: Dataprofiler
    story by taylorfturner · 2022-09-28 · 0 comments
  6. ▲ 2
    DataProfiler: What's in Your Data?
    story by taylorfturner · 2022-09-27 · 1 comment
  7. I spent some time working on something called DataProfiler python library https://github.com/capitalone/DataProfiler The gist is that you can point to any common dataset and load it directly into pandas. from dataprofiler import Data data…
    comment by lettergram · 2022-08-05 · in Loading Data into Pandas: Tips and Tricks You May or May Not Know
  8. …built a similar tool in Python to load any delimited file, json, parquet and Avro with one command: https://github.com/capitalone/DataProfiler Effectively loads anything into a dataframe. After that you can profile the data, merge, save,…
    comment by lettergram · 2021-12-22 · in Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
  9. The thing is, tensorflow has more ability to run cross platform. I help maintain https://github.com/capitalone/DataProfiler Our sensitive data detection library is exported to iOS, android, and Java; in addition to Python. We also run…
    comment by lettergram · 2021-12-14 · in PyTorch vs. TensorFlow in 2022
  10. I really should write up how we did delimiter and quote detection in this library: https://github.com/capitalone/DataProfiler It turns out delimited files IMO are much harder to parse than say, JSON. Largely because they have so many…
    comment by lettergram · 2021-12-04 · in Fast CSV Processing with SIMD
  11. …any large overhead. Having said that, we opted to write our own parallelization for this library: https://github.com/capitalone/DataProfiler As opposed to using the dask frame. Effectively, the dataframe processing wasn’t the bottleneck…
    comment by lettergram · 2021-11-18 · in Dask – A flexible library for parallel computing in Python
  12. My team has worked on a library for a similar purpose: https://github.com/capitalone/DataProfiler Load any document, profile and monitor the profiles for changes that would impact downstream applications. Very common problem, you all are…
    comment by lettergram · 2021-11-15 · in Launch HN: Metaplane (YC W20) – Datadog for Data
  13. It would be possible to build a similar system via a library my team has built: https://github.com/capitalone/dataprofiler Effectively, you can monitor changes between profiles: data1 = dp.Data("file_a.csv") # Load a CSV file profile1 =…
    comment by citilife · 2021-10-07 · in Show HN: Graphsignal – Production Model Monitoring: Anomalies, Drift, Exceptions
  14. …works to load JSON, Parquet, AVRO and loads it into a dataframe. Not CLI exactly, but pretty easy: https://github.com/capitalone/dataprofiler Can Install: pip install dataprofiler[ml] --user How it works: csv_data = Data('your_file.csv') #…
    comment by citilife · 2021-08-25 · in Miller CLI – Like Awk, sed, cut, join, and sort for CSV, TSV and JSON
  15. …solution would be awesome. I can attest this is a real problem. Here's the library we built: https://github.com/capitalone/DataProfiler In this space, Prodigy really dominates: https://prodi.gy/ We actually built our own internal system…
    comment by citilife · 2021-08-09 · in Launch HN: Lightly (YC S21): Label only the data which improves your ML model
  16. Telm.ai (YC S21) - Real-time data quality monitoring Looks interesting! I worked on https://github.com/capitalone/DataProfiler We are looking to monitor correlation changes over time, see if sensitive data gets entered, track schema…
    comment by citilife · 2021-07-23 · in Launch HN: Exams, tasks, K8, eCommerce, cell sites, health, data quality, travel
  17. …data and detect / validate schemas, etc as well as the other items provided by pandas-profiling. https://github.com/capitalone/DataProfiler That being said, we noted the same thing. It shouldn't matter what modeling you use. It's the data…
    comment by citilife · 2021-07-22 · in Modeling libraries don’t matter (2020)
  18. …tool, utilizing a CNN. It works on structured (and unstructured) data and provides additional info. https://github.com/capitalone/DataProfiler Cool part, is you can “extend” the intern name-entity recognition model by refitting with the…
    comment by lettergram · 2021-06-16 · in PyWhat: Identify Anything
  19. …probably be able to automatically generate some of these blocks. You could use something like this: https://github.com/capitalone/DataProfiler With enough data, you could probably further automate this process with AI. Good luck! Feel free…
    comment by lettergram · 2021-06-11 · in Show HN: A flow based data processing editor
  20. ▲ 5
    Ask HN: What is the best tool to infer data type of tabular data?
    story by mahalel · 2021-06-08 · 7 comments
  21. Hope this helps someone: https://github.com/capitalone/DataProfiler We’re working to automate much of that as well in a Python library. The end goal is to: 1. point at any dataset and load it with one command 2. Calculate statistics and…
    comment by lettergram · 2021-05-29 · in Automated Data Wrangling
  22. ▲ 86
    Show HN: DataProfiler – What's in your data? Extract schema, stats and entities
    story by lettergram · 2021-05-10 · 11 comments
  23. ▲ 1
    Show HN: DataProfiler – A Replacement for Pandas-Profiling
    story by lettergram · 2021-05-04 · 0 comments
  24. ▲ 3
    Show HN: DataProfiler – Sensitive Data Detection and Profiling
    story by citilife · 2021-05-03 · 0 comments
  25. ▲ 1
    Show HN: DataProfiler – Extract schema, statistics and PII / NPI detection
    story by citilife · 2021-04-30 · 0 comments
  26. ▲ 4
    Show HN: The DataProfiler – What's in your data?
    story by lettergram · 2021-04-27 · 0 comments
  27. I maintain a similar project, load any CSV, manipulate and get stats, detect sensitive data, etc https://github.com/capitalone/DataProfiler My question, how do you do header detection? That's a _very_ difficult problem.
    comment by lettergram · 2021-04-21 · in Show HN: sheet2dict – simple Python XLSX/CSV reader/to dictionary converter
  28. …repo I linked you can see the citation for an earlier model on synthetic and real world datasets: https://github.com/capitalone/DataProfiler#references https://arxiv.org/pdf/2012.09597.pdf So I don’t really understand the hostility.
    comment by lettergram · 2021-03-27 · in Apt Encounters of the Third Kind
  29. We just open sourced one of our libraries used to detect PII: https://github.com/capitalone/DataProfiler Amazon and Microsoft also have their own offerings, but can be quite expensive for network packets (and pretty slow). Most projects /…
    comment by lettergram · 2021-03-26 · in Apt Encounters of the Third Kind