Hacker GitHub

← All repos

apache/spark

View on GitHub ↗ ★ 44,385 Scala

Apache Spark - A unified analytics engine for large-scale data processing

31 mentions on Hacker News by 28 people: 2 stories, 29 comments · first 2014-09-12 · latest 2026-01-28

Mentions per year 31 mentions · busiest year 2015 (8)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 3
    Add Support for PyCapsule to Pyspark
    story by devin-petersohn · 2026-01-28 · 0 comments
  2. …not collecting all data at once. It won't be released probably until May/June but it's there: https://github.com/apache/spark/commit/ecf179c3485ba8bac72af...
    comment by devin-petersohn · 2026-01-28 · in Pandas 3.0
  3. …script that wraps `mvn` or `sbt` (which are build tools that do not provide this functionality) https://github.com/apache/spark/blob/290b4b31bae2e02b648d2c5... - Automatic sandboxing of your build steps in CGroup/NS containers, to ensure…
    comment by lihaoyi · 2024-12-09 · in Bazel 8.0
  4. ▲ 4
    Spark SQL is getting pipe syntax
    story by tanelpoder · 2024-09-17 · 0 comments
  5. …cache the result this way. There's a more detailed explanation in the old comment to the method: https://github.com/apache/spark/pull/24068/files#diff-5de773... .
    comment by IvanVergiliev · 2021-06-21 · in Accidentally exponential behavior in Spark
  6. …don't have a Github account or for some other reason can't read the patch. Between that patch and https://github.com/apache/spark/pull/24910 you can see why the code is what it is.
    comment by renewiltord · 2021-06-21 · in Accidentally exponential behavior in Spark
  7. Good read - fwiw if this is your blog some of your links are broken and think they are local - https://heap.io/blog/%E2%80%9Dhttps://github.com/apache/spar...
    comment by hobs · 2021-06-21 · in Accidentally exponential behavior in Spark
  8. …to ditching that. The Hadoop ecosystem has definitely been dragging its feet on JDK bumps. [0] https://github.com/apache/spark/blob/master/resource-manager...
    comment by cmckn · 2020-09-15 · in JDK 15
  9. Arrow is tightly integrated in Spark's Scala code: https://github.com/apache/spark/search?q=arrow&unscoped_q=ar... Andy Grove is building a "distributed compute platform implemented in Rust, using the Apache Arrow memory model":…
    comment by MrPowers · 2020-07-27 · in Apache Arrow 1.0
  10. …which a language like TypeScript has to provide is only feasible by hand-writing the parser. [0] https://github.com/apache/spark/blob/master/sql/catalyst/src... [1] https://github.com/prestodb/presto/blob/master/presto-parser... [2]…
    comment by rajnathani · 2020-03-05 · in Why not to use (f)lex, yacc or bison
  11. …queries against JSON derived Datasets with some work. Starting point for some reading maybe? https://github.com/apache/spark/blob/e2d3983de78f5c80fac066b...
    comment by EdwardDiego · 2020-02-26 · in Show HN: Querying JSON documents using SQL-like language in Scala
  12. …still depends on Hadoop for a lot: - Using Parquet files = parquet-mr which is tied to Hadoop MR https://github.com/apache/spark/tree/master/sql/core/src/mai... - Using S3 instead of HDFS = Hadoop S3a connector Even if you don't run HDFS…
    comment by atomicity · 2019-12-09 · in The sad state of sysadmin in the age of containers (2015)
  13. …I have seen has an attached shell script to set up dependency path. Here is a first example: https://github.com/apache/spark/blob/121f9338cefbb1c800fabfe... At least half of that is figuring out what the classpath should be. And the worst…
    comment by theamk · 2019-11-03 · in Why to use ‘python -m pip’
  14. …larooLabs/wallaroo/blob/0.1.2/examples... With Apache Spark, the Word count example is a lot shorter: https://github.com/apache/spark/blob/master/examples/src/mai...
    comment by wdroz · 2017-10-12 · in Go Python, Go: Stream Processing for Python
  15. I find Apache Spark to be exceptionnaly well written and easy to read. (in Scala). https://github.com/apache/spark
    comment by cerisier · 2017-07-24 · in Ask HN: Best-architected open-source business applications worth studying?
  16. …across projects: https://github.com/hammerlab/sbt-parent/blob/1.7.5/src/main/... Spark's POMs (e.g. https://github.com/apache/spark/blob/v2.1.0/pom.xml ) vs. SBT project folder ( https://github.com/apache/spark/tree/v2.1.0/project ) are…
    comment by rdubz · 2017-04-06 · in Building Scala Projects: Maven vs. SBT
  17. …The Spark documentation and interpreter are good places to start. https://github.com/apache/spark/tree/master/examples/src/mai... (Scala) and https://github.com/apache/spark/tree/master/examples/src/mai... (Python)
    comment by lifebeyondfife · 2016-08-30 · in Ask HN: Best way to start and master Hadoop?
  18. …from RDD, looking at the master branch for MLlib in github it looks like ALS is still using RDDs? https://github.com/apache/spark/blob/master/mllib/src/main/s... Admittedly I'm not a spark expert though, thanks for bringing this up!
    comment by idewanck · 2016-08-09 · in Bayesian Optimization for Collaborative Filtering with MLlib
  19. What's wrong with exporting to csv? Seems like it's supported: https://github.com/apache/spark/blob/master/sql/core/src/mai...
    comment by aduffy · 2016-08-01 · in Spark 2.0.0 Released
  20. …[2] https://github.com/graphframes/graphframes [3] http://stackoverflow.com/a/37150604/877069 [4] https://github.com/apache/spark/pull/2144
    comment by nchammas · 2016-06-06 · in Microsoft announces major commitment to Apache Spark
  21. …all open source. They generate Java source code. You can read more at the prototype pull request: https://github.com/apache/spark/pull/10735 (I couldn't find a spec doc). If I understand correctly they insert a `WholeStageCodegen`[1]…
    comment by rusanu · 2016-05-23 · in Spark as a Compiler: Joining a Billion Rows per Second on a Laptop
  22. You don't indent after a package e.g. https://github.com/apache/spark/blob/master/core/src/main/ja... And it's fair enough to indent with a main since it is so infrequent. Ideally you only have one main entry point.
    comment by threeseed · 2016-01-10 · in GCC 6 Will Warn You About Misleading Code Indentations
  23. …course, Node They are? (keyword => "leading") https://github.com/docker/docker/graphs/contributors https://github.com/apache/spark/graphs/contributors I find this claim dubious. Maybe I'm not digging hard enough.
    comment by mbesto · 2015-09-10 · in What the IBM Acquisition of StrongLoop Means for the Node.js Community
  24. …language landscape is diverse and evolving quickly, good times now and great times ahead... [1] https://github.com/apache/spark/blob/master/sql/catalyst/src...
    comment by virtualwhys · 2015-07-27 · in Ask HN: Has anyone here programmed in Kotlin? What do you think about it?
  25. …maybe there are no dragons? Or if there are, the big players don't seem overly bothered. [1] https://github.com/apache/spark/blob/master/sql/catalyst/src... [2] https://www-03.ibm.com/press/us/en/pressrelease/47107.wss
    comment by virtualwhys · 2015-07-25 · in Ask HN: Has anyone here programmed in Kotlin? What do you think about it?
  26. …shuffle and observed some decent speedups: * https://issues.apache.org/jira/browse/SPARK-7081 * https://github.com/apache/spark/pull/5868#issuecomment-10183... However, I guess reduceByKey (and friends) don't benefit yet. Their SGD…
    comment by choppaface · 2015-06-12 · in Announcing Apache Spark 1.4
  27. That part is actually coming: https://github.com/apache/spark/pull/5713
    comment by rxin · 2015-05-31 · in RDDs are the new bytecode of Apache Spark
  28. Since this will be released in Spark 1.3, you can track the dev. in the 1.3 branch on Github https://github.com/apache/spark/tree/branch-1.3
    comment by zodvik · 2015-02-18 · in Introducing DataFrames in Spark for Large Scale Data Science
  29. Btw, there is an new PR by Josh Rosen to fix SPARK-4454 here: https://github.com/apache/spark/pull/4660
    comment by nchammas · 2015-02-17 · in Introducing DataFrames in Spark for Large Scale Data Science
  30. …https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in that time two releases have swung by bringing a tonne of features. I ended up having…
    comment by super_sloth · 2015-02-17 · in Introducing DataFrames in Spark for Large Scale Data Science
  31. …to support importing CSV data as SchemaRDDs: https://issues.apache.org/jira/browse/SPARK-2360 https://github.com/apache/spark/pull/1351
    comment by JoshRosen · 2014-09-12 · in Announcing Apache Spark 1.1