apache/spark on Hacker News

…not collecting all data at once. It won't be released probably until May/June but it's there: https://github.com/apache/spark/commit/ecf179c3485ba8bac72af...

comment by devin-petersohn · 2026-01-28 · in Pandas 3.0

…script that wraps `mvn` or `sbt` (which are build tools that do not provide this functionality) https://github.com/apache/spark/blob/290b4b31bae2e02b648d2c5... - Automatic sandboxing of your build steps in CGroup/NS containers, to ensure…

comment by lihaoyi · 2024-12-09 · in Bazel 8.0

…cache the result this way. There's a more detailed explanation in the old comment to the method: https://github.com/apache/spark/pull/24068/files#diff-5de773... .

comment by IvanVergiliev · 2021-06-21 · in Accidentally exponential behavior in Spark

…don't have a Github account or for some other reason can't read the patch. Between that patch and https://github.com/apache/spark/pull/24910 you can see why the code is what it is.

comment by renewiltord · 2021-06-21 · in Accidentally exponential behavior in Spark

Good read - fwiw if this is your blog some of your links are broken and think they are local - https://heap.io/blog/%E2%80%9Dhttps://github.com/apache/spar...

comment by hobs · 2021-06-21 · in Accidentally exponential behavior in Spark

…to ditching that. The Hadoop ecosystem has definitely been dragging its feet on JDK bumps. [0] https://github.com/apache/spark/blob/master/resource-manager...

comment by cmckn · 2020-09-15 · in JDK 15

Arrow is tightly integrated in Spark's Scala code: https://github.com/apache/spark/search?q=arrow&unscoped_q=ar... Andy Grove is building a "distributed compute platform implemented in Rust, using the Apache Arrow memory model":…

comment by MrPowers · 2020-07-27 · in Apache Arrow 1.0

…which a language like TypeScript has to provide is only feasible by hand-writing the parser. [0] https://github.com/apache/spark/blob/master/sql/catalyst/src... [1] https://github.com/prestodb/presto/blob/master/presto-parser... [2]…

comment by rajnathani · 2020-03-05 · in Why not to use (f)lex, yacc or bison

…queries against JSON derived Datasets with some work. Starting point for some reading maybe? https://github.com/apache/spark/blob/e2d3983de78f5c80fac066b...

comment by EdwardDiego · 2020-02-26 · in Show HN: Querying JSON documents using SQL-like language in Scala

…still depends on Hadoop for a lot: - Using Parquet files = parquet-mr which is tied to Hadoop MR https://github.com/apache/spark/tree/master/sql/core/src/mai... - Using S3 instead of HDFS = Hadoop S3a connector Even if you don't run HDFS…

comment by atomicity · 2019-12-09 · in The sad state of sysadmin in the age of containers (2015)

…I have seen has an attached shell script to set up dependency path. Here is a first example: https://github.com/apache/spark/blob/121f9338cefbb1c800fabfe... At least half of that is figuring out what the classpath should be. And the worst…

comment by theamk · 2019-11-03 · in Why to use ‘python -m pip’

…larooLabs/wallaroo/blob/0.1.2/examples... With Apache Spark, the Word count example is a lot shorter: https://github.com/apache/spark/blob/master/examples/src/mai...

comment by wdroz · 2017-10-12 · in Go Python, Go: Stream Processing for Python

I find Apache Spark to be exceptionnaly well written and easy to read. (in Scala). https://github.com/apache/spark

comment by cerisier · 2017-07-24 · in Ask HN: Best-architected open-source business applications worth studying?

…across projects: https://github.com/hammerlab/sbt-parent/blob/1.7.5/src/main/... Spark's POMs (e.g. https://github.com/apache/spark/blob/v2.1.0/pom.xml ) vs. SBT project folder ( https://github.com/apache/spark/tree/v2.1.0/project ) are…

comment by rdubz · 2017-04-06 · in Building Scala Projects: Maven vs. SBT

…The Spark documentation and interpreter are good places to start. https://github.com/apache/spark/tree/master/examples/src/mai... (Scala) and https://github.com/apache/spark/tree/master/examples/src/mai... (Python)

comment by lifebeyondfife · 2016-08-30 · in Ask HN: Best way to start and master Hadoop?

…from RDD, looking at the master branch for MLlib in github it looks like ALS is still using RDDs? https://github.com/apache/spark/blob/master/mllib/src/main/s... Admittedly I'm not a spark expert though, thanks for bringing this up!

comment by idewanck · 2016-08-09 · in Bayesian Optimization for Collaborative Filtering with MLlib

What's wrong with exporting to csv? Seems like it's supported: https://github.com/apache/spark/blob/master/sql/core/src/mai...

comment by aduffy · 2016-08-01 · in Spark 2.0.0 Released

…[2] https://github.com/graphframes/graphframes [3] http://stackoverflow.com/a/37150604/877069 [4] https://github.com/apache/spark/pull/2144

comment by nchammas · 2016-06-06 · in Microsoft announces major commitment to Apache Spark

…all open source. They generate Java source code. You can read more at the prototype pull request: https://github.com/apache/spark/pull/10735 (I couldn't find a spec doc). If I understand correctly they insert a `WholeStageCodegen`[1]…

comment by rusanu · 2016-05-23 · in Spark as a Compiler: Joining a Billion Rows per Second on a Laptop

You don't indent after a package e.g. https://github.com/apache/spark/blob/master/core/src/main/ja... And it's fair enough to indent with a main since it is so infrequent. Ideally you only have one main entry point.

comment by threeseed · 2016-01-10 · in GCC 6 Will Warn You About Misleading Code Indentations

…course, Node They are? (keyword => "leading") https://github.com/docker/docker/graphs/contributors https://github.com/apache/spark/graphs/contributors I find this claim dubious. Maybe I'm not digging hard enough.

comment by mbesto · 2015-09-10 · in What the IBM Acquisition of StrongLoop Means for the Node.js Community

…language landscape is diverse and evolving quickly, good times now and great times ahead... [1] https://github.com/apache/spark/blob/master/sql/catalyst/src...

comment by virtualwhys · 2015-07-27 · in Ask HN: Has anyone here programmed in Kotlin? What do you think about it?

…maybe there are no dragons? Or if there are, the big players don't seem overly bothered. [1] https://github.com/apache/spark/blob/master/sql/catalyst/src... [2] https://www-03.ibm.com/press/us/en/pressrelease/47107.wss

comment by virtualwhys · 2015-07-25 · in Ask HN: Has anyone here programmed in Kotlin? What do you think about it?

…shuffle and observed some decent speedups: * https://issues.apache.org/jira/browse/SPARK-7081 * https://github.com/apache/spark/pull/5868#issuecomment-10183... However, I guess reduceByKey (and friends) don't benefit yet. Their SGD…

comment by choppaface · 2015-06-12 · in Announcing Apache Spark 1.4

That part is actually coming: https://github.com/apache/spark/pull/5713

comment by rxin · 2015-05-31 · in RDDs are the new bytecode of Apache Spark

Since this will be released in Spark 1.3, you can track the dev. in the 1.3 branch on Github https://github.com/apache/spark/tree/branch-1.3

comment by zodvik · 2015-02-18 · in Introducing DataFrames in Spark for Large Scale Data Science

Btw, there is an new PR by Josh Rosen to fix SPARK-4454 here: https://github.com/apache/spark/pull/4660

comment by nchammas · 2015-02-17 · in Introducing DataFrames in Spark for Large Scale Data Science

…https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in that time two releases have swung by bringing a tonne of features. I ended up having…

comment by super_sloth · 2015-02-17 · in Introducing DataFrames in Spark for Large Scale Data Science

…to support importing CSV data as SchemaRDDs: https://issues.apache.org/jira/browse/SPARK-2360 https://github.com/apache/spark/pull/1351

comment by JoshRosen · 2014-09-12 · in Announcing Apache Spark 1.1