Spark 2.0.0 support #115
Spark 2.0.0 support #115
Conversation
This commit makes the default version of Spark "2.0.0-preview" and consists of various configuration file changes and a couple of method changes. We should remove the -preview from the project files once 2.0.0 is made generally available (so we won't be relying on the preview builds). * Several changes have been made including downloading Akka for streaming-tests * Scala 2.11.8 is used * config/py now looks for $SPARK_HOME instead of /root * foreachRDD is used instead of foreach for a DStream * awaitTerminationOrTimeout is used instead of awaitTermination for a StreamingContext * json4s render call is removed owing to API changes
|
There's a problem here, if we do sbt package from the mllib directory then run the 25 mllib tests, there are four failures as certain test properties have been removed, thanks to Yves Leaute for pointing this out over an email. I'll resolve this by updating the PR and will also adjust the version setting in config.py to prevent artifact resolution problems |
|
Hi Robert, great work with HiBench and Spark-Perf... Ever though of doing something similar for SparkBench? https://github.com/SparkTC/spark-bench |
config/config.py.template
Outdated
| # * Don't use PREP_MLLIB_TESTS = True; instead manually run `cd mllib-tests; sbt/sbt -Dspark.version=1.5.0-SNAPSHOT clean assembly` to build perf tests | ||
|
|
||
| MLLIB_SPARK_VERSION = 2.0 | ||
| MLLIB_SPARK_VERSION = 2.0.0 |
There was a problem hiding this comment.
My environment is a little customized by the time I run this code (so line numbers don't match), but I am seeing this:
Loading configuration from <snip>/spark-perf/config/config.py
Traceback (most recent call last):
File "./bin/../lib/sparkperf/main.py", line 40, in <module>
config = imp.load_source("config", "", cf)
File "", line 410
MLLIB_SPARK_VERSION = 2.0.0
zhang051
left a comment
zhang051
left a comment
There was a problem hiding this comment.
I would like to run spark-perf with spark-2.0.0 support. Is there a beta- version of spark-perf (except updating individual files) that I can download?
Thanks,
Shuxia
4 participants
A few API changes and lots of project file changes
The v2p0 folder contains the code that we'll run with Spark 2 so contains the API changes, it's hard to tell what's new as they're entirely new files but to summarise I have