Argthtjtr

I'm trying to install Spark on a Linux box. I downloaded it from http://spark.apache.org/docs/latest/building-spark.html and am trying to build it with this command:

root# build/mvn -e -X -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -Phive-thriftserver -DskipTests clean package

The build seems to start fine:

Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T07:57:37-04:00)

Maven home: /some_path_here/spark-1.5.0/build/apache-maven-3.3.3

Java version: 1.7.0_05, vendor: Oracle Corporation

Java home: /usr/local/mytools-tools/java/jdk64/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-573.1.1.el6.x86_64", arch: "amd64", family: "unix"

[DEBUG] Created new class realm maven.api

[DEBUG] Importing foreign packages into class realm maven.api

But then it fails:

[debug] Recompiling all 8 sources: invalidated sources (8) exceeded 50.0% of all sources

[info] Compiling 8 Java sources to /some_path_here/spark-1.5.0/launcher/target/scala-2.10/classes...

[debug] Attempting to call javac directly...

[debug] com.sun.tools.javac.Main not found with appropriate method signature; forking javac instead

[debug] Forking javac: javac @/tmp/sbt_6c9436e4/argfile

[error] Cannot run program "javac": error=20, Not a directory

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO] 

[INFO] Spark Project Parent POM ........................... SUCCESS [  2.056 s]

[INFO] Spark Project Launcher ............................. FAILURE [  4.832 s]

and so on.

I'm pretty sure I have JAVA_HOME and PATH defined appropriately.

This box has multiple versions of Java installed, which might be related to the problem.

I fixed this by turning off Zinc, per this answer to a related question: https://stackoverflow.com/a/32766960

A 'quirk' with Spark builds is that it can download its own version of Maven if it determines it is required.

When you run ./build/mvn clean package you are not running Maven directly, you are running a Spark proprietary script. The first thing that script does is check if your mvn --version is new enough for the version that the project determines it needs (which is set in the pom.xml file).

This is an important point because if you're running an old version of maven, Spark may download an additional maven version and install it and use that instead.

Some key things:

When you run ./build/mvn clean package, check which version of maven it is using
When maven runs it does its own traversal to figure out which JAVA_HOME is used
Before trying to run the spark build, check JAVA_HOME is set as a variable
Check that the JAVA_HOME version is a full jdk, not just a jre
Update your Maven to the latest version (or check it is at least as new as the version set by in the pom.xml in the root directory
Thanks