Home / ubuntu / Questions / 972108
In Process
SriniShine
Asked: 2017-11-03 04:41:44 +0800 CST

Installing Spark on Hadoop 2.5

  • 772

I'm running Hadoop on Ubuntu on a Oracle VM on Windows 7. I'm trying to install Spark on an existing Hadoop installation. For demonstration purposes I do not want to change the existing Hadoop version. Following are the versions of software in my VM.

Ubuntu : 14.04

Java : Java 1.7 and OpenJDK runtime environment (IcedTea 2.5.2)

Hadoop : 2.5.0

I figured out that for Hadoop I need to install Spark 1.6.2. When I try to download Spark 1.6 it doesn't give me the option for Hadoop 2.5 but has options for 2.4 and 2.6. Will Spark 1.6 work on Hadoop 2.4?

I would like to know the appropriate platform requirements for Spark, Scala and Java to install Spark on Hadoop 2.5.

apache2

1 Answers

  1. You can build spark for any hadoop version by installing maven and follow below steps:

    1. Download spark source code.

    2. Run below commands:

      tar -xvzf spark-1.6.2.tgz

      cd spark-1.6.2

      mvn -Pyarn -Phadoop-2.5 -Dhadoop.version=2.5.0 -DskipTests clean package

    • 1