I'm working on a Hadoop installation done by someone else.
Ubuntu 18.04.1 LTS
Hadoop version 3.0.3
Java open jdk 11 (auto mode) and 8 (manual mode)
Currently Hadoop runs with Java 11 but I want to configure Hadoop to run with Java 8 because Hive doesn't work with Java 11.
Following is the Java configuration in my virtual machine.
hadoop@hadoop-VirtualBox:~$ whereis java
java: /usr/bin/java /usr/share/java /usr/share/man/man1/java.1.gz
hadoop@hadoop-VirtualBox:~$ update-alternatives --list java
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
hadoop@hadoop-VirtualBox:~$ java -version
openjdk version "10.0.2" 2018-07-17
OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1)
OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1, mixed mode)
hadoop@hadoop-VirtualBox:~$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 manual mode
2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
hadoop@hadoop-VirtualBox:~$ ls -lh /usr/lib/jvm/
total 8.0K
lrwxrwxrwx 1 root root 25 Apr 8 14:16 default-java -> java-1.11.0-openjdk-amd64
lrwxrwxrwx 1 root root 21 Apr 21 14:41 java-1.11.0-openjdk-amd64 -> java-11-openjdk-amd64
drwxr-xr-x 9 root root 4.0K Aug 23 12:28 java-11-openjdk-amd64
lrwxrwxrwx 1 root root 20 Apr 26 16:59 java-1.8.0-openjdk-amd64 -> java-8-openjdk-amd64
drwxr-xr-x 7 root root 4.0K Aug 17 08:24 java-8-openjdk-amd64
I was looking at this thread to configure Java however in the .bashrc or /etc/environment file there is no entry on JAVA_HOME. The only place JAVA_HOME is specified is in the /usr/local/hadoop/etc/hadoop/hadoop-env.sh file. In the file JAVA_HOME is specified as follows:
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
With the above configuration Hadoop runs without any problem. I tried changing the JAVA_HOME in the hadoop-env.sh file to /usr/lib/jvm/java-8-openjdk-amd64 and tried running Hadoop. But this gives an error
Unrecognized option: --add-modules Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.
I tried changing the priority of the Java version using sudo update-alternatives --config java along with the hadoop-env.sh alteration but it still gives the same error.
In order to change the Java version do I need to add the path to .bashrc and /etc/environment as well? I would like to know what changed needs to be done.
Make sure that you have a verified backup first.
A lot of the hadoop ecosystem only supports Java 8. Scala, Spark, etc. However, when installing Hadoop, JAVA_HOME needs to be setup first.
You will need to uninstall Java 11 before installing Java 8. For a Hadoop system, it will be better to receive an error message to help identify where the issue is, then to have different JVMs running (Java 11 and Java 8). Note: I am assuming that Hadoop is running automatically at system startup, so you will definitely need to remove Java 11.
Then you will need to go thru each of the Hadoop configurations files to double check that JAVA_HOME has not been configured. For Ubuntu this is generally
/usr/local/hadoop/etc/hadoopbut can be anything that the previous admin wanted ($HADOOP_HOME/etc/hadoop). A grep of then entire directly will speed this up. This goes for each add-on product such as Hive, HBase, etc. You may be lucky in that only hadoop-env.sh has JAVA_HOME configured.Also check the init startup scripts in /etc/init.d
After rebooting Hadoop, review the log files. I would also shutdown and restart each service one at a time and review the output.