Home / ubuntu / Questions

Questions[hive](ubuntu)

Martin Hope
SriniShine
Asked: 2018-09-08 02:04:44 +0800 CST
  • 4

I'm trying to run Hive 3.1 with Hadoop 3.0. Following is my system configuration:

Ubuntu 18.04.1 LTS
Hadoop  version 3.0.3
Hive 3.1.0
Derby 10.14.2

When I execute the show tables; query I get the following error.

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

Following is the detail error in the hive log file.

2018-09-05T11:38:25,952  INFO [main] conf.HiveConf: Found configuration file file:/usr/local/apache-hive-3.1.0-bin/conf/hive-site.xml
2018-09-05T11:38:30,549  INFO [main] SessionState: Hive Session ID = 826ec55c-7fca-4fff-baa5-b5a010e5af89
2018-09-05T11:38:35,948  INFO [main] SessionState:
Logging initialized using configuration in jar:file:/usr/local/apache-hive-3.1.0-bin/lib/hive-common-3.1.0.jar!/hive-log4j2.properties Asy$
2018-09-05T11:38:47,015  INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/hadoop
2018-09-05T11:38:47,069  INFO [main] session.SessionState: Created local directory: /tmp/mydir
2018-09-05T11:38:47,096  INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/hadoop/826ec55c-7fca-4fff-baa5-b5a010e5af89
2018-09-05T11:38:47,104  INFO [main] session.SessionState: Created local directory: /tmp/mydir/826ec55c-7fca-4fff-baa5-b5a010e5af89
2018-09-05T11:38:47,122  INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/hadoop/826ec55c-7fca-4fff-baa5-b5a010e5af89/_$
2018-09-05T11:38:47,125  INFO [main] conf.HiveConf: Using the default value passed in for log id: 826ec55c-7fca-4fff-baa5-b5a010e5af89
2018-09-05T11:38:47,126  INFO [main] session.SessionState: Updating thread name to 826ec55c-7fca-4fff-baa5-b5a010e5af89 main
2018-09-05T11:38:50,476  INFO [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] metastore.HiveMetaStore: 0: Opening raw store with implementatio$
2018-09-05T11:38:50,695  WARN [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set$
2018-09-05T11:38:50,714  INFO [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] metastore.ObjectStore: ObjectStore, initialize called
2018-09-05T11:38:50,717  INFO [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] conf.MetastoreConf: Found configuration file file:/usr/local/apa$
2018-09-05T11:38:50,719  INFO [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] conf.MetastoreConf: Unable to find config file hivemetastore-sit$
2018-09-05T11:38:50,720  INFO [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] conf.MetastoreConf: Found configuration file null
2018-09-05T11:38:50,722  INFO [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] conf.MetastoreConf: Unable to find config file metastore-site.xml
2018-09-05T11:38:50,722  INFO [826ec55c-7fca-4fff-baa5-b5a010e5af89 main] conf.MetastoreConf: Found configuration file null

hive-site.xml

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:derby:;databaseName=metastore_db;create=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
  </property>

Environment variables in .profile (I am trying to configure an installation done by someone else so the environment variables are set up in .profile instead of .bashrc even though Hadoop is run manually)

#HIVE
export HIVE_HOME=/usr/local/apache-hive-3.1.0-bin
export HIVE_CONF_DIR=/usr/local/apache-hive-3.1.0-bin/conf
export PATH=$HIVE_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:/usr/local/hadoop/lib/*:.
export CLASSPATH=$CLASSPATH:/usr/local/apache-hive-3.1.0-bin/lib/*:.

#DERBY
DERBY_HOME=/usr/local/db-derby-10.14.2.0-bin
export PATH=$PATH:$DERBY_HOME/bin
export CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derby.jar:$DERBY_HOME/lib/derbytool$

The error messages indicates about a metastore-site.xml file which I did not come across the configuration of hive.

18.04 hadoop hive
Martin Hope
SriniShine
Asked: 2018-08-28 15:17:33 +0800 CST
  • 2

I'm working on a Hadoop installation done by someone else. 

Ubuntu 18.04.1 LTS
Hadoop version 3.0.3
Java open jdk 11 (auto mode) and 8 (manual mode)

Currently Hadoop runs with Java 11 but I want to configure Hadoop to run with Java 8 because Hive doesn't work with Java 11.

Following is the Java configuration in my virtual machine. 

hadoop@hadoop-VirtualBox:~$ whereis java
java: /usr/bin/java /usr/share/java /usr/share/man/man1/java.1.gz

hadoop@hadoop-VirtualBox:~$ update-alternatives --list java
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

hadoop@hadoop-VirtualBox:~$ java -version
openjdk version "10.0.2" 2018-07-17
OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1)
OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1, mixed mode)

hadoop@hadoop-VirtualBox:~$ sudo update-alternatives --config java

There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1101      auto mode
  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1101      manual mode
  2            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode


hadoop@hadoop-VirtualBox:~$ ls -lh /usr/lib/jvm/
total 8.0K
lrwxrwxrwx 1 root root   25 Apr  8 14:16 default-java -> java-1.11.0-openjdk-amd64
lrwxrwxrwx 1 root root   21 Apr 21 14:41 java-1.11.0-openjdk-amd64 -> java-11-openjdk-amd64
drwxr-xr-x 9 root root 4.0K Aug 23 12:28 java-11-openjdk-amd64
lrwxrwxrwx 1 root root   20 Apr 26 16:59 java-1.8.0-openjdk-amd64 -> java-8-openjdk-amd64
drwxr-xr-x 7 root root 4.0K Aug 17 08:24 java-8-openjdk-amd64

I was looking at this thread to configure Java however in the .bashrc or /etc/environment file there is no entry on JAVA_HOME. The only place JAVA_HOME is specified is in the /usr/local/hadoop/etc/hadoop/hadoop-env.sh file. In the file JAVA_HOME is specified as follows:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

With the above configuration Hadoop runs without any problem. I tried changing the JAVA_HOME in the hadoop-env.sh file to /usr/lib/jvm/java-8-openjdk-amd64 and tried running Hadoop. But this gives an error

Unrecognized option: --add-modules Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

I tried changing the priority of the Java version using sudo update-alternatives --config java along with the hadoop-env.sh alteration but it still gives the same error.

In order to change the Java version do I need to add the path to .bashrc and /etc/environment as well? I would like to know what changed needs to be done.

java 18.04 hadoop hive
Martin Hope
Hari Krishna K
Asked: 2018-07-17 04:47:50 +0800 CST
  • 0

I am trying to load my MySQL tableLab5 to Hive using Sqoop. Sqoop ran successfully and has taken the data from mysql into hdfs. But throwing the below bolded error about HIVE_CONF_DIR not set. I have set the HIVE_CONF_DIR in ~/.bashrc and ~/.profile as:

export HIVE_HOME=/usr/local/hive
export HIVE_CONF_DIR=$HIVE_HOME/conf

In spite of doing that I am facing the below HIVE_CONF_DIR error. Can you please help me out with this error ASAP? 

18/07/16 13:27:38 **INFO mapreduce.Job: Job job_1531743854195_0001 completed successfully**
18/07/16 13:27:38 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=566628
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=408
        HDFS: Number of bytes written=179248
        HDFS: Number of read operations=16
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=8
    Job Counters 
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=12437
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=12437
        Total vcore-milliseconds taken by all map tasks=12437
        Total megabyte-milliseconds taken by all map tasks=12735488
    Map-Reduce Framework
        Map input records=1000
        Map output records=1000
        Input split bytes=408
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=338
        CPU time spent (ms)=3300
        Physical memory (bytes) snapshot=713752576
        Virtual memory (bytes) snapshot=7925747712
        Total committed heap usage (bytes)=458752000
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=179248
18/07/16 13:27:38 INFO mapreduce.ImportJobBase: Transferred 175.0469 KB in 18.8594 seconds (9.2817 KB/sec)
18/07/16 13:27:38 INFO mapreduce.ImportJobBase: Retrieved 1000 records.
18/07/16 13:27:38 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table tableLab5
Mon Jul 16 13:27:38 IST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
18/07/16 13:27:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tableLab5` AS t LIMIT 1
18/07/16 13:27:38 INFO hive.HiveImport: Loading uploaded data into Hive
18/07/16 13:27:38 **ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.**
18/07/16 13:27:38 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
    at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)
    at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)
    at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)
    at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
    at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)
    ... 12 more
mysql hadoop hive
Martin Hope
Aashish Kumar
Asked: 2018-07-11 10:34:43 +0800 CST
  • 0

I installed Hadoop successfully and it's working fine, but when I installed Hive and ran the hive command in terminal, I got this error

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 64
    at java.util.jar.JarFile.match(java.base@9-internal/JarFile.java:983)
    at java.util.jar.JarFile.checkForSpecialAttributes(java.base@9-internal/JarFile.java:1017)
    at java.util.jar.JarFile.isMultiRelease(java.base@9-internal/JarFile.java:399)
    at java.util.jar.JarFile.getEntry(java.base@9-internal/JarFile.java:524)
    at java.util.jar.JarFile.getJarEntry(java.base@9-internal/JarFile.java:480)
    at jdk.internal.util.jar.JarIndex.getJarIndex(java.base@9-internal/JarIndex.java:114)
    at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:640)
    at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:632)
    at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
    at jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(java.base@9-internal/URLClassPath.java:631)
    at jdk.internal.loader.URLClassPath$JarLoader.<init>(java.base@9-internal/URLClassPath.java:606)
    at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:386)
    at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:376)
    at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
    at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:375)
    at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:352)
    at jdk.internal.loader.URLClassPath.getResource(java.base@9-internal/URLClassPath.java:218)
    at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:463)
    at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:460)
    at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
    at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@9-internal/BuiltinClassLoader.java:459)
    at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@9-internal/BuiltinClassLoader.java:406)
    at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:364)
    at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:184)
    at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419)
    at sun.launcher.LauncherHelper.loadMainClass(java.base@9-internal/LauncherHelper.java:585)
    at sun.launcher.LauncherHelper.checkAndLoadMain(java.base@9-internal/LauncherHelper.java:497)
Unable to determine Hadoop version information.
'hadoop version' returned:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 64
    at java.util.jar.JarFile.match(java.base@9-internal/JarFile.java:983)
    at java.util.jar.JarFile.checkForSpecialAttributes(java.base@9-internal/JarFile.java:1017)
    at java.util.jar.JarFile.isMultiRelease(java.base@9-internal/JarFile.java:399)
    at java.util.jar.JarFile.getEntry(java.base@9-internal/JarFile.java:524)
    at java.util.jar.JarFile.getJarEntry(java.base@9-internal/JarFile.java:480)
    at jdk.internal.util.jar.JarIndex.getJarIndex(java.base@9-internal/JarIndex.java:114)
    at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:640)
    at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:632)
    at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
    at jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(java.base@9-internal/URLClassPath.java:631)
    at jdk.internal.loader.URLClassPath$JarLoader.<init>(java.base@9-internal/URLClassPath.java:606)
    at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:386)
    at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:376)
    at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
    at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:375)
    at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:352)
    at jdk.internal.loader.URLClassPath.getResource(java.base@9-internal/URLClassPath.java:218)
    at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:463)
    at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:460)
    at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
    at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@9-internal/BuiltinClassLoader.java:459)
    at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@9-internal/BuiltinClassLoader.java:406)
    at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:364)
    at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:184)
    at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419)
    at sun.launcher.LauncherHelper.loadMainClass(java.base@9-internal/LauncherHelper.java:585)
    at sun.launcher.LauncherHelper.checkAndLoadMain(java.base@9-internal/LauncherHelper.java:497)

My .bashrc file is:

#Hadoop variables
export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
#end of Hadoop variable declaration


#HIVE variables
export HIVE_HOME=/usr/lib/hive
export HIVE_CONF_DIR=/usr/lib/hive/conf
export PATH=$PATH:$HIVE_HOME/bin
export CLASSPATH=$CLASSPATH:/usr/local/hadoop/lib/*:
export CLASSPATH=$CLASSPATH:/usr/local/hive/lib/*:

I've been stuck with it for a week now, unable to find working solution.

apache2 java hadoop hive