Home / server / Questions / 609893
In Process
Dolan Antenucci
Asked: 2014-07-04 06:53:07 +0800 CST

HBase issue with *some* regionservers connecting

  • 772

I have an HBase cluster that is working, and I'm attempting to add some new servers to the cluster, but "SocketException: Invalid argument" and "FailedServerException: This server is in the failed servers list" errors keep getting generated in the logs.

2014-07-02 22:28:01,140 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.SocketException: Invalid argument
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupConnection(HBaseClient.java:392)
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:438)
    at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1141)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:988)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
    at com.sun.proxy.$Proxy10.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:141)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2040)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2086)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:748)
    at java.lang.Thread.run(Thread.java:701)
2014-07-02 22:28:31,764 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: <MY_MASTER_SERVER>/<MY_MASTER_NAME>:<MY_MASTER_PORT>
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:427)
    at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1141)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:988)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
    at com.sun.proxy.$Proxy10.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:141)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2040)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2086)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:748)
    at java.lang.Thread.run(Thread.java:701)

So far I can't find any differences between the old and new servers:

  • both running Ubuntu 12.04 with all the latest updates and Cloudera's CDH4 for HBase
  • neither have /etc/hosts have entries for master HBase (although I tried adding one on new servers, but still having same issue)
  • firewalls should be configured the same without any local network restrictions (NOTE: on new servers, I can telnet to port 60000, my HBase Master's port, without any errors)

While debugging this, I saw a mention online about IPv6 configuration possibly causing issue, but as far as I know, both old and new servers have whatever default configurations Ubuntu uses for this.

Any ideas on how I can debug further and/or what the issue may be?

ubuntu

0 Answers