HBase local cluster failed to startup after shutdown in Java client

2016-07-13T21:49:59

I have a Maven project compiled with Java 7 uses Spring IoC that suppose to test the use in hbase-client jar.

The dependencies are as follows:

<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-client</artifactId>
    <version>1.0.0-cdh5.5.4</version>
</dependency>

<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-server</artifactId>
    <version>1.0.0-cdh5.5.4</version>
</dependency>

I have a test (using JUnit) that creates a local cluster, create a table and load some data, then client is connecting to cluster, execute lookup and perform shutdown to the cluster, then tries to startup the cluster again to check the client's re-connection mechanism.

The problem is, that during the startup of the cluster after being shutdown, it gets an exception, that after long researching, looks like the shutdown process hasn't completed successfully.

Any help to figure out how to shutdown properly the cluster would be great.

The code context:

@RunWith(SpringJUnit4ClassRunner.class)
public class TestHBaseUserOfflineReconnection
{
    @Value("${userTableName}")
    private static String userTableName = "TestTable";

    @Autowired
    @Qualifier("hbaseUserOfflineReconnectDao")
    private UserDao userOfflineReconnectDao;

    @Autowired
    private DemographicBenchmark benchmark;

    private Table htable;

    private static LocalHBaseCluster hbaseCluster;

    private static MiniZooKeeperCluster zooKeeperCluster;

    private static Configuration configuration;

    static Connection conn = null;

    @BeforeClass
    public static void setup() throws IOException, InterruptedException
    {
        // delete the default local folder for that HBase stores its files
        String userName = System.getProperty("user.name");
        FileUtils.deleteDirectory(new File("/tmp/hbase-" + userName));
        initHbase();
    }

    public static void initHbase() throws IOException, InterruptedException
    {
        configuration = HBaseConfiguration.create();
        zooKeeperCluster = new MiniZooKeeperCluster(configuration);
        zooKeeperCluster.setDefaultClientPort(2181);
        zooKeeperCluster.startup(new File("target/zookeepr-" + System.currentTimeMillis()));

        hbaseCluster = new LocalHBaseCluster(configuration, 1);
        hbaseCluster.startup();
    }

    @Before
    public void initeHTable() throws IOException
    {
        configuration.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());

        conn = ConnectionFactory.createConnection(configuration);

        HTableDescriptor table = new HTableDescriptor(TableName.valueOf(userTableName));

        table.addFamily(new HColumnDescriptor("cf"));

        conn.getAdmin().createTable(table);

        htable = conn.getTable(TableName.valueOf(userTableName));
    }

    public static void shutdown() throws IOException
    {
        hbaseCluster.shutdown();
        hbaseCluster.waitOnMaster(0);
        zooKeeperCluster.shutdown();
    }

@Test
    public void testHBaseReconnection() throws IOException, TkException, InterruptedException
    {
      // do some lookups with the client, and all goes well..

      shutdown();
      initHbase(); // HERE's I GET THE EXCEPTION

      // some more code...
      shutdown(); // after test finished, closing the cluster
    }
}

The exception I get:

ERROR 2016-07-13 16:48:03,849 [B.defaultRpcServer.handler=4,queue=1,port=46727] org.apache.hadoop.hbase.master.MasterRpcServices: Region server localhost,44545,1468417682471 reported a fatal error: ABORTING region server localhost,44545,1468417682471: Unhandled: Region server startup failed Cause: java.io.IOException: Region server startup failed at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:2827) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1317) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:852) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=Server already exists! at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:228) at org.apache.hadoop.hbase.metrics.BaseSourceImpl.(BaseSourceImpl.java:75) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.(MetricsRegionServerSourceImpl.java:66) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.(MetricsRegionServerSourceImpl.java:58) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createServer(MetricsRegionServerSourceFactoryImpl.java:46) at org.apache.hadoop.hbase.regionserver.MetricsRegionServer.(MetricsRegionServer.java:38) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1301)

... 2 more

Copyright License:
Author:「Avihoo Mamka」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/38353610/hbase-local-cluster-failed-to-startup-after-shutdown-in-java-client

About “HBase local cluster failed to startup after shutdown in Java client” questions

I have a Maven project compiled with Java 7 uses Spring IoC that suppose to test the use in hbase-client jar. The dependencies are as follows: &lt;dependency&gt; &lt;groupId&gt;org.apache.hba...
I am trying to connect with a securized hbase with kerberos. It is an hbase deployed into a hdp3 cluster. Exactly, I am trying to access with a java client from a host that is outside the cluster. ...
We have installed a small Hadoop cluster with 3 machines. We are using Hortonworks. We use the following code to connect to Hbase from java client. Configuration config = HBaseConfiguration.creat...
I am trying to start to HBase in IDEA by running HMaster class, but I got this error: java.lang.RuntimeException: Failed suppression of fs shutdown hook: Thread[Thread-8,5,main] at org.apache....
I am trying to connect hbase using JAVA HBase Rest Client. However it's giving following error. The Hbase Rest uses kerberos authentication, and so I have created a kerberos ticket and trying to
The official doc shows in detail how to config and start the hbase cluster, but says little about a shutdown. I have two tactics to stop the entire cluster: stop all the region server first and...
I'm running a multi-node cluster; I'm using hadoop-1.0.3(on both), Hbase-0.94.2(on both) and zookeeper-3.4.6(only master) master:192.168.0.1 slave:192.168.0.2 Hbase is not running perfectly and I
I have a remote Java Client using HBase Java API connectin to HBase. My Java client is running on Wondows, and trying to connect to my DEV Hadoop cluster. I added the hbase-site.xml from the hbase/...
I am trying to connect use Java HBase Rest Client to connect to my Hbase Server and it's showing authentication error. If I use the connection object and use scanner API, I can retrive the data, Ho...
I am using HBASE 0.94.8 standalone mode in Ubuntu. Its working fine i am able to do every operations in Hbase-shell. But after i logged of my system its giving following error 15/07/28 15:10:30 E...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.