This is a collection of administrative scripts to find and fix problems with HBase clusters.
These are based on bin/add_table.rb from hbase 0.20.3, with the following exceptions:
- list_regions.rb (from scratch)
This is just a health check - it lists any regions that aren’t listed in .META. by looking for end keys that don’t have a corresponding start key.
/path/to/hbase/bin/hbase org.jruby.Main ./find_missing_regions.rb /hdfs/path/to/table
For example, if you have your hbase data in /hbase/ on HDFS, and you want to find missing regions in the ‘mytable’ table:
/path/to/hbase/bin/hbase org.jruby.Main ./find_missing_regions.rb /hbase/mytable
This does the same as find_missing_regions.rb, however, it additionally:
-
crawls HDFS to find the files corresponding to the missing regions
-
inserts the regions back into .META.
After inserting a missing region, all regions after the missing region are unreachable for a short while (< 40 seconds when we tried this, but your milage may vary), eg if you have the following regions:
-
^-a
-
a-b
-
b-c
-
c-$
But .META. only lists:
-
^-a
-
b-c
-
c-$
After inserting ‘a-b’ again, ‘b-c’ and ‘c-$’ will be unreachable for a short while.
/path/to/hbase/bin/hbase org.jruby.Main ./fix_missing_regions.rb /hdfs/path/to/table
This is just a diagnostic tool; it doesn’t fix the problem. This picks up situations like when the following regions are in .META.:
-
^-a
-
a-c
-
a-b
-
b-c
-
c-$
a-b and b-c overlap with a-c in this case.
/path/to/hbase/bin/hbase org.jruby.Main ./find_overlapping_regions.rb TableName
list_regions.rb can list all regions in a table, their online/offline state, and what server they are on. It also optionally filters by online/offline state.
This reads directly from .META., so is handy for checking that a ‘disable’ has really worked across all the region servers - for a use case, see: web.archiveorange.com/archive/v/gMxNAJ9pszuhbGFmgyE9
/path/to/hbase/bin/hbase org.jruby.Main ./list_regions.rb TableName [all|online|offline]
If none out of all, online, or offline is specified, the default is ‘all’ (unfiltered).
scrape_online_regions.rb lists all the regions that are online in a specified table for a given list of regionserver hosts. The regionservers must be running the web interface on port 60030.
This scrapes the web interface, as we had a situation where .META. said every single region for our table was offline, however, the region servers themselves were still serving some regions - this led to problems after doing an alter. In other words:
- if you want to check what the master thinks, use list_regions.rb - if you want to check what the region servers think, use scrape_online_regions.rb
This is a regular ruby script, not a JRuby script.
./scrape_online_regions.rb MyTableName host1 host2 host3 ...
migrate.rb runs in two modes, client and server. On the master of the HBase to stream from, set up the server:
/path/to/hbase/bin/hbase org.jruby.Main ./migrate.rb -m server -p 2000
On the master of the HBase to stream to, start the client.
/path/to/hbase/bin/hbase org.jruby.Main ./migrate.rb -m client -s server-host.example.com -p 2000 -t 1340755200 -i 3599