1 java. io. IOException: java. io. IOException: java. lang. IllegalArgumentException: offset (0) + length (8) exceed the capacity of the array: 4
This occurs when simple incr operations are performed. The reason is that the int length is vlen = 4 in the put operation. Not applicable to add operations, but can only be changed to long-type vlen = 8.
2 org. apache. hadoop. hbase. client. retriesExhaustedWithDetailsException: Failed 1 action: NotServingRegionException: 1 time, servers with issues: 10. xx. xx.37: 60020, or org. apache. hadoop. hbase. notServingRegionException: Region is not online: These two kinds of errors occur. Regions in Transition occurs in master-status for up to 10 minutes and remains in PENDING_OPEN State, leading to request blocking. Currently, the machine 10. xx. xx.37 is deprecated and runs stably overnight without any congestion caused by split. Suspected machine problems. The Hmaster log shows that this region server keeps open close without any split or flush operations.
The full name of RIT is region in transcation. each hbase master's open or close operation on region inserts a record into the Master's RIT, because the master's operations on region must be atomic, the open and close operations of region are completed with the assistance of Hmaster and region server. so in order to meet the coordination, rollback, and consistency of these operations. hmaster uses the RIT mechanism and the Node status in Zookeeper to ensure operation security and consistency.
OFFLINE, // region is in an offline statePENDING_OPEN, // sent rpc to server to open but has not begunOPENING, // server has begun to open but not yet doneOPEN, // server opened region and updated metaPENDING_CLOSE, // sent rpc to server to close but has not begunCLOSING, // server has begun to close but not yet doneCLOSED, // server closed region and updated metaSPLITTING, // server started split of a regionSPLIT // server completed split of a region
It is further found that the load balance problem region server is repeatedly closed by open, refer to the http://abloz.com/hbase/book.html#regions.arch.assignment to restart the region server normal
Later, region not on line was Thrown during code execution, because "Thrown by a region server if it is sent a request for a region it is not serving ."
Why is an offline region constantly requested? In addition, this error is concentrated on three of the 150 region servers, and the server log will be tracked. The region will be shut down by CloseRegionHandler, and it will be re-opened in about 20 minutes, after it is disabled, the client still requests this closed region?
3. The setting switch does not take effect if it is not written to hbase.
The code is launched at the beginning, and a switch is added. If there is a problem with hbase, the switch is disabled. However, if a problem occurs, the program becomes stuck. Currently, the reason is that the retry Mechanism is constantly extended. The 60-second timeout and 10 retries from 1 to 32 seconds. in case of a problem, the switch is useless.
You need to configure the rpc timeout parameter and retry time to solve it.
4 flush, split, and compact causes stop-the-world
After a long flush split operation occurs, the hbase server cannot respond to the request. You need to adjust the region size and test the number of flush requests.
5. hbase parameter settings
Hbase. regionserver. handler. count
Considering the I/O capability of the sas disk, set it to 50.
Hbase. hregion. memstore. block. multiplier
When the memstore size is a multiplier multiple of hbase. hregion. memstore. flush. size, block read/write is flush. The default value is 2.
6 region server crush
The cause of Regionserver crash is that the connection timeout between Regionserver and zookeeper is caused by a long GC time.
The internal timeout of Zookeeper is as follows:
MinSessionTimeout is measured in milliseconds. The default value is tickTime.
MaxSessionTimeout, in milliseconds, is 20 times tickTime by default.
(TickTime is also a configuration item. Is the minimum time unit of the internal control time logic of the Server)
If the sessionTimeout sent from the Client exceeds the min-max range, the server automatically intercepts it to min or max and creates a Session object for the Client.
The default tickTime is 2 s, that is, the maximum timeout value of the client is 40 s. It is useless to set zookeeper. session. timeout of regionserver to 60 s in a timely manner.
Changes:
- Change the tickTime of the zookeeper cluster to 9 s, and the maximum timeout value is 180 s. At the same time, change zookeeper. session. timeout to 120 s to avoid timeout caused by GC.
- Add the parameter hbase. regionserver. restart. on. zk. expire to true. Modify the parameter to restart regionserver after timeout between regionserver and zookeeper, instead of shutting down regionserver.
7. deadlock caused by code Problems
A master slow query log has been queried for 2 hours, resulting in slow server response and failure to cope with large writes. The reason for investigation is that the getColumns operation extracts tens of thousands of data records without paging. There are currently no problems with the change of the program paging by about 500 records.
8 operation too slow
2012-07-26 05:30:39, 141 WARN org. apache. hadoop. ipc. HBaseServer: (operationTooSlow): {"processingtimems": 69315, "ts": 9223372036854775807, "client": "10.75.0.109: 34780", "starttimems": 1343251769825, "queuetimems ": 0, "class": "HRegionServer", "responsesize": 0, "method": "delete", "totalColumns": 1, "table": "trackurl_status_list ", "families": {"sl": [{"timestamp": 1343251769825, "qualifier": "zzzn1VlyG", "vlen": 0}]}, "row ": ""}
It takes 69315 s to delete a row of data.
In addition, the magic is that the row is "", and the row cannot be set to null, but an empty string can be added. A round of tests were conducted.
It takes 700 ms to delete a column that does not exist with an empty row-key.
It takes 5 ms to delete an existing column with an empty row-key.
It takes 3 ms for non-empty row-key to delete any column
It is not clear whether it is a bug or how to transfer an empty row-key. The current policy is to avoid null row-key operations on the code end.
9 responseTooSlow
2012-07-31 17:52:06,619 WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): {"processingtimems":1156438,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3dbb29e5), rpc version=1, client version=29, methodsFingerPrint=-1508511443","client":"10.75.0.109:35245","starttimems":1343727170177,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
Description of reference to hbase: The output is tagged with operation e.g.(operationTooSlow)If the call was a client operation, such as a Put, Get, or Delete, which we expose detailed fingerprint information for. If not, it is tagged(responseTooSlow)And still produces parseable JSON output, but with less verbose information solely regarding its duration and size in the RPC itself.
Currently, the batch delete operation on multiple columns of a key is canceled to avoid blocking and no new problems are found.
10 output error
2012-07-31 17:52:06, 812 WARN org. apache. hadoop. ipc. HBaseServer: IPC Server Responder, call get ([B @ 61574be4, {"timeRange": [0, 9223372010954775807], "totalColumns": 1, "cacheBlocks": true, "families ": {"c": ["ALL"]}, "maxVersions": 1, "row": "zOuu6TK"}), rpc version = 1, client version = 29, methodsFingerPrint =-1508511443 from 10.75.0.151: 52745: output error
11 rollbackMemstore Problems
This log is frequently used:
10:21:49, 887 DEBUG org. apache. hadoop. hbase. regionserver. HRegion: rollbackMemstore rolled back 0 keyvalues from start: 0 to end: 0
The method is interpreted as: Remove all the keys listed in the map from the memstore. this method is called when a Put has updated memstore but subequently fails to update the wal. this method is then invoked to rollback the memstore.
It is strange that the start and end indexes are both 0.
Loop in the method: for (int I = start; I <end; I ++ ){
Therefore, it is empty data and empty rollback. Further investigation required
12 a new region server is launched, resulting in region not on line
Request region from the wrong region server
13 The requested region does not exist. reestablishing the tablepool does not work.
Request timestamp 1342510667
Latest region rowkey related timestamp 1344558957
It is found that the attribute for maintaining the region location table is in HConnectionManager.
Get Get, delete Delete, incr Increment is processed in the ServerCallable class withRetries
Scenario 1 if an error occurs (SocketTimeoutException ConnectException RetriesExhaustedExcetion), The regionServer location is cleared.
Scenario 2 numRetries if set to 1, the loop is executed only once, connect (tries! = 0) connect (false), that is, reload = false. location update is not performed. If it is numRetries> 1, it will be retrieved again.
Get Gets List, put Put or Puts List, delete Deletes List, call processBatch of HConnectionManager for processing. If the batch get or put or delete operation results are found to be faulty, refresh regionServer location
Set numRetries to> 1 time. Here I am going to solve the problem three times.
14 zookeeper. RecoverableZooKeeper (195): Possibly transient ZooKeeper exception: org. apache. zookeeper. KeeperException $ ConnectionLossException: keeperrorcode = ConnectionLoss for/hbase/master
This occurs in my standalone test. Whether it is to start hbase from ide or bin, it can be normally connected from shell and cannot be connected from the test program. The zookeeper port is 2181, the client port must be independent of zookeeper,
In the end, the configuration of port 21818 is changed to port 2181, which is running normally. This change must be performed in a single-host environment.
<property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> <description>Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect. </description> </property>