Upgrade Phoenix4.7 to 4.12 problems encountered

Recently due to the business need to use the Phoenix Spark module, but Phoenix4.10 the following version and Spark2.1.1 incompatible, so plan to upgrade Phoenix.
Download Address:
http://archive.apache.org/dist/phoenix/apache-phoenix-4.12.0-HBase-1.1/bin/ Apache-phoenix-4.12.0-hbase-1.1-bin.tar.gz
After decompression, modify the soft connection of the previous Phoenix to the version 4.12 folder, then paste the Phoenix-4.12.0-hbase-1.1-server.jar under the folder into the Lib directory under the HBase path of each node and restart HBase. According to reason, the next thing is all right, but things are not as simple as imagined, first Phoenix sqlline the can ' t find locations exception, and then start the HBase shell, execute command appears

ERROR:org.apache.hadoop.hbase.PleaseHoldException:Master is initializing at
        Org.apache.hadoop.hbase.master.HMaster.checkInitialized (hmaster.java:2177) at
        Org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors (masterrpcservices.java:821) at
        Org.apache.hadoop.hbase.protobuf.generated.masterprotos$masterservice$2.callblockingmethod (MasterProtos.java : 48468) at
        Org.apache.hadoop.hbase.ipc.RpcServer.call (rpcserver.java:2112)
        at Org.apache.hadoop.hbase.ipc.CallRunner.run (callrunner.java:101) at
        Org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop (rpcexecutor.java:130) at
        Org.apache.hadoop.hbase.ipc.rpcexecutor$1.run (rpcexecutor.java:107) at
        Java.lang.Thread.run (thread.java:745 )

Next is the process of troubleshooting, the monitoring tool to see the state of hmaster is healthy, but all the Regionserver node health is red, and 16020 and 16030 ports are inaccessible, so went to look at the Regionserver log, The discovery reported

Java.lang.ClassNotFoundException:org.apache.hadoop.hbase.regionserver.LocalIndexMerger at Java.net.URLClassLoade R.findclass (urlclassloader.java:381) at Java.lang.ClassLoader.loadClass (classloader.java:424) at Sun.misc.
        Launcher$appclassloader.loadclass (launcher.java:331) at Java.lang.ClassLoader.loadClass (classloader.java:357) At Org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors (coprocessorhost.java:155) at org .apache.hadoop.hbase.regionserver.regionservercoprocessorhost.<init> (regionservercoprocessorhost.java:68 ) at Org.apache.hadoop.hbase.regionserver.HRegionServer.run (hregionserver.java:893) at java.lang.Thread.ru N (thread.java:745) Java.lang.RuntimeException:HRegionServer aborted at Org.apache.hadoop.hbase.regionserver.HReg Ionservercommandline.start (hregionservercommandline.java:68) at Org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run (Hregionservercommandline.java:87) at Org.apache.hadoop.util.ToolRunner.run (toolrunner.java:70) at Org.apache.hadoop.hbase.util. Servercommandline.domain (servercommandline.java:126)

After finding related questions on the Internet, see the release note on the official website

Phoenix-4.8.0 Release Notes PHOENIX-3164 is a relatively serious bug this affects the Query Server deployed with "security Enabled (Kerberos or Active Directory). Due to another late-game change in the 4.8.0 release as a issue with the use of Hadoop ' s usergroupinformation CLA SS, every "client session" to the Phoenix Query Server with security enabled would result in a new instance of the Phoenix JDBC driver Phoenixconnection (and other related classes).

This ultimately results in a new connection to ZooKeeper for each "client session". Within a short amount of time of active use with the Phoenix Query Server Creating a new ZooKeeper connection for each "cl Ient session ", the number of ZooKeeper connections would have grown rapidly likely triggering ZooKeeper ' s built-in denial o F Service Protection (MAXCLIENTCNXNS). This would cause all future connections to ZooKeeper by the host running the Phoenix Query Server to be dropped. This would prevent all HBase client API calls which nEed to access ZooKeeper from completing.  As part of PHOENIX-1734 we have changed the local index implementation to store index data in the separate column families In the same data table. So while upgrading, Phoenix at server, we need to remove below local index related configurations from Hbase-site.xml an D Run upgrade steps mentioned here <property> <name>hbase.master.loadbalancer.class</name> <valu
  E>org.apache.phoenix.hbase.index.balancer.indexloadbalancer</value> </property> <property> <name>hbase.coprocessor.master.classes</name> <value> Org.apache.phoenix.hbase.index.master.indexmasterobserver</value> </property> <property> < Name>hbase.coprocessor.regionserver.classes</name> <value> Org.apache.hadoop.hbase.regionserver.localindexmerger</value> </property>

After the 4.8 release, installing Phoenix requires the deletion of these three configuration items in the Hbase-site.xml file, since the three configuration items in the 4.7 and previous versions will result in the "security enabled" case, each build and Phoenix Server session, a new connection instance is created, and creating a large number of connections within a short period of time triggers the zookeeper built-in protection mechanism and denies the connection.

In short, HBase is assigned to HBase when it restarts. The Phoenix-4.12.0-hbase-1.1-server.jar package in the Lib directory reads the Hbase-site.xml file, which results in the discovery of these three outdated configuration items and then causes the exception to be removed from the three configuration items.