solrCloud 4.9 分布式叢集部署及注意事項

標籤：des   style   blog   http   color   java   使用   os   

環境搭建

一.zookeeper

參考：http://blog.chinaunix.net/uid-25135004-id-4214399.html

現有4台機器 10.14.2.201 10.14.2.202 10.14.2.203 10.14.2.204安裝zookeeper叢集 在所有機器上進行1、下載安裝包解壓 tar xvf zookeeper-3.4.5.tar.gz  -C /export/ cd /export/ ln -s zookeeper-3.4.5 zookeeper2、設定環境變數 vim /etc/profile#Set ZooKeeper Enviromentexport ZOOKEEPER_HOME=/export/zookeeperexport PATH=$PATH:$ZOOKEEPER_HOME/bin3、載入環境變數 source   /etc/profile4、配置zookeepercd /export/zookeeper/confcp zoo_sample.cfg  zoo.cfgvim zoo.cfg tickTime=2000initLimit=10syncLimit=5dataDir=/export/zookeeper/datadataLogDir=/export/zookeeper/logclientPort=2181server.1=10.14.2.201:2888:3888server.2=10.14.2.202:2888:3888server.3=10.14.2.203:2888:3888server.4=10.14.2.204:2888:38885、dataDir和日誌目錄dataLogDirmkdir -p /export/zookeeper/datamkdir -p /export/zookeeper/log6、在10.14.2.201上在dataDir目錄下建立myid檔案echo 1 > /export/zookeeper/data/myid myid中的數字與zoo.cfg中的"server.數字=ip或者網域名稱:port1:port2"相對應，設定檔中10.14.2.201對應的server.數字是1 所以myid的數字應為17、在10.14.2.202上在dataDir目錄下建立myid檔案echo 2 > /export/zookeeper/data/myid 8、在10.14.2.203上在dataDir目錄下建立myid檔案echo 3 > /export/zookeeper/data/myid 9、在10.14.2.204上在dataDir目錄下建立myid檔案echo 4 > /export/zookeeper/data/myid 在所用機器上執行10、修改許可權chown hadoop.hadoop -R /export/zookeeper/chown hadoop.hadoop -R /export/zookeeper11、啟動zookeeper/export/zookeeper/bin/zkServer.sh start12、觀察zookeeper狀態   1> 10.14.2.201   # /export/zookeeper/bin/zkServer.sh status   JMX enabled by default   Using config: /export/zookeeper/bin/../conf/zoo.cfg   Mode: follower   2> 10.14.2.202   # /export/zookeeper/bin/zkServer.sh status   JMX enabled by default   Using config: /export/zookeeper/bin/../conf/zoo.cfg   Mode: follower    3> 10.14.2.203   # /export/zookeeper/bin/zkServer.sh status  JMX enabled by default  Using config: /export/zookeeper/bin/../conf/zoo.cfg  Mode: leader    4> 10.14.2.204   # /export/zookeeper/bin/zkServer.sh status   JMX enabled by default   Using config: /export/zookeeper/bin/../conf/zoo.cfg   Mode: follower   13、此時zookeeper已安裝並啟動完畢

　　

二.solrCloud

參考：http://lucien-zzy.iteye.com/blog/2002463

1.安裝tomcat，配置連接埠。部署solr.war，將solr\example\lib\ext下的jar拷貝到solr\WEB-INF\lib下。

2.建立

/home/solr/solrcloud

/home/solr/solrcloud/config-files (放置solr\example\solr\collection1\conf 下的所有檔案)

/home/solr/solrcloud/solr-lib (放置solr\WEB-INF\lib下的所有jar包)

3.配置中文分詞器

4.建立solr資料目錄

/home/solr/solr-cores

產生solr.xml，如下：

<?xml version="1.0" encoding="UTF-8" ?><solr persistent="true"><logging enabled="true"><watcher size="100" threshold="INFO" /></logging><cores defaultCoreName="collection1" adminPath="/admin/cores" host="${host:}" hostPort="8080" hostContext="${hostContext:solr}" zkClientTimeout="${zkClientTimeout:15000}"></cores></solr>

5.建立

tomcat/conf/Catalina

tomcat/conf/Catalina/localhost

並建立solr.xml

<?xml version="1.0" encoding="UTF-8"?><Context docBase="/soft/apache-tomcat-7.0.37/webapps/solr" debug="0" crossContext="true"><Environment name="solr/home" type="java.lang.String" value="/home/solr/solr-cores" override="true"/></Context>

docBase根據實際情況配置；

solr/home為之前建立的目錄：/home/solr/solr-cores

6.修改tomcat/bin/cataina.sh 檔案，在最上方加入：JAVA_OPTS="-DzkHost=master:2181,slave1:2181,slave2:2181"，或直接使用ip。

將以上配置發布到叢集其他機器上。

7.SolrCloud是通過ZooKeeper叢集來保證設定檔的變更及時同步到各個節點上，所以，需要將設定檔上傳到ZooKeeper叢集中。執行如下操作：

java -classpath .:/home/solr/solrcloud/solr-lib/* org.apache.solr.cloud.ZkCLI -cmd upconfig -zkhost 192.168.91.128:2181,192.168.91.129:2181,192.168.91.130:2181 -confdir /home/solr/solrcloud/config-files/ -confname myconf  

-cmd upconfig:上傳設定檔

-confdir:設定檔的目錄

-confname:指定對應的名稱

java -classpath .:/home/solr/solrcloud/solr-lib/* org.apache.solr.cloud.ZkCLI -cmd linkconfig -collection collection1 -confname myconf -zkhost 192.168.91.128:2181,192.168.91.129:2181,192.168.91.130:2181

-cmd linkconfig:為指定collection"綁定"設定檔

-collection

-confname

8.檢查zookeeper的配置資訊

zkCli.sh -server ip:port

ls /configs/myconf

ls /live_nodes

9.啟動tomcat，建立collection

curl ‘http://192.168.91.128:8080/solr/admin/collections?action=CREATE&name=mycollection&numShards=3&replicationFactor=1‘

action：CREATE

name：collection名稱

numShards：分區數量

replicationFactor：複本集數量

也可以通過zk查詢collection的情況：ls /collections/mycollection

 

-------------------------------

1.ZK CLI

ZooKeeper‘s Command Line Utility (CLI).

zkcli Parameters

Short	Parameter Usage	Meaning
	-cmd <arg>	CLI Command to be executed: bootstrap, upconfig, downconfig, linkconfig, makepath, get, getfile, put,putfile, list or clear.  This parameter is mandatory
-z	-zkhost <locations>	ZooKeeper host address.  This parameter is mandatory for all CLI commands.
-c	-collection <name>	For linkconfig: name of the collection.
-d	-confdir <path>	For upconfig: a directory of configuration files.
-h	-help	Display help text.
-n	-confname <arg>	For upconfig, linkconfig: name of the configuration set.
-r	-runzk <port>	Run ZooKeeper internally by passing the Solr run port; only for clusters on one machine.
-s	-solrhome <path>	For bootstrap or when using -runzk: the mandatory solrhome location.

ZooKeeper CLI Examples

Below are some examples of using the zkcli CLI:
Uploading a Configuration Directory

java -classpath example/solr-webapp/WEB-INF/lib/*org.apache.solr.cloud.ZkCLI -cmd upconfig -zkhost 127.0.0.1:9983-confdir example/solr/collection1/conf -confname conf1 -solrhome example/solr

Put arbitrary data into a new ZK file

java -classpath example/solr-webapp/WEB-INF/lib/*org.apache.solr.cloud.ZkCLI -zkhost 127.0.0.1:9983 -put /data.txt ‘some data‘

Put a local file into a new ZK file

java -classpath example/solr-webapp/WEB-INF/lib/*org.apache.solr.cloud.ZkCLI -zkhost 127.0.0.1:9983 -putfile /data.txt /some/local/file.txt

Linking a Collection to a Configuration Set

java -classpath example/solr-webapp/webapp/WEB-INF/lib/*org.apache.solr.cloud.ZkCLI -cmd linkconfig -zkhost 127.0.0.1:9983-collection collection1 -confname conf1 -solrhome example/solr

Bootstrapping All the Configuration Directories in solr.xml

java -classpath example/solr-webapp/webapp/WEB-INF/lib/*org.apache.solr.cloud.ZkCLI -cmd bootstrap -zkhost 127.0.0.1:9983-solrhome example/solr

Scripts

There are scripts in example/cloud-scripts that handle the classpath and class name for you if you are using Solr out of the box with Jetty. Commands then become:

sh zkcli.sh -cmd linkconfig -zkhost 127.0.0.1:9983-collection collection1 -confname conf1 -solrhome example/solr

 

 

2.Managing collections via the collections API

CREATE---

http://localhost:8983/solr/admin/collections?action=CREATE&name=mycollection&numShards=3&replicationFactor=4

name：collection的名稱

numShards：指定分區數量（slices）

replicationFactor：副本數量

maxShardsPerNode：預設值為1，注意三個數值：numShards、replicationFactor、liveSolrNode，一個正常的solrCloud叢集不容許同一個liveSolrNode上部署同一個shard的多個replic，因此當maxShardsPerNode=1時，numShards*replicationFactor>liveSolrNode時，報錯。因此正確時因滿足以下條件：
numShards*replicationFactor<liveSolrNode*maxShardsPerNode

createNodeSet：

collection.configName：指定該collection使用那份config，這份config必須存在於zk中。

DELETE---

http://localhost:8983/solr/admin/collections?action=DELETE&name=mycollection

RELOAD---

http://localhost:8983/solr/admin/collections?action=RELOAD&name=mycollection

SPLIT SHARD---

http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=<collection_name>&shard=shardId

 

3.Creating cores via CoreAdmin

curl ‘http://localhost:8983/solr/admin/cores?action=CREATE&name=mycore&collection=collection1&shard=shard2‘

 

 

-------------------------------

Q1:Multiple schemas in the same SolrCloud ?

java -classpath .:/home/solrcloud/solr-lib/* org.apache.solr.cloud.ZkCLI -cmd upconfig -zkhost 10.200.51.190:2224,10.200.51.192:2224,10.200.51.194:2224 -confdir /home/solrcloud/fox-config-files/ -confname foxconf

指定設定檔，上傳到zk。

curl ‘http://10.200.51.194:2223/solr/admin/collections?action=CREATE&name=fox4&numShards=1&replicationFactor=5&collection.configName=foxconf‘

為建立的collection指定collection.configName。