Druid Timing Database Upgrade process

Druid upgrade process

Currently the Druid cluster version is 0.11.0, the new version 0.12.1 has support for Druid SQL and Redis, taking into account the Druid new features and performance improvements, Therefore, you need to upgrade Druid from 0.11.0 to 0.12.1, the following will be a detailed description of the Druid upgrade step, please follow this step to upgrade strictly in order to avoid some unpredictable problems.

1. Druid Upgrade Package

Druid official website Download druid-0.12.1-bin.tar.gz and mysql-metadata-storage-0.12.1.tar.gz

2. Configure Druid-0.12.1

• Decompression druid-0.12.1-bin.tar.gz

[[email protected]]$ tar -zxvf druid-0.12.1-bin.tar.gz[[email protected]]$ rm -rf druid-0.12.1-bin.tar.gz

• Decompression mysql-metadata-storage-0.12.1.tar.gz

[[email protected]]$ tar -zxvf mysql-metadata-storage-0.12.1.tar.gz -C druid-0.12.1/extensions/[[email protected]]$ rm -rf mysql-metadata-storage-0.12.1.tar.gz

3. Configure Common.runtime.properties

[[email protected] druid-0.11.0]$ cd conf/druid/_common[[email protected] _common]$ vi common.runtime.properties

# If you specify ' druid.extensions.loadlist=[] ', Druid won ' t load a extension from file system.# If you don ' t specify ' d Ruid.extensions.loadList ', Druid would load all the extensions under root extension directory.# more info:http://druid.io/ docs/latest/operations/including-extensions.htmldruid.extensions.loadlist=["Druid-kafka-eight", " Druid-hdfs-storage "," Druid-histogram "," druid-datasketches "," Druid-lookups-cached-global "," Mysql-metadata-storage "]# If you had a different version of Hadoop, place your Hadoop client jar files in your HADOOP-DEP Endencies directory# and uncomment the line below to point to your directory. #druid. extensions.hadoopdependenciesdir=/my/ dir/hadoop-dependencies## logging## Log All runtime properties on startup. Disable to avoid logging properties on startup:druid.startup.logging.logproperties=true## zookeeper# druid.zk.service.host=172.16.xxx.xxx:2181druid.zk.paths.base=/druid## Metadata storage## for Derby server on your Druid Coordinator (only viable in A cluster with a single coordinator, no Fail-over): #druid. metadata.storage.type=derby# druid.metadata.storage.connector.connecturi=jdbc:derby://localhost:1527/var/druid/metadata.db;create=true# druid.metadata.storage.connector.host=localhost#druid.metadata.storage.connector.port=1527# for MySQL: druid.metadata.storage.type=mysqldruid.metadata.storage.connector.connecturi=jdbc:mysql://172.16.xxx.xxx:3306/ druiddruid.metadata.storage.connector.user=rootdruid.metadata.storage.connector.password=123456# for PostgreSQL: #druid. metadata.storage.type=postgresql#druid.metadata.storage.connector.connecturi=jdbc:postgresql:// db.example.com:5432/druid#druid.metadata.storage.connector.user=. #druid. metadata.storage.connector.password= ... # # # deep storage## for local disk (with viable in a cluster if the is a network mount): #druid. Storage.type=local#druid. storage.storagedirectory=var/druid/segments# for hdfs:druid.storage.type=hdfsdruid.storage.storagedirectory=/ druid/segments# for S3: #druid. storage.type=s3#druid.storage.bucket=your-bucket#druid.storage.basekey=druid/segments#druid.s3.accesskey=...# druid.s3.secretkey=...## Indexing Service logs## for local disk (with viable in a cluster if this is a network mount): #dru id.indexer.logs.type=file#druid.indexer.logs.directory=var/druid/indexing-logs# for HDFS:druid.indexer.logs.type =hdfsdruid.indexer.logs.directory=/druid/indexing-logs# for S3: #druid. indexer.logs.type=s3# druid.indexer.logs.s3bucket=your-bucket#druid.indexer.logs.s3prefix=druid/indexing-logs## Service discovery# druid.selectors.indexing.servicename=druid/overlorddruid.selectors.coordinator.servicename=druid/coordinator## monitoring#druid.monitoring.monitors=["Io.druid.java.util.metrics.JvmMonitor"]druid.emitter= loggingdruid.emitter.logging.loglevel=info# Storage type of double columns# ommiting This'll leads to index double as Flo At the storage layerdruid.indexing.doublestorage=double

4. Copying the HDFs configuration file

[[email protected] _common]$ cp core-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/[[email protected] _common]$ cp hdfs-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/[[email protected] _common]$ cp mapred-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/[[email protected] _common]$ cp yarn-site.xml /alidata/server/druid-0.12.1/conf/druid/_common/

5. Enable the Druid SQL feature

[[email protected] broker]$ vi runtime.properties

druid.service=druid/brokerdruid.host=172.16.XXX.XXXdruid.port=8082# HTTP server threadsdruid.broker.http.numConnections=5druid.server.http.numThreads=9# Processing threads and buffersdruid.processing.buffer.sizeBytes=256000000druid.processing.numThreads=2# Query cache (we use a small local cache)druid.broker.cache.useCache=truedruid.broker.cache.populateCache=truedruid.cache.type=localdruid.cache.sizeInByte=2000000000# enable druid sql and httpdruid.sql.enable=truedruid.sql.avatica.enable=truedruid.sql.http.enable=true

Remarks: Runtime.properties new property in broker, Overlord, coordinator, historical, Middlemanager, etc. druid.host=ipaddress

6. Update Middlemanager task Execution number capacity

[email protected] middleManager]$ vi runtime.properties

druid.service=druid/middleManagerdruid.host=172.16.XXX.XXXdruid.port=8091# Number of tasks per middleManagerdruid.worker.capacity=20# Task launch parametersdruid.indexer.runner.javaOpts=-server -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManagerdruid.indexer.task.baseTaskDir=var/druid/task# HTTP server threadsdruid.server.http.numThreads=9# Processing threads and buffers on Peonsdruid.indexer.fork.property.druid.processing.buffer.sizeBytes=256000000druid.indexer.fork.property.druid.processing.numThreads=2# Hadoop indexingdruid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmpdruid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.7.3"]

7. Upgrade historical

[[email protected] druid-0.12.1]$ nohup >> nohuphistorical.out java `cat conf/druid/historical/jvm.config | xargs` -cp conf/druid/_common:conf/druid/historical:lib/* io.druid.cli.Main server historical &

8. Upgrade Overlord

[[email protected] druid-0.12.1]$ nohup >> logs/nohupoverlord.out java `cat conf/druid/overlord/jvm.config | xargs` -cp conf/druid/_common:conf/druid/overlord:lib/* io.druid.cli.Main server overlord &

9. Upgrade Middlemanager

• Prohibit overlor from assigning tasks to Middlemanager of specified services

http://<MiddleManager_IP:PORT>/druid/worker/v1/disable

• View the specified Middlemanager task list

http://<MiddleManager_IP:PORT>/druid/worker/v1/tasks

• Start Middlemanager

[[email protected] druid-0.12.1]$ nohup >> logs/nohupmiddleManager.out java `cat conf/druid/middleManager/jvm.config | xargs` -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager &

• Enable Overlord to assign tasks to the specified Middlemanager

http://<MiddleManager_IP:PORT>/druid/worker/v1/enable

10. Upgrade Broker

[[email protected] druid-0.12.1]$ nohup >> nohupbroker.out java `cat conf/druid/broker/jvm.config | xargs` -cp conf/druid/_common:conf/druid/broker:lib/* io.druid.cli.Main server broker &

11. Upgrade Coordinator

[[email protected] druid-0.12.1]$ nohup >> nohupcoordinator.out java `cat conf/druid/coordinator/jvm.config | xargs` -cp conf/druid/_common:conf/druid/coordinator:lib/* io.druid.cli.Main server coordinator &

At this point, Druid is finished upgrading from version 0.11.0 to version 0.12.1.

Druid Timing Database Upgrade process