Presto Common Configuration Detail _hadoop

Presto Common configuration details

Download Presto-server installation package

http://repo2.maven.org/maven2/com/facebook/presto/presto-server/

Presto update soon, this is presto-server-0.179.tar.gz

1, decompression

Tar zxfv presto-server-0.179.tar.gz

2, Presto-server path, create etc directory

mkdir etc

3, config.properties configuration file, master and slave are different

Master configuration

Coordinator=true
discovery-server.enabled=true
node-scheduler.include-coordinator=true
http-server.http.port=8888
query.max-memory=500gb
query.max-memory-per-node=76gb
discovery.uri= http://hadoopslave10:8888

failure-detector.threshold=0.10

exchange.http-client.request-timeout=30m

http-server.threads.max=1500
task.max-worker-threads=64

exchange.min-error-duration=30m
exchange.max-error-duration=150m

query.max-length=100000000

Slave configuration

Coordinator=false
http-server.http.port=8889
query.max-memory=500gb
query.max-memory-per-node= 76GB
discovery.uri=http://hadoopslave10:8888

exchange.http-client.request-timeout=30m

http-server.threads.max=1500
task.max-worker-threads=64

exchange.min-error-duration=30m
exchange.max-error-duration=150m

query.max-length=100000000                  

The difference is that master sets

Coordinator=true
Discovery-server.enabled=true
Node-scheduler.include-coordinator=true

To configure a resource queue:

Modify Presto/etc/config.properties, add the following configuration
Query.queue-config-file=/home/presto/presto-server/etc/queue_config.json

{"
  queues": {"
    User.${user}": {"
      maxconcurrent": 5,
      "maxqueued":
    {},
    "test": {
      " Maxconcurrent ": 2,
      " maxqueued ": 3
    },
    " admin ": {
      " maxconcurrent ": M,
      " maxqueued ":
    ,
    "global": {
      "maxconcurrent": "
      maxqueued": 1000
    }
  },
  "rules": [
    { 
      ] User ":" Bob ",   #bob为管理员, up to 100 queries at a time, 100 in the waiting queue
      " queues ": [" admin "]
    },
    { 
      " user ":" Guest ", #guest用户使用test的策略, up to 2 queries at a time, with 3 in the waiting queue
      " queues ": [" Test "]
    },
    {
      " queues ": [
        " User.${user} ",
        " global "
      ] 
    }   
  ]   

2, jvm.config configuration file

-server
-xmx128g
-xx:+useconcmarksweepgc
-xx:+explicitgcinvokesconcurrent
-xx:+ cmsclassunloadingenabled
-xx:+aggressiveopts
-xx:+heapdumponoutofmemoryerror
-xx:o Noutofmemoryerror=kill-9%p
-xx:reservedcodecachesize=150m

3, log.properties configuration file

Com.facebook.presto=debug

4, node.properties documents

Node.environment=datamart
node.id=hadoopmaster1
node.data-dir=/home/presto/presto-server-0.179/data                                                

Note node.id Each node is not the same, usually the same as hostname

5, etc directory, create catalog directory

mkdir Catalog

6, hive.properties configuration file

CONNECTOR.NAME=HIVE-HADOOP2
hive.metastore.uri=thrift://hadoopmaster1:9083
hive.config.resources=/etc/ Hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
hive.allow-drop-table=true
Hive.storage-format=parquet
hive.metastore-cache-ttl=0s
hive.metastore-refresh-interval=1s
hive.max-partitions-per-writers=1000

7, cluster restart script, each node configuration password-free

# Restart slaves ssh hadoopmaster1 "/home/presto/presto-server-0.179/bin/launcher restart" & ssh hadoopmaster2 "/hom" E/presto/presto-server-0.179/bin/launcher restart "& SSH hadoopslave1"/home/presto/presto-server-0.179/bin/ Launcher restart "& SSH hadoopslave2"/home/presto/presto-server-0.179/bin/launcher restart "& SSH Hadoopslave3 "/home/presto/presto-server-0.179/bin/launcher restart" & SSH hadoopslave4 "/home/presto/presto-server-0.179/ Bin/launcher restart "& SSH hadoopslave5"/home/presto/presto-server-0.179/bin/launcher restart "& SSH Hadoopslave6 "/home/presto/presto-server-0.179/bin/launcher restart" & SSH Hadoopslave7 "/home/presto/ Presto-server-0.179/bin/launcher restart "& SSH Hadoopslave8"/home/presto/presto-server-0.179/bin/launcher Restart "& SSH hadoopslave9"/home/presto/presto-server-0.179/bin/launcher restart "& SSH hadoopslave11"/home/ Presto/presto-server-0.179/bin/launcher restart "& SSH hadoopslave12"/home/presto/presto-server-0.179/bin/launcher restart "& SSH hadoopslave13"/home/presto/presto-server-0.179/bin/launcher Restart "& SSH hadoopslave14"/home/presto/presto-server-0.179/bin/launcher restart "& SSH hadoopslave15"/home /presto/presto-server-0.179/bin/launcher Restart "& #restart master ssh hadoopslave10"/home/presto/ Presto-server-0.179/bin/launcher Restart "&