（三）storm-kafka源碼走讀之如何構建一個KafkaSpout

標籤：storm   kafka   大資料   即時處理   

上一節介紹了config的相關資訊，這一節說下，這些參數分別是什麼，在zookeeper中的存放路徑是怎樣的，之前QQ群裡有很多不知道該怎麼傳入正確的參數來new 一個kafkaSpout，其主要還是參數傳遞正確就可。

看SpoutConfig的建構函式

public SpoutConfig(BrokerHosts hosts, String topic, String zkRoot, String id) {        super(hosts, topic);        this.zkRoot = zkRoot;        this.id = id;    }

需要一個BrokerHosts，看代碼：

public class ZkHosts implements BrokerHosts {    private static final String DEFAULT_ZK_PATH = "/brokers";    public String brokerZkStr = null;    public String brokerZkPath = null; // e.g., /kafka/brokers    public int refreshFreqSecs = 60;    public ZkHosts(String brokerZkStr, String brokerZkPath) {        this.brokerZkStr = brokerZkStr;        this.brokerZkPath = brokerZkPath;    }    public ZkHosts(String brokerZkStr) {        this(brokerZkStr, DEFAULT_ZK_PATH);    }}

需要brokerZKStr,這個其實就是hosts列表，多個host以逗號隔開，因為zookeeper解析string時是以逗號分隔的，這裡附上zookeeper的解析代碼

public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher,            boolean canBeReadOnly)        throws IOException    {        LOG.info("Initiating client connection, connectString=" + connectString                + " sessionTimeout=" + sessionTimeout + " watcher=" + watcher);        watchManager.defaultWatcher = watcher;        ConnectStringParser connectStringParser = new ConnectStringParser(                connectString);        HostProvider hostProvider = new StaticHostProvider(                connectStringParser.getServerAddresses());        cnxn = new ClientCnxn(connectStringParser.getChrootPath(),                hostProvider, sessionTimeout, this, watchManager,                getClientCnxnSocket(), canBeReadOnly);        cnxn.start();    }

其中主要StringParser做解析的，看俺如何解析的就知道了

public ConnectStringParser(String connectString) {        // parse out chroot, if any        int off = connectString.indexOf('/');        if (off >= 0) {            String chrootPath = connectString.substring(off);            // ignore "/" chroot spec, same as null            if (chrootPath.length() == 1) {                this.chrootPath = null;            } else {                PathUtils.validatePath(chrootPath);                this.chrootPath = chrootPath;            }            connectString = connectString.substring(0, off);        } else {            this.chrootPath = null;        }        String hostsList[] = connectString.split(",");        for (String host : hostsList) {            int port = DEFAULT_PORT;            int pidx = host.lastIndexOf(':');            if (pidx >= 0) {                // otherwise : is at the end of the string, ignore                if (pidx < host.length() - 1) {                    port = Integer.parseInt(host.substring(pidx + 1));                }                host = host.substring(0, pidx);            }            serverAddresses.add(InetSocketAddress.createUnresolved(host, port));        }    }

好了，這裡就說到這了。

剛才說到brokerZKStr需要，還有一個參數就是zkpath，這個可以自己定，也有個預設值 “/brokers”

SpoutConfig還有個zkroot，這個zkroot其實就是Consumer端消費的資訊存放地方，好了給個例子：

String topic = “test”;  //        String zkRoot = “/kafkastorm”; //        String spoutId = “id”; //讀取的status會被存在，/kafkastorm/id下面，所以id類似consumer group                BrokerHosts brokerHosts = new ZkHosts("10.1.110.24:2181,10.1.110.22:2181"); // 這裡使用預設的/brokers        SpoutConfig spoutConfig = new SpoutConfig(brokerHosts, topic, zkRoot, spoutId);        spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme()); // 下一節介紹 scheme                /*spoutConfig.zkServers = new ArrayList<String>(){{ //只有在local模式下需要記錄讀取狀態時，才需要設定            add("10.118.136.107");        }};        spoutConfig.zkPort = 2181;*/                spoutConfig.forceFromStart = true;         spoutConfig.startOffsetTime = -1;//從最新的開始消費         // spoutConfig.metricsTimeBucketSizeInSecs = 6;        builder.setSpout(SqlCollectorTopologyDef.KAFKA_SPOUT_NAME, new KafkaSpout(spoutConfig), 1);

這裡就成功建了一個KafkaSpout，如果項目運行成功的話，

可以到zk master上看下相關資訊，

./bin/zkCli.sh -server 10.1.110.24:2181

而對於StaticHosts來說，看官方解釋：

StaticHosts

This is an alternative implementation where broker -> partition information is static. In order to construct an instance of this class you need to first construct an instance of GlobalPartitionInformation.

    Broker brokerForPartition0 = new Broker("localhost");//localhost:9092    Broker brokerForPartition1 = new Broker("localhost", 9092);//localhost:9092 but we specified the port explicitly    Broker brokerForPartition2 = new Broker("localhost:9092");//localhost:9092 specified as one string.    GlobalPartitionInformation partitionInfo = new GlobalPartitionInformation();    partitionInfo.addPartition(0, brokerForPartition0);//mapping form partition 0 to brokerForPartition0    partitionInfo.addPartition(1, brokerForPartition1);//mapping form partition 1 to brokerForPartition1    partitionInfo.addPartition(2, brokerForPartition2);//mapping form partition 2 to brokerForPartition2    StaticHosts hosts = new StaticHosts(partitionInfo);

個人認為是需要開發人員，自己知道partition與broker之間的對應關係，正確關聯起來。而storm-kafka 0.9.0.1的版本是，不需要指定，我只需要傳入zkServer list，partition總數，由kafkautil利用兩個for（遍曆所有broker和partition）迴圈，臨時產生Consumer去串連消費一下試試，如果有資料，那麼就把partitionId和brokerHost關係存到Map中去。可想而知0.9.3-rc1為什麼要改成這樣了。如果該broker沒有該partition資訊，後果會怎樣？？？筆者沒有測試過，有測試過的請留言，說一下情況。Reference

http://www.cnblogs.com/fxjwind/p/3808346.html

（三）storm-kafka源碼走讀之如何構建一個KafkaSpout