Storm (2)-Log Stream processing

Introduction

This chapter would present an implementation recipe for an enterprise log storage and a search and analysis solution based On the Storm processor. Log data processing isn ' t necessarily a problem that needs solving again; It is, however, a good analogy.

Stream processing is a key architectural concern in the modern enterprise; However, streams of data is often semi-structured at best. By presenting a approach to enterprise log processing, this chapter are designed to provide the reader with all the key El Ements to achieve the capability on any kind of data. Log data is also extremely convenient a academic setting given its sheer abundance. A key success factor for any stream processing or analytics effort are a deep understanding of the actual data and sourcing Data can often be difficult.

It is, therefore, important that the reader considers how the architectural blueprint could was applied to other forms of D ATA within the enterprise.

The following diagram illustrates all the elements so we'll develop in this chapter:

You'll learn how to create a log agent so can be distributed across all the nodes in your environment. You'll also learn to collect these log entries centrally using Storm and Redis, and then analyze, index, and count the L OGs, such that we'll be able to search them later and display base statistics for them.

Creating a log Agent

1. Download and config Logstash to steam local node log into the topologywgetHttps//Logstash.objects.dreamhost.com/release/logstash-1.1.7-monolithic.jar2. Create thefileof Shipper.confinput {file{type="syslog"Path= ["/var/log/messages","/var/log/system.*","/var/log/*.log"]}}output {#output events to stdout forDebugging. Feel FreeTo remove it stdout {} redis {host="localhost"data_type="List"Key="Rawlogs"    }}3. Start a local instance of Redis, and Thenstart Logstash Java-jar logstash-1.1.7-monolithic.jar-f shipper.conf

View Code

Storm (2)-Log Stream processing