Splunk Test report

Splunk use test report

I. technical components and principles

1. indexer indexes local or remote log data.

Working mechanism:

You can index log data of any format with a timeline. This index is used to disrupt data and put it into events based on the timestamp. Each events contains the timestamp, host, source, and source type attributes. A log row is an event. xml logs may be divided into multiple events. When a user searches, these events are searched by splunk and returned to the user.

Events: a single piece of data in splunk, similar to a record in a log file or other data input. when splunk eats data, it breaks the data up into individual pieces and gives each piece a timestamp, host, source, and source type. often, a single event corresponds nicely to a single line in your inputs, but some inputs have multiline events (for example, XML logs) and some inputs actually have multiple events on a single line. when you run a search, events are what you get back.

Event processing ):

There are two phases: resolution and indexing. Data that enters splunk will be put into the parsing pipeline as a block (generally 10 K. During the parsing process, these blocks will be disrupted. The parsing process includes the following actions:

· Extracting a set of default fields for each event, including host, source, and sourcetype.

· Grouping ing character set encoding.

· Identifying line termination using linebreaking rules. While using events are short and only take up a line or two, others can be long.

· Identifying timestamps or creating them if they don't exist. At the same time that it processes timestamps, splunk identifies event boundaries.

· Splunk can be set up to mask sensitive event data (such as credit card or social security numbers) at this stage. It can also be configured to apply custom metadata to incoming events.

There are other operations in the index pipeline:

· Breaking all events into segments that can then be searched upon. You can determine the level of segmentation, which affects indexing and searching speed, search capability, and efficiency of disk compression.

· Building the index data structures.

· Writing the raw data and index files to disk, where post-indexing compression occurs.

 

It simply describes the important process. The parsing MPs queue includes the following pipelines: parsing, merging, and typing.

Event Data: index data obtained after indexing Log Data

Event: A record of log data.

Index: The index contains two types of data:

· The raw data in compressed form ("rawdata ")

· Indexes that point to the raw data ("index files ")

These data are stored in directories called buckets. These directories are stored by time, which better support splunk's olddata processing.

· A splunk "Index" contains compressed raw data and associated indexes.

· A splunk index resides internal SS partition age-designated index directories.

· An index directory is a bucket.

These buckets are rolled for a long time. Level:

· Hot

· Warm

· Cold

· Frozen

As buckets age, they "roll" from one stage to the next. newly indexed data goes into a hot bucket, which is a bucket that's both searchable and actively being written. after the hot bucket reaches a certain size, it becomes a warm bucket, and a new hot bucket is created. warm buckets are searchable, but are not actively written. there are warm buckets.

Once splunk has created some maximum number of warm buckets, it begins to roll the warm buckets to cold based on their age. always, the oldest warm bucket rolls to cold. buckets continue to roll to cold as they age in this manner. after a set period of time, cold buckets roll to frozen, at which point they are either archived or deleted. by editing attributes in indexes. conf, you can specify the bucket aging policy, which determines when a bucket moves from one stage to the next.

Bucket stage	Description	Searchable?
Hot	Contains newly indexed data. Open for writing. One or more hot buckets for each index.	Yes.
Warm	Data rolled from hot. There are warm buckets.	Yes.
Cold	Data rolled from warm. There are using cold buckets.	Yes.
Frozen	Data rolled from cold. splunk deletes frozen data by default, but you can also archive it.	No.

These buckets can be called databases by stage. Such as hotdb, warmdb...

The database can be partitioned. The size of each bucket can be configured.

For detailed index directory results, see:

Http://docs.splunk.com/Documentation/Splunk/4.3/Admin/HowSplunkstoresindexes

2. Forwarder

Deployed on the terminal server to forward log data to indexer. You can also switch to another splunkserver or non-splunkserver. There are two types:

· Universal forwarders. These have a very light footprint and forward only unparsed data.

· Heavy forwarders. These have a larger footprint but can parse, and even index, data before forwarding it.

There can also be three types. For details, see

Http://docs.splunk.com/Documentation/Splunk/4.3/Deploy/Typesofforwarders

The receiver is generally indexer. It can receive data from one or more forwarder. The receiver may also be another forwarder. Similar to scribe. The difference is that data may be parsed and indexed.

This structure provides basic support for data Federation, Server Load balancer, and data routing. See: http://docs.splunk.com/Documentation/Splunk/4.3/Deploy/Forwarderdeploymenttopologies

3. Search head

This is used for distributed search. For example, if there is a large amount of data and many users search for the data concurrently, then index loading on different indexers will help you complete search queries on different servers, achieve the effect of traffic distribution and load reduction. The components that distribute search query requests to different indexers are the search head.

For details, see

Http://docs.splunk.com/Documentation/Splunk/4.3/Deploy/Whatisdistributedsearch

Several figures:

This distrishows a simple distributed search scenario for horizontal scaling, with one search head searching processing SS three peers:

 

In this distrishowing a distributed search scenario for access control, a "security" Department search head has visibility into all the indexing search peers. each search peer also has the ability to search its own data. in addition, the Department a search peer has access to both its data and the data of department B:

 

Finally, this distrishows the use of load balancing and distributed search to provide high availability access to data:

 

4. Deployment Server

Center Configuration Manager. Each splunk instance can be used as a Deployment Server.

The deployment sever handles configuration and content updates to existing splunk installations. You cannot use it for initial or upgrade installations of splunk Components

Splunk instances that are remotely configured by deployment servers are called deployment clients. A splunk instance can be both a deployment server and client at the same time.

Terms:

Term	Meaning
Deployment Server	A splunk instance that acts as a centralized Configuration Manager. It pushes Configuration updates to other splunk instances.
Deployment Client	A remotely configured splunk instance. It has es updates from the Deployment Server.
Server class	A deployment configuration category shared by a group of deployment clients. A deployment client can belong to multiple server classes.
Deployment app	A unit of content deployed to one or more members of a server class or classes.
Multi-tenant Environment	A deployment environment involving multiple deployment servers.

Ii. Important Functions of splunk

1. Data Import: You can use data types to import files, directories, Apache logs, and other files. You can also use regular expressions to define the timestamp format. Data can also be collected using UDP, TCP, or custom scripts.

2 Index Library:

You can create an index database. The default database structure is used by default:

Main Directory: $ splunk_db/index_name/DB

Colddb: $ splunk_db/index_name/colddb

Thawed dB path (restore data): $ splunk_db/index_name/thaweddb

Maximum total quantity

Statistics on the index database (total database size, ratio, first and last time, and Application)

3. Forward and receive

You can configure forwarding and receiving on the Management Interface (that is, forwarder and indexer mentioned above) through IP: Port

4. Support distributed search: the node's Host IP Address: port is required, and the Administrator's user name and password enable splunk to pass remote search and verification.

5. Supports any search in the search box to search for the application's log data directly and supports interactive search. (Some Words in the log are highlighted by the mouse and will be automatically added to the search criteria)

6. Field search is supported. Host, source, and sourcetype are provided by default. It can be customized. You can also add display fields on the search page at any time. Why is this not specified during user upload.

7. You can search by timeline. The timeline shows the frequency of hit searches. You can also adjust the timeline for further search.

8. Search supports operations such as HTTP code, which is very convenient to use. Very meticulous

9. Provides a knowledge base and allows you to add eventtype, fields, and tags.

10 supports searching schedules and searching plans.

11 alarm notification supported

12. The results are displayed in a variety of ways, including displaying source data, field tables, and graphics. User-friendly feature settings

13. supports dynamic creation of dashboards and other views and reports, and online printing.

14. Quick Start Guide for users to familiarize themselves with the System

Iii. Advantages and Disadvantages

1 advantages:

It is very convenient to search logs;

Supports distributed search, scalability, fault tolerance, and powerful load balancing capabilities

Supports graphical search results and reports

Support searching by scheduling

Alarm notification supported

Timeline only displays search trends and results

2. Disadvantages

Weak in log statistics

Index data and search data are stored in indexers. If indexer fails, data will be lost. However, forwarder can choose to back up data locally.

After reading the official technical system, it is still not very sure. If the log generation is too large, forwarder will forward it to one or more indexers, but it is unknown whether the search performance will decline significantly, although the search head is distributed to multiple indexers.