Splunk meeting Review: The Key to big data is machine learning. splunk Review

Splunk meeting Review: The Key to big data is machine learning. splunk Review

Jonathan Allen, author

The Splunk User Conference is coming to an end. During the three-day conference, over 160 topics were discussed, covering security, operation, business intelligence, and even the Internet of Things, the same central topic appears at the meeting over and over again: the Key to big data is machine learning.

Storage is no longer a problem. From dedicated storage hardware running Hadoop compatible nodes to clusters composed of hundreds of computers using common hard disks, we certainly have the ability to handle such storage problems. On the other hand, analysis and visualization tools such as Splunk have also emerged. If you know what you are looking for, these tools can quickly give you the answer you need.

But what should you look? For the vast majority of grass-roots suppliers, the answer is in machine learning. Whether you are talking about network traffic, user behavior, or consumption trends, it doesn't matter. The way you can truly gain insight into what you monitor is to find patterns and correlations in data. Although human operators can follow the trial and error methods, they believe that results can be obtained through computer training and are faster and free of prejudice.

Of course, this does not mean that humans are outdated. It is important that someone confirms relevance not just a coincidence and identifies a way to take action on the information. This is exactly where the visualization tools mentioned above can play a role.

Main Use Cases of big data and machine learning

Although the potential of big data is almost unlimited, it is inevitable that one or two industries will take the lead. If you ask me again in a year, I may say different things, but the current prediction is that both security and operation are at the forefront.

As long as the company is larger than those that only accept cash, the security of information needs to be considered. Even if they do not have intellectual property rights, they are processing sensitive information, such as credit card numbers. It is vital for the company to have a way to reliably detect and block ongoing defaults. The machine learning-based security product promises to provide this capability, and its ease of use is close to "turnkey engineering, the buyer only needs to take the key for acceptance.

Similarly, operational analysis will become popular. Now you can buy tools to monitor your network, decode data packets, or precisely present to you how a given REST call passes through the server's intermediate layer all the way to the database or file system, then compare it with the behavior of a week, a month or a year ago. This is not a concept of the future, but something ready for use today that can run within a week.

Research in other fields will continue, but it will not be so fast. Fraud detection is very important, but most companies rely on their financial institutions to design and implement necessary control measures. I don't expect many commercial and ready-made products in this regard.

Business Intelligence is another area of research that will see a lot of money. But the algorithms that Coca-Cola and PepsiCo use to determine the next popular taste do not look like general or Ford's algorithms that predict how many models of each size will have. So far, the use of commercial products for big data may be limited to basic analysis and visualization.

Other Meeting ideas

All in all, Splunk held a very good meeting. Everything is well organized. Everyone, from beginners to the most advanced data mining engineers, will discuss related topics. My only complaint is that the topic discussion is not recorded. With so much content, people will miss one or two important topics due to conflicts.

Even if you are not interested in Splunk, this is an important meeting for anyone interested in big data, machine learning, and related topics.

Splunk Conference Recap: The Key to Big Data is Machine Learning