Analytics for hackers: how to think about event data

There's this new and really powerful type of data: event data. Well, it's not reallyNew. I bet you are already familiar with event data, even if you don't know it by that name.

Event Data is everywhere

Events are happening all around us. in our apps, cars, appliances, servers, and even in our brains. this is a very exciting time because we can now collect and analyze events at massive scale. more and more things are becoming Internet-connected, making it easier to collect data from all sorts of places. what discoveries will we make about our apps, our behaviors, our society, and ourselves?

I'll leave that question open and focus on an easier one for now. What is event data?

Entity Data

The easiest way to learn the conceptEvent DataIs to compare it to another type of data:Entity Data. If you 've ever worked with an application database, or an Excel spreadsheet, you know about Entity Data. Entity Data looks like this:

Entity table example (enemies table)

Entity data is stored in tables. entities are things like users, products, accounts, posts, levels, etc. there is a separate table for each of type of entity, and each table has columns to hold properties about the entities. there is one row in the table for each entity. in this example, the entities are enemies.

Most databases have been designed to store Entity Data. They're sometimes referred to as relational databases.

Entity Data is really good for capturing the current state of you application. Things like users, number of each type of product, accounts payable, etc. You can very quickly lookup information about any entity.

One characteristic of entity databases is that they are normalized. data is rarely duplicated. for example, you might have a table for accounts, with attributes like the account name, type, category, etc. accounts have owned users associated with them, but you wouldn't store information about those users in the accounts table. instead, you wocould include a key in each user record which links to its account. from a data storage (disk usage) perspective, this is very efficient.

Entity Data example from the excellent Wikipedia article on joins

 

One drawback to this data model is that in order to run analysis on the entities, for example, sort employees by department name, you must pull in data from multiple tables. at Large Scale, these operations take time.

 

Event Data

Now let's look at the characteristics of the new data type, event data.

Event Data example: "publish" event

Event Data doesn't just describe entities; it describes actionsperformed by entities. this example abve describes the action of publishing this blog post. you can imagine we have a collection of events called publishes where we track an event for each new post.

What makes this publish data "event data "? Event Data has three key pieces of information. I first saw these identified by Ben Johnson in his speakerdeck on event data (HE callit "behavior data ").

1. Action

2. Timestamp

3. State

The action is the thing that's happening (e.g. "publish "). the timestamp is self explanatory: the point in time the thing happened. the State refers to all of the other relevant information we know about this event, including information about entities related to the event, such as the author.

Let's look at a more complex event. This is a "death" type event:

Event Data example: Minecraft death event

Here is an example data point for a player death in the game Minecraft. Imagine we are recording every player death that happens in the game.

If you haven't played Minecraft before, all you need to know is thatIt's amazingAnd, similar to keep video games, there are a lot of ways in which the player can experience death: falling from great heights, starvation from not eating enough pork chops, drowning, clumsily stumbling into lava, zombies scaring the crap out of you in a cave, etc.

Let's say we want to analyze these player deaths in Minecraft. perhaps we want to find out the most common type of death, the average player age at the time of death, the most lethal enemies, or any number of death-related questions. perhaps we are trying to find out if the game is too difficult on certain levels, or if the new villain we introduced is causing way more destruction than we 'd imagined, or if there is any correlation between types of users and types or frequency of deaths.

We can find out all of these things using the simple event data model shown above. The event data model has a few special qualities:

1. The data is rich

2. The data is denormalized

3. The data is nested

4. The data is schemaless

 

Event Data is rich

This death event has a lot of properties-information about the player at the time of death, the cause of death, the enemy. events can have hundreds of properties; they seek to describe not just one entity, but all of the entities involved in an action.

In fact, I think we shocould add even more data to the Minecraft death example. Information like: Location of the death, game settings, and software version. Can you think of more?

Perhaps we'll discover that players on difficulty level "hard" experience death by skeletons at a rate of 10x those on "normal" difficulty. is it just me or does that skeleton look extremely smug?

 

Event Data is denormalized

Unlike a relational database, you see the same data repeated over and over again in an event database. things like user attributes, app version, or difficulty settings might be repeated on every single event, even if they rarely change. although it's not super intuitive (and downright unnerving to SQL veterans), this redundancy is the only way to capture a representation of the application stateAt the time of the event. Contrast this with relational (entity) databases. in Entity databases, properties (e.g. player settings) are updated and the previous values are lost forever. event databases give us the ability to capture that entity data at a point in time. event databases are not an alternative to entity databases. an event database is a great companion to an entity database.

 

Event Data is nested

Event data can have lots and lots of properties; most databases optimized for event data allow you to store it using nested JSON. This is really helpful when you have same properties and multiple entities to describe.

 

Event Data is schemaless

As we discussed, event data allows you to capture State at the time of an event. but different properties will be important for different types of Minecraft deaths. for example, starvation, drowning, or lava deaths don't involve an "enemy" and might have their own unique properties like lava temperature. in other words, the death events don't follow a strict schema. we'll add or remove data depending on the type of death. event databases are designed to handle any number of arbitrary properties that you send.

 

Event data at scale

Minecraft is a huge game. 11 million people have bought the game and they are still selling over 10,000 downloads per day! That's a lot of user records. Consider the volume of event data points compared to entity data points. For every one user there are still actions.

Entity Data captures the current state of things. event data captures the history of actions that happen over time. Event Data happens at a much greater scale than Entity Data.

Thankfully, we are now in an era where data storage is cheap enough that it's possible to store event data in a cost alternative tive way. plus, there are all sorts of tools and cloud services that make tracking this type of data easier than ever.

 

Summary

Event Data is a powerful data format that allows us to track and analyze things that happen around us. Here's a handy Table comparing entity & event data.

 

Obama Wetzler wrote this post on June 26,201 3

Analytics for hackers: how to think about event data