How to Build Your Own Blockchain Part 4.2 — Ethereum Proof of Work Difficulty Explained__Ethereum

We’re back at it in the Proof of Work difficulty spectrum, this time going through how Ethereum’s difficulty changes over time. This is part 4.2 of the part 4 series, where part 4.1 was about Bitcoin’s PoW difficulty, and the following 4.3 will be about jbc’s PoW difficulty.

TL;DR

To calculate the difficulty for the next Ethereum block, you calculate the time it took to mine the previous block, and if that time difference was greater than the goal time, then the difficulty goes down to make mining the next block quicker. If it was less than the time goal, then difficulty goes up to attempt to mine the next block quicker.

There are three parts to determining the new difficulty: offset, which determines the standard amount of change from one difficulty to the next; sign, which determines if the difficulty should go up or down; and bomb, which adds on extra difficulty depending on the block’s number.

These numbers are calculated slightly differently for the different forks, Frontier, Homestead, and Metropolis, but the overall formula for calculating the next difficulty is

target = parent.difficulty + (offset * sign) + bomb

Other Posts in This Series Part 1 — Creating, Storing, Syncing, Displaying, Mining, and Proving Work Part 2 — Syncing Chains From Different Nodes Part 3 — Nodes that Mine Part 4.1 — Bitcoin Proof of Work Difficulty Explained Pre notes

For the following code examples, this will be the class of the block.

class Block():  def __init__(self, number, timestamp, difficulty, uncles=None):    self.number = number    self.timestamp = timestamp    self.difficulty = difficulty    self.uncles = uncles

The data I use to show the code is correct was grabbed from Etherscan.

The code doesn’t include edge cases in calculating the difficulty either. For the most part, edge cases aren’t involved in calculating the difficulty target. By not including them, it makes the following code much easier to understand. I’ll talk about the edge cases at the end to make sure they’re not completely ignored.

Finally, for the beginning fork, I talk about what the different variables and functions perform. For later forks, Homestead and Metropolis, I only talk about the changes.

Also, here’s all the code I threw in a Github repo! If you don’t want to write all the code yourself, you should at least clone it locally and run it yourself. 1000% feel free to make pull requests if you want to add more parts to it or think I formatted the code wrong. The Beginning — Frontier

In the beginning, there was Frontier. I’ll jump right in by giving a bullet point section going over the config variables. DIFF_ADJUSTMENT_CUTOFF — Represents the seconds that Ethereum is aiming to mine a block at. BLOCK_DIFF_FACTOR — Helps calculate how much the current difficulty can change. EXPDIFF_PERIOD — Denotes after how many blocks the bomb amount is updated. EXPDIFF_FREE_PERIODS — How many EXPDIFF_PERIODS are ignored before including the bomb in difficulty calculation.

And now descriptions of the functions.

calc_frontier_sign — Calculates whether the next difficulty value should go up or down. In the case of Frontier, if the previous block was mined quicker than the 13 seconds DIFF_ADJUSTMENT_CUTOFF, then the sign will be 1, meaning we want the difficulty to be higher to make it more difficult with the goal that the next block be mined more slow. If the previous block was mined longer than 13 seconds, then the sign will be -1 and the next difficulty will be lower. The overall point of this is that the goal for block mining time is ~12.5 seconds. Take a look at Vitalik Buterin’s post where he talks about choosing 12 seconds at the minimum average block mining time.

calc_frontier_offset — Offset is the value that determines how much or how little the difficulty will change. In the Frontier, this is the parent’s difficulty integer devided by the BLOCK_DIFF_FACTOR. Since it’s devided by 2048, which is 2^11, offset can also be calculated by parent_difficulty >> 11 if you want to look at it in terms of shifting bits. Since 1.0 / 2048 == 0.00048828125, it means that the offset will only change the difficulty by 0.0488% per change. Not that much, which is good because we don’t want the difficulty to change a ton with each different mined block. But if the time becomes consistently under the 13 seconds cutoff, the difficulty will slowly grow to compensate.

calc_frontier_bomb — The bomb. The bomb adds an amount of difficulty that doubles after every EXPDIFF_PERIOD block is mined. In the Frontier world, this value is incredibly small. For example, at block 1500000, the bomb is 2 ** ((1500000 // 100000) - 2) == 2 ** 15 == 32768. The difficulty of the block is 34982465665323. That’s a huge difference meaning that the bomb took on zero affect. This will change later.

calc_frontier_difficulty — Once you have the values for sign, offset, and bomb, the new difficulty is (parent.difficulty + offset * sign) + bomb. Let’s say that the the time it took to mine the parent’s block was 15 seconds. In this case, the difficulty will go down by offset * -1, and then add the small amount of the bomb at the end. If the time to mine the parent’s block was 8 seconds, the difficulty will increase by offset + bomb.

In order to understand it fully, go through the code that follows and look at the calculations.

config = dict(  DIFF_ADJUSTMENT_CUTOFF=13,  BLOCK_DIFF_FACTOR=2048,  EXPDIFF_PERIOD=100000,  EXPDIFF_FREE_PERIODS=2,)def calc_frontier_offset(parent_difficulty):  offset = parent_difficulty // config['BLOCK_DIFF_FACTOR']  return offsetdef calc_frontier_sign(parent_timestamp, child_timestamp):  time_diff = child_timestamp - parent_timestamp  if time_diff < config['DIFF_ADJUSTMENT_CUTOFF']:    sign = 1  else:    sign = -1  return signdef calc_frontier_bomb(parent_number):  period_count = (parent.number + 1) // config['EXPDIFF_PERIOD']  period_count -= config['EXPDIFF_FREE_PERIODS']  bomb = 2**(period_count)  return bombdef calc_frontier_difficulty(parent, child_timestamp):  offset = calc_frontier_offset(parent.difficulty)  sign = calc_frontier_sign(parent.timestamp, child_timestamp)  bomb = calc_frontier_bomb(parent.number)  target = (parent.difficulty + offset * sign) + bomb  return offset, sign, bomb, target

The Middle — Homestead

The Homestead fork, which took place at block number 1150000 in March of 2016, has a couple big changes with calculating the sign.

calc_homestead_sign — Instead of having a single number, DIFF_ADJUSTMENT_CUTOFF which is different for the Homestead fork, that makes the difficulty go up or down, Homestead takes a slightly different approach. If you look at the code, you’ll see that that there are groupings of the sign rather than either 1 or -1. If the time_diff between grandparent and parent is in [0,9],  sign will be 1, meaning that difficulty needs to increase. If the time_diff is [10,19], the sign will be 0 meaning that the difficulty should stay as it is. If the time_diff is in the range [20, 29], then the sign becomes -1. If time_diff is in the range [30,39], then the sign is -2, etc.

This does two things. First, Homestead doesn’t want to equate a block that took 50 seconds to mine as being the same as a block that took 15 seconds to mine. If it took a block 50 seconds, then the next difficulty in fact does need to be easier. Secondly, instead of DIFF_ADJUSTMENT_CUTOFF representing the goal time, this switches the aim point to be the mid point of the range of time_diffs with a sign of 0. [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. Meaning ~14.5 seconds, not including the bomb.

config = dict(  HOMESTEAD_DIFF_ADJUSTMENT_CUTOFF=10,  BLOCK_DIFF_FACTOR=2048,  EXPDIFF_PERIOD=100000,  EXPDIFF_FREE_PERIODS=2,)def calc_homestead_offset(parent_difficulty):  offset = parent_difficulty // config['BLOCK_DIFF_FACTOR']  return offsetdef calc_homestead_sign(parent_timestamp, child_timestamp):  time_diff = child_timestamp - parent_timestamp  sign = 1 - (time_diff // config['HOMESTEAD_DIFF_ADJUSTMENT_CUTOFF'])  return signdef calc_homestead_bomb(parent_number):  period_count = (parent_number + 1) // config['EXPDIFF_PERIOD'] # or parent.number + 1 >> 11 if you like bit notation  period_count -= config['EXPDIFF_FREE_PERIODS']  bomb = 2**(period_count)  return bombdef calc_homestead_difficulty(parent, child_timestamp):  offset = calc_homestead_offset(parent.difficulty)  sign = calc_homestead_sign(parent.timestamp, child_timestamp)  bomb = calc_homestead_bomb(parent.number)  target = (parent.difficulty + offset * sign) + bomb  return offset, sign, bomb, target

The Current — Metropolis

There are a couple differences from Homestead. First is that the DIFF_ADJUSTMENT_CUTOFF is now 9, which means that, without uncles, the target block time is midpoint of [9, 10, 11, 12, 13, 14, 15, 16, 17] or ~13 seconds.

The second takes into account whether or not there are uncles included in the block. And uncle in Ethereum language refers to a point in time where two nodes mine a child block from the same grandparent. So if you’re mining a child block from a parent that has a “sibling”, you’re able to pick one of the siblings to mine from, but also include that you noticed the other block. In that case, Ethereum wants to make the new difficulty larger, buy another offset, to make sure that there is a less likely chance for two natural forks to get much longer.

Now the biggest difference is dissolving the impact of the bombs. Check out the code for calc_metropolis_bomb where not only do we subtract the value of EXPDIFF_FREE_PERIODS, but also METROPOLIS_DELAY_PERIODS which is 30 time periods. A huge number. Instead of talking about the bombs here, I’ll have a section devoted to that after this.

config = dict(  METROPOLIS_DIFF_ADJUSTMENT_CUTOFF=9,  BLOCK_DIFF_FACTOR=2048,  EXPDIFF_PERIOD=100000,  EXPDIFF_FREE_PERIODS=2,  METROPOLIS_DELAY_PERIODS=30,)def calc_metropolis_offset(parent_difficulty):  offset = parent_difficulty // config['BLOCK_DIFF_FACTOR']  return offsetdef calc_metropolis_sign(parent_timestamp, child_timestamp):  if parent.uncles:    uncles = 2  else:    uncles = 1  time_diff = child_timestamp - parent_timestamp  sign = uncles - (time_diff // config['METROPOLIS_DIFF_ADJUSTMENT_CUTOFF'])  return signdef calc_metropolis_bomb(parent_number):  period_count = (parent_number + 1) // config['EXPDIFF_PERIOD']  period_count -= config['METROPOLIS_DELAY_PERIODS'] #chop off 30, meaning go back 3M blocks in time  period_count -= config['EXPDIFF_FREE_PERIODS'] #chop off 2 more for good measure  bomb = 2**(period_count)  return bombdef calc_metropolis_difficulty(parent, child_timestamp):  offset = calc_metropolis_offset(parent.difficulty)  sign = calc_metropolis_sign(parent_timestamp, child_timestamp)  bomb = calc_metropolis_bomb(parent.number)  target = (parent.difficulty + offset * sign) + bomb  return offset, sign, bomb, target

Going Deeper with the Bombs

If you look at one of the difficulty charts from online, you’ll see a recent amount of huge increasing jumps every 100000 blocks, and then a giant drop off about a month ago. Screenshot time for those not wanting to click the link:

Each horizontal line indicates a 3 second change in time it takes to mine a block.

What’s the point of having a bomb like this? A big goal of Ethereum is to get rid of Proof of Work, which requires energy and time to create and validate a new block, into Proof of Stake, which is described in the Ethereum Wiki. In order to force nodes to move to the Proof of Stake implementation, a “bomb” that doubles its impact on the difficulty every 100000 blocks would soon make it take so long to mine a new block, the nodes running on the old fork won’t be able to run anymore. This is where the term “Ice Age” comes from; the block chain would be frozen in time.

This is a good way to manage future changes, but also we run into the issue that the new Proof of Stake implementation, called Casper, wasn’t ready in time before the giant difficulty spikes seen in that graph. That’s where the Metropolis fork came into play — it eliminated the effect the bomb has on the difficulty, but in a few years it will come back into play, where the switch to Casper will (hopefully) be ready to roll. That being said, predicting when features like this fork are ready for production and adoption is difficult, so if Casper isn’t ready to be pushed quick enough, you can create another fork that moves the bomb back in time again.

Bomb Math

Besides the description of what the bombs do to the difficulty as seen on those graphs, it’s worth it to show the math behind why the difficulty is rising to those levels, and what that means for how long it takes to mine a block.

Going back