How to Build Your Own Blockchain Part 4.1 — Bitcoin Proof of Work Difficulty Explained__blockchain

If you’re wondering why this is part 4.1 instead of part 4, and why I’m not talking about continuing to build the local jbc, it’s because explaining Bitcoin’s Proof of Work difficulty at a somewhat lower level takes a lot of space. So unlike what this title says, this post in part 4 is not how to build a blockchain. It’s about how an existing blockchain is built.

My main goal of the part 4 post was to have one section on the Bitcoin PoW, the next on Ethereum’s PoW, and finally talk about how jbc is going to run and validate proof or work. After writing all of part 1 to explain how Bitcoin’s PoW difficulty, it wasn’t going to fit in a single section. People, me included, tend get bored in the middle reading a long post and don’t finish.

So part 4.1 will be going through Bitcoin’s PoW difficulty calculations. Part 4.2 will be going through Ethereum’s PoW calculations. And then part 4.3 will be me deciding how I want the jbc PoW to be as well as doing time calculations to see how long the mining will take.

The sections of this post are: Calculate Target from Bits Determining if a Hash is less than the Target Calculating Difficulty How and when block difficulty is updated Full code Final Questions TL;DR

The overall term of difficulty refers to how much work has to be done for a node to find a hash that is smaller than the target. There is one value stored in a block that talks about difficulty — bits. In order to calculate the target value that the hash, when converted to a hex value has to be less than, we use the bits field and run it through an equation that returns the target. We then use the target to calculate difficulty, where difficulty is only a number for a human to understand how difficult the proof of work is for that block.

If you read on, I go through how the blockchain determines what target number the mined block’s hash needs to be less than to be valid, and how that target is calculated. Other Posts in This Series Part 1 — Creating, Storing, Syncing, Displaying, Mining, and Proving Work Part 2 — Syncing Chains From Different Nodes Part 3 — Nodes that Mine Part 4.2 — Ethereum Proof of Work Difficulty Explained Calculate Target from Bits

In order to go through Bitcoin’s PoW, I need to use the values on actual blocks and explain the calculations, so a reader can verify all this code themselves. To start, I’m going to grab a random block number to work with and go through the calculations using that.

>>>import random>>> random.randint(0, 493928)111388

Block number 11138 it is! Back in time to March of 2011 we go.

We’re going to start with assigning variables with the bits and difficulty values from the block, as well as the hash, so we can test its validity at the end.

>>> bits = '453062093'>>> difficulty = float('55,589.52'.replace(',',''))>>> block_hash = '00000000000019c6573a179d385f6b15f4b0645764c4960ee02b33a3c7117d1e'

Next part shows how to go from the bits field to the target. There are at least two different ways to do this — the first involves string manipulation and string to integer conversion, and the other uses bit manipulation.

For the first string manipulation part, we convert the bits string to an integer and then to a hex string. The first two characters of the hex_bits string are in one variable, which I call shift, and the remaining six are called the value.  From there, we use those values in the integer equation that will calculate the integer value of the target, which we can convert to a hex string to look at how pretty it is!

Note that ‘L’ on the back of the target and hex(target) is only Python telling you that the numbexr is too long to be stored as an int, and that it’s stored as a long.

>>> hex_bits = hex(int(bits))>>> hex_bits'0x1b012dcd'>>> shift = '0x%s' % hex_bits[2:4]>>> shift'0x1b'>>> shift_int = int(exponent, 16)27>>> value = '0x%s' % hex_bits[4:]>>> value'0x012dcd'>>> value_int = int(coefficient, 16)77261>>> target = value_int * 2 ** (8 * (shift_int - 3))>>> target484975157177710342494716926626447514974484083994735770500857856L>>> hex_target = hex(target)>>> hex_target'0x12dcd000000000000000000000000000000000000000000000000L'

It seems a little fake when we slice a string to calculate the target like we do above. When looking through the c++ bitcoin implementation, it shows the other, more technical, main method of using bits to calculate target. Here’s the Python implementation.

I’ll go through the lines and print values here to show a little better what’s going on. One thing to know before reading is that a character in a hex string represents 4 bits. So when we shift the bits variable by 24 bits with bits >> 24 that affectively removes the last 6 characters of the hex string leaving the first two. To get the value variable, we use the bitwise and operator &  to have the final 23 bits be the value. The 23 bits is usually (defined by the number of bits in 0x7fffff) the final 6 characters of the hex string,

Now, the 8 * (shift - 3) value determines how many bits we’re going to shift value over. In this case, it’s determined to be 192 bits. If we divide the 192 by 4, we’ll have the number of zeros characters that will be behind the value in the hex string representation of the target number. You’ll see that it’s 48 zeros. Another thing to think of if this is confusing is that value << y is the same as value * 2**y. That’s how the equations are related.

>>> bits = '453062093'>>> hex(int(bits))'0x1b012dcd'>>> shift = bits >> 24>>> shift27>>> hex(shift)'0x1b'>>> value = bits & 0x007fffff>>> hex(value)'0x12dcd'>>> 8 * (shift - 3)192>>> 192 / 4 #each character in a hex string represents 4 bits48>>> value <<= 8 * (shift - 3)>>> hex(value)'0x12dcd000000000000000000000000000000000000000000000000L'>>> hex(value).count('0') - 1 #don't count the leading zero48

The full function version of this is only a few lines long.

>>> def get_target_from_bits(bits):...   shift = bits >> 24...   value = bits & 0x007fffff...   value <<= 8 * (shift - 3)...   return value...>>> target = def get_target_from_bits(int(bits))>>> hex(target)'0x12dcd000000000000000000000000000000000000000000000000L'

So nice. If you look way above to see the string manipulation code it’s way more confusing and has so many extra lines than the function above. It’s great to go through both to really understand the meaning, but then stick with the simple one. Moving on. Determining if a Hash is less than the Target

As always it seems, there are two simple ways to see if a block’s hash is valid according to the target. The first way is to use full, 64 character, 256 bit, hex strings and compare those. This works great since we’ll be able to see this easily eye to eye. To show this is the case, we’re going to pad the hex_target with the zeros to fill all 64 characters.

>>> hex_target = hex(target)>>> len(hex_target)56>>> len(hex_target[2:-1]) #[2:-1] removes the leading '0x' and ending 'L'53>>> num_padded_zeros = 64 - hex_target_len>>> num_padded_zeros11>>> padded_hex_target = "0x%s%sL" % ('0' * (64-num_padded_zeros), hex_target[2:-1])>>> padded_hex_target'0x0000000000012dcd000000000000000000000000000000000000000000000000L'

Finally, we want to verify that the block’s hash is less than the target. And also, since strings can be considered less than or greater than, we’ll go with that.

>>> len(block_hash)64>>> padded_block_hash = '0x%sL' % block_hash>>> padded_block_hash0x00000000000019c6573a179d385f6b15f4b0645764c4960ee02b33a3c7117d1eL>>> padded_hex_target0x0000000000012dcd000000000000000000000000000000000000000000000000L>>> assert padded_block_hash < padded_hex_target>>>

In this case, you’ll see the hex target is larger than the block’s hash. Comparing which string is larger is as simple as either < or >. If you’re wondering how Python can determine that the characters like 'f' > '1', see  the questions below.

The other way to confirm is by using the integers for both values. But as you’ll see, it’s very difficult, if not impossible, to eyeball which is value larger, and how much larger one value is to than the other.

>>> block_hash_int = int(block_hash, 16)>>> block_hash_int41418456048005639864974238890271849696605172030151526454492446L>>> target484975157177710342494716926626447514974484083994735770500857856L>>> assert(block_hash_int < target)>>> target - block_hash_int443556701129704702629742687736175665277878911964584244046365410L

Those ints are so huge that looking at one by itself is pretty much impossible to know if it’s easy or difficult to be lower than. But since block_hash_int is lower than target int by one character, you can see that’s the case. When you subtract those two giant numbers you get another giant number.

Both are valid methods, but being able to look at the hex strings is better than looking at the integers. But really, when the code is running, there’s no real benefit to pick one over the other. Calculating Difficulty

The most important part of difficulty is to remember from above that difficulty is simply a human representation of the target. I talked about how integer values of targets are pointless since it’s hard to humanly compare two values that are giant integer. Likewise, it’s tough to compare two targets to each other using the hex strings. The difficulty values in a block are to make that easier.

Below, difficulty_one_target is defined as the easiest allowed target value. To calculate the difficulty of another block, we simply divide the difficulty_one_target by the target we’ve calculated using bits. You’ll see that difficulty_one_target will be larger than any other targetnumber since targets go lower with more difficulty. When you divide the larger numerator by a smaller denominator, you’ll have a greater than one value.

>>> difficulty_one_target = 0x00ffff * 2**(8*(0x1d - 3))>>> difficulty_one_target26959535291011309493156476344723991336010898738574164086137773096960L #calculated using >>> pad_leading_zeros(hex(difficulty_one_target)) #pad_leading_zeros is function defined below'0x00000000ffff0000000000000000000000000000000000000000000000000000'>>> calculated_difficulty = difficulty_one_target / float(target) #float() to make it decimal devision55589.518126868665 #this is the same as the difficulty on the block.>>> allowed_error = 0.01>>> assert abs(float(block_difficulty) - calculated_difficulty) <= allowed_error>>>

Feel free to go back and read this over slowly. It took me a long time to figure out the relationship between bits, difficulty, and target. Whether we need ints, hex values, or strings to verify. There are a bunch of ways to do this, and hopefully I’ve listed them all here. How and when block difficulty is updated

The final big part of Bitcoin’s proof of work is to show how and when the difficulty is changed. Pretty much every post on this topic will say that the chain will recalculate the required new difficulty every 2016 blocks. But they don’t dive too deep at all into the topic. That’s what I’m going to do here.

The first part I have to start with is how to start with a target, and go back to bits. The