EStateBox For CRDTs

EStateBox For CRDTs
EStateBox

ProJect Address:

GitHub

According to the implementation of statebox, estatebox is mainly used to solve version conflicts in distributed systems. In Dynamo algorithms, CRDTs (Conflict-free Replicated Data Types) is a better way to resolve conflicts.CRDTsData types can be summarized:

• Conformity rate

• Exchange rate satisfied

• Idempotence

  InEStateBoxMedium:

  • Anop()Must be repeatable:F(Arg, F(Arg, Value)) =:= F(Arg, Value)
  • If{fun(), [term()]}Form is used,fun()Shocould be a reference to an exported function.
  • F(Arg, Value)Shocould return the same type as Value.

    Of course, in other cases, these conditions cannot be fully met.RiakOfVClock.

    Overview:

    EStateBoxIs a data structure in whichFinal consistencyIn the system, suchriak, You can use a definite method to solve parallel conflicts.EStateBoxIt is only a collection of events. The collection of such events will lead to a unique result. Therefore, these events must meet certain conditions, as shown above. AndＲiak VClockSimilar to each other.ProcedureAndOperands.

    Status:

    InＭochi ＭediaThe platform has been used in multiple backend services.

    Theory

    EStateBoxContains a current value and an event queue, which is{timestamp(), op()}Ordered list. When there are two or moreEStateBoxQuiltEStateBox.merge/1Merge, event queue islists.umerge/1Merge,OperationExecuted to update the latestEStatBoxWill generate a newEStateBox.

    • op()Is{fun(), [term()]}Except the last parameter, all parameters are specified in this list. For example:{ordsets:add_element/2, [a]};

    • op()It can also be{module(), atom(), [term()]}Tuples;

    • op()The list can also indicateMultiple operations

      The following are some security scenariosopExample:

      • op()It must be idempotent:fn(arg, fn(arg, value)) =:= f(arg, value)

      • If{fn(), [term()]}Available,fnIt must be customizable.

      • fn(arg, value)Should Return andvalueSame Value

        Inerlang, The following functions are safe to use:

        • {fun ordsets:add_element/2, [SomeElem]}And{fun ordsets:del_element/2, [SomeElem]}

        • {fun ordsets:union/2, [SomeOrdset]}And{fun ordsets:subtract/2, [SomeOrdset]}

        • {fun orddict:store/3, [Key, Value]}

          Some of them are not usable:

          • {fun orddict:update_counter, [Key, Inc]}BecauseF(a, 1, [{a, 0}]) =/= F(a, 1 , F(a, 0}]))Is not idempotent. Optiomizations

            To preventEStateBoxToo large to waste unnecessary memory. Here there are two functions used to cropQeueuRespectively:

            • truncate(n, stateBox)Returns lessnEvent queues

            • expire(age, stateBox)ReturnslastmodifiedBaselineＡgeInternalmilliseconds.

              Merge

              EStateBoxIs based ontimestampTo merge, select the latest one, iftimestampIn the same way, one is automatically selected. This method is easy to handle, but it is not a good method (Is there a better way? ^> ^), Because the clock is not necessarily synchronized in a distributed environment.

              op1 -> value: []op2 -> value: [1]op3-> value: [1, 2]op4 -> value: [1, 2, 3]op5 -> value: [1, 2, 3, 4]op6 -> value: [1, 2, 3, 4, 5]op7 -> value: [1, 2, 3, 4, 5, 6]op8 -> value: [1, 2, 3, 4, 5, 6, 7]op9 -> value: [1, 2, 3, 4, 5, 6, 7, 8]op10 -> value: [1, 2, 3, 4, 5, 6, 7, 8, 9]Queue:[ {1, {&:ordsets.add_element/2, [1]}},  {2, {&:ordsets.add_element/2, [2]}},  {3, {&:ordsets.add_element/2, [3]}},  {4, {&:ordsets.add_element/2, [4]}},  {5, {&:ordsets.add_element/2, [5]}},  {6, {&:ordsets.add_element/2, [6]}},  {7, {&:ordsets.add_element/2, '\a'}},  {8, {&:ordsets.add_element/2, '\b'}},  {9, {&:ordsets.add_element/2, '\t'}},  {10, {&:ordsets.add_element/2, '\n'}}]

              Merge Process

              @spec merge([StateBox]) :: StateBoxdef merge([state]), do: statedef merge(unordered) do  %StateBox{value: v, last_modified: t} = newest(unordered)  queue = unordered |>    Enum.map(fn(%StateBox{queue: q}) -> q end) |>   :lists.umerge  new(t, apply_queue(v, queue), queue)end

              MergeFirst, findStateBox, And thenumergeDropmultiple statebox, And finallyapply_queue(v, queue), Re-operate the operation once, because at this timequeueIs obtained after mergingqueueSo the calculated result is allNode (statebox)Value, which also meetsFinal consistencyIn fact, the operations here already includevBut this does not affect the final result.

              Merge Queue:[{1, {&:orddict.store/3, [:c, :c]}}, {1, {&:orddict.store/3, [:key, :a]}}, {1, {&:orddict.store/3, [:key, :b]}}]value: [c: :c, key: :b]

              CounterData Struct

              counterThe data structure of is as follows:

               @type op :: EStateBox.op() @type timestamp() :: EStatebox.Clock.timestamp() @type timedelta() :: EStateBox.timedelta() @type counter_id() :: EStatebox.Identity.entropy() | :acc @type counter_key() :: {timestamp(), counter_id()} @type counter_op() :: {counter_key(), integer()} @type counter() :: [counter_op()]

              That iscounter-> [{{timestamp,entropy|:acc},Integer}]

              Function Analysicf_inc_acc (value, age, key = {timestamp, _ id })

              Returns an auto-increment or addedcounter StateBox Event

              value --> counter = into `inc(timestamp, key)`    |    | Age  --> accumute(Age, counter)

              @params

              Value: it is a delta, an integer, that is, the maximum time of the superimposed value age: counter events. This value is used together with the timestamp in the key, it will be used in TA = (timestamp-Age), and the previous value of TA will be calculated as key: is the key of a counter event

              This function returnsop,opThe function body isEStateBox.Counter.op_inc_acc/4To insert a timestamp less than: acccounter eventIs not allowed.

              Test Case:

                  test "f_inc_acc_test" do    | # we should expect to get unique enough results from our entropy and    | # timestamp even if the frequency is high.    | fincacc =  1..1000 |> enum.map(fn(_) -> estatebox.counter.f_inc_acc(1, 1000) end)    | ctr = :lists.foldl(&estatebox.apply_op/2, [], fincacc)    | assert 1000 === estatebox.counter.value(ctr)    end

              Inc (counter_key, Integer, counter): counter

                Return a new counter with the given counter event, If there is an ":acc" at or before the  timestamp of the given key then this is a a no-op  @spec inc(counter_key, Integer, counter) ::  counter  def inc({t1, _}, _, counter = [{{t0, :acc}, _} | _]) when t1 <= t0, do: counter  def inc(key, value, counter), do: :orddict.store(key, value, counter)

              @params

              Key: counter id, in the format of {timestamp, counter_id}, counter_id = entropy |: acc, counter_id can be repeated value: operand counter: counter to be operated

              Add an additionalevent counterTo the specifiedcounter.

              Test Case

                 test "inc_test" do   | c0 = []   | c1 = EStateBox.Counter.inc({1, 1}, 1, c0)   | c2 = EStateBox.Counter.inc({2, 2}, 1, c1)   | assert 0 === EStateBox.Counter.value(c0)   | assert 1 === EStateBox.Counter.value(c1)   | assert 2 === EStateBox.Counter.value(c2)   | c1 = EStateBox.Counter.inc({3, 1}, 1, c1)   | assert 2 === EStateBox.Counter.value(c1)   end

              Merge ([counter]): counter-> merge (counter) --> prune (counter) -->: listsumerge (counter)

                 @doc """    Merge the given list of counters and return a new counter    with the union of that history.    """   def merge([counter]), do: counter   def merge(counters), do: :orddict.for_list(merge_prune(counters))

              mergeBased on the sameIdOfevent counterMerge drop.

              Note

              Suchmerge, Data loss occurs during network partitioning, unless you can determineCoordination uniqueFor details, seeTry the last example

              Example

               test "merge test" do    | c0 = []    | c1 = EStateBox.Counter.inc({1, 1}, 1, c0)    | c2 = EStateBox.Counter.inc({2, 2}, 1, c1)    | assert 2 === EStateBox.Counter.value(EStateBox.Counter.merge([c0, c1, c2]))    | assert 1 === EStateBox.Counter.value(EStateBox.Counter.merge([c0, c1, c1]))    | assert 1 === EStateBox.Counter.value(EStateBox.Counter.merge([c1]))    | assert 1 === EStateBox.Counter.value(EStateBox.Counter.merge([c0, c1])) end

              • Old counter test