Programming clojure-concurrency

Document directory

• 6.1 The problem with locks
• 6.2 refs and software transactional memory
• 6.3 use atoms for uncoordinated, synchronous updates
• 6.4 Use agents for asynchronous updates
• 6.5 managing per-thread state with vars

Clojure concurrency, which brother writes compared system, http://www.blogjava.net/killme2008/archive/2010/07/archive/2010/07/14/326027.html

 

Concurrency is a fact of life and, increasingly, a fact of software.

Why concurrency?
• Expensive computations may need to execute in parallel on multiple cores.
• Tasks that are blocked waiting for a resource.
• User interfaces need to remain responsive While locking Ming longrunning tasks.
• Operations that are logically independent are easier to implement

What is the concurrency problem? Undoubtedly, it is status synchronization.
If you run different jobs like pure FP, there is no problem.
However, when coordination and synchronization are required, this problem becomes complicated.
To solve this problem, clojure provides a powerful concurrent library and divides the States to be synchronized into four categories, using different APIs to handle

Clojure provides a powerful concurrency library, consisting of four APIs that enforce different concurrency models: refs, atoms, agents, and vars.
•RefsManage coordinated, synchronous changes to shared state.
•AtomsManage uncoordinated, synchronous changes to shared state.
•AgentsManage asynchronous changes to shared state.
•VarsManage thread-local state.

 

6.1 The problem with locks

In traditional languages, the status is mutable by default. Therefore, when concurrent writing is in place, all States that may be written concurrently must be locked. if you accidentally miss out, there will be a big problem. locking is not a simple task. race conditions and deadlocks are complicated... it can be said that this solution has no aesthetic feeling at all. Is there a better solution?

Yes, There is. in clojure, immutable state is the default. most data is immutable. the small parts of the codebase that truly benefit from mutability are distinct and must explicitly select one or more concurrency APIs. using these APIs, you can split your models into two layers:

•A functional modelThat has no mutable state. Most of your code will normally be in this layer, which is easier to read, easier to test, and easier to run concurrently.
•A mutable ModelFor the parts of the application that you find more convenient to deal with using mutable state (despite its disadvantages ).

Of course, this is one of the core strengths of FP, so we can properly handle the variable state.
The advantage of clojure is that all States are immutable by default, so you only need to pay attention to a small part of the truly mutable states (by default, you need to consider full codebase, any omission will cause a big problem ). the mutable state is explicitly independent for ease of management.

 

6.2 refs and software transactional memory

Most objects in clojure are immutable.
When you really want mutable data, creating a mutable reference (REF) to an immutable object.
The clojure object itself is immutable. If you need variable data, you can create a variable reference and direct it to different immutable objects.

Refs supports synchronous change state, and supports changing multiple states in transaction at the same time.

Create ref

For player applications, a song itself is a constant object, but the current playing of the song is a changing state
Create ref, current-track

(def current-track (ref "Mars, the Bringer of War")) 

Read contents of the reference, you can callDeref (@ reader macro ),

(deref current-track) 
"Mars, the Bringer of War" 
@current-track 
"Mars, the Bringer of War"

Modify ref, ref-Set

(ref-set reference new-value)

An error is reported when you directly call REF-set to modify the reference. This is a good protection mechanism to prevent misoperation.
In clojure, transaction can be used for encapsulation, but lock must be used for general languages, depending on the implementation method.

Because refs are mutable, you must protect their updates. In particular, you wocould use a lock for this purpose.
In clojure, you can use a transaction. transactions are wrapped inDosync:

(dosync & exprs)
(dosync (ref-set current-track "Venus, the Bringer of Peace")) 
"Venus, the Bringer of Peace"

In the above example, the ref switching is completed, and the song object itself has not changed.

Transactional properties, ensure ACI, not d
Like database transactions, STM transactions guarantee some important properties:
• Updates are atomic.
• Updates are consistent.
• Updates are isolated.
Databases provide the additional guarantee that updates areDurable.
Because clojure's transactions areIn-memory transactions, Clojure does not guarantee that updates are durable.

Transaction contains multiple statements,

(def current-track (ref "Venus, the Bringer of Peace")) 
(def current-composer (ref "Holst")) 

(dosync 
(ref-set current-track "Credo") 
(ref-set current-composer "Byrd"))

Read-and-write, alter, commute

Ref-set, directly overwriting write, relatively simple
More commonly used is read-and-write. For example, to update a simple accumulators, you must first know the current value.

(alter ref update-fn & args...)  ;ref = update-fn(ref, &args) 

Messager application, update message

(defn add-message [msg] 
  (dosync (alter messages conj msg)))
 

How STM works: MVCC
Clojure's STM uses a technique called multiversion concurrency control (MVCC), which is also used in several major databases.
This mechanism is also widely used in databases, such as couchdb. For more information, see practical clojure-introduction.
At the same time, clojure ensures MVCC space utilization efficiency through persistent data structures
This is why clojure can easily implement transaction and ensure ACI, because all updates are visible only when the reference is switched.

Therefore, the read-and-write conflict must be resolved. How can this problem be solved if another transaction modifies the value during the read-write process?

This means that the current transaction will be forced to retry to ensure the execution sequence in the transaction.

 

What if you don't care that another transaction altered a reference out from under you in the middle of your transaction?

If another transaction alters a ref that you are trying to commute, the STM will not restart your transaction. Instead, it will simply run your commute function again, out of order.

If you can tolerate another transaction alter reference in the transaction process, use commute
When a conflict occurs, the entire transaction will not be restarted, but the commute will be run again. This means that the commute update can be executed at any time (otherwise there will be a problem)
This makes it easier for the STM system to perform reorder optimization and tradeoff, in exchange for higher concurrency and better performance.

(commute ref update-fn & args...)

Do not use commute unless otherwise specified, because the alter logic must be correct, and misuse of commute may cause errors.

Validation to Refs, adding Constraints

Validation function to the messages reference that guarantees that all messages have non-Nil values for: sender and: text:

(def validate-message-list 
    (partial every? #(and (:sender %) (:text %))))
(def messages (ref () :validator validate-message-list))

 

6.3 use atoms for uncoordinated, synchronous updates

Atoms are a lighter-weight mechanic than refs.
Where multiple ref updates can be coordinated in a transaction, atoms allowUpdatesOf a single value, uncoordinated with anything else.

Atoms is a lightweight refs with higher efficiency. It only allows updating a single state.
Therefore, transaction encapsulation is not required to reduce overhead.

Atoms do not participant in transactions and thusDo not require a dosync. To set the value of an atom, simply callReset!

(def current-track (atom "Venus, the Bringer of Peace"))
(reset! current-track "Credo") ;ref-set

Why reset! And swap! Add all !?

(swap! an-atom f & args) ;alter
(def current-track (atom {:title "Credo" :composer "Byrd"}))
(swap! current-track assoc :title "Sancte Deus")

 

6.4 Use agents for asynchronous updates

Agent for asynchronous updates

(def counter (agent 0))

Update the agent command,Send. Very good image, asynchronous is to send the past to return.

(send agent update-fn & args)
(send counter inc)

Notice that the call to send does not return the new value of the agent,ReturningInsteadAgent itself.

If you want to be sure that the agent has completed the actions you sent to it, you can callAwaitOrAwait-:

(await & agents) 
(await-for timeout-millis & agents) 

Validating agents and handling errors

The agent can also add constraints like ref.

(DEF counter (Agent 0: validator number ?))

If the valid condition is not met, an error is returned. But for all Asynchronization, how does one handle errors?

(Send counter (FN [_] "boo "))

Check the agent and find an error

@ Counter
Java. Lang. Exception: agent has errors

Use Agent-errors to view specific error information.

(Agent-errors counter)
(# <Illegalstateexception...>)

Finally, clear the error through clear-agent-errors, and check the agent again. No error is reported.

(Clear-agent-errors counter)
@ Counter
0

 

Including agents in transactions

Transactions shoshould not have side effects, because clojure may retry a transaction an arbitrary number of times.
However, sometimes you want a side effect when a transaction succeeds.
Agents provide a solution.
If you send an action to an agent from within a transaction, that action will be sent exactly once, if and only if the transaction succeeds.

Because of the conflict, transaction may be retried many times. Therefore, transaction cannot contain side effects, such as file writing and IO. Otherwise, transaction will be executed many times, we all know the implementation of clojure transaction, and it is impossible to perform rollback operations like MySQL.
Therefore, the agent acts as a good solution, because only when the transaction is successful will the agent action be sent once.

Here we talk about side effects, which are generally Io operations and time-consuming. Therefore, it is not suitable to send with send. You need to use send-off.

For more information about how the agent works, see this blog,Http://www.blogjava.net/killme2008/archive/2010/07/archive/2010/07/archive/2010/07/19/326540.html

The agent itself is a common Java object, which maintains a state and a queue internally
Then the thread will fetch the action in the queue and process it. These threads are placed in the thread pool.
Two thread pools,
Fixed-size thread pool (number of CPU cores plus 2) to process the action sent by send
There is no size limit (depending on the memory) on the thread pool, processing the action sent by send-off
Most importantly, these thread pools are shared by all threads. Therefore, sending with send will block updates of other agents.
The implementation is not complex. A typical producer-consumer Mode

 

6.5 managing per-thread state with vars

Refer to this blog
Http://www.blogjava.net/killme2008/archive/2010/07/archive/2010/07/archive/2010/07/archive/2010/07/23/326976.html

There are several methods for binding var,

Root binding, defined by def, shared by all threads

Local binding, defined by let, that is, static lexical binding, because it only works in lexical Scope

Thread-local dynamic binding, defined by binding, the binding within the thread, not limited to lexical Scope

Let and binding are different through examples,

(DEF Foo 1)
(Defn print-Foo [] (println Foo ))
(Let [Foo 2] (print-Foo); is not in the same lexical scope, so let does not work
1
(Binding [Foo 2] (print-Foo); in the same thread, binding takes effect.
2

This is because the let binding is static. It does not change the value of the variable Foo, but uses a lexical scope Foo to "mask" the external Foo value.
Binding stores a binding value in threadlocal of the thread outside the root binding of the variable, so you can see it in this thread, but not limited to lexical scope.

 

Binding is used to temporarily modify the function logic

Add memorize to fib only within the thread range

user=> (defn fib [n]
         (loop [ n n r 1]
            (if (= n 1)
                r
                (recur (dec n) (* n r)))))  
user=> (binding [fib (memoize fib)] 
                (call-fibs 9 10))
3991680

Used occasionally, dynamic binding has greatPower. But it shocould not become your primary mechanic for extension or reuse. functions that use dynamic bindings areNot pure FunctionsAnd can quickly lose the benefits of clojure's functional style.
Use it with caution. Not pure introduces complexity, which is inconsistent with the FP's original intention.

 

Working with Java callback APIs

Several Java APIs depend on callback event handlers.
XML parsers such as sax depend on the user implementing a callback handler interface.

These callback handlers are written with mutable objects in mind. Also, they tend to be single-threaded.
In clojure, the best way to meet such APIs halfway is to use dynamic bindings. this will involve mutable references that feel almost like variables, but because they are used in a single-threaded setting, they will not present any concurrency problems.

Clojure provides the set! Special form for setting a thread-local dynamic binding:
(Set! VaR-symbol New-value)

Set! Shocould be used rarely. In fact, the only place in the entire clojure core that uses set! Is the clojure implementation of a sax contenthandler.

Clojure can be set! To modify the thread-local dynamic binding, but use set with caution !, It is only used to process Java callback.