How to eliminate mnesia overload events

Document directory

• Synchronous writes
• Mnesia application configuration
• How it works

If you're using mnesia disc_copies tables and doing a lot of writes all at once, you're 've probably run into the following message

=ERROR REPORT==== 10-Dec-2008::18:07:19 ===
Mnesia(node@host): ** WARNING ** Mnesia is overloaded: {dump_log, write_threshold}

This warning event
Can get really annoying, especially when they start happening every
Second. But you can eliminate them, or at least drastically reduce their
Occurance.

Synchronous writes

The first thing to do is make sure to use sync_transaction
Or sync_dirty. Doing synchronous writes will slow down your writes in
Good way, since the functions won't return until your record (s) have
Been written to the transaction log. The alternative, which is
Default, is to do asynchronous writes, which can fill transaction log
Far faster than it gets dumped, causing the above error report.

Mnesia application configuration

If synchronous writes aren't enough, the next trick is to modify 2 obscure configuration parameters
.
The mnesia_overload event generally occurs when the transaction log
Needs to be dumped, but the previous transaction log dump hasn't
Finished yet. Tweaking these parameters will make the transaction log
Dump less often, and the disc_copies tables dump to disk more often.Note
:
These parameters must be set before mnesia is started; changing them
Runtime has no effect. You can set them thru the command line or in a config
File.

Dc_dump_limit

This variable controls how often disc_copies tables are dumped from
Memory. The default value is 4, which means if the size of the log is
Greater than the size of table/4, then a dump occurs. To make table
Dumps happen more often, increase the value. I 've found setting this
40 works well for my purposes.

Dump_log_write_threshold

This variable defines the maximum number of writes to the transaction
Log before a new dump is already med. The default value is 100, so a new
Transaction Log dump is saved med after every 100 writes. If you're
Doing hundreds or thousands of writes in a short period of time, then
There's no way mnesia can keep up. I set this value to 50000, which is
Huge increase, but I have enough RAM to handle it. If you're worried
That this high value means the transaction log will rarely get dumped
When there's very few writes occuring, there's also
Dump_log_time_threshold configuration variable, which by default dumps
The log every 3 minutes.

How it works

I might be wrong on the theory since I didn't actually write or design mnesia
,
But here's my understanding of what's happening. Each mnesia Activity
Is recorded to a single transaction log. This transaction log then gets
Dumped to table logs, which in turn are dumped to the table file on
Disk. by increasing the dump_log_write_threshold, transaction log dumps
Happen much less often, giving each dump more time to complete before
The next dump is triggered. and increasing dc_dump_limit helps ensure
That the table log is also dumped to disk before the next transaction
Dump occurs.

 

Ref: http://streamhacker.com/2008/12/10/how-to-eliminate-mnesia-overload-events/