An alternative to memorystream

ArticleDirectory

• Capacity
• Speed (access time)

Introduction

This article describes the ambiguous outofmemoryexception that is often triggered when memorystream is used to process large data sets, and introduces a class --Memorytributary, which can be used to replace. Net's built-in memorystream and support the processing of large data.

Background

When trying to use memorystream to process large data (in the order of tens of MB), it usually causes an outofmemoryexception. Is this because, as it is named, the system memory limit is exceeded? But it is actually the virtual address space of the process.

When a process applies for memory from windows, the memory manager does not allocate address space from Ram, but the "page"-storage block (usually 4 KB) it can exist in Ram, disks, or any memory manager that can decide where to store them. The page is mapped to the address space of the process. Therefore, if a process tries to access the memory from [0xaf758000], it is actually starting to access the [496th page, whether or not it happens to be on the 496th page. You can allocate sufficient memory of any size to the process, as long as the disk space is sufficient, and you can use a large part of it as the virtual address space, which is suitable for any time. In this allocation process, a large number of small fragments are generated (this refers to the storage block ).

This is because the process address space is fragmented: most of them are occupied by the operating system, such as applications.ProgramImages, libraries, and other pre-allocated spaces. Once the Memory Manager allocates the requested memory equivalent to the page size, it must be mapped to the (virtual) address space in the process. When there is not enough continuous space in the remaining address space, the page cannot be mapped, leading to memory allocation failure, thus triggering an outofmemoryexception.

This process does not exhaust space and addresses. It runs on consecutive addresses. To see this (if you are using a 64-bit system), compile the following program on the x86 target platform and run it, and then compile it as the x64 target platform to see the difference between the two.

 1   Static   Void Main ( String [] ARGs)
 2 {
 3 List < Byte []> Allocations = New List < Byte []> ();
 4  
5       For ( Int I = 0 ; True ; I ++)
 6 {
 7           Try 
 8 {
 9 Allocations. Add ( New   Byte [I * 10 ]);
10 }
 11           Catch (Outofmemoryexception E)
 12 {
 13 Console. Write ( String . Format ( "  Saved med {0} allocations  " , I ));
 14 }
 15 }
 16 }

. Net currently implements the memorystream using a byte array as the backup storage. When the content to be written exceeds the length of the array, the capacity will be automatically doubled (Old Chen note: this is the memory allocation mechanism of the. NET array itself ). Based on the program behavior, memorystream based on this backup storage mechanism will soon need more memory than the available virtual address space.

Code Use Cases

This solution does not apply for more continuous memory to store the data contained in the stream. Memorytributary uses a dynamic set of 4 kb internally as the backup storage for on-demand distribution.

Memorytributary is derived from stream, so you can use it like other streams, similar to memorystream.

Memorytributary is just an alternative to memorystream, but it cannot be used to replace memorystream in any situation. The following are some considerations:

1. Memorytributary does not fully implement all constructors of memorystream (because the capacity of memorytributary is not artificially limited ). The initial memorytributary capacity starts from a byte.
2. Memorytributary is a subclass of stream, rather than memorystream. Therefore, it cannot be used where only memorystream members are accepted.
3. MemorytriburaryThere is no backup storage to implement getbuffer (). The similar function is toarray (), but use it with caution!

Note the following when using memorytributary:

1. Blocks are allocated as needed during access (for example, during read or write operations ). Check the position again before reading to ensure that the read operation is performed within the stream range.

   1   //  Create a new memorytributary instance: The length is 0, the position is 0, and no memory is allocated.  
     2 Memorytributary d = New Memorytributary ();
    3  
    4   // Because the length is 0,-1 is returned and no memory is allocated.  
     5   Int A = D. readbyte ();
    6  
   7   //  Length is now set to 10000 bytes, but no memory is allocated!  
     8 D. setlength ( 10000 );
    9  
    10   // Three memory blocks are allocated.  ,
     11   // But B is undefined, because they have not completed initialization.  
     12   Int B = D. readbyte ();

3. The memory is allocated to consecutive blocks. That is to say, if the first block is block 3, Block 1 and block 2 will be automatically allocated;
5. Memorytributary contains a toarray () method, but this is not safe;
7. As an alternative, memorytributary uses the readfrom () and writeto () Methods to operate big data.

Performance indicators

Memorystream and memorytributary are both unpredictable in terms of capacity and speed, because they depend on many factors. An important factor is the memory fragmentation of the current process. A process with a large amount of memory allocated will cause a large amount of memory fragmentation.

Capacity

Stream	Average crash critical value (MB)
Memorystream	488
Memorytributary	1272

Speed (access time)

Streaming test execution time comparison (MS)
Read/write capacity (MB)	Memorystream	Memorytributary (4 kb block)	Memorytributary (64 kB block)	Memorytributary (1 MB block)
10	10	13	11	7
	3	5	3	3
	3	6	3	3
	3	5	3	3
	4	5	3	3
	3	6	3	3
100	100	148	123	52
	34	54	42	35
	34	48	35	34
	35	47	36	35
	34	48	36	35
	35	51	35	35
500	516	390	290	237
	167	222	184	170
	168	186	154	167
	167	187	151	168
	167	186	151	168
	167	185	153	168
1,000	1185	1585	1299	485
	347	547	431	344
	343	463	350	345
	338	462	350	345
	3377	461	349	345
	339	465	351	343

Code download

Go to the original address download: http://www.codeproject.com/Articles/348590/A-replacement-for-MemoryStream