IOS development-use NSInputStream to read large files row by row

IOS development-use NSInputStream to read large files row by row

Sometimes, when reading a file, we may directly read the file into the memory at one time and then split it by row. This is good for small files, but it is not ideal when the files are large. Therefore, we can use lazily read to read files row by row and process each row of data in the background without affecting user operations. You can use NSInputStream to implement this function.

Generally, the following steps are required to read files one by one:

1. Create a buffer to store data processed each time.

2. Create a Stream object and use Stream to read a piece of data and store it in the buffer zone.

3. process this piece of data. Each time a linefeed is found, the data (including linefeeds) is sent out from the buffer zone for processing.

4. After the linefeed is not found in the data segment, the remaining data (if any) is still saved in the buffer zone.

5. Return to 2. repeat this process until the Stream object is closed after the file is read.

This category is the main character here.

#import 
 
  typedef void(^HandleBlock)(NSInteger lineNumber, NSString *line);typedef void(^CompletionBlock)(NSInteger numberOfLines);@interface CYZFileReader : NSObject/** *  init a reader object using the URL of a file * *  @param aFileURL a NSURL object to represent a file's URL * *  @return a reader object */- (id)initWithFileAtURL:(NSURL *)aFileURL;/** *  init a reader object using the file's name. *  The file must be a local file. * *  @param fileName     local file's name *  @param extention    the extention of file * *  @return a reader object */- (id)initWithLocalFileName:(NSString *)fileName withExtension:(NSString *)extention;/** *  enumerate every line in file using a self-defined handle block and *  handle completion using a self-defined completion block * *  @param block           a block to enumerate every line and handle data *  @param completionBlock a block to handle completion event */- (void)enumerateLinesUsingBlock:(HandleBlock)block completionBlock:(CompletionBlock) completionBlock;@end
 

This class has the following private attributes:

@interface CYZFileReader () 
 
  @property (strong, nonatomic) NSInputStream     *inputStream;@property (strong, nonatomic) NSOperationQueue  *queue;@property (strong, nonatomic) NSURL             *fileURL;@property (strong, nonatomic) NSMutableData     *reminder;@property (assign, nonatomic) NSInteger         lineNumber;@property (copy, nonatomic) NSData              *delimiter;@property (copy, nonatomic) HandleBlock         callBack;@property (copy, nonatomic) CompletionBlock     completionBlock;@end
 

Among them, inputStream is the file read into the stream, and queue is the operation queue, which is used to put the data processing operations in the background. FileURL is the URL of the file to be read, and reminder is the intermediate buffer to store a piece of data that is read each time. LineNumber indicates the row currently read. After the file is read, the value is the total number of rows. Delimiter is the delimiter. Here, because it is Row-by-row reading, "\ n" data format can also be customized. There are two operation blocks. For the definition, see. h.

The methods defined in the interface are very simple. The fileURL and delimiter values are assigned in the initialization method.

- (id)initWithFileAtURL:(NSURL *)aFileURL{    if (![aFileURL isFileURL]) {        return nil;    }        self = [super init];    if (self) {        self.fileURL = aFileURL;        self.delimiter = [@"\n" dataUsingEncoding:NSUTF8StringEncoding];    }    return self;}

The other is similar. The enumerate method uses custom blocks to traverse each row of an operating file. In fact, the real operation is not in this method, but in the inputStream proxy. Therefore, only simple assignment and initialization are required.

- (void)enumerateLinesUsingBlock:(HandleBlock)block completionBlock:(CompletionBlock)completionBlock{    //initial the NSOperationQueue whice can only be sequencial    if (self.queue == nil) {        self.queue = [[NSOperationQueue alloc] init];        self.queue.maxConcurrentOperationCount = 1;    }        NSAssert(self.queue.maxConcurrentOperationCount == 1, @"Cannot read file concurrently");    NSAssert(self.inputStream == nil, @"Cannot progress multiple input stream in parallel");        self.callBack = block;    self.completionBlock = completionBlock;        //we use NSInputStream to read file    //here the delegate should be retained(NO ARC) or the global variable(ARC)    self.inputStream = [NSInputStream inputStreamWithURL:self.fileURL];    self.inputStream.delegate = self;    [self.inputStream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];    [self.inputStream open];}

After setting the inputStream proxy, you can perform corresponding operations on the event of stream in the proxy.

#pragma mark - NSStreamDelegate- (void)stream:(NSStream *)aStream handleEvent:(NSStreamEvent)eventCode{    switch (eventCode) {                    case NSStreamEventOpenCompleted:            break;                    case NSStreamEventErrorOccurred:            NSLog(@"NSStreamEventErrorOccureed: error when reading file");            break;                    case NSStreamEventEndEncountered: {            [self emitLineWithData:self.reminder];          //handle last part of data            self.reminder = nil;            [self.inputStream close];            self.inputStream = nil;            [self.queue addOperationWithBlock:^{                self.completionBlock(self.lineNumber + 1);  //invoke the completion block            }];            break;        }                    case NSStreamEventHasBytesAvailable: {            NSMutableData *buffer = [[NSMutableData alloc] initWithLength:4 * 1024];            NSUInteger length = (NSUInteger)[self.inputStream read:[buffer mutableBytes] maxLength:[buffer length]];            if (length > 0) {                [buffer setLength:length];                __weak id weakSelf = self;                [self.queue addOperationWithBlock:^{                    [weakSelf processDataChunk:buffer];                }];            }            break;        }                    default:            break;    }}

Two methods are called here. One is the emitLineWithData function that "sends" data out of the buffer for processing. The idea is to convert the incoming data into a string and upload it to the callback block by combining the current number of rows.

- (void)emitLineWithData:(NSData *)data{    //get current line number    NSUInteger lineNumber = self.lineNumber;    //add current line number    self.lineNumber += 1;        //invoke the block to handle these data    if (data.length > 0) {        //get content of current line        NSString *line = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];                self.callBack(lineNumber, line);    }}

The other is the progressDataChunk method, which is called in the case of NSStreamEventHasBytesAvailable, that is, the data that is read each time is passed into the buffer zone (note that the last time may be left behind) the data is then processed separately by row. Here we use a custom data category method to split data based on a certain character and paste this part of code later.

- (void)processDataChunk:(NSMutableData *)buffer{    if (self.reminder == nil) {        self.reminder = buffer;    } else {        //last chunk of data have some data (part of last line) reminding.        [self.reminder appendData:buffer];    }        //separate self.reminder to lines and handle them    [self.reminder obj_enumerateComponentsSeparatedBy:self.delimiter usingBlock:^(NSData *data, BOOL isLast) {                //if it isn't last line. handle each one        if (isLast == NO) {            [self emitLineWithData:data];        } else if (data.length > 0) {            //if last line has some data reminding, save these data            self.reminder = [data mutableCopy];        } else {            self.reminder = nil;        }    }];}

The code in the block above indicates that if the current row of data is not the last row (this indicates that this is a complete row), it will be sent for processing. If the last row has data (indicating that the data is disconnected in the middle), it is retained to the buffer zone. If it is the last row but there is no content (just to read the end of the row), clear the buffer.

Finally, paste the separate class code of data.

@implementation NSData (EnumerateComponents)- (void)obj_enumerateComponentsSeparatedBy:(NSData *)delimiter usingBlock:(EnumerateBlock)block{    //current location in data    NSUInteger location = 0;        while (YES) {        //get a new component separated by delimiter        NSRange rangeOfDelimiter = [self rangeOfData:delimiter                                             options:0                                               range:NSMakeRange(location, self.length - location)];                //has reached the last component        if (rangeOfDelimiter.location == NSNotFound) {            break;        }                NSRange rangeOfNewComponent = NSMakeRange(location, rangeOfDelimiter.location - location + delimiter.length);        //get the data of every component        NSData *everyComponent = [self subdataWithRange:rangeOfNewComponent];        //invoke the block        block(everyComponent, NO);        //make the offset of location        location = NSMaxRange(rangeOfNewComponent);    }        //reminding data    NSData *reminder = [self subdataWithRange:NSMakeRange(location, self.length - location)];    //handle reminding data    block(reminder, YES);}

It is simply the process of retrieving and processing fields according to the range. You can see the comments I added.

Pay attention to the delegate setting when using it. Here, the delegate of inputStream is set to self. Therefore, you must define the object of this class as a global variable (ARC) or retain (no arc ).

Refer to the blog: crash problem after setting delegate for ios