20. python file processing,
We generally follow this logic when processing files: open files, operate files, and save and close files.
But in python, there are several steps: create a file object, operate on the file object (read, write, and so on), and close the file.
Because the differences between file operations in python2.x and python3.x are large, 3. x can accept more parameters.
Therefore, the following content is applicable to python2.x, which is precisely python2.7.
Next we will analyze them one by one:
1. Create a file object
There are two ways to create a file object. The first one is to use the factory function file (name [, mode [, buffering])-> file object, the other is to call the built-in function open (name [, mode [, buffering])-> file object.
In fact, there is no essential difference between the two. In fact, open is also implemented by calling file, but python officially recommends that we use open to create file objects, so we also comply with official suggestions, the following demonstration is based on open.
First, when we use open to create a file object, we need some parameters. The name is required and it acceptsStringIndicates the file name. The file name can use an absolute or relative path, while the mode indicates that the file name acceptsStringIndicates the mode used to create this file object, and different modes will also affect the file. The following is a summary table about the modes:
| File Mode |
Operation |
| R |
Open in read-only mode,Default |
| RU or Ua |
Open in Read mode and support for common line breaks (PEP 278) |
| W |
Open in write mode (clear if necessary) |
| A |
Open in append mode (starting from EOF, create a file if necessary) |
| R + |
Open in read/write mode |
| W + |
Open in read/write mode (see w) |
| A + |
Open in read/write mode (see) |
| Rb |
Open in binary read mode |
| Wb |
Open in binary write mode (see w) |
| AB |
Open in binary append mode (see) |
| Rb + |
Open in binary read/write mode (see r +) |
| Wb + |
Open in binary read/write mode (see w +) |
| AB + |
Open in binary read/write mode (see a +) |
The following is a further explanation of these models:
R: as the name suggests, reading is performed on the object created in the same mode, but cannot be written. If the file to be opened does not exist, an IOError exception is thrown.
W: open in write-only mode. In this mode, you can only perform write operations, but not read operations. If the opened file exists, clear the original file and open it again, if the opened file does not exist, create the file and open it again. This mode is relatively dangerous, because no matter whether the file is stored or not, the last operation is an empty file. At this time, the write will start from the beginning, that is, the file pointer is at the beginning of the file.
A: It is opened in append mode. In this mode, the file cannot be read, but the write operation can be performed. If the opened file exists, open the file and move the file pointer to the end of the file. At this time, any newly written content will be at the end of the file. If the file does not exist, create a new file and write it from the beginning. Unlike w, the original file is not cleared when the opened file exists, but is appended to write.
R +: r itself cannot perform write operations. After it is extended to r +, data can be written. The r + open file does not empty the file. When writing, the pointer starts at the beginning, that is, the content in my file is 123456. When I use r + to write the 'abc' string, the file content is changed to abc456. Similarly, if the object does not exist, an exception is thrown.
W +, a +: Because w and a do not have the read permission, you can perform read operations after the plus sign is expanded. Other actions are the same.
U: supports common line breaks. different platforms use different symbols to indicate the end of a row, for example, \ n, \ r, or \ r \ n. However, if you write only one method to process line breaks, you cannot do the same on other platforms. It is too troublesome to write a method for each platform. So python introduced UNS in Python 2.3. When you use the 'U' flag to open a file, all the row delimiters (or the row Terminator, no matter what it was originally) are input through Python (for example: when read () is returned, NEWLINE (\ n) is replaced ). (The 'ru 'mode also supports the 'rb' option ). This feature also supports files that contain different types of row terminologies. The newlines attribute of the file object records the row terminator of the file that it "saw.
B: The binary mode is enabled in binary mode by default in linux. Therefore, this option is optional for linux. However, if the binary mode is required, we recommend that you write it in to improve the cross-platform capability. This mode can be used to write not only strings but also buffer objects.
Buffering indicates the buffer used to indicate access to files. 0 indicates no buffer, 1 indicates only one row of data is buffered, and any other value greater than 1 indicates that the given value is used as the buffer size. If this parameter is not provided or a given negative value is used, the system uses the default buffer mechanism, which is used for row Buffering on any electronic Reporter (tty) devices, while other devices use normal buffering. Generally, use the default system mode.
To sum up, creating a file object does not mean reading the file content, which is different from the definition of opening a file in our daily life. When opening a file in Python, you get the file handle, that is, the file operation entry. to read the file content, you also need to read it into the python memory, that is, the so-called input.
2. Input
The so-called input is to read the file content into python, there are the following methods.
1. read ([size])-> read at most size bytes, returned as a string.
If the size (in bytes) is negative or is not given, the EOF is read all the time, that is, the end of the file. ReturnsString(Including line breaks ). Note: In non-blocking mode, data may be returned even if it is lower than the required value, even if there is no size parameter.
This method is one-time reading, that is, loading 1 GB of files will directly occupy 1 GB of memory, so it is not suitable for reading large files.
f = open('test.txt')a = f.read()print aprint repr(a)
File Content:
Code output:
Note the line break here. The file uses three carriage return characters, and there are three line breaks.
We have used the repr () function in python encoding to obtain the encoding of a string. Here we use this function to see what encoding the opened file is.
First, my encoding declaration is: # coding = UTF-8
Encoding Declaration: # coding = gbk
It is found that the returned strings are the same no matter what the encoding declaration is. The encoding statement does not affect the encoding of the string read from the file.
Then, we use UTF-8 encoding to write a file:
#! /Usr/bin/env python # coding = utf-8f = open('test1.txt ', 'w') f. write ('first line \ n Second line \ n third line \ n') f. close ()
Read the data again:
Same as above.
Use gbk to write a file:
#! /Usr/bin/env python # coding = gbkf = open('test2.txt ', 'w') f. write ('first line \ n Second line \ n third line \ n') f. close ()
Read it again:
The results are different.
Note: The encoding of the string read from the file object is irrelevant to the python encoding declaration. It is only related to the encoding used for saving the file.
The python encoding declaration only affects the strings created in python. Therefore, based on this feature, I create a corresponding encoded string in python and save it to the file, all characters in the file are encoded differently. I hope you will not be confused.
Here, we can see how python converts the file encoding.
Encoding conversion is actually for strings, and all the built-in methods used in the string are:
S. decode ([encoding [, errors])-> object
This method returns the decoded string.
Encoding -- the encoding to use, such as "UTF-8" (default ). Errors -- set a solution for handling different errors. The default value is 'strict ', indicating a UnicodeError caused by an Encoding Error. Other possible values include 'ignore', 'replace ', 'xmlcharrefresh', 'backslashreplace', and any value registered through codecs. register_error.
Sample Code:
f = open('test2.txt','r')a = f.read()b = a.decode(encoding='gbk')print bprint repr(b)
It can be seen that After decoding, the Unicode string is obtained.
Next let's take a look at how to encode it.
S. encode ([encoding [, errors])-> object
This method returns the encoded string.
Encoding -- the encoding to use, such as "UTF-8" (default ). Errors -- set a solution for handling different errors. The default value is 'strict ', indicating a UnicodeError caused by an Encoding Error. Other possible values include 'ignore', 'replace ', 'xmlcharrefresh', 'backslashreplace', and any value registered through codecs. register_error.
Sample Code:
f = open('test2.txt','r')a = f.read()b = a.decode(encoding='gbk')c = b.encode(encoding='utf-8')print cprint repr(c)
The result is a UTF-8 encoded string, that is, the conversion of gbk --> Unicode --> UTF-8 is completed. Now, let's take a look at the content in python encoding to understand the meaning of Unicode as a bridge.
f = open('test2.txt','r')a = f.read()f.close()b = a.decode(encoding='gbk')c = b.encode(encoding='utf-8')f = open('test2.txt','w')f.write(c)f.close()
The above code can be used to complete file encoding and conversion. Of course, there is still some room for optimization. Here, it is only used as a logic demonstration.
The following describes the input method.
2. readline ([size])-> next line from the file, as a string.
Read a row at a time, retain the line break, and return a String object. When the size (-1 by default, indicates that the read-to-row Terminator is not negative, it is used to limit the maximum number of bytes to be read. When the number of bytes is set to less than the actual number of bytes in a row, an incomplete row is returned. If it is exceeded, it will not be affected. If it is EOF, that is, the end of the file, an empty string is returned. With the for loop, you can perform row-by-row traversal. Because the return is a string, you can also use the string method for the returned object, such as codec.
f = open('test2.txt', 'r')for x in range(2): print f.readline()
Because the print keyword adds a line break by default, and the file has another line break, two line breaks will appear, and the output line is displayed. You can add oneCommaCancel the default print action.
f = open('test2.txt', 'r')for x in range(2): print f.readline(),
In addition, pay attention to one problem. You do not need to think about the following loop:
F = open('test2.txt ', 'R') for x in f. readline (): # It is equivalent to reading a row and traversing each byte print x # print a byte. In UTF-8, three bytes are required for displaying Chinese characters, the output is garbled print repr (x) # check whether it is a byte.
Each time a row is read, the memory consumption is naturally low, but it is difficult to know the number of lines in the file, so it is not an ideal Traversal method.
3. readlines ([size])-> list of strings, each a line from the file.
The method does not return a string like the other two input methods. It reads all (remaining) rows and uses them as a string.List. The optional parameter size indicates the size of the returned large bytes. If it is greater than 0, all returned rows should be about size bytes (probably slightly larger than this number, because the buffer size needs to be pooled. For example, the buffer size can only be a multiple of 4 K. If the size is 15 k, it may return 16 k ).
Because a list containing all rows is returned, you can traverse it directly. Note the difference with the above.
f = open('test2.txt', 'r')for x in f.readlines(): print x,
However, it also reads the entire file at a time, so the memory usage is also very large.
At this time, there is a more efficient xreadlines method, and the parameters are the same, but in essence it is a generator, that is, each call returns a row, and reading is performed row by row during iteration, higher efficiency.
However, this is not the best method. The best method is to directly iterate the file object:
f = open('test2.txt', 'r')for x in f: print x,
This is a new feature introduced in Python and after file iteration. file objects become their own iterators, which means users do not have to call read *() method To iterate each row of a file in the for loop. In addition, we can also use the next method of the iterator. file. next () can be used to read the next row of the file. Like other iterators, Python also triggers a StopIteration exception after all row iterations are completed. However, the for Loop will automatically call the next method and handle exceptions caused by iterations. Therefore, direct iteration of file objects becomes the best practice.
Another obsolete method is readinto (), which reads the number of bytes to a writable buffer object, which is the same type as the objects returned by the discarded buffer () built-in function. (Since buffer () is no longer supported, readinto () is discarded .)
3. Output
1. write (str)-> None. Write string str to file.
Write a string to the file, which is not demonstrated here.
2. writelines (sequence_of_strings)-> None. Write the strings to the file.
Take a string list as a parameter and write them into the file. The line terminator is not automatically added. Therefore, you must add the line terminator to the end of each line before calling writelines.
Note that there is no "writeline ()" method here, because it is equivalent to calling the write () method using a single line string ending with a row Terminator.
4. Movement of pointers in the file
The pointer in the file is relative to the cursor we usually see. It represents the position of various operations. For example, when we use r + to open a file and write something, the cursor is at the beginning of the file, so the example at the beginning will overwrite the original characters, when Mode a is opened, the cursor is at the end of the file. Therefore, in Mode a, the content written will be appended at the end of the file.
1. seek (offset [, whence])-> None. Move to new file position.
Move the cursor position in the file from whence (0 indicates the start of the file; 1 indicates the current position, the current position is determined by the open mode, or the cursor has been moved before; 2 indicates the end of the file) offset off byte. When off is positive, it moves to the right. If it is negative, it moves to the left. When the cursor starts with a file, it can only be positive; when it ends with a file, it can only be negative. Although some files support the cursor exceeding the end, it is better to not exceed.
2. tell ()-> current file position, an integer (may be a long integer ).
Returns an integer (which may be a long integer) indicating the position of the current cursor.
3. truncate ([size])-> None. Truncate the file to at most size bytes.
Truncates a file to a large size byte. The default value is the current file location (size = file. tell ()). The so-called truncation is the reservation at the front of the cursor, and the rest is removed.
5. Close save
1. close ()-> None or (perhaps) an integer. Close the file.
Close the file. If the file is not closed, all operations on the file are saved in the buffer. Only when the file is closed can the content in the buffer be written to the disk.
2. flush ()-> None. Flush the internal I/O buffer.
Save the content in the buffer to the disk without closing the file.
6. Other Methods
1. isatty ()-> true or false.
Determine whether the file is a tty-like device
2. x. next ()-> the next value, or raise StopIteration
Returns the next row of the file (similar to file. readline (). The StopIteration exception is thrown when no other rows exist.
7. file object related attributes
File. closed True indicates that the file has been closed; otherwise, it is False.
File. encoding used by the encoding file-When Unicode strings are written into data, they will automatically use file. encoding is converted to a byte string; if file. use the default encoding when encoding is set to None.
The access mode used when the file. mode file is opened
File. name file name
File. when newlines does not read the row delimiter, it is None, and only one row separator is a string. When the file has multiple types of row Terminator, is a list containing all the currently encountered row terminologies.
If file. softspace is 0, a space character is added after a data output, and 1 indicates no space is added. This attribute is generally not required by programmers and is used internally by the program.
We will summarize the file operations and revise the errors or supplements.