Quickly determine whether a file is a text file or a binary file

Determine whether a file is a text file or a binary file can help the compression software for different files to select the corresponding algorithm, then how to determine whether a file is text or binary it.

Older versions of PKZIP are used in the following way: If more than 80% of the bytes fall to 7. In the range of 127, it is considered a text file, otherwise it is a binary file. The code is as follows:

int Isplaintext (const char *filename) {FILE *FP = fopen (filename, "RB"); long file_length; Long Plain_text_char_count = 0 ; Long Binary_char_count = 0; int Read_len; unsigned char byte; Fseek (FP, 0, Seek_end); File_length = Ftell (FP); Fseek (FP, 0, Seek_set); while ((Read_len = Fread (&byte, 1, 1, FP)) > 0) {if (Byte >= 7) plain_text_char_count++; else Binary_char_count ++; } fclose (FP); Return (Plain_text_char_count/file_length > 0.8)? 1:0; }

But there are some drawbacks to this approach, such as some of the files contain 128. 255 of letters, such as Greek letters, or this file using other language encoding including Chinese characters and other multibyte characters even using Unicode encoding can not be judged. So here's a new method for zlib, which can be used to judge Ascii,unicode, as well as some other wide character sets.

This algorithm puts a byte of data into three lists:

White list:

9 (TAB), ten (LF), (CR), 32..255

Grey list:

7 (BEL), 8 (BS), one (VT), (FF), (SUB), (ESC)

Blacklist:

0..6, 14..31

The method of judging is:

If a file contains at least one white-list byte and does not contain a byte in the blacklist, then it is a text file, otherwise it is a binary file.

The principle of the algorithm is very simple, the blacklist of the bytes of the file is almost only a binary file, and ordinary text files are almost impossible to appear in these bytes, even the Unicode and other multibyte encoding will also set a high mark character to achieve compatible ASCII.

The code is as follows:

int isPlainText2 (const char *filename) {FILE *FP = fopen (filename, "RB"); long white_list_char_count = 0; int read_len; u nsigned Char byte; while (Read_len = Fread (&byte, 1, 1, FP)) > 0) {if (byte = = 9 | | byte = = | | | byte = = 13 | | (Byte >= && byte <= 255)) white_list_char_count++; else if ((Byte <= 6) | | (Byte >= && byte <= 31)) return 0; } fclose (FP); Return (White_list_char_count >= 1)? 1:0; }

Reference: <<a Fast Method for identifying Plain Text files>> by Cosmin Truta updated in 2006-may-28