In Linux, how does one open a large text file?

In the "Big Data" era, we often encounter situations where there are larger or larger GB of text files. Suppose we need to manually search for and edit these large files, or to solve some specific problems, we need to manually analyze multiple log files on GB. Traditional text editing software is not very effective in processing such large files. When we try to open a large file, it is often frustrated by insufficient memory.

If you are a savvy System Administrator, you may use a combination of cat, tail, grep, sed, awk, and other commands to open and edit a text file. In this tutorial, I will talk about how to open and edit a large text file in Linux.

Vim LargeFile plug-in

Vim text editor has a large number of plug-ins or scripts), which can expand VIM functions. One of the Vim plug-ins is the LargeFile plug-in.

The LargeFile plug-in allows you to quickly load and edit large files. It is implemented by disabling VIM functions such as events, rollback, and syntax highlighting.

To install the LargeFile plug-in on VIM, first confirm whether VIM has been installed.

In Debian, Ubuntu, or Linux Mint:

$ sudo apt-get install vim 

In the Fedora, CentOS, or RHEL system:

$ sudo yum install vim-enhanced 

You can download the LargFile plug-in from Vim website. The latest version number is 5. The downloaded file will be saved as Vimball and ended with. vba ).

To install the plug-in your home directory, use VIM to open the. vba file, as shown below.

$ gunzip LargeFile.vba.gz$ vim LargeFile.vba 

Enter ": so %" in the VIM form and press enter to install the plug-in your home directory.

Enter ": q" to exit VIM.

This plug-in will be installed in ~ /. Vim/plugin/LargeFile. vim. Now you can use VIM as usual.

When a "large" file is loaded in VIM, this plug-in is used to disable events, rollback, syntax highlighting, and other functions. By default, files larger than 100 MB are considered as "large files" by the plug-in ". To change the default settings, You can edit the settings ~ /. Create a vimrc file if it does not exist ).

To minimize the size of a large file to 10 MB ~ /. Add in vimrc

  
  

   
   
let g:LargeFile=10 
  
  

Although LargeFile can speed up file loading, VIM itself does not support editing a large file, because it will load the entire file into the memory at once. For example, if you use VIM to load a 1 GB file, it will occupy a lot of memory and swap space, as shown in the top output.

Therefore, if your file is significantly larger than the physical memory of your Linux system, consider other options, as described below.

Glogg log Resource Manager

If you only need to view a text file and do not edit it, you can consider glogg. It is an independent log analyzer based on the graphic user interface. This glogg analyzer supports filtering and filtering the text files to be opened through regular expressions and wildcards, so that users can only see the content they actually follow.

Install glogg On Debian (Wheezy or later), Ubuntu, or Linux Mint:

$ sudo apt-get install glogg 

Install glogg in the Fedora (17 or later) system:

$ sudo yum install glogg 

Use glogg to open a text file:

$ glogg test.log 

Glogg can quickly open a large text file. It took me about 12 seconds to open a 1 GB log file.

In the "Text" area, you can enter a regular expression and click the "Search" button. It supports expression case-sensitive Search and automatic refresh. After searching, the filtered results are displayed at the bottom of the form.

Compared with VIM, glogg is more lightweight. After loading a 1 GB log file, it only uses 83 MB of physical memory.

JOE style editor

JOE is a lightweight terminal-based text editor released under GPL. JOE is a rare text editor that supports large files. It can open and edit files larger than the physical memory.

In addition, JOE supports a variety of powerful text editing functions, such as non-destructive editing, regular expression search and replacement, unlimited Undo/Redo, and syntax highlighting.

Install JOE On Debian, Ubuntu, or Linux Mint:

$ sudo apt-get install joe 

Install JOE on Fedora, CentOS, or RHEL systems:

$ sudo yum install joe 

To open and edit a text file, run:

$ joe test.log 

Compared with the glogg mentioned above, loading a large text file with JOE takes a period of nearly 30 seconds to load a 1G file. However, considering that full-text editing is required, this is tolerable. Once the file is loaded, you can edit the file in fast terminal mode.

JOE's memory consumption is impressive. Load and edit a 1 GB text file, which only takes 47 MB of physical memory.

If you still know other methods to open/edit large text files in Linux, please share with us!

Via: http://xmodulo.com/2013/12/open-large-text-file-linux.html

Translator: runningwater Proofreader: jasminepeng