Pipe-limited Fluent PV

What is PV?

is not page view, is the pipe viewer, pipe peeping abbreviation. This is the source of the site on Google code, if needed, can visit the website of PV.

The official manual page of this thing (man PV or PV (1)) says that PV is a tool that allows pipeline users to see what is going on in the pipeline. You can see the progress of pipeline processing and so on.

What's PV?

In fact, the command line parameters of PV (1) are very rich. There are many things you can do in addition to pipeline observation. Here are some examples:

1. View data flow rate in a pipeline

2. To view the time of data flow in a pipeline

3. Calculate the expected time to complete by giving the expected size of the data

4. Data transfer rate limiting for flow within the pipeline

Why pipe flow limiting

We often have a variety of current limit requirements, such as when copying data across the network, such as when we write data to disk. To avoid excessive network bandwidth or disk bandwidth , we have to do something to limit the flow.

And the current limit is a very troublesome thing, some tools finite flow functions, such as SCP, rsync and so on, some are not, such as CP, TAR, NC. This makes our automation scripts extremely difficult.

One of the most exciting features of PV is the ability to limit the flow rate of data in a pipeline by parameters, which is the following:

-L rate,--rate-limit rate

We can use K, m, G suffix to denote the magnitude of thousand, trillion, JI, data size is "byte". Like what:

Pv-l 300k

This means that PV limits the flow of data in the pipeline to 300K bytes per second.

Why pipe flow limit?

Because we know that in order to maintain the network of courtesy, in the production environment, try not to fill the bandwidth, and sometimes some operations, will be full of bandwidth. For example, we're copying a huge file at full speed, which is likely to hit some of the weakest links in the network. And we often use the pipeline in the automation script, if we can use the pipeline limit, then we can use the local pipeline to limit the flow first, and then use various forms of IO, thus greatly optimizing the various IO links, reduce overhead. The pipeline limit itself basically allows us to limit the flow of arbitrary commands that support standard input and output, so that the range of use is almost limitless.

Get PV

On most Redhat-based servers, you can install them directly with Yum:

sudo yum install PV

Slackware can go to slackbuilds.org to get the production Package TXZ installation package. The most laborious can go to the above google code download source code, and then compile the installation, the standard way to compile the installation is:

Tar zxvf. CD PV .... /configure make make Install

The process.

Usage and examplesI want to see if a slow query has transmitted data.

I have an unpleasant query, about a few minutes to run, and get more data, I would like to try to perform a bit, see how long it will be:

Time Psql-u qa-h somehost.qunar.com-d vacation-f t3.sql | Pv-t-R >/dev/nul

The query is saved in T3.sql, I link to the test library for testing, and this SQL will copy the data to the standard output. Then, I use PV to view the standard output pipe of the front psql, I let PV record time:

-T

Option to record the transfer rate:

-R

Options.

Then use the time command at the beginning to see how long this command actually used.

I want to transfer a big data to RTOOLS1, but I need a current limit of 100k/seconds.

You can use a command similar to the following:

Tar Cf-result.data | Pv-e-t-b-L 100k| SSH [email protected] ' tar xvf-'

Pipe-limited Fluent PV