Useful command-line tools recommended for Developers

absrtact: Good tools are invaluable for locating problems, and can help us stop problems at the outset, making our work more efficient in general. This article introduces 6 very powerful and flexible tools that can make your life easier with these tools.

As a web developer, the days are not so easy when there are problems with the program. The problem may be everywhere, maybe the request you sent was wrong, it might have been a wrong reply, it might have been a problem with a third-party library you were using, or an external API call failed. Good tools are invaluable for locating problems, and can help us stop problems at the outset, which in general make our work more efficient. Command-line tools are especially useful because they are easy to automate and script, and can be combined and reused in a variety of different ways. This article introduces 6 very powerful and flexible tools that can make your life easier with these tools.

Curl

Curl is a network transport tool that is very similar to wget. The main difference is that wget is saved to a file by default, and curl directs the output to the command line. This makes it very easy for curl to view the contents of the site. For example, we can get our native IP address from the ifconfig.me site:

1. $ Curl Ifconfig.me
2. 93.96.141.93

Curl's-I (view header) and-I (Show header only) options make it useful to debug HTTP responses and see what the server actually sends you:

1. $ curl-i news.ycombinator.com
3. http/1.1 OK
5. content-type:text/html; charset=utf-8
7. Cache-control:private
9. Connection:close

The-l option is also useful to allow curl to automatically track redirects. Curl supports HTTP Basic authentication, cookies, manual setting of headers, and more useful features.

Ngrep

If you want to analyze network packets, we have Wireshark, which has thousands of settings, filters, and configuration options. It also has a command line version of Tshark. If it's just for simple tasks, I think Wireshark is too heavyweight, so unless I need more powerful features, I usually use ngrep to handle it. Ngrep allows you to handle network packets in a manner similar to grep processing files.

For web traffic, you almost always want to add the-w byline option, which preserves line breaks, while the-Q option suppresses output from some mismatched packets. Here is an example of crawling all packets containing a GET or POST request:

1. Ngrep–q–w byline "^ (get| POST). * "

You can also pass in additional message filtering options, such as restricting the matching message to a specific host, IP, or port. Here we make a filter for all traffic flowing through Google, only for 80 ports and the message contains "search".

1. Ngrep–q–w byline "Search" host www.google.com and Port 80

Netcat

Netcat or NC, this is a self-proclaimed network of Swiss saber-like tools. It's simple but also very powerful and flexible, allowing you to create any network connection. Here we take a look at how to use it as a port scanner:

1. $ nc–z example.com 20-100
3. Connection to example.com port [Tcp/ssh] succeeded!
5. Connection to example.com port [Tcp/ssh] succeeded!

In addition to being able to create arbitrary connections, Netcat can also listen for incoming connections. Here we use this function of NC and then with tar to quickly and efficiently copy files between servers. On the server side, run:

1. $NC –l 9090 | tar–xzf–

Run on client:

1. $tar –czf dir/| NC Server 9090

We can use Netcat to expose any application through the network, where we expose the shell through port 8080:

1. $ Mkfifo Backpipe
3. $ nc–l 8080 0<backpipe |/bin/bash > Backpipe

Now we can access the server through any client: Linode

1. $ NC example.com 8080
3. Uname–a
5. Linux li228-162 2.6.39.1-linode34 # #1 SMP Tue June 10:29:24 EDT i686 gnu/linux

Although the following two examples have some opportunistic ingredients (in reality you should use a tool like rsync to copy files, telnet to the server with SSH), they still show the power and flexibility of netcat, and you can accomplish many different things by combining netcat and other applications.

Sshuttle

Sshuttle allows you to establish a secure tunnel for your traffic through any server that can be accessed by SSH. Setup and use are simple and do not require you to install any software on the server or modify any local proxy settings. When you are in a non-secure public WiFi or other untrusted network, SSH allows traffic to go through a secure tunnel, which avoids the intrusion of tools like Firesheep or Dsniff. All network traffic, including DNS requests, will be sent to your SSH server:

1. $ sshuttle–r <server>--dns 0/0

If you provide the-daemon parameter, Sshuttle will run in the background as a daemon. In combination with some other options, you can use alias form to quickly and easily start and stop a secure tunnel:

1. Alias tunnel= ' sshuttle–d–pidfile=/tmp/sshuttle.pid–r <server>--dns 0/0 '
3. Alias Stoptunnel= ' [[-f/tmp/sshuttle.pid]] && kill ' Cat/tmp/sshuttle.pid '

You can also use Sshuttle to solve some of the problems of geo-location based on IP, and now there are many services that use this kind of application such as BBC iplayer need you in the UK, and turntable, need you in the United States. To circumvent this limitation, you need access to a server located in that country. Amazon has free e2c micro-instances in many countries, or you can find a cheap VPS that is available in almost any country in the world.

In this case, we don't need to get all the traffic to go through the tunnel, we just need the traffic on the target service.

Unfortunately Sshuttle only accepts IP addresses as parameters and does not support host names, so we have to use dig to parse the hostname first.

1. $ sshuttle–r <server> ' Dig +short <hostname> '

Siege

Siege is an HTTP benchmark test tool. In addition to the load test feature, it also has a handy-g option, which is very similar to Curl's-il, but here siege also outputs the request header. Here is an example for Google (for brevity I took out some headers)

2. $ siege-g www.google.com
6. get/http/1.1
10. Host:www.google.com
14. user-agent:joedog/1.00 [en] (X11; I; Siege 2.70)
18. Connection:close
22. http/1.1 302 Found
26. location:http://www.google.co.uk/
30. content-type:text/html; Charset=utf-8
34. Server:gws
38. content-length:221
42. Connection:close
46. get/http/1.1
50. Host:www.google.co.uk
54. user-agent:joedog/1.00 [en] (X11; I; Siege 2.70)
58. Connection:close
62. http/1.1 OK
66. content-type:text/html; Charset=iso-8859-1
70. X-xss-protection:1; Mode=block
74. Connection:close

The siege is really in the load test of the server. Just like AB (an Apache HTTP Service benchmark tool), you can send concurrent requests to the site and see how it handles traffic. With the following command, we will launch 20 concurrent connections to Google in 30 seconds and finally get a nice test report.

2. $ siege-c20 Www.google.co.uk-b-t30s
6. ...
10. Lifting the server siege ... done.
14. transactions:1400 Hits
18. availability:100.00%
22. Elapsed time:29.22 secs
26. Data transferred:13.32 MB
30. Response time:0.41 secs
34. Transaction rate:47.91 trans/sec
38. throughput:0.46 mb/sec
42. concurrency:19.53
46. Successful transactions:1400
50. Failed transactions:0
54. Longest transaction:4.08
58. Shortest transaction:0.08

One of the most useful features of siege is that it can take a file of a record URL as input and then "click" on those URLs instead of just a single page. This is useful for load testing because you can reproduce real traffic to the site and observe how the site handles it without having to repeatedly click the same URL over and over again. Here's an example of how you can use siege to reproduce Apache's logging of another server for load testing:

1. $ cut-d '-f7/var/log/apache2/access.log > Urls.txt
3. $ siege-c<concurrency rate>-b-f urls.txt

Mitmproxy

Mitmproxy is an SSL-enabled HTTP man-in-the-middle proxy tool that allows you to check HTTP and HTTPS traffic and support direct rewrite requests. This app has been in some privacy scandals about iOS apps, including the path's address book upload scandal. Its ability to rewrite requests directly is also used for iOS platforms, where applications include forging a false high-score value in Gamecenter.

Mitmproxy is not just to see what the mobile app is sending or to fake a few high score values. It can also solve many problems for web developers. For example, we need to keep clicking the F5 key or clearing the cache to ensure that we see the latest content, and now you can run:

1. $ mitmproxy--anticache

This automatically removes the header associated with the cache control and ensures that you always get to the refreshed content. Unfortunately, this does not automatically set the forwarding function for you as Sshuttle, so you will need to modify global or browser-specific proxy settings after starting mitmproxy.

Another very powerful feature of Mitmproxy is the ability to record and reproduce the interactive process of HTTP. An example of a wireless network login is given in the official documentation. The same technique can be used as a basic web testing framework. For example, to confirm that your user registration process is working correctly, you can turn on the recording session:

1. $ mitmdump-w User-signup

Then run through the user registration process, so far should be able to work as expected. Use CTRL + C to stop recording. At any moment we can reproduce the interactive process we just recorded and check the 200 status code:

1. $ mitmdump-c User-signup | tail-n1 | grep && echo "OK" | | echo "FAIL"

If there is a problem at any point in the registration process, we will see the fail message instead of OK. You can create a complete set of such test suites and often perform such tests to ensure that you are prompted when the site suddenly fails.

Useful command-line tools recommended for Developers