Search and count text files for duplicate or multiple content

When editing text or configuration files in the Linux shell, there can often be the requirement that identical entries in the files occur only once, so that to check how many times a line was duplicated, especially in files with a larger number of lines, this does not have to be done manually, help provide the use of the filters sort and uniq in the Linux bash.

This command counts duplicate lines and sorts the output in the bash.

$ sort FILE | uniq --count

  Replace the FILE placeholder with the real file name.

Only duplicate lines should be shown.

$ sort FILE | uniq --count --repeated

Nothing is displayed if there are no duplicate lines in the file.

Watching network connections to Linux host continuously using netstat and ss

Netstat – derived from Network Statistics, is a command-line utility used by system administrators to analyze network statistics. It displays a whole range of statistics, such as open ports and corresponding addresses on the host system, route table and masked connections.

This article shows how “netstat” and “ss” can be used to display current connections on a Linux host system for analysis in near real time.

How to install net-tools to use netstat

On many modern Linux distributions, netstat is replaced by the new ss utility, if it is not pre-installed, netstat can be installed afterwards. The package that contains netstat is net-tools.

$ yum install net-tools     [CentOS/RHEL]
$ apt install net-tools     [Debian/Ubuntu]

The netstat command runs through filters so that only the remote addresses are displayed, and with the watch command it is carried out continuously at intervals. The output shows the current https requests on a web server.

$ watch -n 5 "netstat -nt | grep :443 | tail -n +3 | awk '{print \$5}' | cut -d: -f1 | sort | uniq -c | sort -n"

Here the remote addresses are showing at an interval of 5 seconds for requests via https (TCP port 443).

If you want to display the current requests of an SMTP relay, port 25 is filtered out and then refresh every 10 seconds.

$ watch -n 10 "netstat -nt | grep :25 | tail -n +3 | awk '{print \$5}' | cut -d: -f1 | sort | uniq -c | sort -n"

Basically, the interval check is possible with any service, any ports and interval times in seconds can be selected.

How to watch IPv6 connections

The connections can also be watching using the new command-line utility, the ss command stand for – another utility to investigate sockets. Basically it directly queries the kernel and can respond much faster than netstat.

$ watch -n 3 "ss -nH | grep :443 | awk '{print \$6}' | sort | uniq -c | sort -n"

The tail and cut filters are no longer used here, as the ss utility has its own filter operators.

The connections for IPv4 and for IPv6 are displayed, each with an IPv4-as-IPv6 address followed by peer source port.

Using multitail there are other features, for example several commands can be divided into windows, as shown in the example with multitail.

$ multitail -R 3 -l "ss -nH | grep :443 | awk '{print \$6}' | sort | uniq -c | sort -n" -cS apache /var/log/apache2/access.log

The output shows the apache.log together with connections on a Debian web server, whereby multitail is split horizontally into two windows, -R 3 specifies the interval of 3 seconds, -l for the external command, here “ss -nH” Suppress header line. The command line tool can be deployed with “apt install multitail”.

Conclusion

This article discusses how the netstat utility can be used to periodically run the netstat command using watch – the netstat command to use the output to check the current requests for services. It is also important to point out that netstat is deprecated and instead ss Utility has taken its place to display refined network statistics. It can also be said that the “older” netstat command has been replaced by both ss and some ip commands.