Tag Archives: Linux How to

Unix Similar multi-user operating systems based on the Linux kernel and essentially on GNU software. Like CentOS, Debian, Ubuntu Fedora.

How to apply cURL in the Linux practice

cURL – Client for URLs Program Library and Command Line Program

cURL is a command line tool that allows you to transfer files from shell or bash scrips via protocols such as http, https, ftp, gopher, sftp, ftps or scp. cURL 1997 developed by Daniel Stenberg is very popular with website developers and is licensed under the open source MIT license.

Unique operating systems include cURL in the standard. Many Embedded Linux systems include cURL, as with Synology, cURL is also ported for Windows and is available in the 64 bit and 32 bit versions. PHP supports the cURL functions with the libcrul library.

cURL offers a wide range of applications, so websites can be remotely controlled and tested, forms can be filled out automatically, information from websites can be checked and processed into other processes. In the Linux Shell Console, the installation of wget is not mandatory, but cURL contains extensive parameters and options, and is already pre-installed.

The following are some examples of how to use cURL in the shell console, so web pages can be retrieved as follows.

$ curl http://example.com/seite.html

This reads the file page.html and outputs it on the standard output. If you want to save the file under the name mypage.html, the following command is used:

$ curl -o mysite.html http://example.com/site.html

Connections to IPv6 hosts are initialized as follows.

$ curl "http://[2001:1234:1234:1::40]/"

  The IPv6 address must be enclosed in square brackets.

View your own Internet IP address with cURL.

$ curl ipline.ch

When a URL has changed, the web page is often redirected to the new URL address, often also http to https redirects. Since cURL does not follow forwards in the standard configuration, you have to pass the parameter -L in this case.

For web pages with self-issued certificate, or outdated TLS v1.0 encryption algorithm, the -k parameter must be applied to allow insecure connections.

$ curl -k https://router.local

When a web site requests user authentication over HTTP, cURL can pass username and password, separated by a colon.

$ curl -u username:password http://www.example-shop.com/

Read the HTTP header with cURL.

$ curl --head http://www.google.com/

The –head parameter instructs cURL to give the output of the HTTP header to the default output. If you want to write the header to the header.txt file, you could do so by pipe operators in the shell, or without the support of the shell by the –dump-header parameter.

$ curl --dump-header headers.txt http://www.google.com/

The current version 7.63 supports the following protocols: DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP.

cURL’s FTP client can also interact with FTP servers.

$ curl ftp://username:password@ftp-server.com

cURL is instructed to output the index of the main directory. Username and password are passed directly with the URL. If you want to list the files subdirectory on the FTP server, it is sufficient to enter the directory to the URL:

$ curl ftp://username:password@ftp-server.com/files/

With cURL an upload is also possible, for this you pass the parameter -T followed by the file name and if necessary.dem path.

$ curl -T meineseite.html -u username:password ftp://ftp-server.com/docs/deineseite.html

Through this command, cURL copies the mypage file.html to the FTP server to the docs directory under the file names yourpage.html.

Preferably, you should use SFTP, where the file.txt is retrieved over an encrypted connection.

$ curl --ftp-ssl ftp://ftp-server.com/datei.txt

The file transfer also goes via SCP, here the file.txt transferred from the directory docs.

$ curl -u username scp://server.com/docs/datei.txt

For automated processes via scripts, the user login is not suitable, here you can authenticate with a private key.

$ curl -u username: --key ~/.ssh/id_rsa scp://server.com/~/datei.txt

The file .txt is transferred from the server’s home directory.

cURL can also be used with Server Message Block (SMB), such as MS LAN Manager and Samba.

$ curl -u "domain\username:passwd" smb://server.server.com/freigabe/datei.txt

Access to file.txt via sharing on a Samba server.

cURL can do even more, the following command line downloads a shell script file and runs it immediately, starting with a period (.).

$ . <(curl -s https://ipline.ch/echo/sysinfo.txt)

This command is executable in a Linux bash shell, system configurations and logs are read out and compiled in one file, and stored as a file under /tmp/sysinfo-*.html.

The following code sample for PHP, the result of search on Google is output with echo $result:

$cSession = curl_init(); 
curl_setopt($cSession,CURLOPT_URL,"http://www.google.com/search?q=think tank");
curl_setopt($cSession,CURLOPT_HEADER, false); 
echo $result;

A look at man curl can be useful, here are all commands with examples of cURL.

curl(1) Curl Manual curl(1)

       curl - transfer a URL


       curl is a tool to transfer dat[options]a[URL...] from or to a server, using one of the supported protocols (DICT, FILE, FTP,
       and TFTP).  The command is designed to work without user interaction.

       curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL
       connections, cookies, file transfer resume, Metalink, and more. As you will see below, the number of features
       will make your head spin!

       curl is powered by libcurl for all transfer-related features. Lake libcurl(3) for details.

       The URL syntax is protocol-dependent. You'll find a detailed description in RFC 3986.

       You can specify multiple URLs or parts of URLs by writing part sets within braces as in:

http://site. "one,two,three".com

or you can get sequences of alphanumeric series by using[] as in:

[1-100]        ftp://ftp.numericals.com/file.t[001-100]xt (with leading zeros)

cURL Online Manual
Sources Link: curl.haxx.se

Protect WordPress from Brute-Force-Attacks with Fail2ban

How To Protect WordPress against Brute-Force-Attacks using Fail2ban

WordPress itself does not offer any protection against brute force attacks. But there are several methods for protection, besides lock-down plugins they blocking IP addresses after every failed login attempt. Another way gives the kernel firewall controlled with Fail2ban. With it the suspicious host is locked out for a certain time.

To Protect your WordPress from Brute-Force-Attacks by Fail2ban, follow this install tutorial.

  To install fail2ban on RHEL then the EPEL repository must be added. Debian provide fail2ban already in the default repository.

$ yum install epel-release

The installation here as root is as follows, a standard configuration is provided, and does not require any further adjustments:

$ yum install fail2ban -y

On Debian and Ubuntu Linux fail2ban is installed as follow.

$ apt install fail2ban -y

After fail2ban is installed on the server, the Apache access_log can be scanned for login attempts. - - [02/Jul/2017:21:23:17 +0200] "POST /xmlrpc.php HTTP/1.1" 200 406 - - [02/Jul/2017:21:23:19 +0200] "GET /wp-login.php HTTP/1.1" 200 6387 - - [02/Jul/2017:21:23:19 +0200] "POST /wp-login.php HTTP/1.1" 200 6561 - - [02/Jul/2017:21:27:55 +0200] "POST /xmlrpc.php HTTP/1.1" 200 406 - - [02/Jul/2017:21:27:56 +0200] "GET /wp-login.php HTTP/1.1" 200 6440 - - [02/Jul/2017:21:27:56 +0200] "POST /wp-login.php HTTP/1.1" 200 6614

So that fail2ban knows which parsers to search for in the log file. The filter is created by editing with vim or other preferred editors. We create the filter file with use the command vi /etc/fail2ban/filter.d/wp-auth.conf

# WordPress brute force auth filter: /etc/fail2ban/filter.d/wp-auth.conf:
# Block IPs trying to auth wp wordpress
failregex = ^<HOST> .*"POST \/wp-login\.php HTTP\/.*$
            ^<HOST> .*"POST \/\/wp-login\.php HTTP\/.*$
            ^<HOST> .*"POST \/xmlrpc\.php HTTP\/.*$
            ^<HOST> .*"POST \/\/xmlrpc\.php HTTP\/.*$
ignoreregex =

The main configuration of fail2ban is located in the file /etc/fail2ban/jail.conf, this is copied to jail.local. Or you insert only the filter by add the configuration with vi /etc/fail2ban/jail.local

enabled = true
filter = wp-auth
action = iptables-multiport[name=AuthFailures, port="http,https"]
logpath = %(apache_access_log)s
bantime = 1800
maxretry = 5

After 5 login attempts (maxretry) is blocked for 30 min.

Before fail2ban is used, a look is given in /etc/fail2ban/jail.conf, under the section [INCLUDES] the paths of the currently used Linux distribution are defined at the line before = paths-distro.conf, here we use CentOS 7 and the appropriate file /etc/fail2ban/paths-fedora.conf

before = paths-centos.conf

We copy the file paths-fedora.conf to paths-centos.conf. All logs should be read under /var/www/*/logs, adjusting the paths to the Apache logs if necessary.

# apache server log
apache_error_log = /var/log/httpd/*error_log
apache_access_log = /var/log/httpd/*access_log

# apache virtual hosts
apache_error_log = /var/www/*/logs/*error_log
apache_access_log = /var/www/*/logs/*access_log

Under Debian the following content is defined in /etc/fail2ban/jail.conf

before = paths-debian.conf

Now restart fail2ban to activate the filter.

 $ systemctl restart fail2ban

Check fail2ban status as follows.

$ fail2ban-client status wp-auth

Output the iptable status of the filter in the chain.

$ iptables -vnL | grep AuthFailures

The iptables output of the filter AuthFailure follows about a lot.

Chain f2b-AuthFailures (1 references)
 pkts bytes target prot opt in out source destination
 252K 32M RETURN all -- * *
 232K 29M RETURN all -- * *

If you have locked yourself out by too many failed attempts and tests. You can unlock yourself again with the following command.

$ fail2ban-client set wp-auth unbanip

  The advantage of this method is that fail2ban blocks suspicious host for all services, many bots will usually go to the next website to continue try attacks, ssh and ftp requests also go nowhere while ICMP requests remain unanswered for the investigated IP address.

Protect WordPress from Brute-Force-Attacks with Fail2ban