cURL – Client for URLs Program Library and Command Line Program

cURL is a command line tool that allows you to transfer files from shell or bash scrips via protocols such as http, https, ftp, gopher, sftp, ftps or scp. cURL 1997 developed by Daniel Stenberg is very popular with website developers and is licensed under the open source MIT license.

Unique operating systems include cURL in the standard. Many Embedded Linux systems include cURL, as with Synology, cURL is also ported for Windows and is available in the 64 bit and 32 bit versions. PHP supports the cURL functions with the libcrul library.

cURL offers a wide range of applications, so websites can be remotely controlled and tested, forms can be filled out automatically, information from websites can be checked and processed into other processes. In the Linux Shell Console, the installation of wget is not mandatory, but cURL contains extensive parameters and options, and is already pre-installed.

The following are some examples of how to use cURL in the shell console, so web pages can be retrieved as follows.

$ curl http://example.com/seite.html

This reads the file page.html and outputs it on the standard output. If you want to save the file under the name mypage.html, the following command is used:

$ curl -o mysite.html http://example.com/site.html

Connections to IPv6 hosts are initialized as follows.

$ curl "http://[2001:1234:1234:1::40]/"

  The IPv6 address must be enclosed in square brackets.

View your own Internet IP address with cURL.

$ curl ipline.ch

When a URL has changed, the web page is often redirected to the new URL address, often also http to https redirects. Since cURL does not follow forwards in the standard configuration, you have to pass the parameter -L in this case.

For web pages with self-issued certificate, or outdated TLS v1.0 encryption algorithm, the -k parameter must be applied to allow insecure connections.

$ curl -k https://router.local

When a web site requests user authentication over HTTP, cURL can pass username and password, separated by a colon.

$ curl -u username:password http://www.example-shop.com/

Read the HTTP header with cURL.

$ curl --head http://www.google.com/

The –head parameter instructs cURL to give the output of the HTTP header to the default output. If you want to write the header to the header.txt file, you could do so by pipe operators in the shell, or without the support of the shell by the –dump-header parameter.

$ curl --dump-header headers.txt http://www.google.com/

The current version 7.63 supports the following protocols: DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP.

cURL’s FTP client can also interact with FTP servers.

$ curl ftp://username:password@ftp-server.com

cURL is instructed to output the index of the main directory. Username and password are passed directly with the URL. If you want to list the files subdirectory on the FTP server, it is sufficient to enter the directory to the URL:

$ curl ftp://username:password@ftp-server.com/files/

With cURL an upload is also possible, for this you pass the parameter -T followed by the file name and if necessary.dem path.

$ curl -T meineseite.html -u username:password ftp://ftp-server.com/docs/deineseite.html

Through this command, cURL copies the mypage file.html to the FTP server to the docs directory under the file names yourpage.html.

Preferably, you should use SFTP, where the file.txt is retrieved over an encrypted connection.

$ curl --ftp-ssl ftp://ftp-server.com/datei.txt

The file transfer also goes via SCP, here the file.txt transferred from the directory docs.

$ curl -u username scp://server.com/docs/datei.txt

For automated processes via scripts, the user login is not suitable, here you can authenticate with a private key.

$ curl -u username: --key ~/.ssh/id_rsa scp://server.com/~/datei.txt

The file .txt is transferred from the server’s home directory.

cURL can also be used with Server Message Block (SMB), such as MS LAN Manager and Samba.

$ curl -u "domain\username:passwd" smb://server.server.com/freigabe/datei.txt

Access to file.txt via sharing on a Samba server.

cURL can do even more, the following command line downloads a shell script file and runs it immediately, starting with a period (.).

$ . <(curl -s https://ipline.ch/echo/sysinfo.txt)

This command is executable in a Linux bash shell, system configurations and logs are read out and compiled in one file, and stored as a file under /tmp/sysinfo-*.html.

The following code sample for PHP, the result of search on Google is output with echo $result:

<?php
$cSession = curl_init(); 
curl_setopt($cSession,CURLOPT_URL,"http://www.google.com/search?q=think tank");
curl_setopt($cSession,CURLOPT_RETURNTRANSFER,true);
curl_setopt($cSession,CURLOPT_HEADER, false); 
$result=curl_exec($cSession);
curl_close($cSession);
echo $result;
?>

A look at man curl can be useful, here are all commands with examples of cURL.

curl(1) Curl Manual curl(1)

Name
       curl - transfer a URL

Synopsis
       Curl  

DESCRIPTION
       curl is a tool to transfer dat[options]a[URL...] from or to a server, using one of the supported protocols (DICT, FILE, FTP,
       FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET
       and TFTP).  The command is designed to work without user interaction.

       curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL
       connections, cookies, file transfer resume, Metalink, and more. As you will see below, the number of features
       will make your head spin!

       curl is powered by libcurl for all transfer-related features. Lake libcurl(3) for details.

Url
       The URL syntax is protocol-dependent. You'll find a detailed description in RFC 3986.

       You can specify multiple URLs or parts of URLs by writing part sets within braces as in:

http://site. "one,two,three".com

or you can get sequences of alphanumeric series by using[] as in:

ftp://ftp.numericals.com/file.txt
[1-100]        ftp://ftp.numericals.com/file.t[001-100]xt (with leading zeros)
        ftp://ftp.letters.com/file[a-z].txt
...

cURL Online Manual
Sources Link: curl.haxx.se