Sneaky Peek at how your web browser communicates with a web server

Sneaky Peek at how your web browser communicates with a web server

Here we are going to learn about Ping, Curl, HTTP Status codes including the ubiquitious '404 Not Found', Request Headers, Response Headers.

Lots of fun extras after the video you can even get the weather in your shell with curl so be sure to scroll down, all you need is a computer with a shell like a Mac or Raspberry Pi.

Really easy to use ping just open a shell and type ping followed by a domain name.

$ ping google.com
PING google.com (142.250.178.14) 56(84) bytes of data.
64 bytes from lhr48s27-in-f14.1e100.net (142.250.178.14): icmp_seq=1 ttl=117 time=11.3 ms
64 bytes from lhr48s27-in-f14.1e100.net (142.250.178.14): icmp_seq=2 ttl=117 time=11.1 ms
64 bytes from lhr48s27-in-f14.1e100.net (142.250.178.14): icmp_seq=3 ttl=117 time=11.6 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 11.140/11.364/11.606/0.190 ms

Curl is just as easy here we go for google.com instead of www.google.com and see the response in the shell is a 301 redirect.

curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

If you want to see the response header too then add the -i option

$ curl -i google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Sat, 02 Oct 2021 14:54:48 GMT
Expires: Mon, 01 Nov 2021 14:54:48 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

You can have some fun with Curl replace london in the following command with another location to see the weather in the shell.

$ curl http://wttr.in/london
Weather report: london

   _`/"".-.     Rain, heavy rain
    ,\_(   ).   +14(12) °C
     /(___(__)  ↑ 24 km/h
     ‚‘‚‘‚‘‚‘   8 km
     ‚’‚’‚’‚’   1.6 mm
                                                       ┌─────────────┐
┌──────────────────────────────┬───────────────────────┤  Sat 02 Oct ├───────────────────────┬──────────────────────────────┐
│            Morning           │             Noon      └──────┬──────┘     Evening           │             Night            │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│               Overcast       │               Overcast       │      .-.      Heavy rain     │      .-.      Light drizzle  │
│      .--.     +13(11) °C     │      .--.     +14(12) °C     │     (   ).    16 °C          │     (   ).    +14(12) °C     │
│   .-(    ).   ↖ 18-23 km/h   │   .-(    ).   ↖ 21-26 km/h   │    (___(__)   ↑ 22-29 km/h   │    (___(__)   ↗ 23-32 km/h   │
│  (___.__)__)  10 km          │  (___.__)__)  10 km          │   ‚‘‚‘‚‘‚‘    5 km           │     ‘ ‘ ‘ ‘   2 km           │
│               0.0 mm | 0%    │               0.0 mm | 0%    │   ‚’‚’‚’‚’    16.8 mm | 94%  │    ‘ ‘ ‘ ‘    0.5 mm | 65%   │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
                                                       ┌─────────────┐
┌──────────────────────────────┬───────────────────────┤  Sun 03 Oct ├───────────────────────┬──────────────────────────────┐
│            Morning           │             Noon      └──────┬──────┘     Evening           │             Night            │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│               Cloudy         │               Cloudy         │    \  /       Partly cloudy  │    \  /       Partly cloudy  │
│      .--.     +12(11) °C     │      .--.     16 °C          │  _ /"".-.     +13(12) °C     │  _ /"".-.     +12(11) °C     │
│   .-(    ).   ↗ 14-19 km/h   │   .-(    ).   ↗ 24-28 km/h   │    \_(   ).   ↗ 19-28 km/h   │    \_(   ).   ↑ 14-20 km/h   │
│  (___.__)__)  10 km          │  (___.__)__)  10 km          │    /(___(__)  10 km          │    /(___(__)  10 km          │
│               0.0 mm | 0%    │               0.0 mm | 0%    │               0.0 mm | 0%    │               0.0 mm | 0%    │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
                                                       ┌─────────────┐
┌──────────────────────────────┬───────────────────────┤  Mon 04 Oct ├───────────────────────┬──────────────────────────────┐
│            Morning           │             Noon      └──────┬──────┘     Evening           │             Night            │
├──────────────────────────────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
\  /       Partly cloudy  │               Cloudy         │               Overcast       │               Overcast       │
│  _ /"".-.     +14(12) °C     │      .--.     16 °C          │      .--.     +14(13) °C     │      .--.     +13(12) °C     │
\_(   ).   ↑ 18-23 km/h   │   .-(    ).   ↑ 20-23 km/h   │   .-(    ).   ↖ 14-20 km/h   │   .-(    ).   ↖ 16-23 km/h   │
│    /(___(__)  10 km          │  (___.__)__)  10 km          │  (___.__)__)  10 km          │  (___.__)__)  10 km          │
│               0.0 mm | 0%    │               0.0 mm | 0%    │               0.0 mm | 0%    │               0.0 mm | 0%    │
└──────────────────────────────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
Location: London [51.509648,-0.099076]

Follow @igor_chubin for wttr.in updates

You can even use curl to learn about curl by the way you can do this for other shell commands for example curl cheat.sh/ls

$ curl cheat.sh/curl
 cheat.sheets:curl
# curl
# Command-line tool for transferring data with URL syntax

# Process a single GET request, and show its output on stdout.
curl http://path.to.the/file

# Download a file and specify a new filename.
curl http://example.com/file.zip -o new_file.zip

# Download multiple files.
curl -O URLOfFirstFile -O URLOfSecondFile

# Download all sequentially-numbered files (1-24).
curl http://example.com/pic[1-24].jpg

# Download a file and follow redirects.
curl -L http://example.com/file

# Download a file and pass HTTP Authentication.
curl -u username:password URL

# Download a file with a Proxy.
curl -x proxysever.server.com:PORT http://addressiwantto.access

# Download a file from FTP.
curl -u username:password -O ftp://example.com/pub/file.zip

# Get an FTP directory listing.
curl ftp://username:password@example.com

# Resume a previously failed download.
curl -C - -o partial_file.zip http://example.com/file.zip

# Fetch only the HTTP headers from a response.
curl -I http://example.com

# Fetch your external IP and network info as JSON.
curl http://ifconfig.me/all/json

# Limit the rate of a download.
curl --limit-rate 1000B -O http://path.to.the/file

# POST to a form.
curl -F "name=user" -F "password=test" http://example.com

# POST JSON Data.
curl -H "Content-Type: application/json" -X POST -d '{"user":"bob","pass":"123"}' http://example.com

# POST data from the standard in / share data on sprunge.us.
curl -F 'sprunge=<-' sprunge.us

 cheat:curl
# To download a file:
curl <url>

# To download and rename a file:
curl <url> -o <outfile>

# To download multiple files:
curl -O <url> -O <url>

# To download all sequentially numbered files (1-24):
curl http://example.com/pic[1-24].jpg

# To download a file and pass HTTP authentication:
curl -u <username>:<password> <url>

# To download a file with a proxy:
curl -x <proxy-host>:<port> <url>

# To download a file over FTP:
curl -u <username>:<password> -O ftp://example.com/pub/file.zip

# To get an FTP directory listing:
curl ftp://username:password@example.com

# To resume a previously failed download:
curl -C - -o <partial-file> <url>

# To fetch only the HTTP headers from a response:
curl -I <url>

# To fetch your external IP and network info as JSON:
curl http://ifconfig.me/all.json

# To limit the rate of a download:
curl --limit-rate 1000B -O <outfile>

# To get your global IP:
curl httpbin.org/ip

# To get only the HTTP status code:
curl -o /dev/null -w '%{http_code}\n' -s -I URL

 tldr:curl
# curl
# Transfers data from or to a server.
# Supports most protocols, including HTTP, FTP, and POP3.
# More information: <https://curl.haxx.se>.

# Download the contents of an URL to a file:
curl http://example.com -o filename

# Download a file, saving the output under the filename indicated by the URL:
curl -O http://example.com/filename

# Download a file, following [L]ocation redirects, and automatically [C]ontinuing (resuming) a previous file transfer:
curl -O -L -C - http://example.com/filename

# Send form-encoded data (POST request of type `application/x-www-form-urlencoded`). Use `-d @file_name` or `-d @'-'` to read from STDIN:
curl -d 'name=bob' http://example.com/form

# Send a request with an extra header, using a custom HTTP method:
curl -H 'X-My-Header: 123' -X PUT http://example.com

# Send data in JSON format, specifying the appropriate content-type header:
curl -d '{"name":"bob"}' -H 'Content-Type: application/json' http://example.com/users/1234

# Pass a user name and password for server authentication:
curl -u myusername:mypassword http://example.com

# Pass client certificate and key for a resource, skipping certificate validation:
curl --cert client.pem --key key.pem --insecure https://example.com