In the realm of command-line tools, curl stands as a versatile and powerful utility. Though it may appear complex at first glance, this beginner's guide will simplify the learning process, making it accessible to newcomers.
By the end of this article, you'll be equipped to utilize curl effectively, whether for retrieving web data, working with APIs, or accomplishing various tasks from the command line.
What is cURL?
cURL is a command-line tool and library for transferring data with URLs. It's a versatile and powerful utility that allows users to interact with various network protocols, making it an invaluable tool for web developers, sysadmins, and anyone working in a command-line environment.
With cURL, you can perform a wide range of tasks, from making simple GET requests to complex operations like sending data, handling authentication, and following redirects. It supports a multitude of protocols, including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, LDAP, and many more, making it a go-to choice for tasks involving data retrieval, web scraping, API testing, and automation of web-related tasks.
Whether you're a beginner or an experienced user, understanding how to use curl effectively can significantly enhance your ability to work with web resources and services directly from the command line.
Installation
Some operating systems ship with curl by default. You can check if your operating system has curl installed by typing the following in your terminal
curl --version
If it fails, you can find the appropriate installation guide for your OS here https://curl.se/download.html
Basic Usage
The most basic usage of cURL is to retrieve web content. To do this, simply provide the URL you want to access as an argument.
curl https://example.com
This command sends a GET request to https://example.com and displays the response body in your terminal.
We can save the output to a file using the -o or --output option.
curl -o example.html http://example.com
It doesn't just have to be HTML. We can use this to download images. For example
curl -o image.jpeg https://images.pexels.com/photos/18271819/pexels-photo-18271819/free-photo-of-architecture.jpeg
This will download a Pexels image and save it as image.jpeg
curl also allows you to see the headers and status codes with the "-v" or --verbose flag.
curl -v https://example.com
If you are only interested in the headers and not the actual response data, you can use the -I flag
curl -I https://example.com
HTTP/2 200
content-encoding: gzip
accept-ranges: bytes
age: 186237
cache-control: max-age=604800
content-type: text/html; charset=UTF-8
date: Fri, 15 Sep 2023 13:40:19 GMT
etag: "3147526947+gzip"
expires: Fri, 22 Sep 2023 13:40:19 GMT
last-modified: Thu, 17 Oct 2019 07:18:26 GMT
server: ECS (nyb/1D2A)
x-cache: HIT
content-length: 648
HTTP Methods
To learn about how to use each of the major HTTP methods, we are going to use curl to interact with the JSON Placeholder Rest API.
Find the documentation for the Rest API here.
GET - Getting a resource
The GET method is used to retrieve data or resources from a server. By default, this is the HTTP method used if nothing is specified. To get a specific post from the JSON Placeholder Rest API, you can use the following curl command:
curl https://jsonplaceholder.typicode.com/posts/1
When you run this command, curl sends a GET request to the provided URL, and the server responds with the details of the post with ID 1.
POST - Creating a resource
The POST method is used to create new data on the server. It's like submitting a form or adding a new entry to a database. To create a new post on the JSON Placeholder Rest API, you can use curl as follows:
curl -X POST -H "Content-Type: application/json" -d '{"title": "New Post", "body": "This is the content of my post", "userId": 1}' https://jsonplaceholder.typicode.com/posts
In this command:
-X POSTspecifies that you're making a POST request.-H "Content-Type: application/json"sets the HTTP header to indicate that you're sending JSON data.-d '{"title": "New Post", "body": "This is the content of my post", "userId": 1}'is the JSON data you're sending.https://jsonplaceholder.typicode.com/postsis the URL where you want to create the new post.
The server will receive this request, create a new post with the provided data, and respond with the details of the newly created post.
PUT & PATCH - Updating a resource
The PUT and PATCH methods are used to update existing resources on the server. PUT typically replaces the entire resource, while PATCH makes partial modifications. To update an existing post with PUT, use this curl command:
curl -X PUT -H "Content-Type: application/json" -d '{"title": "Updated Title", "body": "Updated content"}' https://jsonplaceholder.typicode.com/posts/1
In this command:
-X PUTspecifies a PUT request.-H "Content-Type: application/json"sets the content type to JSON.-d '{"title": "Updated Title", "body": "Updated content"}'is the JSON data with the changes.https://jsonplaceholder.typicode.com/posts/1is the URL of the post to be updated.
If you prefer to make partial changes using PATCH, you can modify the command like this:
curl -X PATCH -H "Content-Type: application/json" -d '{"title": "Revised Title"}' https://jsonplaceholder.typicode.com/posts/1
Both of these commands will update the specified post's title and content on the server.
DELETE - Deleting a resource
The DELETE method is used to remove a resource from the server. It's like deleting a file or removing a record from a database. To delete a post using curl, you can use the following command:
curl -X DELETE https://jsonplaceholder.typicode.com/posts/1
Here:
-X DELETEindicates a DELETE request.https://jsonplaceholder.typicode.com/posts/1is the URL of the post you want to delete.
When you execute this command, curl sends a DELETE request to the specified URL, and the server will remove the post with ID 1 from its database.
Handling Headers and Cookies
For the next two sections, we will be using this simple express server with three endpoints
const express = require("express");
const cookieParser = require("cookie-parser");
const multer = require("multer");
const PORT = 8000;
const app = express();
app.use(cookieParser());
const upload = multer({ dest: "./uploads" });
app.get("/headers", (req, res) => {
res.json(req.headers);
});
app.get("/cookies", (req, res) => {
res.cookie("name", "codewithflash");
res.json(req.cookies);
});
app.post("/upload", upload.single("file"), (req, res) => {
res.json({
filename: req.file.filename,
});
});
app.listen(PORT, () => {
console.log(`Server is listening on port ${PORT}`);
});
/headers responds with the request headers
/cookies responds with the request cookies
/upload is a multipart/form-data endpoint for uploading files.
Handling Headers
HTTP headers are pieces of additional information sent along with your request to provide details or preferences to the server. These can include authentication tokens, content types, user agents, and more. When using curl, you can include headers using the -H or --header option.
Let's look at a few examples of setting headers
Setting a custom user agent header
curl -H "User-Agent: My-Custom-Agent" http://localhost:8000/headers-H "User-Agent: My-Custom-Agent"sets a custom User-Agent header named "My-Custom-Agent" for the request.We can see it in the response
{"host":"localhost:8000","accept":"*/*","user-agent":"My-Custom-Agent"}Sending Authorization Header for Token-Based Authentication
curl -H "Authorization: Bearer YOUR_TOKEN" http://localhost:8000/headers
-H "Authorization: Bearer YOUR_TOKEN" includes an Authorization header for token-based authentication.
This produces the following response
{"host":"localhost:8000","user-agent":"curl/7.88.1","accept":"*/*","authorization":"Bearer YOUR_TOKEN"}
Multiple Headers
We can also send multiple headers by just having multiple
-Hflagscurl -H "Content-Type: application/json" -H "Authorization: Basic YOUR_API_KEY" http://localhost:8000/headers
-H "Content-Type: application/json"sets the content type as JSON.-H "Authorization: Basic YOUR_API_KEY"sets an Authorization header with a Basic authentication scheme and your API key.{"host":"localhost:8000","user-agent":"curl/7.88.1","accept":"*/*","content-type":"application/json","authorization":"Basic YOUR_API_KEY"}
Handling Cookies
Cookies are small pieces of data stored on the client-side, allowing servers to identify and remember users between different requests.
With curl, we can send cookies using key-value pairs and/or a file.
The -b or --cookie option in curl allows you to send cookies along with your HTTP requests. You can provide the cookie data in the form of key-value pairs
curl -b "session_id=1234; user_token=abcd" http://localhost:8000/cookies
This responds with
{"session_id":"1234","user_token":"abcd"}
Sometimes, too, the server also sets cookies on the response which may be necessary for subsequent requests.
You can handle cookie data when receiving a response from the server using the -c or --cookie-jar option.
In the sample server above, the server sets the "name" cookie.
curl -c ./cookies -b "session_id=1234; user_token=abcd" http://localhost:8000/cookies
curl will create a file, "cookies" and save it disk.
One subsequent requests, we can send it along with the command below
curl -b "session_id=1234; user_token=abcd" -b "./cookies" http://localhost:8000/cookies
We are able to send the cookies as part of the request with the "-b" flag.
{"name":"codewithflash","session_id":"1234","user_token":"abcd"}
Uploading Files
Uploading files to a server or an endpoint using cURL is a common and convenient task. One effective way is by employing the -F or --form option, which simulates a form submission or a "multipart/form-data" request. Within this approach, the file parameter, preceded by @, designates the file to be uploaded, specifying its path. For instance, consider this example:
curl -F "file=./file.txt" -X POST http://localhost:8000/upload
In this command:
-F "file=./file.txt"instructs cURL to upload the file namedfile.txtfrom the current directory.file=./file.txtpoints to the local file to be uploaded.
By utilizing this method, you can effortlessly transfer files to the server, making it an invaluable tool for developers, administrators, and anyone needing to move files in a streamlined and efficient manner.
I hope this is enough for you to get started using curl
Happy cURLing!😀