How to URL Encode Data for curl — Escape Special Characters in Shell

Curl is the universal tool for testing and debugging APIs. But URL encoding with curl has a unique challenge: you have to deal with shell escaping AND URL encoding at the same time.

A simple request like:

curl "https://api.example.com/search?q=hello world"

fails because the shell sees the space as a command separator. And even if you quote it:

curl "https://api.example.com/search?q=hello world"

the URL itself is invalid because spaces are not allowed.

This guide covers every approach to safely URL-encode data for curl — from simple quoting to scriptable encoding functions, across Linux, macOS, and Windows.

---

The Core Problem

When you run curl in a shell, you are dealing with two layers of encoding:

1. Shell escaping — the shell interprets (or preserves) special characters before passing them to curl
2. URL encoding — curl sends the URL string as-is to the server, which expects valid percent-encoding

If the URL contains &, the shell treats it as a background process command. If it contains $, the shell tries variable expansion. If it contains spaces, argument parsing breaks.

These two layers create confusion that leads to malformed requests, especially when working with dynamic values.

---

Approach 1: Quoting the URL

Single quotes prevent all shell interpretation:

curl 'https://api.example.com/search?q=hello world'

With single quotes, the shell passes the exact string to curl. But the URL still contains a literal space, which is invalid. The server may accept it, reject it, or interpret it differently.

Better:

curl 'https://api.example.com/search?q=hello%20world'

This works because the shell passes the string unchanged and the URL is valid.

When to Use Single Quotes

• The URL is known at the time of writing
• You manually encode special characters
• No shell variable interpolation needed

When Single Quotes Fail

You cannot include a single quote inside single quotes:

curl 'https://example.com?q=it's broken'
# This is a syntax error

Use double quotes or escape sequences instead.

---

Approach 2: Double Quotes with Encoding

Double quotes allow variable expansion:

QUERY="hello world"
curl "https://api.example.com/search?q=${QUERY}"

But ${QUERY} still contains a raw space. The server receives:

https://api.example.com/search?q=hello world

This URL is invalid. You need to encode the value first.

Shell Variable Encoding Function

urlencode() {
  local string="$1"
  local strlen=${#string}
  local encoded=""
  local pos c o

  for ((pos = 0; pos < strlen; pos++)); do
    c="${string:$pos:1}"
    case "$c" in
      [-_.~a-zA-Z0-9] ) o="${c}" ;;
      * ) printf -v o '%%%02x' "'$c"
    esac
    encoded+="${o}"
  done
  echo "${encoded}"
}

Usage:

QUERY="hello world & special!"
ENCODED=$(urlencode "$QUERY")
curl "https://api.example.com/search?q=${ENCODED}"

This produces:

https://api.example.com/search?q=hello%20world%20%26%20special%21

---

Approach 3: Using --data-urlencode (Recommended)

Curl provides a built-in option that handles URL encoding automatically:

curl -G "https://api.example.com/search" \
  --data-urlencode "q=hello world & special!"

The --data-urlencode option:

1. Encodes the value
2. Appends it to the URL as a query parameter (when combined with -G)

Without -G, --data-urlencode sends the data as a POST body with Content-Type: application/x-www-form-urlencoded.

Multiple Parameters

curl -G "https://api.example.com/search" \
  --data-urlencode "q=hello world" \
  --data-urlencode "category=books & media" \
  --data-urlencode "page=1"

This produces:

https://api.example.com/search?q=hello%20world&category=books%20%26%20media&page=1

How --data-urlencode Handles Different Formats

# Encode the value after =
curl -G "https://example.com/search" --data-urlencode "q=hello world"

# Encode the entire string (including key)
curl -G "https://example.com/search" --data-urlencode "name=value"

The option parses on the first = sign. The key is not encoded; the value is.

---

Approach 4: POST with Form Data

For POST requests, --data-urlencode sends properly encoded form data:

curl -X POST "https://api.example.com/submit" \
  --data-urlencode "name=John Doe" \
  --data-urlencode "email=john@example.com" \
  --data-urlencode "message=Hello & welcome!"

This sends:

Content-Type: application/x-www-form-urlencoded

name=John%20Doe&email=john%40example.com&message=Hello%20%26%20welcome!

---

Approach 5: POSIX printf Encoding

On systems without a custom urlencode function, printf can encode bytes:

urlencode_printf() {
  local string="$1"
  local encoded=""

  for ((i = 0; i < ${#string}; i++)); do
    local char="${string:$i:1}"
    case "$char" in
      [a-zA-Z0-9.~_-]) encoded+="$char" ;;
      *) encoded+=$(printf '%%%02X' "'$char") ;;
    esac
  done

  echo "$encoded"
}

Usage:

ENCODED=$(urlencode_printf "hello world & special")
curl -G "https://api.example.com/search" --data-urlencode "q=$ENCODED"

---

Approach 6: Python One-Liner

If Python is available, it is the most reliable cross-platform approach:

ENCODED=$(python3 -c "import urllib.parse; print(urllib.parse.quote('$1'))")

Better as a reusable function:

urlencode_python() {
  python3 -c "import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1]))" "$1"
}

QUERY="hello world & special"
ENCODED=$(urlencode_python "$QUERY")
curl "https://api.example.com/search?q=${ENCODED}"

---

Approach 7: Node.js One-Liner

urlencode_node() {
  node -e "process.stdout.write(encodeURIComponent('$1'))"
}

QUERY="hello world & special"
ENCODED=$(urlencode_node "$QUERY")
curl "https://api.example.com/search?q=${ENCODED}"

---

Approach 8: jq with JSON

If you are working with JSON APIs, jq can help:

curl -G "https://api.example.com/search" \
  --data-urlencode "q=hello world"

# Or construct the full URL with jq
URL=$(jq -nr --arg q "hello world" \
  '"https://api.example.com/search?q=\($q|@uri)"')

curl "$URL"

The @uri format specifier in jq percent-encodes the value.

---

Handling Special Shell Characters

Variables with $

NAME="John & Doe"
# Wrong — shell expands $NAME
curl "https://example.com?name=${NAME}"

# Correct with single quotes (but no variable expansion)
curl 'https://example.com?name=John%20%26%20Doe'

# Correct with encoding
ENCODED=$(urlencode "$NAME")
curl "https://example.com?name=${ENCODED}"

Exclamation Marks in Bash

In interactive bash, ! triggers history expansion inside double quotes:

curl "https://example.com?q=hello!"  # May fail due to history expansion

# Use single quotes or escape with \!
curl 'https://example.com?q=hello%21'

Backticks and $()

# Wrong — command substitution
curl "https://example.com?q=$(date)"

# Correct if you intend it
curl "https://example.com?q=$(urlencode "$(date)")"

---

Full Script Example

#!/usr/bin/env bash

# URL-encode a string
urlencode() {
  local string="$1"
  local length="${#string}"
  local encoded=""

  for ((i = 0; i < length; i++)); do
    local char="${string:$i:1}"
    case "$char" in
      [a-zA-Z0-9.~_-]) encoded+="$char" ;;
      *) printf -v hex '%%%02X' "'$char"
         encoded+="$hex" ;;
    esac
  done

  echo "$encoded"
}

# curl wrapper with automatic encoding
curlenv() {
  local method="GET"
  local url=""
  local -a data_args=()

  while [[ $# -gt 0 ]]; do
    case "$1" in
      -X|--request) method="$2"; shift 2 ;;
      -d|--data) data_args+=("--data-urlencode" "$2"); shift 2 ;;
      *) url="$1"; shift ;;
    esac
  done

  if [[ ${#data_args[@]} -gt 0 ]]; then
    if [[ "$method" == "GET" ]]; then
      curl -G "$url" "${data_args[@]}"
    else
      curl -X "$method" "$url" "${data_args[@]}"
    fi
  else
    curl "$url"
  fi
}

# Examples
curlenv -X GET "https://api.example.com/search" \
  -d "q=hello world & special" \
  -d "page=1"

---

Common Mistakes with curl and Encoding

Mistake 1: URL in Double Quotes with Unencoded &

curl "https://example.com/search?q=hello&page=1"

The & tells the shell to run the command in the background. Curl receives only https://example.com/search?q=hello.

Mistake 2: Forgetting -G with --data-urlencode

# Without -G, this sends a POST
curl --data-urlencode "q=hello world" "https://example.com/search"

# With -G, it appends to the URL
curl -G --data-urlencode "q=hello world" "https://example.com/search"

Mistake 3: Encoding the Entire URL

# Wrong — encodes :, /, ?
ENCODED=$(urlencode "https://example.com/search?q=hello")
curl "$ENCODED"

Only encode parameter values, not the entire URL structure.

Mistake 4: Not Quoting at All

curl https://example.com/search?q=hello world

The shell splits on the space and curl receives two arguments.

---

Testing curl Encoding

# Use -w to inspect the actual request
curl -G "https://httpbin.org/get" \
  --data-urlencode "q=hello world & special" \
  -w "\n%{url_effective}\n"

httpbin.org/get echoes back the request parameters, letting you verify the encoding.

# Or use --trace for full request dump
curl -G "https://example.com/api" \
  --data-urlencode "q=hello world" \
  --trace - 2>&1 | head -20

---

Best Practices for curl URL Encoding

Prefer --data-urlencode with -G

This is the most reliable approach. Curl handles the encoding internally.

Use Single Quotes When Possible

Single quotes prevent all shell interpretation. Encode the values manually.

Encode in a Separate Step

ENCODED=$(urlencode "$RAW_VALUE")
curl "https://example.com?q=${ENCODED}"

Test with Echo First

ENCODED=$(urlencode "hello world & special")
echo "curl 'https://example.com?q=${ENCODED}'"

Be Consistent Across Requests

If you are scripting, use one approach consistently. Mixing --data-urlencode, manual encoding, and raw URLs creates hard-to-debug inconsistencies.

---

Related Resources

For language-specific encoding approaches:

• How to Encode a URL in JavaScript JavaScript URL Encoding

• How to URL Encode a Query String in Python Python URL Encoding

• Common URL Encoding Mistakes Developers Keep Making Common Mistakes

• URL Encoding Explained with Real API Examples URL Encoding with Real API Examples

---

FAQ

How do I URL encode data with curl?

Use --data-urlencode with -G for GET requests, or --data-urlencode alone for POST requests.

Why does curl fail when I include & in the URL?

The & character is interpreted by the shell as a background command separator. Quote the URL or use --data-urlencode.

What is the difference between --data and --data-urlencode?

--data sends the string as-is (with basic shell escaping only). --data-urlencode percent-encodes the value automatically.

How do I send a POST request with URL-encoded form data in curl?

curl -X POST https://example.com/submit \
  --data-urlencode "name=John Doe" \
  --data-urlencode "email=john@example.com"

How do I encode data for a GET request in curl?

Use the -G flag with --data-urlencode:

curl -G "https://example.com/search" \
  --data-urlencode "q=hello world"

How do I URL encode a variable in bash?

Use a shell function like urlencode() that iterates over characters and percent-encodes unsafe bytes. Or use Python's urllib.parse.quote().

Does curl support %20 vs + for spaces?

Curl's --data-urlencode produces %20. If you need +, encode manually using a custom function.

How do I debug what curl is actually sending?

Use curl -w '%{url_effective}' to see the final encoded URL, or curl --trace - for full request details.

---

Final Thoughts

URL encoding with curl has a learning curve because shell escaping and URL encoding interact in confusing ways. The safest approach is to use --data-urlencode with -G for GET requests, or --data-urlencode alone for POST form data.

If you need to encode values in a script, write a reusable urlencode function or call Python's urllib.parse.quote. Avoid manual string concatenation with unquoted special characters.

For quick testing of encoded payloads, the URL Encoder/Decoder tool helps verify what curl will actually send.

---