------------------------------------------------------------------------------- URLencoding or Percent Encoding Encoding data so it can be placed within a URL, or in post data of the form https://.../?var1=value1&var2=value2&var3=value3 Where the var and values are seperatally URLencoded so as to preserve the '&' and '=' seperators Basically characters that could be missinterpreted need to be converted to the for "%xx" where xx is the character code in hex. An exception is space which is encoded as either "+" (URL form) or as "%20" Not all characters need to be encoded (RFC3986)... And depends on how 'special' the character is in the application. See https://en.wikipedia.org/wiki/Percent-encoding Reserved Characters (must be encoded) "!#$&'()*+,/:;=?@[]" Unreserved Characters (optionally encoded) [-_.~a-zA-Z0-9] There exists a non-standard encoding for Unicode characters: %uxxxx However RFC3987 Internationalized Resource Identifier specifies Unicode should be UTF-8 and encoded as binary %xx codes. In POST data the values (after the '?' is set in the body of the request using the data type "application/x-www-form-urlencoded" ------------------------------------------------------------------------------- Encoding URLencoding A Test string string='My data_+/?=&%_-!@#$^*().~' Unicode test (should become "%C3%BC") string="ΓΌ" --- BASH... WARNING: Neither bash method handles unicode or binary input # Pure Bash urlencode (looped) urlencode() { # WARNING: does not handle unicode! for (( pos=0 ; pos<"${#1}" ; pos++ )); do local c="${1:$pos:1}" case "$c" in [-_.~a-zA-Z0-9]) printf "%c" "$c" ;; \;) printf "%c" ';' ;; # optional - make more readable \ ) printf "%c" '+' ;; # optional * ) printf '%%%02x' "'$c" ;; esac done } urlencode "$string" # Pure Bash urlencode (series - minimal) urlencode_series() { o=${1//%/%25} # must be first o=${o//&/%26} # variable seperator o=${o////%2F} # URI path seperator o=${o///%3E} # XML close brace (optional) o=${o//\?/%3F} # start GET data o=${o//+/%2B} # before space o=${o// /+} # spaces, could be "%20" printf "%s" "$o" } urlencode_series "$string" --- External Command Alturnatives NOTE: All this methods substitute "%20" instead of "+" for spaces and coverts almost all non alpha-numerics to percent form perl -MURI::Escape -e 'print uri_escape($ARGV[0]);' "$string" php -r 'echo urlencode("'"$string"'");' Python Just encode a individual string... python3 -c "import urllib.parse; print(urllib.parse.quote(input()))" \ <<<"$string" Pyhton can do the urlencoding while putting together the query string, But then uses '+' for spaces. import urllib.parse queryString = { 'name' : 'Jason Smith', 'age_group' : '(18-24)'} urllib.parse.urlencode( queryString ) # => 'name=Jason+Smith&age_group=%2818-24%29' A parameter 'safe' can be used to specify characters that are safe to not URLencode into percents. import urllib.parse queryString = { 'name' : 'Jason Smith', 'age_group' : '(18-24)'} urllib.parse.urlencode( queryString , safe='()') # => 'name=Jason+Smith&age_group=(18-24)' JQ (watch the EOL)... Does not encode characters "_-!*().~" encodes space as %20 Always adds a newline to output (not to encoded string) jq -Rr @uri <<<"$string" OR if there are linefeeds in the data to be encoded... printf '%s' "$string" | jq -sRr @uri OR jq -rn --arg x "$string" '$x|@uri' Just encode ALL bytes to %xx (3 times bigger, handles unicode) printf "%s" "$string" | { xxd -p |tr -d \\n | sed 's/../%&/g'; } echo ------------------------------------------------------------------------------- Decoding URLencoding perl -pe 'tr/+/ /; s/%([a-fA-F0-9][a-fA-F0-9])/chr hex $1/eg;' # Example.... # Input: %24argon2i%24v%3D19%24m%3D4096%2Ct%3D3%2Cp%3D1%24%2FFzf # Output: $argon2i$v=19$m=4096,t=3,p=1$/Fzf BASH string='%24argon2i%24v%3D19%24m%3D4096%2Ct%3D3%2Cp%3D1%24%2FFzf' partial="${string/+/ /}" printf '%b' "${partial//%/\\x}" -------------------------------------------------------------------------------