Using curl with the Amazon Product Request API

bsilvereagle

Feb 6, 2017 • 4 min read

Motivation

Amazon provides a handy Product Advertising API that allows you to fetch the price, reviews, description, and image (among other things) for products. There are several examples on how to use the API for Ruby, PHP, Python, and Java. But sometimes it's useful to use curl from the shell.

Amazon provides a guide and a few examples for the Product Advertising API. It is easy enough to follow until step 8.

8. Calculate an RFC 2104-compliant HMAC with the SHA256 hash algorithm using the string above with this example AWS secret key: 1234567890. For more information about this step, see documentation and code samples for your programming language.

It takes jumping through some hoops in order to calculate the RFC 2104-compliant HMAC with the SHA256 hash algorithm with Unix utilities.

Creating the Request

There are several types of requests exposed by the API, allowing searching for items, cart manipulation, or requesting info about particular items. The Amazon documentation uses the following request, which will be used throughout this post:

http://webservices.amazon.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Operation=ItemLookup&ItemId=0679722769&ResponseGroup=ItemAttributes,Offers,Images,Reviews&Version=2013-08-01

It should be easy enough to see that the request is an ItemLookup for the item with ASIN 0679722769 (Ulysses by James Joyce) and should return ItemAttributes, Offers, Images, and Reviews. There are a plethora of ResponseGroups available to request.

We'll store the request arguments as ARGS

ARGS="Service=AWSECommerceService&AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&AssociateTag=mytag20&Operation=ItemLookup&ItemId=0679722769&ResponseGroup=ItemAttributes,Offers,Images,Reviews&Version=2013-08-01"

Adding the timestamp

The current UTC timestamp needs added to the arguments list, which is easy enough to do with the date command.

TIMESTAMP=$(date -u "+%Y-%m-%dT%H:%M:%SZ")
ARGS="$ARGS&Timestamp=$TIMESTAMP"

URL Encoding ',' and ':'

The commas and colons can be substituted for their URL encoded counterparts with sed.

ARGS=$(echo $ARGS | sed 's/:/%3A/g;s/,/%2C/g')

Line break the argument pairs

The ampersands in the argument string can be "translated" to newlines with tr.

ARGS=$(echo $ARGS | tr '&' '\n')

Sorting the argument pairs

By default, sort sorts alphabetically. However, the sorting needs to happen in byte order, that is all uppercase parameters come before lowercase parameters. By setting LC_ALL=C sort will sort in byte order. Note that $ARGS is enclosed in double quotes so the newlines are interpreted properly.

ARGS=$(echo "$ARGS" | LC_ALL=C sort)

Rejoin with Ampersands

The newlines can be translated back to ampersands with tr. echo is setup to not print a trailing newline, which would be replaced with an ampersand.

ARGS=$(echo -n "$ARGS" | tr '\n' '&')

Create string to sign

The string to sign is made by appending the type of HTTP request, the target URL, and the REST endpoint.

STRING=$'GET\nwebservices.amazon.com\n/onca/xml\n'$ARGS

Create signature

The infamous Step 8. The request string needs to be signed with a AWS secret key. The command line utility openssl is capable of computing the HMAC with the SHA256 algorithm. The output from openssl then needs base64 encoded. Again, echo must be configure to not print a trailing newline.

SIGNATURE=$(echo -n "$STRING" | openssl dgst -sha256 -hmac "$SECRET_KEY" -binary | openssl enc -base64)

URL encode the signature

Base64 encoding allows '+' and '=' which need URL encoded.

SIGNATURE=$(echo $SIGNATURE | sed 's/+/%2B/g;s/=/%3D/g;')

Create Request URL

The API URL, arguments, and signature need combined into one URL.

REQUEST="http://webservices.amazon.com/onca/xml?$ARGS&Signature=$SIGNATURE"

Make curl post

curl -s $REQUEST

If successful, the API will return XML.

<?xml version="1.0" ?>
<ItemLookupResponse
	xmlns="http://webservices.amazon.com/AWSECommerceService/2013-08-01">
	<OperationRequest>
...
		<Item>
			<ASIN>0679722769</ASIN>
			<DetailPageURL>https://www.amazon.com/Ulysses-James-Joyce/dp/0679722769%3FSubscriptionId%3DAKIAJYK4ZM7MKHMVJ2RA%26tag%3Dfrdmtoplay-20%26linkCode%3Dxm2%26camp%3D2025%26creative%3D165953%26creativeASIN%3D0679722769</DetailPageURL>
			<ItemLinks>
				<ItemLink>
					<Description>Technical Details</Description>
...
		</Item>
	</Items>
</ItemLookupResponse>

Parsing the output

The Product Advertising API returns XML which can be parsed in bash with a few tricks. The first is going to be a user defined function read_xml() which reads tags into $ENTITY and the corresponding content into $CONTENT

read_xml () {
    local IFS=\>
    read -d \< ENTITY CONTENT
}

Used in a while loop, read_xml can be used to search for specific XML entities, for example the PageURL.

while read_xml; do
	if [[ $ENTITY = "DetailPageURL" ]]; then
		URL=$CONTENT
	fi
done < curl -s $REQUEST
echo $URL

The complete bash script:

# User Defined Parameters
AFFILIATE_TAG=mytag-20
ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
ASIN=0679722769
SECRET_KEY=12345678

# Request arguments
ARGS="Service=AWSECommerceService&AWSAccessKeyId=$ACCESS_KEY&AssociateTag=$AFFILIATE_TAG&Operation=ItemLookup&ItemId=$ASIN&ResponseGroup=ItemAttributes,Offers,Images,Reviews&Version=2013-08-01"  

# Append Timestamp
TIMESTAMP=$(date -u "+%Y-%m-%dT%H:%M:%SZ")  
ARGS="$ARGS&Timestamp=$TIMESTAMP"  

# URL Encode
ARGS=$(echo $ARGS | sed 's/:/%3A/g;s/,/%2C/g')  

# Create line broken list of argument pairs
ARGS=$(echo $ARGS | tr '&' '\n')

# Sort arguments in ABCabc order
ARGS=$(echo "$ARGS" | LC_ALL=C sort)  

# Rejoin the arguments into one line
ARGS=$(echo -n "$ARGS" | tr '\n' '&')

# Build request string
STRING=$'GET\nwebservices.amazon.com\n/onca/xml\n'$ARGS

# Create signature
SIGNATURE=$(echo -n "$STRING" | openssl dgst -sha256 -hmac "$SECRET_KEY" -binary | openssl enc -base64)  

# URL Encode signature
SIGNATURE=$(echo $SIGNATURE | sed 's/+/%2B/g;s/=/%3D/g;')  

# Create Request
REQUEST="http://webservices.amazon.com/onca/xml?$ARGS&Signature=$SIGNATURE"

# Make request
curl -s $REQUEST

Was this information useful? Are you going to buy something from Amazon in the next 24-hours? Please consider using my Amazon Affilliate URL as a way of saying thanks. Or, feel free to donate BTC (1DNwgPQMfoWZqnH78yt6cu4WukJa3h8P1f) or ETH (0xf3c4a78c24D34E111f272Ac2AC72b1f01ba52DF3).