AWS Secrets Manager and the principle of least-privilege
AWS Secrets Manager is a great place to store secrets that are needed by somebody else (machines or people) if you are running your infrastructure on AWS.
You can give subjects (users, groups, or roles) access only to specific secrets following the principle of least privilege.
There’s an AWS guide which will help you with the process: Allow read access to specific secrets in AWS Secrets
Below, I’m going to walk you through the process of storing and using API keys for Amplitude Export API.
Storing API keys
This shows how to give users access to the specific API (keys).
I assume you already have an Amplitude project with corresponding API keys but this is really just an example so you definitely don’t need an Amplitude account to follow along.
1. Save API keys in Secrets Manager
First, store the Amplitude API keys in Secrets Manager under well-defined key, lets say my-app/prod/amplitude-api
. The secret value will look like this:
{
"api-key": "abcdefg",
"secret-key": "1234567890"
}
Then, copy Secret ARN
(can be found at the top of the AWS Secrets Manager console - 'Secret details' section). For example:
arn:aws:secretsmanager:{AWS_REGION}:{AWS_ACCOUNT_ID}:secret:config/my-app/prod/amplitude-api-N9iZ3I
2. Create the IAM policy
module "amplitude-api-policy" {
source = "terraform-aws-modules/iam/aws//modules/iam-policy"
version = "~> 4.3"
name = "amplitude-api-policy"
path = "/"
description = "This policy allows you to access Amplitude API keys"
policy = <
Create the IAM policy using the specific ARN - this example uses terraform:
3. Create the IAM group and attach the policy
Again, with Terraform:
module "amplitude-api-users-group" {
source = "terraform-aws-modules/iam/aws//modules/iam-group-with-policies"
version = "~> 4.3"
name = "amplitude-api-users"
group_users = [
"john.doe"
]
attach_iam_self_management_policy = false
# attach the policy defined above to the group
custom_group_policy_arns = [
module.amplitude-api-policy.arn,
]
}
With this policy in place, you can be sure they have access only to the specific API secrets - nothing more, nothing less.
Using API keys
We have the API keys safely stored in the Secrets Manager and every member of the amplitude-api-users
group can now fetch them via standard AWS APIs.
Fetch API keys with AWS CLI
Quick way to test that you have access to API keys is with AWS CLI:
aws secretsmanager get-secret-value --secret-id "config/my-app/prod/amplitude-api"
# this should return something like this:
{
"ARN": "arn:aws:secretsmanager:{AWS_REGION}:{AWS_ACCOUNT_ID}:secret:config/my-app/prod/amplitude-api-N9iZ3I",
"Name": "config/my-app/prod/amplitude-api",
"VersionId": "ae7b2ec0-df1c-416f-8037-7b03517c2670",
"SecretString": "{\"api-key\":\"abcdefg\",\"secret-key\":\"1234567890\"}", (1)
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": "2022-02-16T10:00:58.597000+01:00"
}
1. SecretString
contains the JSON we stored in Secrets Manager before
Use API keys with Clojure
Here’s a complete code for fetching API keys via AWS API and using them to download events from Amplitude:
(ns amplitude.export
"Playing with Amplitude Export API:
https://developers.amplitude.com/docs/export-api#export-api-parameters
It can be used to export raw events data unlike the UI
where you can typically only export to chart data or users.
See https://community.amplitude.com/instrumentation-and-data-management-57/how-do-i-pull-a-specific-event-with-all-event-properties-and-export-to-csv-491"
(:require [clj-http.client :as http]
[clojure.data.json :as json]
[clojure.java.io :as io]
[clojure.string :as str]
[cognitect.aws.client.api :as aws]
[cognitect.aws.credentials :as credentials]))
(def secrets-client
(delay (aws/client {:api :secretsmanager
:credentials-provider (credentials/profile-credentials-provider "my-aws-profile")})))
(defn fetch-api-keys []
(let [{:keys [SecretString] :as _result} (aws/invoke @secrets-client
{:op :GetSecretValue
:request {:SecretId "config/my-app/prod/amplitude-api"}})]
(json/read-str SecretString)))
(def api-keys (delay (fetch-api-keys)))
;; check the docs
(comment
(keys (aws/ops @secrets-client))
(aws/doc @secrets-client :GetSecretValue)
,)
(defn download-events! [output-file start end]
(println "Exporting Amplitude events between" start "and" end)
(time (let [{:strs [api-key secret-key]} @api-keys
response (http/get (format "https://amplitude.com/api/2/export?start=%s&end=%s" start end)
{:basic-auth [api-key secret-key]
:as :stream})]
(io/copy (:body response) (io/file output-file))
(println "Amplitude events exported to" output-file))))
(defn- read-events! [json-file]
(map json/read-str (line-seq (io/reader json-file))))
(defn read-all-events! [directory]
(let [read-all-xf (comp (filter #(str/ends-with? (.getName %) ".json"))
(mapcat read-events!))]
(into [] read-all-xf (file-seq (io/file directory)))))
(comment
;; download all the events from 31.1.2022
(download-events! "all-events.zip" "20220131T00" "20220131T23")
;; Now go to the file and unzip it manually
;; Also gunzip all the files from the extracted zip archive.
;; Then you can read all the downloaded events stored in given folder
(def all-events
;; it can take almost 10 seconds to read the data from JSON files consuming about 150 MB disk space
(->> (time (read-all-events! "amplitude-events-directory"))))
,)
The perils of MFA
The approach I described above works fine but there’s a problem if you are using the IAM policy to enforce multi-factor
The trouble is that this policy explicitly denies almost all the actions if the user is not using MFA. It becomes a problem when you try to fetch the secret through aws cli - you can get a rather confusing error:
aws secretsmanager get-secret-value --secret-id "config/my-app/prod/amplitude-api"
An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed
What does it have to do with KMS?
Under the hood, Secrets Manager uses a KMS key for encryption and decryption of secrets. This means it has to have
proper permissions for the KMS key:
-
kms:GenerateDataKey
-
kms:Decrypt
If you use the default aws-managed KMS key it should all be good because they automatically create an IAM policy allowing every member of the associated AWS account to access KMS. But the "Deny" in the aforementioned mfa
enforcement policy overwrites the "Allow" rule and thus leads to the permission error.
Temporary session tokens to the rescue!
After spending a couple of days on this problem and posting a question about it, I’ve finally managed to solve it through the use of temporary session tokens for aws cli.
You can automate this relatively easily with a bash script and a custom aws profile.
If you use 1password or another password manager with MFA and CLI support, you can then use the script like this:
eval $(op signin my) && aws_mfa.sh -u -m $(op get totp 'My 1password aws login item name')
This will generate temporary security credentials valid for 12 hours and saves them into the mfa
profile (check ~/.aws/credentials
).
Fetching secrets then works again, but you must specify the profile
aws --profile mfa secretsmanager get-secret-value --secret-id "config/my-app/prod/amplitude-api"
The complete script:
#!/bin/bash
#########################################################################################################
# Bash script to create temporary session tokens with MFA for AWS KMS. #
# #
# After running this script, ~/.aws/config and ~/.aws/credentials are updated automatically containing #
# the newly created session tokens and configs. #
# Source: https://aws.amazon.com/premiumsupport/knowledge-center/authenticate-mfa-cli/ #
# #
# Requirements: #
# aws-cli: #
# See: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html #
# jq: #
# $ sudo apt-get install jq # linux #
# $ brew install jq # macOs #
# #
# Usage: #
# $ aws_mfa.sh -u -m #
# #
#########################################################################################################
# Exit immediately if there's an error.
set -e
# Constants.
AWS_CREDENTIAL_KEYS=("aws_access_key_id" "aws_secret_access_key" "aws_session_token")
PROFILE_NAME="mfa"
CONFIG_REGION_KEY="region"
CONFIG_REGION_VALUE="eu-west-1"
# Parameters.
username=""
mfa_code=""
# Help menu.
function help() {
echo "Initializes AWS KMS with a temporary session token using MFA."
echo "Usage: aws_mfa.sh -u -m "
echo ""
echo "Options:"
echo " -h Prints the help menu."
echo " -u Sets the username [required]."
echo " -m Sets the MFA code [required]."
}
# Read arguments.
while getopts "hu:m:" opt; do
case ${opt} in
h)
help
;;
u)
username=$OPTARG
;;
m)
mfa_code=$OPTARG
;;
\?)
help
;;
esac
done
# Check username and MFA code.
if [ -z "$username" ]
then
echo "Missing username, please provide a valid username with -u."
exit 64
fi
if [ -z "$mfa_code" ]
then
echo "Missing MFA code, please provide a valid MFA code with -m."
exit 64
fi
# Get serial number for the user.
mfa_arn=$(aws iam list-virtual-mfa-devices | jq -c ".VirtualMFADevices[] | select(.User.UserName == \"$username\") | .SerialNumber" | tr -d '"')
if [ -z "$mfa_arn" ]
then
echo "User not found: ${username}, could not init AWS."
else
echo "Obtained MFA ARN"
fi
# Update credentials: detele existing session token and add new one under given profile.
echo "Updating ~/.aws/credentials"
aws_credential_values=($(aws sts get-session-token --serial-number ${mfa_arn} --token-code ${mfa_code} | jq -c ".Credentials.AccessKeyId, .Credentials.SecretAccessKey, .Credentials.SessionToken" | tr -d '"'))
for i in "${!aws_credential_values[@]}"; do
aws configure set --profile ${PROFILE_NAME} ${AWS_CREDENTIAL_KEYS[$i]} ${aws_credential_values[$i]}
done
# Update config.
echo "Updating ~/.aws/config"
aws configure set --profile ${PROFILE_NAME} ${CONFIG_REGION_KEY} ${CONFIG_REGION_VALUE}
Links