Engineering

AWS Secrets Manager and the principle of least-privilege

Written by Juraj Martinka | Apr 5, 2022 9:42:28 AM

AWS Secrets Manager is a great place to store secrets that are needed by somebody else (machines or people) if you are running your infrastructure on AWS.

 

You can give subjects (users, groups, or roles) access only to specific secrets following the principle of least privilege.

 

There’s an AWS guide which will help you with the process: Allow read access to specific secrets in AWS Secrets

Manager.

 

Below, I’m going to walk you through the process of storing and using API keys for Amplitude Export API.

 

 

 

Storing API keys

 

This shows how to give users access to the specific API (keys).

 

I assume you already have an Amplitude project with corresponding API keys but this is really just an example so you definitely don’t need an Amplitude account to follow along.

 

 

1. Save API keys in Secrets Manager

 

First, store the Amplitude API keys in Secrets Manager under well-defined key, lets say my-app/prod/amplitude-api. The secret value will look like this:

 

{
  "api-key": "abcdefg",
  "secret-key": "1234567890"
}

 

Then, copy Secret ARN (can be found at the top of the AWS Secrets Manager console - 'Secret details' section). For example:

 

arn:aws:secretsmanager:{AWS_REGION}:{AWS_ACCOUNT_ID}:secret:config/my-app/prod/amplitude-api-N9iZ3I


2. Create the IAM policy

 

module "amplitude-api-policy" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-policy"
  version = "~> 4.3"

  name        = "amplitude-api-policy"
  path        = "/"
  description = "This policy allows you to access Amplitude API keys"
  policy      = <

 

Create the IAM policy using the specific ARN - this example uses terraform:

 

 

3. Create the IAM group and attach the policy

 

Again, with Terraform:

 

module "amplitude-api-users-group" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-group-with-policies"
  version = "~> 4.3"

  name = "amplitude-api-users"

  group_users = [
    "john.doe"
  ]

  attach_iam_self_management_policy = false

  # attach the policy defined above to the group
  custom_group_policy_arns = [
    module.amplitude-api-policy.arn,
  ]
}

 

With this policy in place, you can be sure they have access only to the specific API secrets - nothing more, nothing less.

 

 

 

Using API keys

 

We have the API keys safely stored in the Secrets Manager and every member of the amplitude-api-users group can now fetch them via standard AWS APIs.

 

 

Fetch API keys with AWS CLI

 

Quick way to test that you have access to API keys is with AWS CLI:

 

aws secretsmanager get-secret-value --secret-id "config/my-app/prod/amplitude-api"

# this should return something like this:
{
    "ARN": "arn:aws:secretsmanager:{AWS_REGION}:{AWS_ACCOUNT_ID}:secret:config/my-app/prod/amplitude-api-N9iZ3I",
    "Name": "config/my-app/prod/amplitude-api",
    "VersionId": "ae7b2ec0-df1c-416f-8037-7b03517c2670",
    "SecretString": "{\"api-key\":\"abcdefg\",\"secret-key\":\"1234567890\"}", (1)
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2022-02-16T10:00:58.597000+01:00"
}

 

1. SecretString contains the JSON we stored in Secrets Manager before


 

Use API keys with Clojure

 

Here’s a complete code for fetching API keys via AWS API and using them to download events from Amplitude:

 

(ns amplitude.export
  "Playing with Amplitude Export API:
  https://developers.amplitude.com/docs/export-api#export-api-parameters

  It can be used to export raw events data unlike the UI
  where you can typically only export to chart data or users.
  See https://community.amplitude.com/instrumentation-and-data-management-57/how-do-i-pull-a-specific-event-with-all-event-properties-and-export-to-csv-491"
  (:require [clj-http.client :as http]
            [clojure.data.json :as json]
            [clojure.java.io :as io]
            [clojure.string :as str]
            [cognitect.aws.client.api :as aws]
            [cognitect.aws.credentials :as credentials]))


(def secrets-client
  (delay (aws/client {:api :secretsmanager
                      :credentials-provider (credentials/profile-credentials-provider "my-aws-profile")})))

(defn fetch-api-keys []
  (let [{:keys [SecretString] :as _result} (aws/invoke @secrets-client
                                                       {:op :GetSecretValue
                                                        :request {:SecretId "config/my-app/prod/amplitude-api"}})]
    (json/read-str SecretString)))

(def api-keys (delay (fetch-api-keys)))


;; check the docs
(comment
  (keys (aws/ops @secrets-client))

  (aws/doc @secrets-client :GetSecretValue)
,)


(defn download-events! [output-file start end]
  (println "Exporting Amplitude events between" start "and" end)
  (time (let [{:strs [api-key secret-key]} @api-keys
              response (http/get (format "https://amplitude.com/api/2/export?start=%s&end=%s" start end)
                                 {:basic-auth [api-key secret-key]
                                  :as :stream})]
          (io/copy (:body response) (io/file output-file))
          (println "Amplitude events exported to" output-file))))

(defn- read-events! [json-file]
  (map json/read-str (line-seq (io/reader json-file))))

(defn read-all-events! [directory]
  (let [read-all-xf (comp (filter #(str/ends-with? (.getName %) ".json"))
                          (mapcat read-events!))]
    (into [] read-all-xf (file-seq (io/file directory)))))


(comment
  ;; download all the events from 31.1.2022
  (download-events! "all-events.zip" "20220131T00" "20220131T23")

  ;; Now go to the file and unzip it manually
  ;; Also gunzip all the files from the extracted zip archive.

  ;; Then you can read all the downloaded events stored in given folder
  (def all-events
    ;; it can take almost 10 seconds to read the data from JSON files consuming about 150 MB disk space
    (->> (time (read-all-events! "amplitude-events-directory"))))
,)

 

 

The perils of MFA

 

The approach I described above works fine but there’s a problem if you are using the IAM policy to enforce multi-factor

authentication.

 

The trouble is that this policy explicitly denies almost all the actions if the user is not using MFA. It becomes a problem when you try to fetch the secret through aws cli - you can get a rather confusing error:

 

aws secretsmanager get-secret-value --secret-id "config/my-app/prod/amplitude-api"

An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed

 

 

What does it have to do with KMS?

 

  • kms:GenerateDataKey

  • kms:Decrypt

 

If you use the default aws-managed KMS key it should all be good because they automatically create an IAM policy allowing every member of the associated AWS account to access KMS. But the "Deny" in the aforementioned mfa

enforcement policy overwrites the "Allow" rule and thus leads to the permission error.

 

 

 

Temporary session tokens to the rescue!

 

After spending a couple of days on this problem and posting a question about it, I’ve finally managed to solve it through the use of temporary session tokens for aws cli.

 

You can automate this relatively easily with a bash script and a custom aws profile.

 

If you use 1password or another password manager with MFA and CLI support, you can then use the script like this:

 

eval $(op signin my) && aws_mfa.sh -u  -m $(op get totp  'My 1password aws login item name')

 

This will generate temporary security credentials valid for 12 hours and saves them into the mfa profile (check ~/.aws/credentials).

 

Fetching secrets then works again, but you must specify the profile

 

aws --profile mfa secretsmanager get-secret-value --secret-id "config/my-app/prod/amplitude-api"

The complete script:

#!/bin/bash

#########################################################################################################
# Bash script to create temporary session tokens with MFA for AWS KMS.                                  #
#                                                                                                       #
# After running this script, ~/.aws/config and ~/.aws/credentials are updated automatically containing  #
# the newly created session tokens and configs.                                                         #
# Source: https://aws.amazon.com/premiumsupport/knowledge-center/authenticate-mfa-cli/                  #
#                                                                                                       #
# Requirements:                                                                                         #
#   aws-cli:                                                                                            #
#     See: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html                #
#   jq:                                                                                                 #
#     $ sudo apt-get install jq     # linux                                                             #
#     $ brew install jq             # macOs                                                             #
#                                                                                                       #
# Usage:                                                                                                #
#   $ aws_mfa.sh -u  -m                                                             #
#                                                                                                       #
#########################################################################################################

# Exit immediately if there's an error.
set -e

# Constants.
AWS_CREDENTIAL_KEYS=("aws_access_key_id" "aws_secret_access_key" "aws_session_token")
PROFILE_NAME="mfa"
CONFIG_REGION_KEY="region"
CONFIG_REGION_VALUE="eu-west-1"


# Parameters.
username=""
mfa_code=""

# Help menu.
function help() {
    echo "Initializes AWS KMS with a temporary session token using MFA."
    echo "Usage: aws_mfa.sh -u  -m "
    echo ""
    echo "Options:"
    echo "  -h    Prints the help menu."
    echo "  -u    Sets the username [required]."
    echo "  -m    Sets the MFA code [required]."
}

# Read arguments.
while getopts "hu:m:" opt; do
    case ${opt} in
        h)
            help
            ;;
        u)
            username=$OPTARG
            ;;
        m)
            mfa_code=$OPTARG
            ;;
        \?)
            help
            ;;
    esac
done

# Check username and MFA code.
if [ -z "$username" ]
then
    echo "Missing username, please provide a valid username with -u."
    exit 64
fi
if [ -z "$mfa_code" ]
then
    echo "Missing MFA code, please provide a valid MFA code with -m."
    exit 64
fi


# Get serial number for the user.
mfa_arn=$(aws iam list-virtual-mfa-devices | jq -c ".VirtualMFADevices[] | select(.User.UserName == \"$username\") | .SerialNumber" | tr -d '"')
if [ -z "$mfa_arn" ]
then
    echo "User not found: ${username}, could not init AWS."
else
    echo "Obtained MFA ARN"
fi

# Update credentials: detele existing session token and add new one under given profile.
echo "Updating ~/.aws/credentials"
aws_credential_values=($(aws sts get-session-token --serial-number ${mfa_arn} --token-code ${mfa_code} | jq -c ".Credentials.AccessKeyId, .Credentials.SecretAccessKey, .Credentials.SessionToken" | tr -d '"'))
for i in "${!aws_credential_values[@]}"; do
    aws configure set --profile ${PROFILE_NAME} ${AWS_CREDENTIAL_KEYS[$i]} ${aws_credential_values[$i]}
done

# Update config.
echo "Updating ~/.aws/config"
aws configure set --profile ${PROFILE_NAME} ${CONFIG_REGION_KEY} ${CONFIG_REGION_VALUE}