Rotating IAM Keys with Felix

You can find Felix on GitHub - the next link to it is pretty far down this post.

IAM keys. Can't live with 'em, can't get your ELB access logs into SumoLogic without 'em.

Well, you can largely live without them if you try hard enough. But if you can't, you should totally rotate them. The CIS AWS Benchmark recommends that keys live no longer than 90 days. Our AWS TAMs have casually thrown out 30 days.

The way I like to phrase it is just, "rotate them all the time to ensure you're ready to rotate them at any time."

I have 100 IAM keys in 100 Travis projects. How am I supposed to rotate them?

So you use TravisCI. And you have 100 projects that all deploy something to AWS. Meaning you have 100 different IAM keys.

Luckily, they are just deployment keys, so you don't really need to rotate them, right?

Right! Well...until you do need to rotate them. Remember earlier this year when Travis accidentally leaked a bunch of secure environment variables (including GitHub OAuth tokens)? And people weren't really sure how to go through all of their projects and update their secrets?

Maybe we should figure out how to rotate these keys regularly so it's not a giant hassle when we actually need to do it.

Travis, meet Felix.

So that's why I wrote Felix.

Felix is a minimal-configuration, flexible, and serverless framework for orchestrating IAM key rotations with third-party services. It currently supports SumoLogic, GitLab, and TravisCI.

How does Felix work? It uses the IAM User path and username to discover information about how and where the key is used. It looks at all users in a specified path (/service/ by default) and uses the rest to discover where the key is used and how it can rotate the key.

Let's take for example, the IAM user /service/travis/maclennann/rake-terraform. Felix sees this user and is able to discover that the key attached to this user is used in TravisCI in the maclennann/rake-terraform project.

Felix generates a new key for this user, passes it to the TravisCI plugin to update the environment variables for that project, and deactivates the old key.

Let's try it!

Felix is super easy to deploy, configure, and run. So let's try it.

Note: This will likely cost you a couple bucks, but we're going to clean up after ourselves.



Let's get your local environment set up first.

() $ git clone
Cloning into 'felix'...
remote: Counting objects: 119, done.
remote: Compressing objects: 100% (63/63), done.
remote: Total 119 (delta 52), reused 119 (delta 52), pack-reused 0
Receiving objects: 100% (119/119), 172.76 KiB | 2.13 MiB/s, done.
Resolving deltas: 100% (52/52), done.
() $ cd felix/
() $ nvm use
Found '/Users/nmaclennan/src/' with version <v6.10.2>
Now using node v6.10.2 (npm v5.4.0)
() $ npm install -s
added 577 packages in 8.468s

Are you familiar with projects using the serverless framework? That's what Felix is. Basically, you can look in the serverless.yml file at the root of the repo to see what's going to happen when you deploy it - it's a glorified CloudFormation template (in a good way).

It's going to create:

  • an S3 bucket to store the code
  • a KMS key to encrypt secret settings
  • an SNS topic to publish results to
  • the Lambda function itself
  • a CloudWatch event to trigger the function every other day

You shouldn't need to change anything in there, but please do go ahead and give everything a look over. You shouldn't run things you don't understand in your account.


Once you're done giving it the once-over, let's deploy it!

Note that the sts alias I'm running here will generate me STS keys for my account using the method I wrote about last year for awsadvent. You should make sure your proper AWS profile, region, etc are all configured however it is you manage that.

() $ sts production
Password for nmaclennan:
Generated STS keys for production...

(production) $ npm run deploy

> felix@0.2.0 deploy /Users/nmaclennan/src/
> sls deploy

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (2.38 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
Serverless: Stack update finished...
Service Information
service: felix
stage: prd
region: us-east-1
stack: felix-prd
api keys:
  rotate: felix-prd-rotate

Nice. That was easy, right?


Felix gets its configuration from the SSM parameter store. So let's get that all set up. Luckily, there's a nice little script right in the repo to help get you started.

You'll need a few things: the ARN of the SNS topic to publish results to (arn:aws:sns:us-east-1:[account id]:FelixReports is the one created when we deployed), the IAM path prefix to find users in (/service/ is the default), and your Travis API key (you can generate it using the travis gem).

(production) $ ./configure.js
Felix Configuration Setup:

Felix uses the EC2 Parameter Store to load its configuration. Let's make sure you
have all of the correct parameters created in the right paths and the right ones
are encrypted.

Plugin: aws
Description: AWS-specific settings for locating IAM users and notifying SNS.
  Parameter: /felix/aws/snsTopic
  Description: The SNS topic to publish Felix reports.
  Value (): arn:aws:sns:us-east-1:[account id]:FelixReports
    /felix/aws/felix/aws/snsTopic => arn:aws:sns:us-east-1:[account id]:FelixReports

  Parameter: /felix/aws/userPath
  Description: The path preflix to look in for IAM users.
  Value (): /service/
    /felix/aws/felix/aws/userPath => /service/

Plugin: gitlab
Description: Gitlab settings for updating IAM keys used by Pipelines jobs.
Do you want to configure this plugin? [y/n]: n

Plugin: sumologic
Description: SumoLogic settings for updating AWS sources.
Do you want to configure this plugin? [y/n]: n

Plugin: travis
Description: Travis CI settings for managing repositories.
Do you want to configure this plugin? [y/n]: y
  Parameter: /felix/travis/token
  Description: The Travis API token.
  Value (): [redacted]                         
    /felix/travis/felix/travis/token => [redacted]

/felix/aws/snsTopic => arn:aws:sns:us-east-1:[account id]:FelixReports
/felix/aws/userPath => /service/
/felix/travis/token => [redacted]


Setting up a user

Alright! We're good to start rotating! Now we just need some users to generate and rotate keys for. And a Travis project to manage those keys within. Once you've chosen one for yourself, let's make that project a user.

$ aws iam create-user --path /service/travis/maclennann/ --user-name rake-terraform@travis
    "User": {
        "UserName": "rake-terraform@travis", 
        "Path": "/service/travis/maclennann/", 
        "CreateDate": "2017-08-31T22:54:27.078Z", 
        "Arn": "arn:aws:iam::[account id]:user/service/travis/maclennann/rake-terraform@travis"

You may notice I appended the user with @travis. This is an optional thing. Since some UIs only show the username, this preserves the context we lose when the path is dropped and prevents collisions if you use multiple services for one project.

But, in short, Felix will manage keys for this used in the Travis project maclennann/rake-terraform.

Try it out!

Note: If you subscribe to the SNS topic that was created on deploy, you can get email notifications about successful rotations (and errors).

Nice! We're good to go. Even though keys generated for this user won't do anything yet, we can still try rotating them in Travis.

You'll notice this project doesn't currently have any variables:

(production) $ travis env list --repo maclennann/rake-terraform
# environment variables for maclennann/rake-terraform

Let's run Felix and get the project some keys!

(production) $ npm run remote

> felix@0.2.0 remote /Users/nmaclennan/src/
> sls invoke -f rotate -l
REPORT RequestId: [guid]	Duration: 1349.49 ms	Billed Duration: 1400 ms 	Memory Size: 256 MB	Max Memory Used: 62 MB	

(production) $ travis env list
# environment variables for maclennann/rake-terraform

Cool! It created keys for us and posted them to Travis. Let's test out rotation, too, while we're here!

(production) $ npm run remote

> felix@0.2.0 remote /Users/nmaclennan/src/
> sls invoke -f rotate -l
REPORT RequestId: [guid]	Duration: 709.72 ms	Billed Duration: 800 ms 	Memory Size: 256 MB	Max Memory Used: 65 MB	

(production) $ travis env list --repo maclennann/rake-terraform
# environment variables for maclennann/rake-terraform

Nice! Look! It changed! The original key started with AKIAIMP and the new one is AKIAJ52. And Felix can do this for all of your keys. Well, the ones in Travis, SumoLogic, or GitLab.


So hopefully you've decided to just go ahead and run Felix forever. But also maybe you want to clean up from your experiment before you roll it out for real.

(production) $ ./node_modules/.bin/sls remove
Serverless: Getting all objects in S3 bucket...
Serverless: Removing objects in S3 bucket...
Serverless: Removing Stack...
Serverless: Checking Stack removal progress...
Serverless: Stack removal finished...


Anyway, check out Felix. Let me know what you think. We're looking to expand the repertoire of services it can interact with. Things like Jenkins Credentials Store, Secret Server, and a couple others have been thrown around, if you're looking for something to do.

This project is also still definitely early in its lifecycle and there are bound to be bugs and some really high-value feature enhancements. If there's something you don't like about the project as it is now (I'll admit, the SumoLogic plugin is pretty slow) or something new you would like to see, feel free to just pop in an issue or even like a PR.

Author image
Cloud | Security | Cloud Security Passionate about providing developers with easy-to-use security and operations tooling.