Occasionally, an incident will require you to have access to the customer’s environment. We recommend using our built in Break Glass workflow to acquire this access, while maintaining trust and confidence with your customers.

The Break Glass feature currently depends on Cloudformation Quickcreate links, and therefore only supports AWS. There is nothing specific to AWS about the best practices we recommend here, though. You can build your own custom Cloudformation stacks, or use an alternative approach for other platforms.

Break-Glass Runbook

To ensure you follow a consistent SOP, we recommend creating a break glass runbook. Having this prepared ahead of time will prevent you from having to scramble for solutions in the heat of the moment. It should address:

  • When it is appropriate to break glass
  • How to contact your customer
  • How to create the Cloudformation Quickcreate link and share it with your customer
  • Instructions for your customer to install the break glass stack
  • How to debug and resolve issues quickly once you have access
  • Instructions for your customer to uninstall the break glass stack when the incident is resolved

Break Glass Permissions

You can define the permissions needed to break glass in your app config. There are four levels of access:

  • Network: Permissions to access, update, and delete the VPC the install is provisioned in.
  • Sandbox: Permissions to access, update, and delete the install sandbox resources.
  • Runner: Permissions to access, update, and delete the install runner resources.
  • Component: Permissions to access, update, and delete the install component resources.

While getting full access may be convenient, you should provide guidance for how much access is needed in your runbook.

Customer-Managed Networks

If your app is hosted in its own, isolated network, then you can probably grant yourself full access to that network and all of its resources. However, if the app is hosted in a customer-managed network, you should be very careful about how much access you have.

A few recommendations you should consider:

  • Set a permissions boundary on the break glass role
  • Scope access based on tags
  • If your resources are deployed in specific subnets, only grant access to those subnets