Overview

Run EnvKey in your own AWS account on the same auto-scaling, highly available, geographically redundant, high-security architecture that powers EnvKey Cloud. It takes about an hour to install (most of that is waiting for resources to spin up).

Your infrastructure isn't on AWS? That's fine. You can connect from anywhere, including other cloud providers.

Don't like AWS? That's fine too. After initial setup, you'll rarely (if ever) need to sign in to your EnvKey AWS account. It runs, scales, and upgrades on its own with zero downtime.

Setup & Deployment

1.) Install the EnvKey UI and CLI on your local machine.

2.) Open the EnvKey UI and click Create A New Organization.

3.) On theChoose Your Host screen, select Business Self-Hosted.

4.) Create a new AWS account. If your organization already has an AWS account, we recommend creating this account under an AWS organization in order to centralize billing and account management. Store the root account sign in credentials somewhere safe, like a password manager.

5.) If billing is not already connected through your AWS organization, you'll need to add a payment method to your account. AWS costs typically run between $200 and $250 per month (depending on options, regions, and savings plan).

6.) Create a CloudTrail logging trail in order to track actions taken in this account.

7.) Enable multi-factor authentication for the root account. Go to the Security Credentials page. In the Multi-factor authentication (MFA) section, click the Activate MFA button and follow along to add multi-factor authentication.

8.) Delete root user access key. Also on the Security Credentials page, go to the Access keys (access key ID and secret access key) section. Delete the access key in the list.

9.) Signed in to your new AWS account, go to the IAM users page in the AWS console and create a new user for yourself. Under Select AWS access type, check the Password - AWS Management Console access box. Leave the other fields in the form as they are, then click the Next: Permissions button. On the Set permissions step, select Attach existing policies directly, then check AdministratorAccess in the list. Then click Next: Tags. Then Next:Review. Then Create User. In the Success box you'll find a url for you to sign in with your new IAM user, and below that a list with your new user in it. Click Show next to the temporary password and copy the password.

10.) Sign out of the root account and sign in with your IAM user account using your new user's sign in url. You'll be asked to set a new password. Store your IAM user credentials somewhere safe, like a password manager.

🚧

No More Root

From now on, all actions taken in the EnvKey AWS account should use an IAM user account, not the root account (this is a security best practice).

11.) Enable multi-factor authentication for your IAM user account. Go to the Security Credentials page. In the Multi-factor authentication (MFA) section, click the Activate MFA button and follow along to add multi-factor authentication.

12.) On your local machine, create a file at $HOME/.aws/credentials if you don't already have one.

13.) Go to the IAM users page in the console. Create a envkey-deploy IAM user and check the Access key - Programmatic access box. Then click the Next: Permissions button. On the Set permissions step, select Attach existing policies directly, then check AdministratorAccess in the list. Then click Next: Tags. Then Next:Review. Then Create User. Next, copy the access key id and secret access key into the file you created in the previous step, with the following format:

[envkey-host]
aws_access_key_id = your-access-key-id
aws_secret_access_key = your-secret-access-key

The envkey-deploy IAM user should be deleted once the deployment has finished, and the envkey-host credentials removed from the $HOME/.aws/credentials file.

14.) Decide on a primary region and a failover region.

📘

Behind-Your-Firewall Mode

If you're going to use Behind-Your-Firewall Mode to connect to EnvKey from a VPC in another AWS account through AWS PrivateLink, EnvKey will run in a single region. It still offers high availability and an in-region failover to Lambda and S3.

You can choose from the following regions:

Virginia • us-east-1
Ohio • us-east-2
Northern California • us-west-1
Oregon • us-west-2
Canada Central • ca-central-1
Sao Paolo • sa-east-1
Singapore • ap-southeast-1
Sydney • ap-southeast-2
Ireland • eu-west-1
London • eu-west-2
Paris • eu-west-3
Frankfurt • eu-central-1
Stockholm • eu-north-1
Mumbai • ap-south-1
Tokyo • ap-northeast-1
Seoul • ap-northeast-2

15.) Get out of the SES sandbox. Go to the Request Production Access page in the SES console. Make sure you're in the SES console for your primary region (check the region selector on the right side of the header at the top of the page). Under Mail type select Transactional. Until your request is approved, your EnvKey instance will be limited to sending 200 emails per day and 1 per second, and will only be able to send to verified email addresses. EnvKey can work around these limitations, but it works better with the sandbox restrictions removed.

16.) Verify a domain in SES. An email address verified by SES is required for the from field of emails sent by EnvKey. And until your request to get out of the SES sandbox is approved, EnvKey can also only send to verified addresses. The best way to cover all your bases is to verify the domain your organization uses for email. So if everyone in your org has [email protected] email addresses, you would verify the widgetco.com domain, allowing you to use any address on that domain for the from field, and to also send to any address on that domain. Go to the Create identity page in the SES console for your primary region, select Domain under Identity type, and then follow the instructions to verify the domain. If needed, you can also verify individual email addresses.

17.) Unless you'd prefer to host EnvKey on a subdomain of a domain you already own (and add DNS records yourself), purchase a domain for your EnvKey host in Route 53.

📘

Behind-Your-Firewall Mode

If you're going to use Behind-Your-Firewall Mode to connect to EnvKey from a VPC in another AWS account through AWS PrivateLink, you'll need to purchase a new domain in Route53 (adding DNS records yourself to a domain you already own won't work).

18.) Create or import wildcard certificates in AWS Certificate Manager for *.your-domain.com in both your primary and failover regions. Verify them if needed with DNS. It may take some time for these certificates to verify. You won't be able to sign in to your organization until they're verified.

19.) Back on your local machine, in the EnvKey UI, click My AWS Account Is Ready.

20.) Fill in the form, then click Install at the bottom of the screen. The EnvKey installer will run in your AWS account. It takes 30-45 minutes to spin up resources.

21.) When installation completes, you'll get an email containing an Init Token. If you're running EnvKey in Behind-Your-Firewall mode, it will also include a VPC Endpoint Service Name. If you're configuring DNS yourself instead of using a Route53 domain, it will also include DNS records.

22.) If you're configuring DNS yourself instead of using a Route53 domain, add the DNS records you received by email to your domain.

23.) If you're running EnvKey in Behind-Your-Firewall mode, you can connect to EnvKey from a VPC in another AWS account (the consumer account) by creating a VPC Endpoint with the Service Name that you received by email alongside your Init Token.

• That will generate a connection request in your EnvKey account. To approve the request, go to the endpoint services page in the VPC section of the console. There should be one service in the list. Select it, then from the Endpoint connections tab, select the endpoint corresponding to the consumer account. Then click Actions > Accept endpoint connection request. Once you've approved that request, EnvKey will be connected to the VPC in the consumer account.

• To access EnvKey from your local machine and sign in, you'll need a VPN connection into the consumer account.

24.) Make a GET request to https://your-subdomain.your-domain.com/, and make sure you get an API OK response. If you don't, you might need to wait for DNS to resolve or certificates to validate.

25.) In the EnvKey UI, paste in the Init Token you received by email to sign in to your new organization.

26.) Now that your installation has succeeded, go to the IAM users page in the console and delete the envkey-deploy IAM user. Then remove the envkey-host credentials from your $HOME/.aws/credentials file.

27.) To invite users, authorize devices, or generate server ENVKEYs, you'll need to purchase a license. In the EnvKey UI, click the Org Settings Menu at the top of the sidebar on the left, and then click Billing. On the Billing screen, find your Billing Id. Send an email to [email protected] that includes your Billing Id. We'll get back to you shortly with an invoice and a service contract. When the contract has been signed and the invoice paid, we'll send you a license. In the EnvKey UI, on the Billing screen, paste your license it into the Set New Licensefield and clickUpdate License.

28.) After your request to get out of the SES sandbox is approved (see step 15), you should increase your EnvKey hosts EmailsPerSecond configuration value to whatever your new limit is. To find your limit, go to the SES dashboard for your primary region, find the Sending limits box, and the Maximum send rate in that box. To update the configuration value, see updating configuration below.

29.) You may also want to setup DKIM for SES to improve the deliverability of emails from EnvKey and remove warnings in some email clients.

If you have any trouble getting your SES production access request approved, please reach out to AWS support. If that doesn't get you anywhere, email [email protected] and we'll help you get it approved.

Administration

EnvKey Business Self-Hosted is almost 100% hands-off. It runs, scales, and upgrades on its own with zero downtime. That said, here's an overview of EnvKey host administration, just in case you need to look under the hood.

Updating configuration

Go to the CloudFormation stacks page for your primary region and find the envkey-fargate-api stack (the stack name will also have a random string at the end). Click on the stack name to see the stack's details, click the Update button, then select Use current template and click Next. The following configuration values can be updated from here:

• VerifiedSenderEmail (from address in emails sent by your EnvKey host)
• ApiTaskCpu (see valid options and combinations here)
• ApiTaskMemory* (see valid options and combinations here)
• ApiMaxTasks (must be 2 or greater; setting to 2 will disable auto-scaling)
• ApiTargetCpuUtilization (controls auto-scaling)
• ApiTargetRamUtilization(controls auto-scaling)
• FailoverLogsInterval (interval that EnvKey host checks for logs from the failover bucket)
• EmailTokenExpirationMs (time before email authentication tokens expire)
• ExternalAuthSessionExpirationMs(time given to users to complete sign in in the IDP portal with SSO)
• EmailsPerSecond (max sending rate before emails are queued--should be set according SES limit after getting out of SES sandbox)

You shouldn't update any other parameters, that aren't listed above, as doing so could break your host. It will take a few minutes for updates to be applied, and they won't cause any downtime.

Logs

All EnvKey logs can be found in CloudWatch under Log groups.

Alerts

If your EnvKey host receives an unexpected error, it will be sent to the 'Infra Alerts Email' you supplied to the EnvKey UI when setting up your host.

You'll also receive an alert for the following events:

• When your EnvKey Fargate cluster exceeds its ApiTargetCpuUtilization parameter, meaning it will probably scale up.
• When your EnvKey Fargate cluster exceeds its ApiTargetRamUtilization parameter, meaning it will probably scale up.
• When either of your database instances has less than 200mb of freeable memory.
• When either of your database instances exceeds 80% CPU utilization.
• When replication lag between the master and secondary database instances exceeds 200ms.

Vertically scaling the database without downtime

1.) By default, EnvKey provisions two medium t4g database instances into an AWS Aurora cluster, which offer a good balance between performance and affordability. It's unlikely that most organizations will need to scale beyond these instances, but if you do need to scale, it's easy to do.

2.) Go to the RDS Console in your primary region, click the Databases tab. You'll see your EnvKey cluster listed to the right.

3.) It's a good idea to take a snapshot of your database before scaling (though it shouldn't be needed). Click on your cluster to go to its settings, then click on the Maintenance & backups tab. Scroll down to the Snapshots section and click Take snapshot. Give it a name like envkey-pre-scaling-snapshot-2022-01-24 and click Take Snapshot.

4.) Next, go back to your cluster at the top of the page, select your reader instance and click Modify. Then select the instance class you want to scale to and confirm. When it's finished scaling up, now do the same for your writer instance. When your writer instance becomes unavailable, the reader instance will automatically be promoted to take its place.

5.) You can also scale down if needed in the same way.

Restoring from a database backup

1.) Go to the RDS Console in your primary region, click the Databases tab. You'll see your EnvKey cluster listed to the right. Click on it to go to your cluster's settings, then click on the Maintenance & backups tab.

2.) Scroll down to Snapshots and find the snapshot corresponding to the date you want to restore your database to. EnvKey stores database backups for the previous 35 days, the maximum allowed by RDS. Click on the snapshot you want to restore to go to the snapshot's details page, then click the Actions dropdown and Restore Snapshot.

3.) Fill in the DB cluster identifier field with something like envkey-restored-2022-01-22.

4.) Virtual private cloud (VPC) and Subnet groupshould use the same VPC and Subnet group as your current cluster (these should be selected as the default option).

5.) Under VPC security group, remove the default Security Group, then click the dropdown below Existing VPC Security Groups and select the security group that includes DbSecurityGroup in its name. It will look something like envkey-vpc-a923m9-DbSecurityGroup-5X4NP83M2NXC8.

6.) For Availability Zone, leave it as No preference.

7.) For DB instance class, select the Burstable classes radio button, then select db.t4g.medium (unless you're using a different instance class for your cluster, in which case you should select that one--see Vertical Scaling above.

8.) Expand the Additional Configuration section, and change both DB cluster parameter group and DB parameter group from the default selection to the parameter groups created for your original cluster.

9.) Leave other settings as they are and click Restore DB cluster.

10.) You'll be taken back to the cluster list and should see your new cluster. Select it, click the Actions dropdown, and click Add Reader. Fill in DB instance identifier with something like envkey-restored-reader, leave other settings as they are, and click Add reader.

11.) When both the writer and reader instances have finished provisioning, go back to the restored cluster's settings and click the Connectivity & security tab. Under endpoints, copy the Endpoint name of the Writer Instance.

12.) Almost done. Now go to the CloudFormation stacks page for your primary region and find the envkey-fargate-api stack (the stack name will also have a random string at the end). Click on the stack name to see the stack's details, click the Update button, and then select Use current template and click Next. Find the DbHost field and copy in the Endpoint name value you copied from your restored cluster's writer instance. Click Next and finish applying the update .

13.) Now you just need to sync EnvKey's failover S3 buckets with the restored state of the database. Run:

envkey host resync-failover

Changing DOS and overage protection limits

EnvKey Business Self-Hosted includes a WAF layer that throttles unusually large bursts (whether malicious or accidental) that could overload the host or run up large bills.

You can update the configuration values that control this throttling by going to the CloudFormation stacks page for your primary region and find the envkey-waf-api and envkey-waf-failover stack. Click on the stack name to see the stack's details, click the Update button, then select Use current template and click Next.

You can also do the same for the envkey-waf-failoverstack in your failover region.

In the envkey-waf-api stack in the primary region, the following parameters can be updated:

• MaxRequestSize (max request size for any request)
• FetchMaxRequestSize (max request size for fetch requests)
• SocketMaxRequestSize (max request size for websocket requests)
• ApiMaxRequestRate (max request rate for any request)
• SmallFetchMaxRequestRate (max request rate for small fetch requests)
• LargeFetchMaxRequestRate (max request rate for large fetch requests)
• SocketMaxConnectionRate (max websocket connection attempt rate)

In the envkey-waf-failover stacks in each region, the following parameters can be updated:

• MaxRequestSize (max request size for any failover request)
• MaxRequestRate (max request rate for any failover request)

While it's not recommended, you can also delete the envkey-waf-api stack from the primary region and the envkey-waf-failover stacks entirely from each region if you'd rather have no throttling in place at all.

Bringing down a host

Create a new envkey-destroy IAM user for automated access. Copy the access key id and secret access key into the file you created in the previous step, with the following format:

[envkey-host]
aws_access_key_id = [your-access-key-id]
aws_secret_access_key = [your-secret-access-key]

Then run:

envkey host destroy

After the destroy command has finished, delete the envkey-destroy IAM user from your AWS account and remove the envkey-host credentials form your $HOME/.aws/credentials file.

📘

Backups

EnvKey database backup snapshots won't be deleted by the host destroy command.

Also note that if you previously restored a backup into a new database cluster, the host destroy command won't be able to delete the restored cluster, so you'll have to delete it manually in the RDS Console.