PwC manages large-scale cloud infrastructure by using IaC for provisioning the required application infrastructure automatically. We have a mature methodology for automating and streamlining infrastructure provisioning and management with IaC. One of the popular cloud-agnostic solutions for IaC is Terraform which PwC uses to build, update and maintain their infrastructure in a flexible and secure manner.
"PwC is an AWS Premier Tier GSI Partner” and the firm leverages its deep partnership with AWS to utilise relevant tools and technologies to enhance IaC.
Terraform maintains the status of the current infrastructure in a state file, which acts as a single place of storage in the environment. Generally, in a large organisation, one or more team members are working on the same IaC projects. When multiple team members are constantly updating locally stored state files, then those files might get into an inconsistent state. Furthermore, they may get corrupted or deleted by accident. Terraform state files can be stored at remote locations to avoid such accidents.
To keep files consistent at the remote location, teams can enable state locking in the state files while working with Terraform scripts. In this regard, PwC is leveraging the process of using an S3 bucket as the remote store for the Terraform state files and DynamoDB for state locking, while deploying the infrastructure using Terraform and AWS. Moreover, the team can keep separate state files for individual environments like testing setup, preproduction or production.
This standard practice has enabled PwC and its clients to facilitate an efficient infrastructure setup and modifying it in a secure manner. Moreover, it has contributed significantly to manage states in a time-efficient manner, thus minimising the occurrence of issues in the future. Therefore, this practice can become a major contributor to an organisation’s success, while managing multiple cloud resources efficiently and securely.
AWS configuration and implementation
In this article, we will highlight the ways in which the PwC infrastructure has been using AWS managed services – like the S3 bucket and DynamoDB for Terraform.
Architecture of Terraform state management in AWS
- PwC set up a dedicated Amazon S3 bucket to store their Terraform state files. This S3 bucket is used as the central repository for its infrastructure’s state information.
- The S3 bucket supports encryption, which reduces the issues of storing sensitive data in Terraform state files.
- Versioning can be enabled on the S3 bucket, allowing PwC to track changes and recover from unexpected issues. Moreover, versioning enables PwC to have a historical record of every state change.
- One can also enable multi-factor authentication (MFA) delete on the S3 bucket so that the files remain safe and can't be deleted by any unauthorised persons.
- PwC used a DynamoDB table for state locking. This table stores the lock status, has a unique lock ID and lock time to live (TTL).
- By using strong locks in DynamoDB, PwC ensures that locks automatically expires after some time, preventing extended locking scenarios.
To implement the above-mentioned solution, organisations would need to ensure that their existing Terraform configuration can be updated to use these resources as the backend for state management. Below is a snippet of a sample backend.tf file:
By using remote configuration for Terraform state files, one can maintain the state file stored in the S3 bucket securely. This is because the S3 bucket is encrypted and enables versioning, and the required resources have AWS identity and access management (IAM) permissions. For every change made to the cloud infrastructure, Terraform will automatically get the latest state file from the S3 bucket, lock it using the DynamoDB table, apply the changes, update the latest state again in the S3 bucket, and then release the lock. State locking happens automatically for all operations that can ‘write’ on the state file.
Best practices
- Separate AWS accounts: Consider using separate AWS accounts for the Terraform state storage resources (S3 and DynamoDB) and store the application infrastructure (like state files) in the shared accounts. This avoids deletion or unauthorised changes to the Terraform state files.
- S3 bucket IAM permissions: S3 bucket permissions: Disallow access to the S3 bucket where the Terraform state is stored using IAM policies. Only allow required users and roles to ‘read’ and ‘write’ to the bucket. DynamoDB table permissions: Restrict access to the DynamoDB table for state locking. Only Terraform itself and authorised users or roles should have access to modify the lock.
- Versioning of state files: Enable versioning on the S3 bucket. This will help in tracking the changes to your Terraform state files over time, making it easier to recover from accidental changes or file corruption.
- Encryption of the S3 bucket: Due to the sensitive nature of the Terraform state files, S3 bucket objects can be encrypted using S3 bucket encryption policies.
- DynamoDB locks: Maintaining strong locks using DynamoDB guarantees that locks are automatically released, therefore preventing extended locking and reducing wait times for other developers.
- Monitoring and alerts: CloudWatch can be used to implement monitoring and alerting on the S3 bucket and DynamoDB table. Amazon Simple Notification Service (SNS) can be used to get instant alerts over email or SMS. This will help alert the concerned parties in case of any unauthorised changes.
- Review and audit permissions: Periodically review and audit the IAM permissions linked with your S3 bucket and DynamoDB table to ensure that only verified users/resources have access to the same.
- Automated backup: Implement automated backup and restore procedures for the S3 bucket and DynamoDB table. This is a part of disaster recovery (DR) solutioning.
- Documentation: Prepare a detailed document outlining the state locking mechanism so that new team members can be updated about the guidelines.
Conclusion
PwC’s commitment to infrastructure stability and scalability is enhanced by the strategic use of Amazon S3 and DynamoDB for Terraform state locking. By implementing this solution, PwC has created a secure solution of reliability, allowing the team to focus on innovation and growth without having to constantly keep track of the safety and consistency of the AWS infrastructure. This solution can also be deployed across multiple cloud providers using their native services to create and maintain a safe and secure infrastructure using Terraform.
