There are virtually no constraints regarding CIDR setups in AWS. However, a careless choice can lead to major connectivity and extensibility issues.
In AWS, the creator of a VPC must specify a CIDR block, from which the IP addresses will be assigned to the inhabiting resources. A subnet is a subrange of the CIDR block of a VPC. Resources can only be launched in subnets - meaning they are an essential part of the design.
Such an architecture implies that multiple choices related to CIDR blocks have to be made during the creation and usage of a VPC in AWS. Firstly, we need to pick a CIDR block for the VPC itself, and then we need to choose a subblock for every subnet. In the long run, there will definitely be more than one subnet in a mature network architecture - subnets can have different configurations for different purposes, and a well-thought usage of subnets can facilitate traffic monitoring.
There’s very few constraints imposed by AWS regarding the CIDR choices, as they wish to support the widest possible range of customers. Some customers might be owning legacy networks, whose integration in the new setup (by a full migration or perhaps a site-to-site connection) might require workarounds, which temporarily go against what is considered best practices.
As a result, best practices are not explicitly enforced. In a dynamic startup environment, this makes space for some poor choices, which will impose operational costs in the future. The most notable examples consist of:
- Using a CIDR block from the public IP address pool
It’s not a very common issue, but we still spot it in the wild from time to time.
Public IP addresses can easily be overlooked. For instance:
- 172.18.3.4 is a private IP address;
- 172.31.3.4 is a private IP address;
- 172.32.3.4 is a public IP address.
Such a choice can have very confusing consequences. If one of your services ever needs to connect to public internet, and if the address resolves to an IP address in your VPC’s pool, the connection will fail, without many debugging hints.
The solution is very simple - RFC 1918 is very clear in regards of the private IP addresses. In practice, we recommend a part of the 10.0.0.0/8 block - that is 2^24 = 16777216 addresses. It will be enough, and addresses starting with 10 are easily distinguishable from public addresses in traffic monitoring.
- Using overlapping CIDR blocks for VPCs
Mature cloud setups usually have more than one VPC. It is a good practice to have multiple accounts (to ensure proper environment isolation), and every account will probably have a VPC or several. Usage of multiple VPCs is not limited to this use case though, there are valid reasons for having more than one VPC in one account.
At some point, there may come a need to arrange connectivity between resources in different VPCs, with the prime example being observability - it is very likely that you will want to ingest data from different VPCs to your observability toolset.
AWS are aware of such a use-case, and they came forward with a solution - VPC Peering. VPC Peering enables communication between different VPCs using the physical, private AWS backbone, meaning your traffic never reaches public internet. It has no single point of failure and no bandwidth limit. In addition, it is free, as long as you don’t cross regions or availability zones. However, to use it, the CIDR blocks of VPCs cannot overlap, because it would lead to ambiguous rules in network routing tables. If you overlook this requirement during the creation of your VPCs, you will be unable to leverage the extremely useful VPC Peering mechanism in the future.
In addition, overlapping IP ranges also hinder the explainability of the future firewall rules (or security groups in AWS terminology). A convenient way to create those is by referencing CIDR blocks - but if a CIDR block’s meaning is ambiguous, a rule is no longer readable. Same principle applies to monitoring - the ability to distinguish VPCs by the IP address of a resource is extremely practical.
- Assigning too little IP addresses to a VPC or a subnet
In the private IP address space that you decide to use, you also need to devote some of the network address bits to distinguish the VPC, and the subnet. If you use too little bits to distinguish VPCs, you will inevitably have to start using overlapping ranges, which will hinder your ability to capitalize on the VPC Peering. If you use too little bits to distinguish subnets, you will hit a hard limit at some point, and will no longer be able to add any new subnets to a VPC. In contrary, using too many bits to distinguish VPCs and subnets will leave you with very little IP addresses for resources launched in the subnets.
A reasonable design could be to use 8 bits to distinguish VPCs (this also lets you easily differentiate VPCs by the second digit in the IPv4 address), and then 6 bits for a subnet. This would leave you with 2^10 = 1024 IP addresses per subnet, which is reasonable in many scenarios.
The remediation of the resulting problems usually requires a redesign of the networking setup. In a typical case, a new VPC is created, and resources are gradually migrated there. The operational overhead is quite large. However, these efforts can be avoided by putting some rudimentary effort into choosing the CIDRs wisely from the beginning - which we highly recommend.