Architecture

The purpose of this lab is to demo hybrid AWS DNS and highlight the steps to setup the necessary R53 endpoints and resolver rules. Hybrid DNS is a common feature across all IT infrastructures and hopefully this blog post will serve as a basic template to get you started if you are tasked with implementing domain resolution between cloud and onprem.

As part of this lab we will implement the architecture below:

alt

Deployment and demo

The Terraform code to deploy the solution is available at aws-hybrid-dns. Feel free to deploy the code and follow along the guide in the section Detailed steps to fully understand the steps involved.

The repo has a tfstate folder where you can deploy the S3 and Dynamodb backend to store your Terraform state. You will need to deploy the code in that folder first before deploying the lab infrastructure.

Once the infrastructure is deployed, you need to complete an extra step before testing the DNS resolution from onprem to cloud and vice-versa.

DNS resolution from onprem to cloud

Attempting to resolve the hostname of the cloud instance from the onprem app server will result in the NXDOMAIN error. You will need to change the nameserver used by the onprem app server from the default AWS one (the +2 IP address) to the IP of the DNS server (onprem-dns) and reboot the server:

alt

The new resolved.conf file:

alt

After you made the change and rebooted the instance you should be able to resolve the cloud R53 record:

alt

DNS resolution from cloud to onprem

When resolving onprem records from the cloud, the R53 resolver will forward queries via the outbound endpoint to the onprem DNS server. You can see this working below:

alt

Detailed steps

In any hybrid DNS setup, there are 2 main components:

  1. Inbound DNS resolution - DNS queries coming from onprem servers towards the cloud DNS server. The onprem DNS server forwards queries to the 2 IP addresses of the R53 inbound endpoint. The inbound endpoint forwards the query to the AWS R53 resolver.
  2. Outbound DNS resolution - DNS queries coming from cloud servers to onprem DNS servers. The AWS R53 resolver notices that the onprem domain is included in a forward rule and forwards the query to the R53 outbound endpoint. The outbound endpoint forwards the query to the target IP of the onprem DNS server.

The sequence of deployed resources is listed below:

Cloud infra

  1. For the cloud infra you need a VPC and 2 subnets. We will deploy our web server in one subnet and the inbound/outbound endpoints in both subnets because they require a minimum of 2.
  2. A separate route table and the route table associations for each of the subnets above.
  3. There will be no Internet Gateway attached to the VPC. You will access the instance via SSM and will need to deploy VPC endpoints (ssm, ec2messages, ssmmessages).
  4. A security group with the necessary inbound rules (ssh, dns, https), an instance role which allows SSM access and an ec2 instance.
  5. A R53 Private Hosted Zone (PHZ) and an A record inside this PHZ for the web server. In the lab the PHZ is cloud.example.com and the A record is web.cloud.example.com.
  6. An inbound R53 resolver endpoint.

Onprem infra

  1. The onprem environment is simulated through a separate VPC. For this demo we will deploy the DNS server and the app server in a single AZ. Therefore you will need a single subnet.
  2. Steps 2,3 are identical to the steps 2,3 from the cloud infra setup.
  3. 2 ec2 instances: an app server and a BIND DNS server. The BIND DNS server will have the cloud.example.com zone configured to forward queries to the 2 R53 inbound endpoints setup in the cloud VPC. The configuration file can be found here.

After you deploy the above resources, you will need to create a VPC peering connection between the 2 VPCs and consequent routes in the cloud and onprem route tables. You will also need to create a R53 outbound endpoint with a forward rule containing the onprem DNS server IP as the target IP. The forward rule will be associated with the cloud VPC.

After completing the above steps you should be able to resolve DNS queries in your hybrid environment.

Things to remember

  1. Do not forget to allow UDP/TCP on higher ports(1024-65535) from onprem in the inbound rules for the security group of the outbound resolver endpoints. This is needed because the onprem DNS server will reply on a higher port.
  2. When setting up R53 resolver rules, do not forget to associate the rule with the VPC from which you want to forward the queries. Wasted some time troubleshooting and enabling flow logs on the outbound endpoints ENIs just to find out that the rule was not associated with my VPC.