Creating a VPC-VPC VPN with OpenS/WAN
AWS offers a managed VPN service. This service is a highly available, scalable and AWS-managed service that lets you create an IPSec-based VPN with just a few clicks. So, why would you build your own IPSec-based VPN instead? There are two possible reasons:
- The AWS VPN service, at this time, is not able to create a VPN against another AWS VPN endpoint. This means that while you can use it to setup a VPN from an AWS VPC to your on-premises datacenter, you cannot use it to setup a VPN between two AWS VPCs. This is not a problem if those VPCs happen to live in the same region: We've got VPC Peering for that. But if you need to connect VPCs from different regions together, the AWS VPN service is not going to help you. (Update: As of Nov 29, 2017, it is finally possible to peer VPCs from different regions together.)
- The VPN endpoint you have on-premises may not be supported by AWS. Of course, the list of AWS-supported VPN endpoints is growing by the day, but what if your endpoint is not supported yet? And note that software solutions such as OpenS/WAN are also not supported at the moment, so if you already built a VPN infrastructure based on OpenS/WAN, the AWS VPN service is not going to help you.
This document covers setting up OpenS/WAN on a Linux Bastion host, in an AWS context. So we're not just going to setup OpenS/WAN, but we're also going to setup the AWS VPC to support routing properly. But before we start, one word of warning: IPSec based VPNs, regardless of whether you are using the AWS VPN service or OpenS/WAN as discussed here, are intended (and great!) for permanent VPNs that need to connect sites/VPCs together. They are NOT really intended or suitable for mobile workers who need to connect an individual system to an existing network. For that last problem, an OpenVPN based solution is a lot better. Watch this space for a future article on OpenVPN.
Scenario
The scenario as used in this document is as follows:
We have two VPCs. One VPC is located in Dublin, Ireland (the "eu-west-1" region) and the other in Frankfurt, Germany (the "eu-central-1" region). Dublin uses 10.2.0.0/16 addresses, while Frankfurt uses 10.3.0.0/16 addresses.
Within each VPC I have defined multiple "public" subnets where our Bastion Host will live. Public subnets in Dublin are 10.2.0.0/24, 10.2.1.0/24 and 10.2.2.0/24, while the public subnets in Frankfurt are 10.3.0.0/24 and 10.3.1.0/24. Note that in this scenario I will only use one of these public subnets as I'll be creating only one Bastion Host.
Each VPC also supports multiple "private" subnets and it's these subnets that need to be connected together. The private subnets in Dublin are 10.2.3.0/24, 10.2.4.0/24 and 10.2.5.0/24. The private subnets in Frankfurt are 10.3.2.0/24 and 10.3.3.0/24. All private subnets already have their own "private" routing table. (This "private" route table does not contain a 0.0.0.0/0 route to the Internet Gateway.)
All instructions given are for the Dublin side of the connection. Dublin will be designed the "left" side of the connection within the context of OpenS/WAN. Unless otherwise noted, the same commands, but with mirror IP addresses and such, will also have to be executed on the Frankfurt side of the connection. Frankfurt will be designed the "right" side of the connection.
Step 1: Setup your VPCs and hosts
This article assumes you already have configured your VPC. At the very least, you need a "public" subnet, which means that any EC2 instances deployed into this subnet will receive a public IP address. The scenario assumes you also have created a "private" subnet, where all your internal hosts will live.
Make sure that you've setup your VPCs so that they do NOT have an overlapping IP address space. If you need to connect two VPCs together that have overlapping IP ranges, well, you're in for a world of hurt and the result is not going to be pretty.
Once your VPCs have been setup, deploy a basic Linux instance into a public subnet on each end. This will become your Bastion Host. Make sure you have SSH access to this Bastion Host, properly protected by a Security Group that limits SSH access to the public IP address you're currently using. See my article on setting up Bastion Hosts for tips on how to do this.
Also deploy a basic Linux instance into a private subnet on each end. This will be used for testing the VPN connection. Again, make sure you have SSH access to this host by using the Bastion Host as a jump host.
Step 2: Perform basic configuration of your bastion hosts
If you are just testing IPSec and do not intend to stop/start your Bastion Hosts during the test, you can get by with the public IP addresses that AWS auto-assigns to your Bastion Host. However, for a permanent solution I strongly suggest you get Elastic IP addresses and associate these with your Bastion Hosts. In this example, the Elastic IP address of my Dublin Bastion host is 52.212.65.198 and the Elastic IP address of my Frankfurt Bastion host is 52.57.23.15. Also make note of the internal IP address that your Bastion Host received. These will be needed in the configuration file, so that OpenS/WAN can identify which end of the connection it's going to be. In this example, the internal IP address of my Dublin Bastion host is 10.2.0.160, while Frankfurt received 10.3.1.79.
All the instructions below will need root permissions. Since you'll be logging in as ec2-user first, you need to become root by executing the following command:
$ sudo su -
You may also want to perform a yum update before you start:
# yum -y updateFor debugging purposes, it helps a lot if you have tcpdump installed:
# yum -y install tcpdump
If you are not using Amazon Linux but another distribution, then you may want to deactivate iptables and SELinux at this stage. Note that Amazon Linux has both iptables and SELinux disabled by default.
In my experience it is perfectly possible to run IPSec in combination with iptables and SELinux, but they may get in the way while we're still testing things. Obviously once everything works you're going to activate these features again.
# service iptables stop # setenforce 0
Step 3: Setup security groups
In each VPC you will need three SGs:
- An SG that is associated with the Bastion Host, and allows incoming IPSec traffic from the other Bastion Host (VPN endpoint).
- An SG that is associated with the Bastion Host, and allows incoming traffic from the private subnets in the local VPC.
- An SG that is associated with the Internal Hosts, and allows incoming traffic from all private subnets in all VPCs.
- Allow traffic from UDP port 500
- Allow traffic from UDP port 4500
- Allow traffic from protocol 50 (AH)
- Allow traffic from protocol 51 (ESP)
- Allow all ICMP (This is not required but useful for testing. They can be removed once everything works.)
When configured correctly, your security group "IPSec from Dublin" inbound rules will look like this (the example is taken from Frankfurt):
Associate this SG with the respective Bastion Host.
Second, your Bastion Hosts need to be able to receive all traffic from their local "private" subnets - the subnets we're trying to connect. We're not going to limit this traffic to specific protocols right now, although you may want to do so in the future. So within each VPC create an SG "ALL from local private subnets" and allow all traffic from these subnets. Associate this SG with the Bastion Host.
An example SG "ALL from local private subnets" (again from Frankfurt) will look like this:
Third, your Internal Hosts need to be able to receive all traffic from other hosts on any private subnet - both from this end and from the other end of the VPN tunnel. Again, we're not going to limit traffic to specific protocols right now, but you may want to do so in the future.
An example "ALL from all private subnets" SG will look like this:
Associate this SG with the Internal Hosts.
Step 4: Configure the VPC Routing Tables to use the VPN tunnel
The VPC needs to know that the other VPC can be accessed by routing traffic through your Bastion Host. So you need to modify the routing table of the VPC (or the routing tables of your individual subnets, if you have multiple route tables for your subnets). The route target for your route will be the ENI (Elastic Network Interface) that's associated with your Bastion Host. So go to the AWS console, and make note of the EC2 instance id of your Bastion Host. It will look like this: i-12345678. You then go to Network Interfaces and look for the ENI that's associated with that EC2 ID. It will look like this: eni-12345678. Now go to Route Tables, select the proper route table and add a route to the IP address range of your counterpart VPC, via the ENI. It will look like this:
Step 5: Disable Source/Destination Checks
You also need to deactivate Source/Destination checks on the ENI of the Bastion Hosts. Source/Destionation checks are by default enabled on every network interface, and mean that the interface can only receive traffic that's intended for its IP address, and can only send traffic originating from its IP address. As your Bastion Host needs to forward traffic that neither originates from it, nor is intended for it, those checks need to be disabled. In EC2 Instances, select the Bastion instance. Click on Actions; Change Source/Dest. Check and make sure this is set to disabled.
Step 6: Allow IP forwarding on the Bastion Hosts
Edit the file /etc/sysctl.conf, so that the ip_forward line looks like this:
net.ipv4.ip_forward = 1To activate this setting:
# sysctl -p /etc/sysctl.conf
Step 7: Install and activate OpenS/WAN
# yum -y install openswanReview the /etc/ipsec.conf file. It should look like this:
# /etc/ipsec.conf - Openswan IPsec configuration file
#
# Manual: ipsec.conf.5
#
# Please place your own config files in /etc/ipsec.d/ ending in .conf
version 2.0 # conforms to second version of ipsec.conf specification
# basic configuration
config setup
# Debug-logging controls: "none" for (almost) none, "all" for lots.
# klipsdebug=none
# plutodebug="control parsing"
# For Red Hat Enterprise Linux and Fedora, leave protostack=netkey
protostack=netkey
nat_traversal=yes
virtual_private=
oe=off
# Enable this if you see "failed to find any available worker"
# nhelpers=0
# You may put your configuration (.conf) file in the "/etc/ipsec.d/" and uncomment this.
include /etc/ipsec.d/*.conf
Note the last line: By default this line is commented out. In order for the OpenS/WAN daemon to include the files in /etc/ipsec.d, that line needs to be uncommented - so you need to remove the '#' from the beginning of the line.
We can now start OpenS/WAN, and make sure it is activated on the next reboot.
# service ipsec start # chkconfig ipsec on
Step 8: Create and activate the Dublin-Frankfurt tunnel
Create an /etc/ipsec.d/dublin-frankfurt.conf file. This file will contain all the necessary information for the connection. The original idea behind OpenS/WAN was that you would only need to create one file which could then be used on both ends of the connection. However, as both VPN instances are behind a NAT gateway, they will need their own internal, private IP on the "left"/"right" line instead of the public IP that the other side of the connection requires. So practically speaking you're going to have to create two almost-but-not-quite-identical files.
The /etc/ipsec.d/dublin-frankfurt.conf file in Dublin will look like this:
conn dublin-frankfurt
type=tunnel
authby=secret
auto=start
pfs=yes
leftid=52.212.65.198 # Public IP address
left=10.2.0.160 # Private IP address
leftsubnets={10.2.3.0/24, 10.2.4.0/24, 10.2.5.0/24}
rightid=52.57.23.15 # Public IP address
right=52.57.23.15 # Public IP address
rightsubnets={10.3.2.0/24, 10.3.3.0/24}
The /etc/ipsec.d/dublin/frankfurt.conf file in Frankfurt will look like this:
conn dublin-frankfurt
type=tunnel
authby=secret
auto=start
pfs=yes
leftid=52.212.65.198 # Public IP address
left=52.212.65.198 # Public IP address
leftsubnets={10.2.3.0/24, 10.2.4.0/24, 10.2.5.0/24}
rightid=52.57.23.15 # Public IP address
right=10.3.1.79 # Private IP address
rightsubnets={10.3.2.0/24, 10.3.3.0/24}
For testing purposes we're going to define a "shared secret" now. This shared secret will be known to both ends of the connection, and will be used to authenticate the other server. This works and is considered secure, but leads to an exponential growth in the number of shared secrets if the number of VPN endpoints grows. At the end of this document I'll present a different method of doing authentication: By using RSA keys.
In order to setup a shared secret, first, verify that your /etc/ipsec.secrets file looks like this:
include /etc/ipsec.d/*.secrets
Create a file /etc/ipsec.d/dublin-frankfurt.secrets with the following content, on both Bastion Hosts:
52.212.65.198 52.57.23.15: PSK "My shared secret"
Obviously "My shared secret" should be replaced with your own shared secret. Using a randomly generated text string of 16 characters or more, at least. And use different shared secrets for each connection.
If everything is setup correctly, you should now be able to bring up the tunnel. This is a two-step process. You first add the tunnel to the OpenS/WAN configuration, and then activate it.
# ipsec auto --add dublin-frankfurt # ipsec auto --up dublin-frankfurtMonitor the /var/log/messages file while you execute these commands. You should be seeing about half a page of output when the tunnel is setup correctly. Once the tunnel is up, you can see this with the ipsec command as well:
# ipsec lookYou can now test your tunnel by logging into the Internal Host, and pinging the other Internal Host at the other end of the connection:
$ ping 10.3.2.135
While you are testing your tunnel, it is very illustrative to run a tcpdump, both on your eth0 device and on the tun0 device of the Bastion Host. Do filter out SSH traffic though, otherwise you will also see the SSH traffic, which will in turn generate more SSH traffic.
# tcpdump -i eth0 -ln not port ssh # tcpdump -i tun0 -ln not port ssh
Step 9: Setting up RSA authentication
As said earlier, we've so far based our VPN security on shared secrets. This is fine if you only have a handful of VPN connections. But if you have dozens of endpoints, you may need hundreds of shared secrets, and this will quickly become unmanageable. A far better solution in that case would be to use RSA public/private keypairs at both ends which are then used for authentication and encryption key negotiation.
At both ends of the connection, generate an RSA key pair:
# ipsec newhostkey --output /etc/ipsec.secrets --bits 1024 --verbose --configdir /etc/ipsec.d
Note: If you get an error "ipsec rsasigkey: keypair geenration failed", then most likely you have a corrupt file somewhere in /etc/ipsec.d. Here's the sequence of commands that seems to solve this:
# ipsec newhostkey --output /etc/ipsec.secrets --bits 1024 --verbose --configdir /etc/pki/nssdb # cp -f /etc/pki/nssdb/key3.db /etc/ipsec.d # service ipsec restart
Check your /etc/ipsec.secrets file. It should look like this:
: RSA {
# RSA 1024 bits ip-10-2-0-160 Tue Oct 18 11:45:49 2016
# for signatures only, UNSAFE FOR ENCRYPTION
#pubkey=0sAQO4TJQhd3T+...
Modulus: 0xb84c94217774fe...
PublicExponent: 0x03
# everything after this point is CKA_ID in hex format when using NSS
PrivateExponent: 0x823...
Prime1: 0x823...
Prime2: 0x823...
Exponent1: 0x823...
Exponent2: 0x823...
Coefficient: 0x823...
CKAIDNSS: 0x823...
}
# do not change the indenting of that "}"
Note that I have trimmed the actual keys for security reasons. Also note that the "include" statement is gone. This means that the /etc/ipsec.d/*.secrets files will no longer be used.
Once you have setup both ends of the connection with an RSA key, you can modify the /etc/ipsec.d/dublin-frankfurt.conf files. We need to change the "authby=secret" line to "authby=rsasig", and we need to add leftrsasigkey/rightrsasigkey lines with the public key of each endpoint to the file. The public keys can be cut&pasted from the /etc/ipsec.secrets files, on the pubkey line.
The /etc/ipsec.d/dublin-frankfurt site will then look like this:
conn dublin-frankfurt
type=tunnel
authby=rsasig
auto=start
pfs=yes
leftid=52.212.65.198 # Public IP address
left=10.2.0.160 # Private IP address
leftsubnets={10.2.3.0/24, 10.2.4.0/24, 10.2.5.0/24}
leftrsasigkey=0sAQO4TJQhd3T+...
rightid=52.57.23.15 # Public IP address
right=52.57.23.15 # Public IP address
rightsubnets={10.3.2.0/24, 10.3.3.0/24}
rightrsasigkey=0sAQPJCIcomXu...
Now reload and restart your tunnel:
# ipsec auto --down dublin-frankfurt # ipsec auto --delete dublin-frankfurt # ipsec auto --add dublin-frankfurt # ipsec auto --up dublin-frankfurt
Step 10: Finishing off
If required, enable iptables and SELinux again. If necessary, apply any settings so that these security features still allow IPSec to work.
Do a full stop/start cycle of both Bastion Hosts, and ensure that the tunnel comes up automatically. (Note that if you did not use Elastic IP addresses, the public IP addresses of your Bastion Hosts will have changed after a stop/start cycle. You may want to reboot them instead.)
Create a backup, snapshot and/or AMI from your Bastion Host so that you can redeploy your Bastion Host quickly if something breaks.
Tune the Security Groups that are associated with your internal (non-bastion) hosts so that only specific traffic is authorized. Not only will this increase the security, but it will also limit the volume and thereby the cost of the data flowing across your VPN connection. At this stage you may also want to remove the ICMP inbound rule from your "IPSec from Frankfurt" and "IPSec from Dublin" SGs.
If you have more than two VPCs, you may need to setup VPNs between each VPC. Note that, just like VPC peering, VPN connections are not transitive: You cannot connect VPC A with VPC C by going via an intermediary VPC B. If you need to connect more than a handful of VPCs together, you may want to take a look at the OpenS/WAN documentation that explains how you can store the RSA keys in DNS. This greatly simplifies VPN setup.