IPsec VPN between Mikrotik RouterOS and an Amazon VPC

This post describes how I configure IPsec tunnels between Mikrotik routers and VPCs (virtual private clouds) hosted at Amazon AWS.

The values below are real when I write this, but not when you read it. I'm using a scratch VPC (with address 10.12.0.0/16) and will change my router's IP address to 192.0.2.4 before publishing.

Amazon's cookbook (the generic version you can download using the button labeled Download configuration) has almost the right order for Mikrotik, but the syntax needs changing.

IPsec proposal

Amazon's cookbook says (reformatted for better readability):

Later in the cookbook, Amazon specifies a second tunnel, identical except for a few points:

Each tunnel may be down for a second or two while keys are being renegotiated. Since there are two tunnels, one will always be available for transmitting payload traffic.

In Mikrotik terms, we set up a proposal (the crypto Amazon wants), one peer for each tunnel (with the preshared keys and IP addresses), and four policies (two tunnels times two policies: All traffic to/from 10.12.0.0/16 and to/from the tunnel endpoints must be encrypted). Here are the Mikrotik commands:

[admin@example] > ip ipsec proposal add auth-algorithms=sha1 comment="what Amazon wants" enc-algorithms=aes-128 lifetime=1h name=aws pfs-group=modp1024

[admin@example] > ip ipsec peer add address=72.21.209.193/32 auth-method=pre-shared-key dh-group=modp1024 dpd-interval=10s dpd-maximum-failures=3 enc-algorithm=aes-128 exchange-mode=main hash-algorithm=sha1 lifetime=8h proposal-check=obey secret=lQYYOz0bTt5CzgSTjwcGffL8GrgchPCx send-initial-contact=yes

[admin@example] > ip ipsec peer add address=72.21.209.225/32 auth-method=pre-shared-key dh-group=modp1024 dpd-interval=10s dpd-maximum-failures=3 enc-algorithm=aes-128 exchange-mode=main hash-algorithm=sha1 lifetime=8h proposal-check=obey secret=OJqz54fyOdO8RCNbsZHp9zafISAB38vu send-initial-contact=yes

[admin@example] > ip ipsec policy add action=encrypt dst-address=10.12.0.0/16 dst-port=any ipsec-protocols=esp level=unique proposal=aws protocol=all sa-dst-address=72.21.209.193 sa-src-address=192.0.2.4 src-address=0.0.0.0/0 src-port=any tunnel=yes

[admin@example] > ip ipsec policy add action=encrypt dst-address=10.12.0.0/16 dst-port=any ipsec-protocols=esp level=unique proposal=aws protocol=all sa-dst-address=72.21.209.225 sa-src-address=192.0.2.4 src-address=0.0.0.0/0 src-port=any tunnel=yes

[admin@example] > ip ipsec policy add action=encrypt dst-address=169.254.255.1/32 dst-port=any ipsec-protocols=esp level=unique proposal=aws protocol=all sa-dst-address=72.21.209.193 sa-src-address=192.0.2.4 src-address=0.0.0.0/0 src-port=any tunnel=yes

[admin@example] > ip ipsec policy add action=encrypt dst-address=169.254.255.5/32 dst-port=any ipsec-protocols=esp level=unique proposal=aws protocol=all sa-dst-address=72.21.209.225 sa-src-address=192.0.2.4 src-address=0.0.0.0/0 src-port=any tunnel=yes

Note that sha1 is specified twice in the Mikrotik configuration. I don't know why.

Amazon recommends overriding don't fragment and adjusting MSS. I don't do any of that. I like to imagine that my endpoints behave sanely, and I avoid applying bandaids during routing.

The tunnels will not work yet, for lack of IP addresses. The Amazon cookbook talks about a tunnel, but Mikrotik doesn't use a separate tunnel, it needs the IP addresses to be set on the right interface. In my case, the packets should go out on 192.0.2.4, so I add both addresses to the right interface:

[admin@example] > ip address print detail
…
1 address=192.0.2.4/24 network=192.0.2.0 interface=ether8 actual-interface=ether8
…

[admin@example] > ip address add address=169.254.255.2/30 disabled=no interface=ether8 network=169.254.255.0

[admin@example] > ip address add address=169.254.255.6/30 disabled=no interface=ether8 network=169.254.255.4

At this point, you've done everything right, but only one tunnel will work due to a Mikrotik bug (well, some people would say it's Amazon going beyond the spec, but I think Amazon's behaviour is sensible).

IPsec testing

At this point IPsec should start working. To test:

[admin@example] > ip ipsec remote-peers print
0 local-address=192.0.2.4 remote-address=72.21.209.193 state=established side=responder established=49s
0 local-address=192.0.2.4 remote-address=72.21.209.225 state=established side=responder established=29s

[admin@example] > ip ipsec installed-sa print
Flags: A - AH, E - ESP, P - pfs
0 E spi=0x5167DCB src-address=72.21.209.193 dst-address=192.0.2.4 auth-algorithm=sha1 enc-algorithm=aes replay=4 state=mature auth-key="d7e244a918f032ce6ab3547bc3e92b2e458076f8" enc-key="fcc804416b2a96a0ace6f3ebde81cbf2" addtime=jul/29/2011 13:17:25 add-lifetime=48m/1h usetime=jul/29/2011 13:17:31 use-lifetime=0s/0s current-bytes=43448 lifebytes=0/0
1 E spi=0xA01040D src-address=72.21.209.225 dst-address=192.0.2.4 …

The second command should print four (relevant) blocks. Too much for me to quote.

You can also ping both 169.254.255.1 and 169.254.255.5. In my case the RTT is 110-120ms. If yours is 1ms, you may not be pinging the right host. Also, the first couple of packets may be rejected because the crypto work isn't done yet. Patience.

Another test is system telnet 169.254.255.1 179, which has to succeed. (179 is Amazon's BGP server. If you can't connect to that you'll have a bit of a problem routing packets properly.)

If any of this breaks, wait 30 seconds, then disable all of your packet filters and try again. Really. I thought I was clever and didn't disable enough, and… well, I'm not clever.

Routing with BGP

Amazon provides two BGP neighbours:

I use a separate BGP instance for Amazon. When I tried to use the same BGP instance as for other work, I had problems with inappropriate next-hop advertisements, Amazon's use of duplicated addresses, etc.

[admin@example] > routing bgp instance add as=65000 client-to-client-reflection=no disabled=no ignore-as-path-len=no name=aws-us-east-test out-filter="" redistribute-connected=no router-id=10.255.12.1

[admin@example] > routing bgp peer add address-families=ip default-originate=if-installed hold-time=30 instance=aws-us-east-test multihop=no name=aws-us-east-test-1 remote-address=169.254.255.1 remote-as=7224 remove-private-as=no

[admin@example] > routing bgp peer add address-families=ip default-originate=if-installed hold-time=30 instance=aws-us-east-test multihop=no name=aws-us-east-test-5 remote-address=169.254.255.5 remote-as=7224 remove-private-as=no

This instance tells Amazon nothing about your network in general and doesn't tell any other routers about 169.254.255/24. Unfortunately it also doesn't tell other routers about the route to Amazon. In my case, I prefer to solve that by advertising the VPC network as appropriate using routing bgp network.

I run with one BGP instance disabled, in order to work around the bug mentioned above:

[admin@example] > routing bgp peer disable aws-us-east-test-5