If you’re not running Amazon Linux with the built in network interface management tools, adding multiple ENIs on the same subnet can be a confusing experience. We use this sometimes to run multiple elastic IPs on separate network interfaces so we can bind to them separately.
We worked through this with Amazon support recently and thought we should share a quick overview of how to do this on Red Hat / CentOS 7.
1. Force your default gateway to be eth0
Edit /etc/sysconfig/network and add:
Not doing this left the default gateway of the main routing table set to the last interface to be configured, which caused some strange behavior.
2. Configure each additional interface you’ve added
In /etc/sysconfig/network-scripts, create an ifcfg-ethX for each new interface.
1. The DEVICE name to match the ENI.
3. Add a custom route for each additional interface
Again in /etc/sysconfig/network-scripts, create a route-ethX file for each interface.
1. The device name
2. Increment the table number
3. The gateway to your VPC subnet’s gateway.
4. Change the source IP to the assigned internal network address of the ENI.
default via 10.0.0.1 dev eth1 table 2
10.0.0.0/24 dev eth1 src 10.0.0.10 table 2
4. Add a custom rule for each additional interface
Also in /etc/sysconfig/network-scripts, create a rule-ethX for each interface.
1. Increment the table number to match route-ethX
2. Change the IP to the assigned internal network address of the ENI.
from 10.0.0.10/32 table 2
Restart the network service and you should be up and running. You can confirm with “ip rule”:
# ip rule
0: from all lookup local
32764: from 10.0.0.10 lookup 3
32765: from 10.0.0.11 lookup 2
32766: from all lookup main
32767: from all lookup default
Note that Amazon suggested a custom route and rule for eth0, but we found allowing eth0 to use the default main routing table not only worked but was more flexible.