sbs20

DIY Devices - the network

2021-05-21 DIY devices Home router Raspberry Pi

Router

TL;DR - go to the appendix

Why? #

I had been interested in buying a raspberry pi back in 2015 and conincidentally had been having internet connectivity issues. I had found that my internet connection was actually fine, but the DNS servers that my router was serving up (my ISP's) were flaky. I'd read about pi-hole and realised I had my excuse to buy the pi. Except I didn't use pi-hole - I wanted to do it myself. A bit of tinkering with dnsmasq and a shell script to download advertising URLs and I had myself a more reliable connection.

From time to time, though, my ISP would reset my router (rude) and I would have two DHCP servers on the network. If my ISP can do that, they can conceivably access my network. There's nothing really particularly secret - I can't really imagine anyone at the ISP taking the time to sniff around, but that's the not the point - if there's a backdoor then someone else might find their way in.

I wanted to build my own router and take control. I also found the wonderful arstechnica guide to building a linux router which was hugely helpful and gave me the confidence to know I was going to do it. And while I didn't have IOT devices I was thinking about it and increasingly interested in the idea of segregating my network with VLANs; partly because by this time I was working for a rather large networking company, but also because fun.

Hardware #

I read a lot about possible hardware. Whatever I got would have to be open - insofar as I could run plain linux on it. Ideally I wanted to go ARM because it's cheaper to buy and cheaper to run. I found things like Solid Run Clearfog, Microtik and the Banana-Pi but the more I read the more I found reports of poor quality and poor support. I'm all for tinkering, but if this was going to be my home router, I need it to work. I also read about poor performance with ARM and looked more closely at x86 (and equivalent) hardware. I found a lot of love for pcengines but it also came at a price.

In the end, after a lot of thinking I went for a Gigabyte EL-20-3050-8GB. It's not available anymore, but has 2 NICs, a Celeron CPU, 2GB RAM and 8GB of onboard MMC. It was under £200.

I also got my hands on a couple of smart switches (ones that can handle VLANs). I originally bought GS108Ts but a year later upgraded to GS110TP-200EUS in order to get POE.

Finally, I decided to use Ubiquiti kit for wireless access points. I have been really happy with this decision. The APs are extremely good value, the controller software is good and stable, and can run happily on a 512mb Debian VM. The APs run using PoE and can also mesh with each other where cables are hard to run.

Design objectives #

I wanted to have 3 LANs:

Having run this for a couple of years I have ended up with:

The Fabric LAN only has access to itself, although most of the things on it span multiple VLANs. The private LAN can access the whole network. The Home and IOT LAN are effectively the same except for what goes in them - they are shielded from each other just to mitigate anything bad happening.

Other than that, I wanted:

Deviations from arstechnica #

Debian instead of Ubuntu #

The arstechnica article uses Ubuntu Server. And so did I initially. But Ubuntu likes to do things its own way and it's a little bit fast moving (unstable) for server applications in my view, and it started to deviate from the norm in terms of network config. So after a few months I ended up rebuilding with Debian Buster. In order to avoid losing internet access, I modified my build scripts to run on a Raspberry Pi (a RPI4 with additional USB NIC) and tested that. Then swapped the proper router for the Pi, rebuilt the router and swapped back.

An interesting aside: the Raspberry Pi wasn't really fast enough. It's great as a backup and for testing - but I was limited to about 50mbit/s throughput which confirms all the earlier research I'd done.

VLANs #

This was one of the hardest things to figure out. I don't have a networking background so it was sometimes hard to search for things without knowing the correct jargon - but as you'll see from the scripts below, it's ultimately not too difficult once you get the hang of it.

Also #

And? #

I'm really pleased with the result. It's fast and reliable. I am no longer concerned about my ISP getting up to no good. I can control my own DNS and DHCP (although I could do that before with my home made pi-hole) but most of all I have extremely granular control of the firewall and IPS, and VLANs.

Is having VLANs worth it? Probably not for most people - but I do run a couple of NAS devices and have data which I care about (as in, I don't want it lost or corrupted). VLANs offer an additional layer of security. It's not that I don't trust my friends not to hack, it's that I don't trust that their devices have zero malware.

Appendix #

/etc/network/interfaces

auto lo
iface lo inet loopback

allow-hotplug enp4s0
iface enp4s0 inet static
    address 192.168.0.4/24
    gateway 192.168.0.1
    dns-nameservers 127.0.0.1

allow-hotplug enp3s0
iface enp3s0 inet static
    address 10.0.1.1/24

allow-hotplug enp3s0.1
iface enp3s0.1 inet static
    address 10.0.1.1/24

allow-hotplug enp3s0.7
iface enp3s0.7 inet static
    address 10.0.7.1/24

allow-hotplug enp3s0.7:0
iface enp3s0.7:0 inet static
    address 10.0.7.2/24

allow-hotplug enp3s0.111
iface enp3s0.111 inet static
    address 10.0.111.1/24

allow-hotplug enp3s0.111:0
iface enp3s0.111:0 inet static
    address 10.0.111.2/24

allow-hotplug enp3s0.222
iface enp3s0.222 inet static
    address 10.0.222.1/24

/etc/nftables.rules

#!/usr/sbin/nft -f
# This file is generated externally and is liable to be overwritten at any
# point. Do not edit this on-box. Or at least, if you do, don't complain about
# it being overwritten.

# References:
#   * https://arstechnica.com/gadgets/2016/04/the-ars-guide-to-building-a-linux-router-from-scratch/
#   * https://www.leaseweb.com/labs/2013/12/setup-linux-gateway-using-iptables/
#   * https://serverfault.com/questions/779115/forward-traffic-between-vlans-with-iptables

########################################
# Clear ################################
########################################
flush ruleset

########################################
# NAT rules ############################
########################################
add table ip nat
add chain ip nat PREROUTING { type nat hook prerouting priority -100; policy accept; }
add chain ip nat INPUT { type nat hook input priority 100; policy accept; }
add chain ip nat OUTPUT { type nat hook output priority -100; policy accept; }
add chain ip nat POSTROUTING { type nat hook postrouting priority 100; policy accept; }

# NAT routing on WAN interface
add rule ip nat POSTROUTING oifname "enp4s0" counter masquerade

# Filters
add table ip filter
add chain ip filter INPUT { type filter hook input priority 0; policy accept; }
add chain ip filter FORWARD { type filter hook forward priority 0; policy accept; }
add chain ip filter OUTPUT { type filter hook output priority 0; policy accept; }

########################################
# Service rules ########################
########################################

# basic global accept rules - loopback, ICMP, traceroute, established all
# accepted. http://shouldiblockicmp.com/
add rule ip filter INPUT iifname "lo" counter accept
add rule ip filter INPUT ip protocol icmp counter accept
add rule ip filter INPUT ct state related,established counter accept

# enable traceroute rejections to get sent out
add rule ip filter INPUT udp dport 33434-33523 counter reject

# SSH - accept from everywhere
add rule ip filter INPUT tcp dport 22 counter accept

# DNS - accept from LAN and VLANS
add rule ip filter INPUT iifname "enp3s0" tcp dport 53 counter accept
add rule ip filter INPUT iifname "enp3s0" udp dport 53 counter accept
add rule ip filter INPUT iifname "enp3s0.7" tcp dport 53 counter accept
add rule ip filter INPUT iifname "enp3s0.7" udp dport 53 counter accept
add rule ip filter INPUT iifname "enp3s0.111" tcp dport 53 counter accept
add rule ip filter INPUT iifname "enp3s0.111" udp dport 53 counter accept
add rule ip filter INPUT iifname "enp3s0.222" tcp dport 53 counter accept
add rule ip filter INPUT iifname "enp3s0.222" udp dport 53 counter accept

# DHCP client requests - accept from LAN and VLANS
add rule ip filter INPUT iifname "enp3s0" udp dport 67-68 counter accept
add rule ip filter INPUT iifname "enp3s0.7" udp dport 67-68 counter accept
add rule ip filter INPUT iifname "enp3s0.111" udp dport 67-68 counter accept
add rule ip filter INPUT iifname "enp3s0.222" udp dport 67-68 counter accept

# drop all other inbound traffic
add rule ip filter INPUT counter drop

########################################
# Forwarding rules #####################
########################################

# forward packets along established/related connections
add rule ip filter FORWARD ct state related,established counter accept

# Internet access ######################
# forward from LAN (enp3s0) to WAN (enp4s0)
add rule ip filter FORWARD iifname "enp3s0" oifname "enp4s0" counter accept
add rule ip filter FORWARD iifname "enp3s0.7" oifname "enp4s0" counter accept
add rule ip filter FORWARD iifname "enp3s0.111" oifname "enp4s0" counter accept
add rule ip filter FORWARD iifname "enp3s0.222" oifname "enp4s0" counter accept

# Allow access everywhere from 0.7 #####
add rule ip filter FORWARD iifname "enp3s0.7" oifname "enp3s0" counter accept
add rule ip filter FORWARD iifname "enp3s0" oifname "enp3s0.7" ct state related,established  counter accept
add rule ip filter FORWARD iifname "enp3s0.7" oifname "enp3s0.111" counter accept
add rule ip filter FORWARD iifname "enp3s0.111" oifname "enp3s0.7" ct state related,established  counter accept
add rule ip filter FORWARD iifname "enp3s0.7" oifname "enp3s0.222" counter accept
add rule ip filter FORWARD iifname "enp3s0.222" oifname "enp3s0.7" ct state related,established  counter accept

## Allow access to MQTT from IOT
add rule ip filter FORWARD iifname "enp3s0.222" oifname "enp3s0.111" ip daddr 10.0.111.48 tcp dport 1883 counter accept

# drop all other forwarded traffic
add rule ip filter FORWARD counter drop

# fail2ban
table ip fail2ban {
    chain input {
            # Assign a high priority to reject as fast as possible and avoid more complex rule evaluation
            type filter hook input priority 100;
    }
}

/etc/systemd/system/router-rules.service

[Unit]
Description = Apply base firewall rules for router functionality

[Service]
Type=oneshot
ExecStart=/etc/nftables.rules

[Install]
WantedBy=network-pre.target

/etc/dnsmasq.d/00-global.conf

except-interface=enp4s0
domain-needed
bogus-priv
no-resolv
server=208.67.222.222
server=208.67.220.220

addn-hosts=/etc/dnsmasq.d/.hosts
local=/my-domain.com/
no-hosts
expand-hosts
domain=my-domain.com
# log-dhcp
# log-queries
log-facility=/var/log/dnsmasq.log

/etc/dnsmasq.d/10-dhcp-main.conf

# The names assigned to each vlan (e.g vlan-system) are descriptions only - they
# should be consistent within this file but are not used otherwise
# DHCP: ranges
dhcp-range=vlan-system,10.0.1.10,10.0.1.250,12h
dhcp-range=vlan-admin,10.0.7.128,10.0.7.192,12h
dhcp-range=vlan-home,10.0.111.10,10.0.111.250,12h
dhcp-range=vlan-iot,10.0.222.10,10.0.222.250,12h

# DHCP: issue the default gateways
dhcp-option=vlan-system,3,10.0.1.1
dhcp-option=vlan-admin,3,10.0.7.1
dhcp-option=vlan-home,3,10.0.111.1
dhcp-option=vlan-iot,3,10.0.222.1

# DHCP: issue DNS servers
dhcp-option=vlan-system,6,10.0.1.1
dhcp-option=vlan-admin,6,10.0.7.1,10.0.7.2
dhcp-option=vlan-home,6,10.0.111.1,10.0.111.2
dhcp-option=vlan-iot,6,10.0.222.1

dhcp-leasefile=/etc/dnsmasq.d/.leases

/etc/dnsmasq.d/20-dhcp-map.conf

# Assign fixed IP addresses using DHCP

# System; VLAN 1
dhcp-host=unifi,10.0.1.100,infinite
dhcp-host=proliant-ilo,10.0.1.122

# Admin; VLAN 7
dhcp-host=vpn,10.0.7.10,infinite

# Services; VLAN 100
# Canon printer
dhcp-host=34:9f:7b:30:c9:43,10.0.100.4,infinite

/etc/dnsmasq.d/30-dns.conf

# Passthrough : https://serverfault.com/a/420748/428070
server=/internet-server.my-domain.com/8.8.8.8

# DNS records
address=/printer.my-domain.com/10.0.100.4
address=/unifi.my-domain.com/10.0.1.100

Finally run this. Don't do it in one go. Copy each line and understand what it is you're doing!

# Run as root
sudo su

# Install packages
apt install nftables vlan dnsmasq
systemctl unmask dnsmasq
systemctl stop dnsmasq

# Disable the dhcp client on the router
if ! grep --quiet 'denyinterfaces \*' /etc/dhcpcd.conf; then
  echo "denyinterfaces *" >> /etc/dhcpcd.conf
fi

# Disable systemd-resolved AFTER we've got dnsmasq
# https://computingforgeeks.com/install-and-configure-dnsmasq-on-ubuntu-18-04-lts/
systemctl disable systemd-resolved
systemctl stop systemd-resolved
rm -f /etc/resolv.conf    
echo "nameserver ${NAMESERVER}" > /etc/resolv.conf

if ! grep --quiet '127.0.1.1' /etc/hosts; then
  echo "127.0.1.1      ${HOSTNAME}" >> /etc/hosts
fi

# Make our rule loader executable
chmod +x /etc/nftables.rules

# Enable ipv4 forwarding in future
sed -i "s/#net.ipv4.ip_forward=.*/net.ipv4.ip_forward=1/g" /etc/sysctl.conf
# ... and now
echo 1 > /proc/sys/net/ipv4/ip_forward

# Fail2ban
apt install fail2ban iptables- -y

# https://wiki.meurisse.org/wiki/Fail2Ban#nftables
rm -f /etc/fail2ban/action.d/nftables-common.local
rm -f /etc/fail2ban/jail.local

cat << EOF > /etc/fail2ban/action.d/nftables-common.local
[Init]
# Definition of the table used
nftables_family = ip
nftables_table  = fail2ban

# Drop packets 
blocktype       = drop

# Remove nftables prefix. Set names are limited to 15 char so we want them all
nftables_set_prefix =
EOF

cat << 'EOF' > /etc/fail2ban/jail.local
# GLOBALS
[DEFAULT]
bantime  = 3600
findtime  = 3600
maxretry = 3

# configure nftables
banaction = nftables-multiport
banaction_allports = nftables-allports
chain     = input

# JAILS
#; sshd is already enabled by default

[recidive]
enabled = true
maxretry = 3
logpath  = /var/log/fail2ban.log
bantime  = 864000  ; 10 days
findtime = 86400   ; 1 day
EOF
systemctl restart fail2ban

systemctl enable router-rules.service
systemctl enable nftables.service

nft list ruleset