New Mikrotik Router

I’m been using a Safe@Office Router/Firewall at home now for several years, but it bothered me. First, the licensing model wasn’t clear–it only allowed limited client connections to the Internet versus limiting the number of VPN connections–which is what I thought the licenses were for from the documentation. At 5 users, one person can just about eat it up (phone, laptop, desktop, Apple TV, iPad). Then anyone else is rejected from reaching the Internet. Not a good situation, and shame on you Safe@office for not being clear about it.

Then the unit started falling over periodically. I believe my new MacBook Pro was tickling a bug in the router causing it to reset spontaneously. It was always sporadic as to when it would happen, so I just groaned and dealt with it for a while.

Well, I’ve been interested in OpenWRT for a while (and have even made some contributions), so I went on the prowl for hardware that I could use. I’ve not been happy with most consumer firewall/routers. In two years time, I went through 3 of them. They all appear have died from overheating, while my previous firewall had lasted over 6 years prior to failing. The quality dropped somewhere along the way.

Project Turris

I spent some time looking for some hardware that I could ultimately use with OpenWRT and/or was more open. Project Turris has some really nice hardware and is built on OpenWRT, but you can’t just purchase a unit for your own use (at least not at the moment).

RouterBoard

Eventually, I came across the RouterBoard. While the bare boards are nice, I couldn’t find a cost effective one that supported Gigabit Ethernet on all the ports. However, one of their prepackaged units does: the RB951Ui-2HnD.

The units from RouterBoard come loaded with the RouterOS operating system, which is really Linux underneath it all. The interface is quite sophisticated, implementing what feels like a full-fledged GUI in the browser, and also supports configuration via SSH. In all seriousness, this board seems every bit as capable as a Cisco router, at a fraction of the cost.

So I took the leap, and decided to run with RouterOS for now. Actually, I bought two units: one to try loading OpenWRT onto, and another to use in production.

Now, I’ve been trying to avoid playing systems administrator at home–I want to work on more interesting things than managing a network. This is solidly a step back in the systems administrator direction, but not terribly so. I have learned a few things along the way though.

Hiccups

I ran into some minor hiccups along the way. First, the DNS server/cache present in RouterOS knows nothing about the default domain. I believe this was so that you could prevent resolution of sites you don’t want your users visiting (not that this will really stop them). But it took a bit to realize this was happening and that I needed to plugin in the fqdn of the host I was adding.

It also took a little bit to make sure I had port forwarding for a couple of ports setup correctly. If you’ve never worked with iptables before, then this can be quite daunting as the chain names and packet flows are complicated. Fortunately, I know enough to be dangerous and Mikrotik had some help on their documentation site for RouterOS.

I also assumed that since the unit could do Gigabit Ethernet that it would naturally support Jumbo (9000 byte) frames. I was wrong. I have a NAS that I use for various kinds of data storage, and I want maximum throughput to the unit, so the MTU for my network connection is set to 9000 bytes. I put a gigabit switch behind the firewall that can cope with jumbo frames, but it ruined my plans for having a single device in my office for the networking bits. I’m not to upset though as I’m prone to hook up and play with new equipment or help others with theirs, so the extra ports are handy to have.

This last bit was really what hurt though. And it was interesting in that it didn’t manage to manifest itself until two months later. I was giving GitLab a look, so I created an account, created a new repository, and was ready to push some data into it. Then SSH hung. It just sat there until it timed out, and I had no real reason why. Adding debugging into the mix didn’t help show anything either. Reducing the available ciphers and MACs did, but I still used the same ones it was negotiating before. :-/

A long story short, it turns out that Path MTU discovery just plain doesn’t work. This is supposed to allow both ends to probe for the maximum frame size without fragmentation, allowing you to make the most effective use of your bandwidth. In this case, both the far end and I are using Jumbo packets, but the links between us don’t support it. Unfortunately, too many routers and firewalls drop ICMP packets all together, and prevent the network stack from discovering an appropriate size for the maximum message size (MSS) in TCP. To make it worse, firewalls often drop fragmented packets (though RouterOS does appear to accept them). It appears that either my packets were fragmenting and getting dropped on the far end, or that the opposite was happening somewhere along the way back to me.

The answer turned out to be surprisingly simple, and is what most consumer firewalls do: MSS clamping. You just add a mangle rule to the firewall saying to change the MSS size to 1500 on the forward chain for the WAN interface. This will force the TCP connections to use a smaller size, preventing fragmentation, and allowing both ends to communicate successfully. It was a time consuming lesson to learn, but yet another one that I’ll put into my Book of Seemingly Esoteric Knowledge.

Other than that, I’ve been really happy with the new router. I hope it lasts!