AquaOctet

Where We Left Off#

In March I wrote up building a router from scratch: a Protectli VP2420, a Quectel RM520N-GL modem card, a MediaTek MT7921 WiFi card, and a move from OPNsense to OpenWRT to stitch it all together.

This is the follow-up. Six months of that box as the only internet connection for work and everything else, through an international move and, as I write this, a hotel room in Germany. Some of the first post held up. Some of it needs correcting. And the most useful upgrade of the whole six months was not a component at all: a SIM card that holds eSIM profiles, which fixed the one part of travel internet the router itself never could.

Short version: the hardware has been boring in the best way. Nearly every hour I lost was to software, carriers, or my own stale config.

The Scorecard#

Corrections to the First Post#

I would rather correct the record than leave optimistic numbers sitting there.

The cellular numbers were generous#

I quoted 80-100 Mbps down on Rakuten. That was never the sustained reality. Measured properly (repeated runs, bound to the cellular interface, at different times of day, from different hardware and locations), LTE looked like this:

Perfectly usable for remote work, though not the 100 Mbps I signed for. Rakuten was still my best option at the time; the honest mean was just a long way from the marketing. Thankfully I worked 22:00 to 06:00 JST, so the congested hours were mostly someone else's problem.

It has never connected to 5G#

The modem is 5G. The plan was 5G. The antenna pair is 5G. The serving cell even advertised it. And the router spent its whole Rakuten life on LTE Band 3, because no n77 cell was ever reachable from where it sat. In April the network flatly refused the 5G registration; by July the modem would search and simply find nothing.

Here is the part I cannot explain: Rakuten's coverage map called the apartment a dead zone, which would settle it, except my phone, on the same carrier in the same room, received Rakuten 5G just fine. The signal was there. The router never saw it.

Relocation, and the Home/Travel Split#

Over the summer we moved back stateside, into a bigger home, and the WiFi card in the Protectli stopped being enough, for two reasons:

• Signal strength. The MT7921 and its two small antennas were sized for an apartment, not for covering an entire house.
• No real multi-band support. The MT7921 is a single radio, serving one band at a time: 2.4 GHz or 5 GHz, never both. A house full of IoT devices wants 2.4 GHz permanently; laptops and phones want 5 GHz. One card cannot give you both.

So WiFi at home now comes from a separate access point: my old Asus router running Asuswrt-Merlin, set to AP mode. The Protectli still owns DHCP, DNS, firewall, routing.. all of it. The Asus just moves packets over the air.

The nice side effect is the split this creates. The Asus is the home WiFi and it never moves, so the IoT devices stay joined to the same network whether the router is on its shelf or in a suitcase. The Protectli's own radio is now the travel WiFi, switched on only when we go nomad. Taking the router on a trip no longer means re-pairing a house full of smart plugs when I get back.

The AP still needed hand-tuning. Left on auto it chose a 160 MHz channel reaching into DFS space. Wider is not better when the signal is weak.

All of it lives in git#

A move like this touches everything at once: new WAN, new AP, radio off at home, radio back on for travel, cellular on or off depending on the country. That is exactly how you end up with a router nobody fully understands, including the person who configured it.

So none of it is done from memory. The whole router is managed from a git repo on my laptop, and the relocation, the AP swap, and the trip below are all just commits: what changed, when, and a known-good state to roll back to. The travel part is literally a checklist (radio on, cellular on, snapshot, and the reverse when we get home), because "why is there no WiFi" in a hotel room is not the moment to be remembering UCI syntax. The rest of the harness, and the hurdles that justified building it, are further down.

Field test: a hotel in Germany#

The checklist got its first real use almost immediately. We took a trip to Germany shortly after the move, and part of this post is being written from the hotel with the router on the desk.

The internet source for the whole trip is the cellular modem and nothing else. No hotel WiFi, no hotel ethernet. How that SIM got there is its own story.

It is doing exactly the job it was built for. Radio back on, every device we brought joins the same SSID it always does, and all of it sits on its own isolated LAN behind our own firewall and DNS rather than loose on the hotel network. Nothing had to be reconfigured per device. The router was the only thing that had to learn where it was.

The one thing that nearly stopped the show was not software at all: the router needs a wall socket, and German sockets are not American ones. For all the planning that went into bands, APNs, and failover, the most important travel accessory turned out to be the correct power plug. It is on the checklist now.

The Hurdles#

Roughly in the order they hurt.

1. The major-version upgrade (24.10 to 25.12)#

One upgrade swapped both the kernel (6.6 to 6.12) and the package manager (opkg to apk). Things I learned:

• Sysupgrade keeps your config, not your packages. I reinstalled 100+ packages afterwards, including the modem and WiFi drivers. Plan for the router to come back up without its WAN.
• LuCI custom commands vanished silently, even though the config file that holds them was preserved.
• WiFi went into a silent setup/teardown loop. No error, no log line. The driver and firmware were fine (I proved that by running hostapd by hand), so I shipped a small init script that bypassed OpenWRT's wifi scripts entirely. It worked. Then upstream fixed the bug, and my workaround and the fixed scripts each started a hostapd at boot and fought each other. I found the AP sitting on 2.4 GHz channel 1 at 20 MHz, days later.
• My backup scripts broke without telling me. They called opkg, which no longer existed, and quietly skipped the package inventory for two months. A backup you have not restored from is a hypothesis.

2. mwan3 and WireGuard both want to own routing#

I ran a full-tunnel WireGuard VPN for a while. The tunnel came up, handshook happily, and carried zero traffic. mwan3's policy rules sit at a higher priority than the main routing table, so every packet was steered out the raw WAN before the tunnel's default route was ever consulted.

The fix was to make the tunnel a first-class mwan3 interface with its own policy. A second, site-to-site tunnel then hit the same family of problem from a different angle: its endpoint got routed inside the first tunnel, because that tunnel already owned the default route when the new one came up.

Two smaller ones from the same config:

• Two per-device rules sat below the catch-all rule and had matched zero packets, ever. Rule order matters and nothing warns you.
• A policy named balanced was shaped as a failover. It never mixed links. The name had me convinced otherwise for weeks.

I eventually turned the full tunnel off. The exit was in Europe, which turned a 40 ms connection into 165 ms and capped it at about 19/10 Mbps. That is a VPN-exit problem, not a router problem, but it is the router that gets blamed.

3. The modem wedges, rarely but completely#

Twice the cellular data path died while the modem still reported itself registered on LTE.

After a SIM swap. I borrowed the SIM for a phone, put it back, and the router got 3-5 Mbps with a suspiciously flat +320 ms of latency. Signal was perfect. The same SIM did 20 Mbps in a phone. Everything on my side (WiFi, routing, shaping) checked out, because the problem was not on my side: the hot-swap re-attach had been handed a throttled session by the carrier core. A full modem reset produced a new carrier-side address and restored 22/16 Mbps at 40 ms.

The signature is worth memorising: single-digit Mbps, constant added latency, clean signal. Reset the modem before you touch a single config file. I now do it after every SIM reinsertion, and I confirm it worked by checking that the carrier-assigned address actually changed.

Overnight, on its own. The kernel logged a transmit queue timeout on wwan0, DHCP renewals failed, the lease expired, and mwan3, correctly, moved everything to the wired WAN. Signal that week had dropped from -85 to -117 dBm RSRP on the same cell, so something physical had shifted; weak RF under load is my best guess at the trigger.

Related gotchas:

• The modem shipped with SIM hot-swap detection disabled. Insert a SIM while it is powered and it reports NOT INSERTED forever. AT+QSIMDET=1,1 fixes that.
• An AT+CFUN=1,1 reboot sent over the serial port can silently do nothing. The QMI-level reset (uqmi --set-device-operating-mode reset) is the one I trust.

I wrapped all of this in a reset-modem.sh that proves the new session is genuinely new. It earned its keep within a week.

4. The 20 Mbps mystery#

Back stateside, the new place has gigabit fibre. I plugged it into the router. Everything topped out at 19 Mbps.

The culprit was me, from March: an SQM (CAKE) shaper on that port, tuned to 20/8 for the slow connection it used to carry. Shaper on: 18.6 / 7.3 Mbps. Shaper off: 884 / 939 Mbps. Nothing was broken. The router was doing precisely what I had asked it to, for a link that no longer existed.

On the cellular side I went the opposite way and added cake-autorate, which adjusts the shaper continuously from measured latency. A fixed cap on a cellular link is always wrong: too low at noon, too high on Sunday evening. Autorate stopped the evening bufferbloat without me retuning anything.

What I Built Around It#

The theme of every hurdle above is state I could not see: a stale shaper, a dead rule, a backup that was not backing up. The move added its own version: a router reshaped by a dozen small changes that existed only in my head. That is why the repo exists. It started as a folder of notes and turned into a small management harness. On top of the travel checklist you have already met:

• Snapshot before and after. One command pulls the full config, package list, and modem and network state off the router, and commits itself. Leaving home, coming home, before any change: snapshot.
• Drift check. I still click around in LuCI like everyone else. The drift script tells me when the live router no longer matches the last snapshot, so those clicks do not get lost.
• Custom files deploy from the repo. Anything I wrote myself gets edited in git and pushed out by a script, never hand-edited on the box.
• The AP too. The Asus is read-only from the repo's side: I configure it in its own UI, and a script captures its radio settings and clients so I have a before and after to compare.
• Upgrade and rollback are scripted, and each one takes a snapshot first. The router's own config is never blindly overwritten: the router is the truth for its settings, the repo is the truth for my custom files.
• Tests. Shellcheck and bats, because bash scripts that touch your only internet connection deserve them.
• Secrets live encrypted in-repo with git-crypt, behind a scanner that refuses to commit anything that looks like a credential.
• A scheduled CI job compares my installed packages against the OpenWRT feeds and keeps a living list of pending updates, flagging the security-relevant ones.

In the interest of honesty: as I write this, that list says 104 updates pending, 7 security-relevant, oldest ignored for 66 days, and I am two point releases behind. Building the dashboard turned out to be easier than acting on it. After the last major upgrade I am in no rush, but the number is there, climbing, which is the point.

The Real Problem Was Never the Router#

The whole reason for this build was: land in a country, put in a local SIM, have good internet. The router part works. "Put in a local SIM" is where trips have gone sideways.

It is hard to boil this down to cost versus performance, because those were never my deciding factors. These were:

Eligibility#

1. Nationality matters. Many plans are closed to foreigners, because the carrier wants a resident locked into a contract.
2. Some SIMs refuse to run in a modem. They check the device's IMEI and expect a phone.
3. Carriers push their own pocket WiFi instead. I do not trust those devices, and they are usually overpriced paperweights.
4. Physical SIMs are going away. They are becoming the premium option, and newer phones ship with no tray at all.
5. eSIMs are everywhere. Sold online, delivered instantly, and the barrier to issuing them is low enough that a friend of mine provisioned one for me in Japan. (The process was explained to me. I retained none of it.)

Time#

1. I work remotely. Unstable internet is a direct hit to my work. Yes, I could simply not travel.
2. Most physical SIMs now need in-person identity verification at a store. I have lost multiple days to vendors unwilling to deal with a foreigner.
3. After all that, I have still been throttled at undisclosed times and intervals, and had complete outages, with no information from the provider.

Service#

1. Most places really have one, maybe two, carriers worth using, because of who owns the towers.
2. Price-to-performance varies wildly between packages, and the fine print is usually where the bad deal lives.
3. Advertised coverage, "5G" or even "4G", may not include the one spot you work from, while a different package does.

Put those together and the answer is obvious: eSIMs solve eligibility and time almost completely. You buy one from a couch, in minutes, with no store visit and no passport photocopy, and if it is bad you buy a different one.

One problem. My router has a SIM slot, not an eSIM chip.

The Fix: a Physical SIM That Holds eSIMs#

It turns out you can buy a physical SIM card that is an eSIM chip: a removable eUICC. It sits in an ordinary SIM slot and looks like an ordinary SIM to whatever it is plugged into, but you can download eSIM profiles onto it and switch between them.

I bought the 9eSIM V3. About $24, holds up to 50 profiles, and works with any standard (GSMA) eSIM provider. Under the hood the companion app is built on NekokoLPA2, an open-source LPA (the "Local Profile Assistant"), which is the piece of software that downloads a profile from a carrier and writes it to the chip. It is MIT-licensed and runs on Android, iOS, macOS, Linux, and Windows, talking to the card through a phone's SIM slot or a USB smart-card reader.

That last part matters to me. The card follows the open eUICC standard, so it is not tied to the vendor's app. If the company disappears, any compatible LPA can still manage it.

How it fits the router#

[ eSIM provider ] -- QR / activation code -->  [ LPA app ]
                                                   |
                                    writes profile to the card
                                     (phone slot or USB reader)
                                                   |
                                             [ 9eSIM card ]
                                                   |
                                     moves into the router's SIM slot
                                                   |
                          [ RM520N-GL ] sees a normal SIM -> QMI -> wwan0

1. Buy an eSIM plan for wherever I am going.
2. Load it onto the card with the app.
3. Put the card in the router. The modem neither knows nor cares that the "SIM" is a downloaded profile.
4. Set the APN for that plan, and, per the lesson above, reset the modem after inserting the card.

Several plans can live on the card at once, so a multi-country trip is one card and a profile switch, not a bag of tiny plastic rectangles.

In the field: Germany#

This is not theoretical. It is how this post is reaching the internet right now. The SIM in the hotel story above is this card, and it is the trip's only uplink:

1. Bought an eSIM plan that fit what we needed.
2. Loaded it onto the card with the app.
3. Card into the router, APN set to exactly what the plan suggested.
4. Bam. Live.

No store, no ID check, no counter staff deciding whether they felt like helping a foreigner today. Compare that with the "multiple days lost" entry above. The whole process took less time than finding the right power adapter.

In use it has been easier than the physical-SIM route, and it performs about the same when it works well, which depends on the plan you bought far more than on the card.

Caveats#

• Modem-unfriendly plans are still modem-unfriendly. An eSIM plan that checks IMEI will be just as unhappy in a router as its plastic equivalent. The card solves getting a plan, not what the plan permits.
• You need something to write profiles with. A compatible phone, or a USB reader and a laptop.
• "Unlocked" does not mean open. My Pixel wrote to the card without complaint. My Japanese-market Sony, also carrier-unlocked, would not: its SIM interface appears to cooperate only with cards it considers authorised, and this one is not. I have not dug into exactly what is being enforced. I will say that Japanese carriers, like a lot of Japanese companies, have a long history of doing strange things in their own private interest, and this smells like one of them. If you plan to use a phone as your writer, test it before the trip.
• APNs are per plan. The modem will not guess. Keep the APN written down next to the profile name.

Would I Build It Again?#

Yes, with different expectations.

• The hardware was the easy part and has needed nothing since March.
• The software needs an owner. OpenWRT is excellent, and it will happily let you build a routing table that defeats itself. Snapshot before every change. Future-you has forgotten what present-you configured.
• Measure, then write it down. Every number I quoted from memory was wrong in the flattering direction.
• The SIM supply chain was the actual bottleneck, and one cheap card fixed more of my travel-internet pain than any component in the original build.

Updated troubleshooting reference#

Until next time,

✧~ Aqua ~✧#

Why Build Your Own Router?#

I have never met a consumer router I fully trusted: cheap plastic, locked firmware, mediocre drivers, and security patches that arrive six months late -- if at all. If you work remotely, route traffic through a VPN, run a homelab, or travel internationally with a SIM card, you want hardware you actually control.

My problem was specific: I needed a travel router that could accept a local SIM card, broadcast WiFi, handle dual-WAN failover, and still be light enough to throw in a bag. No consumer box does all of that well, so I built one from a small-form-factor x86 machine running open-source firmware.

This is the exact path I took from unboxing a Protectli VP2420 to a fully operational dual-WAN 5G + WiFi 6 router, including the gotchas and dead-ends along the way.

The Hardware Stack#

Here is what I ended up with, and why I picked each piece.

Why the VP2420?#

The VP2420 ships with four Intel I225-V 2.5GbE NICs, fanless passive cooling, and -- crucially -- two separate M.2 slots: one B-Key for LTE/5G modems and one E-Key (2230) for WiFi. Both can coexist, no tradeoffs.

The Elkhart Lake J6412 is no powerhouse, but for routing, NAT, and firewall duty it's more than sufficient. AES-NI hardware acceleration handles VPN tunnels without breaking a sweat.

Why the Quectel RM520N-GL?#

The RM520N-GL is a global 5G module covering Band 3 and n77 (Rakuten Japan), AT&T/T-Mobile bands (North America), and Telstra bands (Australia). If you move between countries, this is the modem that doesn't force you to choose a region.

More importantly: it communicates via QMI protocol, which OpenWRT supports natively. This is the key distinction that makes the rest of this guide work.

Why the MT7921 over Intel AX200/AX210?#

Intel WiFi cards look attractive on paper -- WiFi 6E, widely available, cheap. The problem is their Linux driver (iwlwifi) has severe limitations in Access Point mode. It's designed for client devices (laptops), not APs. On OpenWRT, Intel cards effectively cap out at 2.4GHz in AP mode and often behave unreliably with multiple clients.

The MediaTek MT7921 uses the open-source mt76 driver with first-class OpenWRT support and full AP mode on both 2.4GHz and 5GHz, so it was the obvious pick.

The OPNsense Problem#

The original plan was to run OPNsense -- it's polished, well-documented, and familiar to anyone coming from pfSense. But when I tried to connect the Quectel RM520N-GL, I hit a wall that turned out to be fundamental rather than configurable.

OPNsense and pfSense do not support QMI or MBIM protocols. Both rely exclusively on PPP for cellular modem communication -- an older, slower serial protocol. Since OPNsense 24.7, even PPP with internal LTE modems has been broken for many users, with debug errors like label not found in the MPD daemon.

The real-world performance difference is severe:

OPNsense is a fine firewall OS if your WAN is Ethernet, but for embedded cellular modems it is simply the wrong tool. OpenWRT has supported this use case from the start.

Installing OpenWRT on the VP2420#

The VP2420 is an x86_64 machine, so you'll use the generic x86 OpenWRT image -- not a device-specific build.

Download the image#

Get the latest stable x86/64 combined image from downloads.openwrt.org. Look for:

openwrt-x86-64-generic-ext4-combined-efi.img.gz

The EFI variant works reliably with the VP2420's BIOS.

Flash and install#

1. Decompress the image:

gunzip openwrt-*.img.gz

2. Flash to a USB drive:

dd if=openwrt-*.img of=/dev/sdX bs=4M status=progress

3. Boot the VP2420 from USB (press F11 at POST for the boot menu)

4. Once booted, identify your internal SSD:

lsblk

5. Flash to internal storage:

dd if=openwrt-*.img of=/dev/sda bs=4M

6. Reboot and remove the USB drive.

By default, OpenWRT only creates a small root partition. After first boot, expand it with parted and resize2fs to use your full SSD. You'll need the space for packages.

First boot access#

Connect your laptop directly to port 1 (eth0) with an Ethernet cable. OpenWRT's default LAN IP is 192.168.1.1.

ssh root@192.168.1.1

No password is set initially -- set one immediately:

passwd
opkg update

Configuring the Quectel Modem via QMI#

With the RM520N-GL installed in the B-Key M.2 slot, OpenWRT should enumerate the device automatically. Verify it's present:

lsusb | grep -i quectel
# Expected: Quectel Wireless Solutions Co., Ltd.

ls /dev/cdc-wdm*
# Expected: /dev/cdc-wdm0

Install QMI tools#

opkg update
opkg install kmod-usb-net-qmi-wwan uqmi luci-proto-qmi

Verify the connection manually first#

# Start the data connection
uqmi -d /dev/cdc-wdm0 --start-network --apn rakuten.jp --keep-client-id wds

# Confirm connected
uqmi -d /dev/cdc-wdm0 --get-data-status
# Expected: "connected"

# Bring up the interface and get an IP
ip link set wwan0 up
udhcpc -i wwan0

# Test connectivity
ping -I wwan0 8.8.8.8

Seeing "disconnected" from --get-data-status? Run --start-network first. The modem won't auto-connect until the interface is formally requested.

Persistent config via UCI#

Once confirmed working, set it up permanently:

uci set network.wwan=interface
uci set network.wwan.proto='qmi'
uci set network.wwan.device='/dev/cdc-wdm0'
uci set network.wwan.apn='rakuten.jp'
uci set network.wwan.pdptype='ipv4v6'
uci set network.wwan.auth='none'
uci commit network

# Add to WAN firewall zone
uci add_list firewall.@zone[1].network='wwan'
uci commit firewall

/etc/init.d/network restart

Check signal quality#

uqmi -d /dev/cdc-wdm0 --get-signal-info

Good signal targets: RSSI above -70 dBm, RSRP above -90 dBm, SNR above 15 dB.

APN reference by carrier#

Adding WiFi 6 with the MT7921#

The MT7921 fits the VP2420's M.2 2230 E-Key slot. You'll also need two antenna cables (IPEX/MHF4) routed to the pre-drilled antenna holes on the VP2420 chassis -- Protectli includes these holes specifically for this purpose.

Install the driver and hostapd#

opkg update
opkg remove wpad-basic-wolfssl   # remove the limited default
opkg install kmod-mt7921e wpad-openssl

reboot

After reboot, verify the card is detected:

dmesg | grep mt7921
iw dev

The interface creation quirk#

On some OpenWRT builds with the MT7921, the phy0-ap0 virtual interface expected by hostapd doesn't get created automatically. Confirm this is your issue:

hostapd -dd /var/run/hostapd-phy0.conf 2>&1 | head -20
# If you see: "Could not read interface phy0-ap0: No such device"
# -> proceed with the fix below

Create the interface manually and test:

iw phy phy0 interface add wlan0 type __ap
ip link set wlan0 up

# Update the hostapd config to use wlan0 instead
sed -i 's/phy0-ap0/wlan0/' /var/run/hostapd-phy0.conf

# Run hostapd -- should show AP-ENABLED
hostapd -dd /var/run/hostapd-phy0.conf 2>&1 | grep -E "ENABLED|ERROR"

To make it persistent across reboots:

cat << 'EOF' > /etc/rc.local
# Create wireless interface for MT7921
iw phy phy0 interface add wlan0 type __ap
ip link set wlan0 up
exit 0
EOF

Configure the wireless network via LuCI#

Navigate to Network > Wireless in LuCI. Click Add on your radio and configure:

• Mode: Access Point
• SSID: your network name
• Band: 5GHz (802.11ax / WiFi 6)
• Channel: 36 or auto
• Width: 80MHz
• Security: WPA2/WPA3 Mixed + strong passphrase
• Network: Assign to lan

Channel 36 at 80MHz width gives the best balance of speed and interference avoidance in dense urban environments. The MT7921 supports 2.4GHz and 5GHz in AP mode -- 6GHz is not available in AP mode on this card.

Dual WAN with mwan3#

With the cellular modem as primary WAN and an Ethernet connection as WAN2, mwan3 handles health checking, failover, and load balancing between both connections.

Install mwan3#

opkg install mwan3 luci-app-mwan3
/etc/init.d/mwan3 enable

Full mwan3 config#

Edit /etc/config/mwan3:

config globals 'globals'
    option mmx_mask '0x3F00'

config interface 'wwan'
    option enabled '1'
    option family 'ipv4'
    list track_ip '8.8.8.8'
    list track_ip '1.1.1.1'
    option track_method 'ping'
    option reliability '1'
    option count '3'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '3'

config interface 'wan2'
    option enabled '1'
    option family 'ipv4'
    list track_ip '8.8.8.8'
    option track_method 'ping'
    option reliability '1'
    option count '3'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '3'

config member 'cellular_primary'
    option interface 'wwan'
    option metric '1'
    option weight '2'

config member 'ethernet_primary'
    option interface 'wan2'
    option metric '1'
    option weight '3'

config policy 'balanced'
    list use_member 'cellular_primary'
    list use_member 'ethernet_primary'

config policy 'cellular_only'
    list use_member 'cellular_primary'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'
    option proto 'all'

config rule 'ipv6_default'
    option family 'ipv6'
    option proto 'all'
    option use_policy 'cellular_only'

Apply and verify:

/etc/init.d/mwan3 restart
mwan3 status

All IPv6 traffic is routed through the cellular interface -- Rakuten provides native IPv6 (240b:: prefix) while most secondary connections are IPv4-only via CGNAT.

Fix LAN connectivity after enabling mwan3#

If clients lose internet access after enabling mwan3, the WAN interfaces likely aren't in the firewall's wan zone:

uci add_list firewall.@zone[1].network='wwan'
uci add_list firewall.@zone[1].network='wan2'
uci commit firewall
/etc/init.d/firewall restart

DNS and Cloudflare Setup#

OpenWRT uses dnsmasq as its local DNS resolver. By default it forwards queries to whatever the WAN interface provides. I override this with Cloudflare's DNS and lock it down so WAN interfaces can't overwrite it.

Configure upstream DNS resolvers#

uci set dhcp.@dnsmasq[0].noresolv='1'
uci del dhcp.@dnsmasq[0].server
uci add_list dhcp.@dnsmasq[0].server='1.1.1.1'
uci add_list dhcp.@dnsmasq[0].server='1.0.0.1'
uci add_list dhcp.@dnsmasq[0].server='2606:4700:4700::1111'
uci add_list dhcp.@dnsmasq[0].server='2606:4700:4700::1001'
uci commit dhcp
/etc/init.d/dnsmasq restart

Prevent WAN interfaces from overwriting DNS#

uci set network.wwan.peerdns='0'
uci set network.wan2.peerdns='0'
uci commit network

Enable DNSSEC#

uci set dhcp.@dnsmasq[0].dnssec='1'
uci set dhcp.@dnsmasq[0].dnsseccheckunsigned='1'
uci commit dhcp
/etc/init.d/dnsmasq restart

Verify:

nslookup cloudflare.com 127.0.0.1

Final Architecture#

[ Internet ]
      |
      +---- 5G/LTE (Rakuten) -------- Quectel RM520N-GL (M.2 B-Key)
      |                                -> QMI -> wwan0 interface
      |
      +---- Ethernet WAN2 ----------- eth3 (VP2420 port 4)
                                       -> DHCP -> wan2 interface
              | both via mwan3 load balancer
[ Protectli VP2420 ]  <-  OpenWRT
      |
      +---- LAN (eth0-eth2)  ->  Wired clients
      |
      +---- WiFi AP (wlan0)  ->  MediaTek MT7921
                                  -> 5GHz 802.11ax / WiFi 6
                                     Bridged to br-lan

DNS:  dnsmasq -> 1.1.1.1 / 1.0.0.1 (Cloudflare + DNSSEC)
IPv6: Routed via cellular (Rakuten native dual-stack)

Performance results#

Key packages installed#

opkg install \
  kmod-usb-net-qmi-wwan uqmi luci-proto-qmi \
  kmod-mt7921e wpad-openssl \
  mwan3 luci-app-mwan3 \
  luci-app-firewall luci-app-opkg \
  tcpdump iperf3 htop

Total cost estimate: VP2420 ($350) + RM520N-GL ($120) + MT7921 ($25) + antennas ($15) + SSD (~$30) = ~$540. Not cheap, but this box will outlast any consumer router and I can rebuild it at the OS level whenever I want.

Troubleshooting quick reference#

Router hostname: KeyToVoid -- Running OpenWRT -- Tokyo, Japan

Most Open WebUI guides assume Docker Compose on a single box. I run my homelab on Docker Swarm, and getting OpenWebUI, Ollama, and ChromaDB working as Swarm services -- especially with a GPU -- took more digging than it should have. Here is the stack file I ended up with, plus the GPU configuration that actually makes Swarm schedule onto your NVIDIA card.

Overview#

Three services in one stack: OpenWebUI for the front end, Ollama for model hosting, and ChromaDB as the vector database for RAG. Everything below works on CPU too; the GPU parts are clearly marked so you can strip them out.

Prerequisites#

1. Basic understanding of Docker Swarm.
2. Docker Swarm configured on your host system.
3. Proper directories or volumes created for data storage:

   mkdir -p data/open-webui data/chromadb data/ollama
   

4. GPU support (optional) with NVIDIA Container Toolkit installed and configured.

Deployment Steps#

Step 1: Prepare Your Environment#

• With GPU Support: Ensure your host system has GPU support configured:

  • Enable CUDA for your OS and GPU.
  • Install the NVIDIA Container Toolkit.
  • Edit /etc/docker/daemon.json to advertise the GPU to Swarm (get the UUID prefix from nvidia-smi -a | grep UUID):

    {
      "runtimes": {
        "nvidia": {
          "path": "nvidia-container-runtime",
          "runtimeArgs": []
        }
      },
      "default-runtime": "nvidia",
      "node-generic-resources": ["NVIDIA-GPU=GPU-<YOUR_GPU_UUID_PREFIX>"]
    }
    

  • Enable GPU resource advertising in /etc/nvidia-container-runtime/config.toml by uncommenting:

    swarm-resource = "DOCKER_RESOURCE_GPU"
    

  • Restart the Docker daemon:

    sudo service docker restart
    

• With CPU Support: Remove the deploy.resources.reservations.generic_resources block from the ollama service in the stack file below. That is the only GPU-specific part.

Step 2: Configure the Docker Stack#

Below is a sample docker-stack.yaml file for deploying the three services:

version: '3.9'
services:
  openWebUI:
    image: ghcr.io/open-webui/open-webui:main
    depends_on:
      - chromadb
      - ollama
    volumes:
      - ./data/open-webui:/app/backend/data
    environment:
      DATA_DIR: /app/backend/data
      OLLAMA_BASE_URLS: http://ollama:11434
      CHROMA_HTTP_PORT: 8000
      CHROMA_HTTP_HOST: chromadb
      CHROMA_TENANT: default_tenant
      VECTOR_DB: chroma
      WEBUI_NAME: Awesome ChatBot
      CORS_ALLOW_ORIGIN: "*"
      RAG_EMBEDDING_ENGINE: ollama
      RAG_EMBEDDING_MODEL: nomic-embed-text-v1.5
      RAG_EMBEDDING_MODEL_TRUST_REMOTE_CODE: "True"
    ports:
      - target: 8080
        published: 8080
        mode: ingress
    deploy:
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3

  chromadb:
    hostname: chromadb
    image: chromadb/chroma:0.5.15
    volumes:
      - ./data/chromadb:/chroma/chroma
    environment:
      - IS_PERSISTENT=TRUE
      - ALLOW_RESET=TRUE
      - PERSIST_DIRECTORY=/chroma/chroma
    ports:
      - target: 8000
        published: 8000
        mode: ingress
    deploy:
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
    healthcheck:
      test: ["CMD-SHELL", "curl localhost:8000/api/v1/heartbeat || exit 1"]
      interval: 10s
      retries: 2
      start_period: 5s
      timeout: 10s

  ollama:
    image: ollama/ollama:latest
    hostname: ollama
    ports:
      - target: 11434
        published: 11434
        mode: ingress
    deploy:
      resources:
        reservations:
          generic_resources:
            - discrete_resource_spec:
                kind: "NVIDIA-GPU"
                value: 0
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
    volumes:
      - ./data/ollama:/root/.ollama

Step 3: Deploy the Stack#

The command is the same for GPU or CPU -- just make sure you modified the stack file first if you are CPU-only:

docker stack deploy -c docker-stack.yaml -d super-awesome-ai

Additional Resources#

• OpenWebUI GitHub
• ChromaDB GitHub
• Ollama GitHub