Installing pfSense 2.6 on ZimaBoard
I backed the ZimaBoard Single Board Server project on Kickstarter in early 2021, a couple of months ago it finally arrived and the first project on the todo list was to try this as a replacement for my overkill pfSense server (Dell R210 II Server) which consumed ~100W compared to ~6W of the Zimabaord, a cost reduction of over £200 per year in electricity costs too. The ZimaBoard comes pre-installed with Casa OS on the onboard 32GB eMMC storage, but that can be overwritten with whatever software we want.
When I backed the project, I also bought an extra NIC as I needed 3 connections if I was going to have a backup WAN link, however, there isn’t a way to mount the PCIe Network Card into the ZimaBoard and keep it secured. The usual riser bracket also has to be removed as it would otherwise foul the case.
Note: This isn’t the setup I finally ended up running, but this is the journey I went on.
Installing pfSense
Here are the steps I followed to get pfSense 2.6.0 up and running on the ZimaBoard:
- Download the latest pfSense USB Image from: https://www.pfsense.org/download/
- Architecture: AMD64
- Installer: USB Memstick Installer
- Console: VGA
- Write the USB Image to a USB Flash drive using Balena Etcher (or similar)
- Attach a display and keyboard to the ZimaBoard
- Insert USB Flash drive now containing pfSense installing image
- Boot the ZimaBoard and follow the wizard to install pfSense, selecting the eMMC storage#
- The ZimaBoard supports up to 2 x SATA drives, depending on the features planned to be used on pfSense, consideration should be given to using a SATA SSD or HDD.
- After installation completes I had to enter the BIOS and change the Boot option to MMC.
Realtek Network Drivers
The two onboard NICs are identified as Realtek PCIe GbE Family Controller and worked out of the box, though FreeBSD 12 which pfSense 2.6 is built on doesn’t include the latest drivers. The left hand NIC was re0 and the other re1
The 2.5GbE network card (Realtek RTL8125B) wasn’t detected after installation and required updated Realtek drivers to be installed.
Updated 4 Dec 2022: Realtek have released driver version 197.
Update 19 Jan 2023: Realtek have released driver version 198.
Installing Realtek version 198 driver
- If there is an old driver version e.g.:
pkg delete realtek-re-kmod-196.04 - then
fetch -v https://pkg.freebsd.org/FreeBSD:12:amd64/latest/All/realtek-re-kmod-198.00_1.pkg - then install the driver:
pkg install -f -y realtek-re-kmod-198.00.pkg - If not already present add the following configuration items to
/boot/loader.conf
if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"- Reboot pfSense
Configuration Tweaks for pfSense
If you are planning on using pfSense as a basic firewall/router you should consider minimising the writes to the eMMC storage, though if you are planning on using additional packages like ntopng it will need more space than can be provided by a RAM Disk
- System -> Advanced -> Miscellaneous -> RAM Disk Settings:
- Check
Use RAM Disks - Reboot pfSense.
- Check
Enable capabilities of the CPU (Intel Celeron N3450)
- System -> Advanced -> Miscellaneous -> Cryptographic & Thermal Hardware:
- Cryptographic Hardware:
AES-NI CPU-based Acceleration - Thermal Sensors:
Intel Core* CPU on-die thermal sensor
- Cryptographic Hardware:
It’s unclear if pfSense is correctly supporting the CPU clock boost capabilities of the CPU, as it shows the clock speed as 1.1GHz. However, to give it a chance of working:
- System -> Advanced -> Miscellaneous -> Power Savings:
- Check:
Enable PowerD - Set the policies to:
Hiadaptive
- Check:
Problems
Networking Issues
I ran this configuration for several weeks and noticed I was having problems with my backup network connection (via Starlink) going offline, sometimes disabling and re-enabling the interface fixed it, sometimes a reboot of Starlink would fix it, sometime pfSense had to be rebooted. dmesg filled upa with a stream of errors:
arpresolve: can't allocate llinfo for 100.64.0.1 on re1Where re1 is the onboard GbE NICs connected to the Starlink, not the 2.5GbE NIC.
DNS errors from unbound were also being constantly logged to /var/log/resolver.log
May 25 22:26:18 pfsense-zimaboard unbound[75327]: [75327:3] error: recvfrom 26 failed: Protocol not available
May 25 22:26:18 pfsense-zimaboard unbound[75327]: [75327:1] error: recvfrom 24 failed: Protocol not availableGiven the move from the Dell R210 II server to the ZimaBoard was both a hardware move and an upgrade from pfSense 2.4 to 2.6, it’s unclear if one or the other was the cause of the problems.
Solution
I decided to stop using any of the Realtek NICs and instead take the Intel Quad port NIC (Intel(R) PRO/1000 ET 82576 (Quad Copper) I had in the Dell server and use that. The card is significantly larger and so I also created a new wall mount to allow the Intel NIC to be secured.
Since switching to the Intel quad port NIC, I’ve not experienced any of the issues above.
BIOS & CPU Issues
- BIOS version information is corrupted
- DMI table is broken (possible cause of BIOS version info)
dmidecode -t processor -t cache
# dmidecode 3.3
Scanning /dev/mem for entry point.
SMBIOS 3.0.0 present.
Invalid entry length (0). DMI table is broken! Stop.- The Intel N3450 CPU has a base clock frequency of 1.1GHz with a burst frequency of 2.2GHz. However, pfSense only shows the clock running at 1.1GHz, it’s unclear if this is just cosmetic or if it really isn’t bursting.
- The speed issue seems similar to a post here on Reddit from 2019
- The supported CPU speeds listed are:
sysctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 1101/0 1100/0 1000/0 900/0 800/0
Thank you for sharing your work. I have always read that the intel nic’s are preferable to the cheaper Realtek ones. Curious if your setup will route your fully allocated wan download speed, and if you have a gigabit connection. Thanks Again
Hi,
My broadband speed is only ~500Mbps via my Cable provider (Virgin Media) and the backup, StarLink connection runs between 200-400Mbps. My setup has no issue running at those speeds. Obviously, there are some plugins to pfSense which can add significant extra load, but the ZimaBoard seems to have more than enough capacity.
Since moving to the Intel Pro 1000 NIC I’ve had less frequent issues with StarLink, but it still sometimes shows the gateway status as Pending but disabling and re-enabling the interface has “fixed” it each time it occurs. I suspect both the onboard Realtek NICs and the 2.5Gbps external NIC would work fine with the latest drivers, the only issue I had was the odd behaviour with StarLink.
In the near future I may go back and try the RealTek based solution again to see if I can narrow down the cause of the odity.
I’ve read somewhere that these were using broadcom NICs not Realtek. I guess not ? I personally haven’t had issues with realtek but I’d use proxmox then virtualize pfsense using virtio those are very good drivers
Definately Realtek NICs. As for virtualising pfSense. Sure you can do that, but it’s not something I like to do, except for lab purposes. If it’s your main router then by virtualising it you’re adding an additional layer of complexity and fragility. But depends on your use case of course.
I’m setting up my ZimaBoard with two 240GB SSD (ada0, ada1). The ZFS Configuration is ‘mirror’.
With this configuration there is no reason to install anything on the ZimaBoard internal memory, correct (eMMC storage#)?
On reboot it should then boot off the mirrored SSD.
Thanks for the great write up.
So long as you ensured you installed on to the SSD and that the boot partition isn’t on the eMMC storage. Load the BIOS and ensure the SSD is the boot disk and confirm it boots.
Hi Martin, the Realtek driver version has been updated to 198 now, so
Thanks, Martin. Post updated.
Any updates on your zimaboard experience with pfsense?
I’m still using it with the Intel NIC, though I know others are running pfsense and other firewall software on their ZB using the onboard Realtek NICs without issue. However if you’ve not purchased a ZB yet, you might want to read a follow on related post: https://www.martinrowan.co.uk/2023/01/is-a-zimaboard-the-right-choice-for-me/
Hello, I have a new 432 board with the latest pfsense image version 2.6.0-RELEASE (amd64) built on Mon Jan 31 19:57:53 UTC 2022 FreeBSD 12.3-STABLE. and using the default ZFS file system.
Did you have any issues with storage issues with the internal mmc storage?
System crash/hang after a few hours.
I have tried reloading fresh image multiple times and using the onboard NIC’c
Console error
pfSense kernel: mmcsd0: Error indicated: 2 Bad CRC
pfSense kernel: sdhci_pci0-slot0: Controller timeout
pfSense kernel: sdhci_pci0-slot0: ============== REGISTER DUMP ==============
pfSense kernel: sdhci_pci0-slot0: Sys addr: 0x287f6610 | Version: 0x00001002
pfSense kernel: sdhci_pci0-slot0: Blk size: 0x00000080 | Blk cnt: 0x00000001
pfSense kernel: sdhci_pci0-slot0: Argument: 0x00000000 | Trn mode: 0x00000012
pfSense kernel: sdhci_pci0-slot0: Present: 0x1fff0000 | Host ctl: 0x00000025
pfSense kernel: sdhci_pci0-slot0: Power: 0x0000000b | Blk gap: 0x00000080
pfSense kernel: sdhci_pci0-slot0: Wake-up: 0x00000000 | Clock: 0x00000007
pfSense kernel: sdhci_pci0-slot0: Timeout: 0x00000007 | Int stat: 0x00000000
pfSense kernel: sdhci_pci0-slot0: Int enab: 0x00000020 | Sig enab: 0x00000020
pfSense kernel: sdhci_pci0-slot0: AC12 err: 0x00000000 | Host ctl2:0x0000000b
pfSense kernel: sdhci_pci0-slot0: Caps: 0x546ec881 | Caps2: 0x00000807
pfSense kernel: sdhci_pci0-slot0: Max curr: 0x00000000 | ADMA err: 0x00000000
pfSense kernel: sdhci_pci0-slot0: ADMA addr:0x00000000 | Slot int: 0x00000000
pfSense kernel: sdhci_pci0-slot0: ===========================================
pfSense kernel: sdhci_pci0-slot0: Tuning failed, using fixed sampling clock
pfSense kernel: mmcsd0: Error indicated: 1 Timeout
pfSense kernel: sdhci_pci0-slot0: Tuning failed, using fixed sampling clock
pfSense kernel: mmcsd0: Error indicated: 1 Timeout
pfSense kernel: mmcsd0: Setting erase start position failed Timeout
Hi, I’m still running pfsense on my Zimaboard without issue. I also checked the logs and no sign of any errors.
I see you’ve posted on the IceWhale discord channel (my alias: shortbloke), that’s probably the best place to get support from the IceWhale team.
Hey, thanks for sharing.
I’ve used what looks like an identical 4 port card as you but the ZimaBoard fails to boot. Did you tweak any bios settings?
No bios changes needed.
Are you saying the ZimaBoard works fine until you plug the card in, then it doesn’t boot?
If so can you test the card in another PC?
Some updates for pfSense 2.7 on ZimaBoard:
– The realtek driver can be upgraded to version 198 with only one command: pkg install realtek-re-kmod, and a reboot. This solved the problem of Realtek interface going offline every 3-8 days (i.e.: “re1: watchdog timeout” continuously logged).
– The CPU clock of pfSense 2.7 on Zimaboard seems to be limited to 1.1GHz:
[2.7.0-RELEASE][[email protected]]/root: sysctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 1101/0 1100/0 1000/0 900/0 800/0
but using “Hiadaptive” powerd mode instead of “Minimum” gives improved performances and I can reach routing at 1Gbps. Maybe “Minimum” modem locks the CPU at speed slower than 1.1GHz.