This is a simple gist with details on how to run a simple btest client on a QNAP TX-431KX NAS. This is likely to work on any QNAP NAS running QTS 5.x but I've only tested it on a TS-431KX.
As a bonus, I dumped a bunch of text from some exploratory digging on the hardware/firmware. I do this kind of thing for fun. Go figure.
This is the process I used on my TS-431KX running QTS 5.0.0.2055.
- Enable the BTest Server on one or more Mikrtoik devices on your network
- There are also public BTest servers out there if you're up for testing your internet bandwidth
- Install
Entware-std 1.03using the QTS App Store viaqnapclub.eurepo- The signature's expired but it'll still work
- Enable SSH on the QNAP device and connect via SSH
- On connect, exit the ASCII menu using
QthenYto get to a proper bash shell
- On connect, exit the ASCII menu using
- Use
opkgto install required packagesgit,gcc,make,automake,autoconf,sed,grep,gawk,nano- I might've missed a few here but any missing packages should be apparent during the compile process
- Make an SSH key and add it to your Github account (if required)
- https://github.com/settings/keys
- This is required because the
opkgversion of git doesn't have HTTPS
- Find a good directory and clone down the
btest-opensourcerepo- (any
/share/CACHEDV*_DATA/dir should work ok) cd /share/CACHEDEV5_DATA/codegit clone [email protected]:samm-git/btest-opensource.gitcd btest-opensource
- (any
- Edit
btest.cto increase MTUcmd.tx_size=1500;->cmd.tx_size=9000;
- Run the
bootstrapscript./bootstrap
- Add
+xto theinstall-shfilechmod +x ./install-sh
- Run the
./configurescript genreated by./bootstrap./configure
- Make the binary
make
- Do that
btestthing./btest -c 192.168.888.1 -t -b 4000M
Max on this platform seems to be 4294M before it loses track of things. This makes behavior very unpredictable if the -b flag isn't set, which forces a starting bitrate. Created a bug in the original project to document this.
That's fine though, just run two or more concurrently. Running three concurrent btest -t processes at -b 4000 pushed aobut 8.2 Gbps of traffic over the SFP+ port on the TS-431, which was definitely useful enough for basic testing.
Also, transmit works a lot faster than receive (about ~70-80% of a single core), although I didn't do comprehensive testing or anything.
The 8.2 Gbps number above was attained using a RB4011iGS+ running firmware 6.49.6. The NAS was connected to a CRS305-1G-4S+ (same firmware) using a generic 10GBE SFP+ copper DAC. This was connected to the router via a Mikrotik S+AO0005 active optical 10GBE SFP+ DAC.
The two other Mikrotik devices I have on hand are both CRS305-1G-4S+ which, while capable of switching 10Gbps traffic just fine, aren't beefy enough to operate as capable BTest servers. When receiving traffic they cap out at about 1.2 Gbps or so.
This NAS is particularly interesting for bandwidth testing since it has an SFP+ port capable of 10GB Ethernet. It runs on a 32-bit armv7l Annapurna Labs Alpine AL214 1.7 GHz quad-core Cortex-A15 CPU, which is capable enough to generate a substantial amount of traffic.
Since there isn't a block diagram or any other detailed documentation for the SOC or the NAS itself, I spent some time looking around to get a better feel for what this SOC is like inside and how QNAP put this thing together. Sharing it here in case this is useful for someone in the future.
Looks like they just ran with (or built on) a reference dtree for the devboard. :)
[/sys/firmware/devicetree/base] # cat model
Annapurna Labs Alpine Dev Board
[/sys/firmware/devicetree/base] # cat compatible
annapurna-labs,alpineal,alpine
[/sys/firmware/devicetree/base] # cat chosen/bootargs
boot_ver=1.73.6 pci=pcie_bus_perf console=ttyS0,115200 root=/dev/ram mtdparts=mx_nand:32M(boot1_kernel),216M(boot1_rootfs2),32M(boot2_kernel),216M(boot2_rootfs2),15M(config);spi0.0:1088K(loader)ro,384K(env)
Although I can't find detailed specs on the AL214 CPU anywhere it looks like this system's SFP+ port is integrated on the SOC itself via an PCIe 2.0 bus.
[/sys/class/pci_bus] # ls -al
drwxr-xr-x 2 admin administ 0 Jul 1 17:27 ./
drwxr-xr-x 53 admin administ 0 Jul 1 17:14 ../
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 0000:00 -> ../../devices/platform/soc/fbc00000.pcie-internal/pci0000:00/pci_bus/0000:00/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 0001:00 -> ../../devices/platform/soc/fd800000.pcie-external0/pci0001:00/pci_bus/0001:00/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 0001:01 -> ../../devices/platform/soc/fd800000.pcie-external0/pci0001:00/0001:00:00.0/pci_bus/0001:01/
[~] # lspci -vt
-+-[0001:00]---00.0-[01]----00.0 Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
\-[0000:00]-+-00.0 Annapurna Labs Ltd. SFP+ 10G Ethernet Adapter
+-01.0 Annapurna Labs Ltd. Gigabit Ethernet Adapter
+-03.0 Annapurna Labs Ltd. Gigabit Ethernet Adapter
+-04.0 Annapurna Labs Ltd. Device 0011
+-04.1 Annapurna Labs Ltd. Device 8011
+-05.0 Annapurna Labs Ltd. Device 0021
+-05.1 Annapurna Labs Ltd. Device 8021
\-09.0 Annapurna Labs Ltd. Device 0031
[~] # lspci -vvD -s 0000:00
0000:00:00.0 Ethernet controller: Annapurna Labs Ltd. SFP+ 10G Ethernet Adapter (rev 01)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 255
Region 0: Memory at fe000000 (64-bit, non-prefetchable) [size=128K]
Region 2: Memory at fe154000 (64-bit, non-prefetchable) [size=4K]
Region 4: Memory at fe140000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE- FLReset+
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [90] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=0001d000
PBA: BAR=0 offset=0001dff0
Capabilities: [100 v1] Vendor Specific Information: ID=1c36 Rev=0 Len=200 <?>
Capabilities: [300 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 3, Total VFs: 3, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 8002
Supported Page Size: 0000001f, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: al_eth
[~] # cat /sys/module/al_eth_drv/version
0.2
[~] # ls /sys/devices/platform/soc
driver_override fd882000.spi/ fd8a8000.pinctrl/
fa100000.nand-flash/ fd883000.uart0/ fd8c0000.serdes/
fb001000.gic_main/ fd884000.uart1/ modalias
fb070000.nb_service/ fd885000.uart2/ of_node@
fb080000.mc/ fd887000.gpio0/ soc:arch-timer/
fb090000.ccu/ fd888000.gpio1/ soc:pmu/
fbc00000.pcie-internal/ fd889000.gpio2/ soc:reboot/
fbe00000.msix/ fd88a000.gpio3/ soc:sata_sw_leds/
fbff5ec0.cpu_resume/ fd88b000.gpio4/ subsystem@
fd800000.pcie-external0/ fd88c000.wdt0/ uevent
fd860a00.thermal/ fd897000.gpio5/
fd880000.i2c-pld/ fd8a8000.pbs/
[~] # find /sys/devices | grep 0000:00:00.0$
/sys/devices/platform/soc/fbc00000.pcie-internal/pci0000:00/0000:00:00.0
[~] # cat /proc/cpuinfo
processor : 0
model name : Annapurna Labs Alpine AL214 Quad-core ARM Cortex-A15 CPU @ 1.70GHz
Speed : 1.7GHz
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc0f
CPU revision : 4
[x4]
Hardware : Annapurna Labs Alpine
Revision : 0000
Serial : 0000000000000000
[~] # cat /proc/version
Linux version 4.2.8 (root@U16BuildServer176) (gcc version 4.8.2 20131014 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2013.10 - Linaro GCC 2013.10) ) #2 SMP Tue May 31 07:18:55 CST 2022
[~] # cat /proc/cmdline
pci=pcie_bus_perf console=ttyS0,115200 root=/dev/ram mtdparts=mx_nand:32M(boot1_kernel),216M(boot1_rootfs2),32M(boot2_kernel),216M(boot2_rootfs2),15M(config);spi0.0:1088K(loader)ro,384K(env) vmalloc=560M zswap.enabled=0 zswap.compressor=lz4 memmap=2M$0x7000000 ramoops.mem_address=0x7000000 ramoops.mem_size=0x200000 ramoops.console_size=0x100000
[/sys/devices/platform/soc/fd882000.spi] # cat /proc/mtd
dev: size erasesize name
mtd0: 00110000 00001000 "loader"
mtd1: 00060000 00001000 "env"
mtd2: 02000000 00020000 "boot1_kernel"
mtd3: 0d800000 00020000 "boot1_rootfs2"
mtd4: 02000000 00020000 "boot2_kernel"
mtd5: 0d800000 00020000 "boot2_rootfs2"
mtd6: 00f00000 00020000 "config"
[~] # ls -alh /sys/class/mtd
drwxr-xr-x 2 admin administ 0 Jul 1 17:27 ./
drwxr-xr-x 53 admin administ 0 Jul 1 17:14 ../
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd0 -> ../../devices/platform/soc/fd882000.spi/spi_master/spi0/spi0.0/mtd/mtd0/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd0ro -> ../../devices/platform/soc/fd882000.spi/spi_master/spi0/spi0.0/mtd/mtd0ro/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd1 -> ../../devices/platform/soc/fd882000.spi/spi_master/spi0/spi0.0/mtd/mtd1/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd1ro -> ../../devices/platform/soc/fd882000.spi/spi_master/spi0/spi0.0/mtd/mtd1ro/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd2 -> ../../devices/virtual/mtd/mtd2/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd2ro -> ../../devices/virtual/mtd/mtd2ro/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd3 -> ../../devices/virtual/mtd/mtd3/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd3ro -> ../../devices/virtual/mtd/mtd3ro/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd4 -> ../../devices/virtual/mtd/mtd4/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd4ro -> ../../devices/virtual/mtd/mtd4ro/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd5 -> ../../devices/virtual/mtd/mtd5/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd5ro -> ../../devices/virtual/mtd/mtd5ro/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd6 -> ../../devices/virtual/mtd/mtd6/
lrwxrwxrwx 1 admin administ 0 Jul 1 19:00 mtd6ro -> ../../devices/virtual/mtd/mtd6ro/
The MTDs can be dumped from userland via nanddump in the the entware nand-tools package. Managed to dump all of the MTDs without much issue. Might be fun to dig into the binaries sometime.
Oh, what the heck. Why not.
mtd0 looks like our main bootloader, complete with an init script and FDTs to get things going.
# binwalk mtd0.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
458824 0x70048 uImage header, header size: 64 bytes, header CRC: 0x35428B04, created: 2019-12-13 06:11:07, image size: 1980 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0xEC0DCBE9, OS: Firmware, CPU: ARM, image type: Script file, compression type: none, image name: "init_script"
773888 0xBCF00 CRC32 polynomial table, little endian
915704 0xDF8F8 Flattened device tree, size: 285 bytes, version: 17
1048648 0x100048 Flattened device tree, size: 19152 bytes, version: 17
mtd1 just looks like a bunch of ASCII data at various offsets, no actual binary stuff. Most interesting thing was baudrate=115200, I guess?
mtd2 dumped without any issues too. Pretty much what I'd expect from a boot kernel.
# binwalk mtd2.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 uImage header, header size: 64 bytes, header CRC: 0xADF824E4, created: 2022-05-30 23:19:00, image size: 26278072 bytes, Data Address: 0x8000, Entry Point: 0x8000, data CRC: 0x533557FC, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-4.2.8"
64 0x40 Linux kernel ARM boot executable zImage (little-endian)
17944 0x4618 gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
7514985 0x72AB69 GIF image data 27946 x
mtd3 dumped fine, but it ran into an ECC bitflip! Neat. Looks like a UBI image of the rootfs.
# nanddump -f ./mtd3.bin /dev/mtd3
ECC failed: 0
ECC corrected: 1
Number of bad blocks: 0
Number of bbt blocks: 0
Block size 131072, page size 2048, OOB size 64
Dumping data starting at 0x00000000 and ending at 0x0d800000...
ECC: 1 corrected bitflip(s) at offset 0x09841000
# file mtd3.bin
mtd3.bin: UBI image, version 1
mtd4 and mtd5 are basically the same, probably just a second slot for OTA-style firmware updates.
mtd6 is also a UBI image, but a its a lot smaller than the others. I'm guessing this is just a bunch of files representing basic config stuff that needs to referenced during/after initial boot but before mounting disks and stuff.
[~] # ls -alh /sys/class/gpio
drwxr-xr-x 2 admin administ 0 Jul 1 17:14 ./
drwxr-xr-x 53 admin administ 0 Jul 1 17:14 ../
--w------- 1 admin administ 32.0k Jul 1 07:54 export
lrwxrwxrwx 1 admin administ 0 Jul 1 17:14 gpio0 -> ../../devices/platform/soc/fd887000.gpio0/gpio/gpio0/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpio28 -> ../../devices/platform/soc/fd88a000.gpio3/gpio/gpio28/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpio3 -> ../../devices/platform/soc/fd887000.gpio0/gpio/gpio3/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:14 gpio32 -> ../../devices/platform/soc/fd88b000.gpio4/gpio/gpio32/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpio4 -> ../../devices/platform/soc/fd887000.gpio0/gpio/gpio4/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpio42 -> ../../devices/platform/soc/fd897000.gpio5/gpio/gpio42/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpio43 -> ../../devices/platform/soc/fd897000.gpio5/gpio/gpio43/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpiochip0 -> ../../devices/platform/soc/fd887000.gpio0/gpio/gpiochip0/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpiochip16 -> ../../devices/platform/soc/fd889000.gpio2/gpio/gpiochip16/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpiochip24 -> ../../devices/platform/soc/fd88a000.gpio3/gpio/gpiochip24/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpiochip32 -> ../../devices/platform/soc/fd88b000.gpio4/gpio/gpiochip32/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpiochip40 -> ../../devices/platform/soc/fd897000.gpio5/gpio/gpiochip40/
lrwxrwxrwx 1 admin administ 0 Jul 1 17:27 gpiochip8 -> ../../devices/platform/soc/fd888000.gpio1/gpio/gpiochip8/
[/sys/class/gpio] # for i in `ls | grep -E "gpio[0-9]" | sort -V | sed -e 's/@//'`; do dir=`cat $i/direction`; val=`cat $i/value`; echo $i $dir $val; done
gpio0 in 1
gpio3 out 0
gpio4 out 1
gpio28 out 1
gpio32 in 1
gpio42 in 1
gpio43 in 1
There's a 4-pin JST header on the mainboard that provides a serial connection, labelled UART0_DEBUG. This provides kernel output during boot.
A 4-pin JST header labelled MCU_ISP is next to this but I didn't play with this one. Its likely for the Weltrend MCU (details below) which probably has onboard flash of its own.
Near the CPU is an 8-pin header labelled AL_SPI1. Although its labed like its SPI related its shape and location make it likely to be an ARM JTAG header. Could do double duty as an SPI header for flashing nearby flash chips depending on what pins its connected to though. Didn't play with this one either.
The DDR3L-1600 (PC3-12800) SDRAM is in an SO-DIMM socket, which is nice. It shipped with a Transcend 2 GB (1Rx8) module but after an RMA swap it was replaced with a Micron MT8KTF25664HZ-1G4M1 2GB (1Rx8) DDR3L-1333 CL11 module after an RMA, which fixed a broken SATA port. I replaced the original 2 GB module with a Patriot PSD38G1600L2S 8 GB, CL11, 1.35V module.
The SFP+ cage looks like its connected directly to the MCU, while the two gigabit Ethernet ports are connected through a pair of Atheros AR8035 10/100/1000 transcievers, one each.
USB is handled by an EtronTech EJ188 USB 3.0 host controller. Not much special there.
There's an unpopulated header which looks like its for a Mini-PCIe socket. Not too remarkable. Just unpopulated with a few unpopulated discrete component pads around it.
Between that socket and the USB chip is a 9FG104EGLF chip. I didn't trace the pins but at a glance it looks like its providing a clean clock for the PCIe bus.
Near the two 4-pin headers mentioned above, there's a Weltrend WT61P803 MCU onboard, which I couldn't find a datasheet for. However, its likely part of the same "Turbo 8052 CPU" family as the WT61P808 and WT61P809. Based on its position on the board, near a SN74AVC4T245 dual-bit transceiver and 74LVC32AD quad 2-input OR gate, a few discrete transisors, the CR2032 RTC battery, and the front panel LEDs/buttons its likely that this MCU's handling LEDs and basic power control stuff. This has a spot of blue ink on it, indicating that its probably pre-flashed prior to pick-and-place during manufacturing.
Primary firmware storage is handled by a Micron SLC NAND chip, FGBA Code NW189, which looks like its a MT29F4G08ABBDAH4. On my unit this is marked with a dab of orange ink on the corner.
Between the NAND and the AL_SPI1 header is a GD25LQ16CSIG 16 Mb Quad-SPI NOR flash chip. This has a dab of green ink on it. This might be exposed to the OS as mtd0/mtd1, given that there's a nor type MTD on an SPI bus under /sys/devices/platform/soc.
Also next to the AL_SPI1 header is a generic looking 24C16 16Kb EEPROM, which looks like an I2C chip. There's a spot of red ink on this one.
On the I2C bus there's a what appears to be a PIC16F722A microcontroller (or something compatible) that I didn't spot while taking photos of the board, which is interesting. Its visible under /sys/class/i2c-dev/i2c-0/device/0-0018. I'm curious if this the Weltrend device above or if its something else living somewhere else on the board?
There's also a PCA9543 I2C bus switch out there somwehere that I didn't spot on the visual inspection, under /sys/class/i2c-dev/i2c-0/device/0-0070.
Then there's the RTC chip, an Epson RX8010 (or compatible), under /sys/class/i2c-dev/i2c-0/device/0-0032.
I also spotted a thermistor by the SATA riser connector. Its labelled RT2 so there might be another somewhere on the board I didn't spot.