The Approximately Monthly Zoomer

Server Upgrade 1: The Case of Constrained Case Constraints

2025-05-14

Thinking outside the box is overrated. Having a set of constraints and trying to get the absolute most out of the situation is usually where true creativity shines. In this case the constraints are literal constraints of a case.

Hardware Upgrade

Every once in a while you encounter a deal too good to pass up on - such was the case when I saw a black friday deal for 64GB of DDR4 memory. One thing lead to another and I also bought a used motherboard and processor. Together with the PSU and SSDs I already had lying around this makes a server.

…

The case! I completely forgot about a case!

Thinking About the Box

My current server resides beneath the television, in the space I’m told has historically been reserved for magnetic tape readers of some sort. Since we don’t need that level of archival backups, this space has been unoccupied - perfect for my server. Having used a small convertible minitower by HP up until now, I never paid much attention to the height of the available space, but as it turns out, regular ATX cases aren’t really available in widths of less than 13cm.

At first, I thought about screwing all parts onto a plastic board and creating a basic cover out of the same material but then I found a cheaper, more readily available material - cardboard. I totally didn’t buy an appropriately sized plastic board and lose it on my way home.

The Signature Box 2

I’m not going to move the server about a lot and it doesn’t have to look particularly nice (just inconspicuous). It should be somewhat shielded from dust and most importantly: it needs to be done yesterday.

I got to cutting, gluing, creasing, bending, folding, taping and painting my new case. I hope its design is more of an insight into the arbitrary deadline I’ve set myself than it is a representation of my crafting abilities.

Cardboard isn’t exactly known for its structural integrity, so the more complex and intricate case design will have to wait until I get my hands on another plastic board.

enp432981s420wlx24017w

Installing Proxmox went fine, their installer does everything it needs to do, nothing to see here. Which is precisely what I wanted, to see nothing, so I turned off the machine, removed the GPU, and restarted it.

I am unsure what exactly was happening but the server didn’t boot anymore - at least not in the way I wanted it to. At first I thought the UEFI wants to show me that the hardware changed but it can’t since there is no GPU (there is no integrated graphics either (Ryzen)). But what I think was actually happening, is that the name of the network interface changes when the GPU is no longer plugged in, something about PCI enumeration, and this confuses Proxmox, which is actually booting but unreachable from the network. There is a random piece of paper somewhere where I’ve written down exactly how I solved this problem, but since the problem is solved and this piece of paper no longer needed, it probably resides somewhere in the pocket dimension where pens teleport to when they fall off the table, never to be seen again. Something about changing the interface name to eth0 I think it was.

What’s in the Box?

My case doesn’t really look particularly nice - one might even call it laughably hideous. Maybe I could take the attention away from the case itself and direct it toward the actual hardware with all of the RGB everywhere. The CPU fan, all four RAM modules, and the motherboard greet you with the finest unicorn themed light show when you turn it on. Unfortunately, how ever pleasant this display of lights may be, it shouldn’t distract from the quotidian televiewing experience, so now I have the pleasure of installing OpenRGB onto Proxmox and create a systemd unit so I can turn off the lights after boot. Such fun. I just need it to run openrgb --mode static --color 000000 on boot, how hard can it be?

[Unit]
Description=Turn off unicorn vomit

[Service]
ExecStart=/usr/bin/opt/openrgb --mode static --color 000000

[Install]
WantedBy=multi-user.target

Turns out you need a metric tonne of dependencies for OpenRGB, many of which I would rather not install on the same OS as Proxmox, not only for security reasons but also for maintainability reasons - the less my proxmox install deviates from the default, the less exotic and unique problems I’ll have to solve. Many weeks of trying different things and procrastinating later I was too frustrated with this whole light thing and just bought a cheap second-hand atx case (that is too wide for my constrained space of course). Not having the server neatly tucked in underneath the television is a visual burden I will have to bear after all, assuaged only by the tenebrous lack of RGB illumination. So much for my constrained case creativity. At least I can now focus on the important part - finally migrating all of my old stuff onto a new server.

Grubbing for Files

2025-01-06

When you turn on your computer, your BIOS is trying to find some bootable files to start your installed operating system. During a Linux install, also when using any sort of encryption, you set up a boot partition for that. This is done so your UEFI can boot something known, which will later handle all your decrypting needs. The size of this partition, 512MB in my case, is set during the installation. Resizing this partition should you run into problems like not having enough space for upgrades is a little complicated but doable. If your drive is unencrypted, that is.

In case you install Linux on your laptop and don’t want anyone who “finds” it to be able to see all your files, this doable task becomes a much more complex undertaking. So of course we won’t be doing any of that and instead we’ll do something easy, dirty, naughty to fix the problem. But why would you even run into problems with your boot partition? How can it suddenly become too small?

The Very Hungry Ramdisk

When upgrading your kernel, the initial ramdisk is installed in your boot partition - a file called initrd.img-6.9.42-amd69 for example is copied to /boot. Usually the older version of the initial ramdisk and some other files are left there until the upgrade is complete and sometimes the current and last version are installed simultaneously in case there is something wrong with the newer one so you can still tell your bootmanager to boot from the old one.

The current initrd size for my debian installation is over 250MB - together with the other files, there isn’t enough space for two versions in the boot partition. With every update, the size of this initrd file grows and grows. Sometimes it is also exarcerbated by using custom or modified kernels (I wish my attention span was long enough to know if I have ever modified my kernel (I probably have at some point (and it’s probably arrowing my knee right now (I just checked and no I didn’t modify it in any way (so the arrowing is not completely self-inflicted))))).

So how do we cope with being too large? We’ll just delete some old stuff from /boot to make space, It’ll be fine.

It was Not Fine

> be me
> apt update && apt upgrade
> "not enough space in /boot, the installation will likely fail, are you sure you want to continue?"
> 🙂
> update-initramfs: failed for /boot/initrd.img-6.9.42-amd69 with 1
> dpkg: error processing package initramfs-tools (--configure)
> oh no! anyway...
> delete old files and move initrd somewhere else
> apt update && apt upgrade
> put back initrd manually
> reboot pc
> "Kernel Panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)"

At Least the Screen isn’t Blue

I’ve been using Linux for a while now and know my way around so I was fully aware that what I was doing was stupid, but determining on just how many layers this is stupid will probably require a few more falls off my bike.

It’s never nice to see your computer fail to boot, but at least this was completely self-inflicted and preventable. After adding a new initrd file to /boot it would be wise to tell your bootloader to actually use that file, wouldn’t it? As it turns out, I should have run update-grub2 after my little stunt because the boot entry was lacking the line that told the bootloader which initial ramdisk to load - no wonder it failed. What I had to do to fix it was to enter grub, press e to edit the entry, add initrd /initrd.img-6.9.42-amd69 at the bottom, and boot with F10. This got me back to my linux install, and from there I was able to run update-grub2 so it can generate the proper entries.

I Have Learned Nothing

Since I wanted to reinstall my OS anyways (or start distrohopping again) and rid my computer from the frankendebian it’s been running for the past 5 years, this would have been the perfect opportunity to do just that but after researching new distros and making a list of things I’d need to reinstall or reconfigure I suddenly lost the motivation to do that. Looks like I’ll be manually moving files from and to /boot for the foreseeable future. What works for the moment is moving all large files from /boot to ~/boot for example, then doing the upgrade, and removing the files from ~/boot if the boot was successful. Unfortunately, my attention span is so bad, the moment my laptop successfully boots, I forget I even did something weird to it. Don’t tell anyone but my ~/boot directory looks like this and at this point I’m too sentimental to delete the files:

config-6.10.11-amd64   config-6.12.5-amd64        initrd.img-6.12.12-amd64   System.map-6.11.9-amd64   vmlinuz-6.11.4-amd64
config-6.1.0-25-amd64  config-6.12.6-amd64        initrd.img-6.12.17-amd64   System.map-6.12.10-amd64  vmlinuz-6.11.5-amd64
config-6.10.9-amd64    config-6.12.9-amd64        initrd.img-6.12.19-amd64   System.map-6.12.11-amd64  vmlinuz-6.11.7-amd64
config-6.11.10-amd64   initrd.img-6.10.11-amd64   initrd.img-6.12.5-amd64    System.map-6.12.12-amd64  vmlinuz-6.11.9-amd64
config-6.11.2-amd64    initrd.img-6.1.0-25-amd64  initrd.img-6.12.6-amd64    System.map-6.12.17-amd64  vmlinuz-6.12.10-amd64
config-6.11.4-amd64    initrd.img-6.10.9-amd64    initrd.img-6.12.9-amd64    System.map-6.12.19-amd64  vmlinuz-6.12.11-amd64
config-6.11.5-amd64    initrd.img-6.11.10-amd64   System.map-6.10.11-amd64   System.map-6.12.5-amd64   vmlinuz-6.12.12-amd64
config-6.11.7-amd64    initrd.img-6.11.2-amd64    System.map-6.1.0-25-amd64  System.map-6.12.6-amd64   vmlinuz-6.12.17-amd64
config-6.11.9-amd64    initrd.img-6.11.4-amd64    System.map-6.10.9-amd64    System.map-6.12.9-amd64   vmlinuz-6.12.19-amd64
config-6.12.10-amd64   initrd.img-6.11.5-amd64    System.map-6.11.10-amd64   vmlinuz-6.10.11-amd64     vmlinuz-6.12.5-amd64
config-6.12.11-amd64   initrd.img-6.11.7-amd64    System.map-6.11.2-amd64    vmlinuz-6.1.0-25-amd64    vmlinuz-6.12.6-amd64
config-6.12.12-amd64   initrd.img-6.11.9-amd64    System.map-6.11.4-amd64    vmlinuz-6.10.9-amd64      vmlinuz-6.12.9-amd64
config-6.12.17-amd64   initrd.img-6.12.10-amd64   System.map-6.11.5-amd64    vmlinuz-6.11.10-amd64
config-6.12.19-amd64   initrd.img-6.12.11-amd64   System.map-6.11.7-amd64    vmlinuz-6.11.2-amd64

At least I finally gave ventoy a try - can definitely recommend! Also I’ve installed NixOS on my gaming pc and per NixOS EULA I now have to tell you that I’m a NixOS user and that you should also install NixOS. Nix OS. Nix OS.

VIScon 2024

2024-10-12

This year’s VIScon Symposium was again filled with interesting talks - a total of 25 in 4 tracks. Since these talks weren’t recorded or the slides publicly available, I wasn’t able to do my usual thing of enjoying the hallway track and watching the talks later, so I did what any muggle without a time-turner would do and just went to one talk per slot.

User Enumeration Based Profiling

If you are forgetful like me or (god forbid) don’t use a password manager, you sometimes forget, that you have already created an account on a specific platform. Sometimes you get a nice error, that says something along the lines of “this email address is already registered”. Surely this cannot be abused for mischievous purposes, right?

Automation, Automation, Automation

Mario thought he could try to abuse this usability feature. He looked around and found some websites that expose whether a specific email already has an account or not and then did what any hacker would do and automated this. In the end he did this for 111 webapps.

If you know exactly which platforms a user frequents, you can probably guess some things about them. The chances of a middle aged female teacher using HN and battle.net are probably low.

The main attribute Mario looked at for his talk is gender. Given a person has an account on sites A,B,C, and D, what is the probability they are male or female?

Lies, Damned Lies, and Statistics

Several approaches of answering these questions were outlined in his talk, some naive, some less so, and some using machine learning of course. The only question I had in the end was: How precise would a prediction based on the email address alone have been compared to these approaches? My thought was that marco.polo@gmail.com and thor420@me.com had a higher probability of being male compared to fairy.maria@hotmail.com and jessica.miller@outlook.com.

How to make Formula 1 drivers obsolete

Students of the AMZ explained how their cars use computer vision and various other sensor inputs to control their autonomous car and win races. It was a very nice overview of the tech stack they use and an approachable explanation of their systems.

Given the (unrelated) shitshow at the A2RL in Abu Dhabi a few months ago, I’d say we’re at least 5 years™ away from making F1 drivers obsolete, although Ferrari is doing their best to stay on a par with the autonomous racing league (next year they’ll be better for sure).

Hacking the SBB

When you use the builtin sharing feature of the SBB app to share your itinerary, the link consists of a part that is always the same and of several (I think it was 6 in this case) characters that change. You don’t have to be a genius to see that this is probably insecure. Lucas decided to just iterate over all possible combinations of these links and scrape all the different itineraries. As it turned out, these links not only show your full journey stop-to-stop, but also your coordinates or address if you used those features.

So what if you can see an anonymous itinerary?

Let’s look at all the people who are going to the airport. Statistically, 30% of those people enter their home address or location to search for an itinerary. You now have a list of adresses of people that are going to the airport and are, possibly, not going to be home for a few days. Do I need to go on?

Breaking and Entering

Julia, allegedly not named after the programming language^{[citation needed]}, penetration tester and VIS alumna, recounted some of her engagements and how she got into places, where she wasn’t supposed to be. Besides the usual smoking-break-tailgaiting she explained how social engineering in general can be used during engagements while casually flexing never having had to use her get-out-of-jail-free card.

A Year in AI

Max and Nicolas brought the audience up to speed in all things AI in The fsued news show all while tracking NVidia’s stock price and the $/MTok of the best™ models throughout the year. Following the style of the fnord Jahresrückblick, it was a lighthearted and informative talk, leading to lots of discussion with and amongst the audience - a wonderful talk to end the day with.

The Hallway Track

Next to the talks there were of course some company booths, one with the obligatory lockpicking challenge. After patiently waiting in line for the people ahead of me to pick the lock or give up, I gave it a try - 44 metric seconds 😎. Although I preferred the quality and diversity of the first two VIScons, it was a good event overall and I’m glad I went. I can’t wait for the next VIScon.

Hardware Hacking 2: Electric Boogaloo

2024-05-10

After dabbling in hardware hacking last time, I think I might have found my most recent passion. I’m switching to an ISP that allows you to have any router of your choosing attached, so I thought I could find one with OpenWrt support and install it for maximum customization. The router should support the IEEE standards 802.11ax and ac for the newer clients on the network and also n for internet of shit and legacy devices from hell (also known as “printers”). Also, it should be somewhat future-proof, support the latest stable branch of OpenWrt, and not break the bank. This lead me to the Asus RT-AX59U, which received OpenWrt support in November of 2023.

Git Blame

At the time of writing the OpenWrt wiki page for this router was lacking to say the least - the only information as to how to install OpenWrt onto the router was the commit message of the commit that added support for it. The general idea was this:

Set up a TFTP server with the intramfs firmware that is reachable by the router.Section 4
Connect to the router via UART.
Interrupt the boot sequence to get to a shell.
Download the intramfs image over TFTP and boot it.
Use sysupgrade to install OpenWrt permanently.

Sounds a little complicated but more than doable, so I went ahead and bought the router for 100 gold, fully confident I could do it. It all worked out fine in the end but I did encounter some unexpected problems.

Is this thing glued?

There are only two screws that are visible from the outside - they are located on the underside of the device, holding the base attached to the device. So I went ahead and unscrewed them. The easily visible bottom screws. I expected the base to fall off or at least be easy to remove, but the amount of force I had to use to remove it lead me to believe I missed some pin I had to push down or some hidden button I had to press. Sure, breaking the base would have been suboptimal, but I could have easily 3D-printed some sort of base if I broke it, so I went ahead and pulled like a maniac. The next two screws were somewhat hidden deep inside the enclosure, but nothing you can’t reach with a regular screwdriver. The second set of screws. With those removed, I again tried to pry away another layer. It was so hard to pull away that I had to ask someone else for help - both grasping the router with both hands and pulling it apart. A rather cartoonish scene, but it worked. I wasn’t sure how to proceed from here and pull away even more of its plastic casing but I didn’t need to since I already found what I needed.

UART

After removing the plastic, you immediately get greeted by four unpopulated pins. The UART header. On the other side, that is visible through a small window, we can see that the pins are labelled “RX, TX, GND, VCC”. The UART header, labelled. Before plugging in a USB TTL adapter, I quickly checked the baud rate with my oscilloscope. This step wasn’t really necessary since modern devices are pretty much always™ communicating at 115200 baud with 8 data bits, no parity bits and 1 stop bit - you’ll also see this referred to as 115200,8N1. Now that we know where to plug in and which parameters to use we can go ahead and attach the cables to the router, plug in the USB cable into a computer on one end, on the other end we enthusiastically plug into a micro USB port and…

A broken micro USB port.

…break it.

Enshittification of Local Business

I try to have all tools I need ready when starting with a project, because when I start, I really don’t want to stop. Unfortunately there weren’t any local businesses that sell USB TTL converters for a reasonable price. 30 gold for a CH340-based adapter? No thank you! Suddenly I became very creative and remembered that I have a raspberry pi somewhere, so I bought a coffee instead and headed home. The raspberry pi has a lot of GPIO pins, surely some of them can be used for UART. Checking some docs we can see that this is indeed the case. Pins 14 and 15 can be used after running raspi-config and enabling the serial console in the interfaces section (disable login shell, enable serial port).

Intermediate Egyptology 201

After waiting for what felt like an eternity for the raspberry pi to boot I was greeted by the minimal raspberry pi os desktop environment. The first thing I did was enable ssh and connect it to my network, so I wouldn’t have to directly interact with it. Within ssh I ran picocom -b 115200 -r -l /dev/serial0, started the router, and got nothing but garbled output with the occasional readable text. So garbled in fact, that it messed with my shell and turned it into a display of ancient Egyptian linguistics. It couldn’t have been the wrong baud rate, I checked with my oscilloscope and there wouldn’t have been entire blocks of text that are readable. I double and triple checked that all pins were properly connected and had good contact and thought that maybe the ssh connection was wonky. Maybe it works on the raspi but doesn’t get sent over ssh properly. When I entered the command on the raspi it was still garbled - but it was less garbled! Success! When I saw it was still garbled, but less so, it immediately clicked. The raspberry pi is overloaded and can’t keep up with Wi-Fi, bluetooth, ssh, a desktop environment, and UART at the same time! I disabled Wi-Fi and bluetooth, dropped down to the login shell with Ctrl + Alt + F2 (or was it F3? I don’t remember) and finally got ungarbled output.

Inspecting the Shell

We can interrupt the boot sequece by pressing the number (1|2|3|4|7|9) at the right time.

Please choose the operation: 
   1: Load System code to SDRAM via TFTP.
   2: Load System code then write to Flash via TFTP.
   3: Boot System code via Flash (default).
   4: Entr boot command line interface.
   7WAN Show April 19, 2024: Load Boot Loader code then write to Flash via Serial.
   9: Load Boot Loader code then write to Flash via TFTP.

You choosed 4

   4: Entr boot command line interface.

U-Boot 2022.04-rc1 (Aug 02 2023 - 10:59:04 +0800)
MT7986>

I choosed [sic] 4 to enter the U-Boot shell and then proceeded to download the OpenWrt intramfs image.

MT7986> setenv ipaddr 192.168.1.1
MT7986> setenv serverip 192.168.1.70
MT7986> tftpboot 0x46000000 rtax59u.bin
switch prereq:0
Using ethernet0@15100000 device
TFTP from server 192.168.1.70; our IP address is 192.168.1.1
Filename 'rtax59u.bin'.
Load address: 0x46000000
Loading: Got ARP REPLY, set eth addr (e1:ee:78:00:81:e5)
#################################################################
     #################################################################
     #################################################################
     #################################################################
     #################################################################
     #################################################################
     #################################################################
     #########################################
     12.6 MiB/s
done
Bytes transferred = 7278084 (6f0e04 hex)
MT7986>

Now we should be able to boot the downloaded image.

MT7986> bootm 0x46000000
## Loading kernel from FIT Image at 46000000 ...

And after a while, we are greeted with this beautiful ascii art:

BusyBox v1.36.1 (2024-03-22 22:09:42 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 23.05.3, r23809-234f1a2efa
 -----------------------------------------------------
root@OpenWrt:/#

Now we can login to the webinterface, install the sysupgrade image, reboot, bob’s your uncle.

No Screwing Around

Usually these kinds of products have some sort of tamper evident warranty sticker somewhere. I only realized where it was when I was reassembling it - it was on one of the screws of the second set removed.

A tamper evident sticker used on a screw.

But don’t worry about these stickers because in most countries the manufacturers can’t actually weasel their way out of a warranty replacement unless they can prove that you broke the device. Funnily enough, I didn’t even put these screws back in since they are not in contact with any metal (so are not used for grounding) and the clips of the plastic case are strong enough to hold up a person, let alone the case itself.

view all articles

1 older