Long time no post! When we announced that we were going to overshoot our October goal we promised some technical posts, and here’s the first one.
WISP includes automatic updates. That can be pretty scary, so naturally there are questions about our processes — what if we push a bad update?! First let’s start with the update channels.
Update channels
There will be three update channels you can select between. Our public update channels are as follows:
- Staging – this is our experimental branch, expect bugs to crawl in
- Beta – we think it’s ready for production but we’re still testing
- Production – this is the stable channel we assume most people will use
While we’ll still have our own internal testing, we’ll be utilising these channels to better test features & identify issues.
You’ll even be able to select different channels for different components. This means if you want to run your SFTP-server on the beta channel, but you want your daemon running on the production channel – you can!
What if it goes wrong?!
This is a common question we get. Because the updates are automatic, what if they go wrong? WISP’s bootstrapper will jump through the hoops you would normally jump through if a manual update goes wrong. Let’s follow along a WISP startup log:
[BOOTSTRAPPER] [INFO] Hello world! [BOOTSTRAPPER] [DEBUG] Registering projects... [BOOTSTRAPPER] [INFO] Project daemon is missing, forcing it to update... [BOOTSTRAPPER] [DEBUG] Running VersionChecker... [BOOTSTRAPPER] [DEBUG] Polling... [BOOTSTRAPPER] [DEBUG] bootstrapper - current: v1.0.0, latest: v1.0.0 [BOOTSTRAPPER] [DEBUG] daemon - current: v0.0.0, latest: v1.0.0 [BOOTSTRAPPER] [INFO] Update available for daemon! (current: v0.0.0, latest: v1.0.0; channel: production) [BOOTSTRAPPER] [INFO] Delaying startup until everything is updated... [BOOTSTRAPPER] [INFO] Updating daemon to version v1.0.0... [BOOTSTRAPPER] [DEBUG] Downloading new version of daemon... [daemon.wisp, package.json, package-lock.json] [BOOTSTRAPPER] [DEBUG] Downloading the file daemon.wisp... [BOOTSTRAPPER] [DEBUG] Verifying signature... [BOOTSTRAPPER] [INFO] File daemon.wisp is intact and signed by Stepan Fedotov <[email protected]>. [BOOTSTRAPPER] [DEBUG] Downloading the file package.json... [BOOTSTRAPPER] [DEBUG] Downloading the file package-lock.json... [BOOTSTRAPPER] [DEBUG] Installing dependencies... [BOOTSTRAPPER] [DEBUG] Executing npm install... [BOOTSTRAPPER] [INFO] Update finished for daemon, new version is v1.0.0. Proceeding to run the process...
So far everything looks normal. Here comes the simulated failure:
[BOOTSTRAPPER] [WARN] Failed starting process for daemon: Error: Process exited with code 127 [BOOTSTRAPPER] [INFO] Starting the updated version failed, rollbacking to previous version...
WISP will automatically rollback when it detects a failed update. But what if even the rollback goes wrong?! What would you do in this situation? Maybe try a fresh reinstall?
[BOOTSTRAPPER] [WARN] ROLLBACKED VERSION FAILED - DELETING PROJECT IN HOPE OF REINSTALL FIXING THE ISSUE.
In the case the rollback also fails, WISP will reinstall itself for you; leaving the game servers intact of course. It will also skip the version which caused the failure in the first place, downloading the last version that was running successfully on the machine.
Okay so what about MITM?!
The eagle eyed amongst you would have noticed in the startup log earlier that we’re using signature verification on each update, to verify it was signed & distributed by our team, here it is again:
[BOOTSTRAPPER] [DEBUG] Downloading the file daemon.wisp... [BOOTSTRAPPER] [DEBUG] Verifying signature... [BOOTSTRAPPER] [INFO] File daemon.wisp is intact and signed by Stepan Fedotov <[email protected]>.
Using signature verification we can protect from MITM attacks on peoples networks.
Okay, well what if your update server gets hacked?!
We have taken every step possible, including physical access keys, to lock everything down like a bank. But we aren’t going to make someones job easier by sharing anymore than that. However, if you look into our teams track record you’ll know we’re very security conscious and almost every design decision is made with security in mind.
Running with the idea that something does happen, though – virtually everything is containerised; minimising the impact on your host machine. We will also have procedures pre-planned for a breach event, and it wouldn’t go unnoticed.
Currently (literally) physical attacks are outside of our threat model, so we feel pretty confident knowing that’s how far things would need to get. But in case someone is worried about that too, none of our public company locations have access anyway.
Footnote
A lot of hosting companies particularly pay for their bandwidth usage. So they’ll be happy to hear our daemon is less than 1mb.
It’s time to say goodbye to manual updates, but that’s all for today folks! More sauce coming soon?
