Awasu
Saturday 5th March 2022 3:09 PM [Tutorial]

I've always been fond of the phrase "cattle, not pets", which refers to the idea that computer servers should be treated as cattle (i.e. you should have no problem killing them), as opposed to pets. One of the most important changes over the 35+ years I've been a professional developer is the rise of automating processes. Back in the day, if you wanted to set up a new server, you did it manually, carefully installing all the software and other dependencies, then even more carefully configuring them, and since you didn't want to have to do that work again[1]And it was often the case that you couldn't re-create them, even if you wanted to, because of all the minor tweaks and changes that were invariably made over time, that didn't get documented., these servers were treated as precious pets. But today, with the rise of technologies such as Ansible and containers, servers are disposable - if one fails, just throw it away and run a script to create a new one.

This approach introduces some new considerations (e.g. managing a fleet of servers, re-creating them when they fail, etc.), giving rise to a new class of software known as container orchestration. The king of these is Kubernetes, and since I recently did a bit of work with this, I wanted to set up a local instance for testing. While there are things like minikube, that let you set up a local cluster on your PC, there's nothing like a proper test environment that mirrors a real production environment as closely as possible.

The trend these days is, of course, to do everything in the cloud, so there's no shortage of information on how to set things up using e.g. AWS or GCP, but rather less on how to set up a bare-metal local cluster, so we'll remedy that here with a set of instructions on how to set up a local Kubernetes instance that has:

  • a single VM that provides the control plane
    It's possible to have a single server manage the control plane and act as a node (i.e. run containers), but in the interests of making this cluster as "real" as possible, we'll separate them out.
  • two more VM's that will act as nodes[2]These will actually run the containers.
    We want two of these, so that we can test things like distributing workloads over multiple servers, automatic failover if a server goes down, etc.
  • dynamically-provisioned disk space
    This gives us persistent disk storage, even as servers come and go.
  • a local Docker registry to store images
    So that we don't have to put them on Docker Hub.

Tutorial index

References

References
1 And it was often the case that you couldn't re-create them, even if you wanted to, because of all the minor tweaks and changes that were invariably made over time, that didn't get documented.
2 These will actually run the containers.
Monday 10th January 2022 2:07 PM [Tutorial]

Yeah, it's been a while :| I've been quietly chugging away in the background on Awasu client work, as well as other non-Awasu projects[1]And to be honest, I haven't felt much like writing :| , but it's been some time since my last mega-tutorial, so let's remedy that with a deep dive into the internals of everyone's favorite source control system, git.

This tutorial assumes that you are familiar with using git (e.g. commits, branches, tags), and we'll take a look at the internals of git and how it works, and in particular, its file formats.

References

References
1 And to be honest, I haven't felt much like writing :|
Monday 4th November 2019 4:51 PM [Tutorial]

A few years ago, I wrote a long series of tutorials showing how to embed Python into a C/C++ program, and periodically threatened to write another series showing how to go the other way i.e. extend Python by calling your own C/C++ code[1]Typically because you want better performance, or because you want to run it multi-threaded, which Python is known to not handle very well..

Well, I've finally made good on that promise and written some tutorials on how to write a Python extension module:

A minimal setup

How to create a minimal Python extension module.

Calling functions

How to define a C/C++ function and call it from Python.

Handling errors

How to pass errors back to the calling Python code.

Callbacks

How to call back into Python from the C/C++ code.

Managing the GIL

How to manage the GIL, for better multi-threaded performance.

Never let it be said that I don't keep my promises! :roll:

References

References
1 Typically because you want better performance, or because you want to run it multi-threaded, which Python is known to not handle very well.
Wednesday 30th October 2019 2:26 PM [General, Tutorial]

Many moons ago, I wrote a tutorial on how to set up an internet gateway on a Banana Pi, complete with DHCP, DNS, VPN, firewall and ad-blocking. It works well, I still use one today, and have even taken it with me on a few long backpacking trips. However, I worry about it being a bit fragile, and fear the day when an over-zealous customs officer decides it looks like something that could trigger a bomb :| , so I was overjoyed when I finally found my holy grail: something that does all of the above, in the form factor of a USB thumb drive.

GL-iNet's GL-USB-150 costs around USD 30, and comes with almost everything I need to get online when I'm on the road. This tutorial will be much shorter than the previous one, because nearly everything is already set up and ready to go :clap:

Getting started

Plug it in, give it 30 or 40 seconds to start up, then open a browser and go to http://192.168.8.1. To login, the default password is goodlife; once you're in, change this under More Settings/Admin Password.

It runs a DHCP server, and your computer will have already been assigned an IP address in the 192.168.8.xxx range.

Go to the Internet page, click on the Scan link, then connect to a WiFi network.

Open another browser window, and confirm that you're online.

Configuring the VPN

Go to the Management tab in the VPN/OpenVPN Client page, and upload your VPN configurations. This will typically be a ZIP of a bunch of .ovpn files, but if you have them, you will also need to include the .crt and/or .pem files.

Unfortunately, the stock firmware has a bug that prevents the ZIP file from being processed correctly, so you will need to upgrade the firmware first. Get the latest version from here[1]Version 3.026 worked for me., then install it via the Upgrade page.

Once the VPN configurations have been installed, you will be able to select which one to use from the VPN/OpenVPN Client page. Check your IP address to confirm that you are going through the VPN.

Installing software

To install additional software, go to the More Settings/Advanced page, and in the new browser window that opens, go to System/Software and update the package lists[2]This doesn't seem to persist after a reboot, so you have to remember to do this every time :-| .

I installed the following packages:

  • bash
  • tmux
  • openssh-sftp-server (so that I can scp files in and out)
  • openssh-client (for a version of ssh that allows forwarding)
  • coreutils (for GNU tools)
  • bind-dig (for dig)
  • mtr (a handy network monitoring tool)

To change your default shell to bash, update /etc/passwd.

The only down-side to this device: while you can just about install a minimal version of Python, the disk is so small, there won't be any room for anything else :-(

Ad-blocking

The only thing missing from this device is an ad-blocker. Since it uses dnsmasq for DNS, rather than bind as the Banana Pi does, the process is slightly different, but not much. Here's the script that I use:

# This script downloads blacklisted ad servers and updates dnsmasq to block them.
#
# The following line needs to be added to /etc/dnsmasq.conf:
#   conf-file=/root/dns-blacklist

BLACKLIST_URL="http://pgl.yoyo.org/as/serverlist.php?hostformat=dnsmasq&mimetype=plaintext"
BLACKLIST_FNAME=/root/dns-blacklist

echo "Downloading the DNS blacklist..."
TMP_FNAME=/tmp/dns-blacklist
wget -O "$TMP_FNAME" "$BLACKLIST_URL"
if [ $? -ne 0 ] ; then exit 1 ; fi
echo

# fixup the entries so that they return "NX Domain"
echo "Updating the DNS blacklist..."
sed -i 's/address/server/g;s/127.0.0.1//g' "$TMP_FNAME"
echo

# install the new DNS blacklist
echo "Installing the DNS blacklist..."
echo "  $TMP_FNAME => $BLACKLIST_FNAME"
mv "$TMP_FNAME" "$BLACKLIST_FNAME"
echo "Restarting dnsmasq..."
/etc/init.d/dnsmasq restart
echo

echo "All done."

The DNS blacklist is downloaded to a temp file, fixed up and then transferred to /root/dns-blacklist. You will need to tell dnsmasq to load this file by adding the following line to /etc/dnsmasq.conf:

conf-file=/root/dns-blacklist

This script can be configured to run periodically, or just run it manually every now and then.

Shutting down

There doesn't seem to be any way to shut down the device cleanly. I'm guessing it's been designed so that people can just pull the thing out of the USB port, but this really irks the sysadmin in me :roll: , so to shutdown cleanly, type the following in the console:

halt

The green LED light stays on, but the device will shutdown.

References

References
1 Version 3.026 worked for me.
2 This doesn't seem to persist after a reboot, so you have to remember to do this every time :-|
Thursday 26th April 2018 6:22 PM [Awasu News]

Another year, another Awasu Day, and it's with great pleasure I can release the latest and greatest in the long line of Awasu releases: Awasu v3.2.

This release is faster, slicker and just damn prettier than ever before, with numerous improvements, in particular, to the search engine[1]Many thanks to Jacek for prodding me into action on this..

Check it out, not least of all because there will be, yet again, no price increase :clap: , and if you purchased in the last 2 years, this release will be included as part of that i.e. completely free.

And once you're up and running, don't forget about one of the key features of Awasu, it's extensibility, with many free plugins here (e.g. skins, Office integration, geo-location and translation tools), as well as a bunch of paid ones e.g. monitoring:

Have fun with this one, and I'm definitely off to the pub... :jig:

References

References
1 Many thanks to Jacek for prodding me into action on this.
Sunday 18th March 2018 9:02 AM [Awasu News]

One thing that was quietly introduced in the recent 3.2.rc1 release was support for running on Linux, under Wine. This is something that I've been looking at for quite a while already, and Awasu has run reasonably well like this, but I've finally bit the bullet and made some changes to the code to help with the process.

Most of the issues relate to the embedded browser. They seem to have included a version of Gecko, tricked out to look like Internet Explorer, which mostly works, but Awasu has some sophisticated interactions with its embedded browser, which may or may not work. All the issues I came across have been documented in the wiki, along with work-arounds and other notes.

I'm one of those people who have become increasingly disillusioned with the direction Windows has taken recently, and while I've used Linux on the server for many years[1]And UNIX before that :eek: , it's never been much fun on the desktop. However, Gnome on Fedora is not bad these days, and given that my recent foray into the iUniverse has left me scarred for life, it looks like Linux might it, and so this version of Awasu will be getting a lot more dog-fooding in the near future... :-)

References

References
1 And UNIX before that :eek:
Saturday 10th March 2018 9:38 AM [Awasu News]

Awasu 3.2.rc1 has been released here. This is a release candidate for 3.2, so it comes as an installer, and all you need to do is run it and it will auto-magically upgrade your installation[1]You must be running Awasu 3.0 or later..

This release contains numerous optimizations and performance improvements to speed up all operations, and... nah, just kidding :-) , all that work's been done already. This release contains a few bug fixes and UI tweaks, but there is one change of note: the Personal Edition now allows SSL feeds i.e. feeds that use the more secure https:// URL's (instead of http://).

There's been a big push over the past year or two to get people to use SSL, in particular with the major browsers slowly starting to flag web sites that don't use it as potentially insecure. SSL feeds had previously been available only in paid versions of Awasu, but can now be used even in the free Personal Edition, as well. It's a small thing, in the grand scheme of things, but it's our contribution to helping make the web a safer place ::-):

Have fun, and if all goes well, this will become the final 3.2 release.

References

References
1 You must be running Awasu 3.0 or later.
Thursday 18th January 2018 6:34 PM [Awasu News]

Awasu 3.1.1 has been released here.

Lots of Awasu goodness in this release, with numerous changes to speed up the search engine, and also to optimize the shutdown process, which is much quicker and snappier now.

This release cycle will be shorter than the others, so get yourself set up with this one; the 3.2 release candidate is just around the corner... :jig:

Monday 11th December 2017 7:49 PM [Awasu News]

Just a quick alpha release to tie up some loose ends from the rather extensive work done in the previous release. Nothing earth-shattering to report, just more of the Awasu goodness we all know and love :-) , and the 3.1.1 beta will be hot on its heels in the new year...

Thursday 30th November 2017 5:11 PM [General]

OMG, this is just so freaking cool!

I can't remember exactly how, but I stumbled on these a few days ago, and I simply had to order a bundle, and they just arrived today :jig:

If you don't understand what it means, vi is a text editor for developers that is, while extremely powerful, notoriously difficult to learn and use, and :wq is what you have to type in to save your file and exit - ":" puts you into "command" mode, "w" writes the file out, and then "q" to quit.

This is, of course, not particularly intuitive and it's a rite of passage for Linux newbies to get insanely frustrated trying to use Ctrl-C to exit the program, to no effect, to the point where the developer of a popular clone felt it would be worthwhile to show a hint about how to exit if the user presses Ctrl-C[1]Although even doing this doesn't always help :-D .

The story of how these stickers came to be can be found here, and if you feel like getting a few for yourself, they're only a buck each! Even shipping to Australia was an extremely reasonable $2.20, so you really have no excuse, no matter where you are in the world.

I'm doing a lot of work right now for a client where I'm stuck on a back-end server, writing Python in vim[2]Lots of print() statements :cry: I got a license for PyCharm, but that turned out to be a complete waste of time :wall: , and while I'm not one for putting stickers on my laptop, one these these will brighten up my day enormously.

Almost enough to overcome the ignominy of having to use a Mac[3]Almost. :-(

References

References
1 Although even doing this doesn't always help :-D
2 Lots of print() statements :cry: I got a license for PyCharm, but that turned out to be a complete waste of time :wall:
3 Almost.