Awasu
Monday 4th November 2019 4:51 PM [Tutorial]

A few years ago, I wrote a long series of tutorials showing how to embed Python into a C/C++ program, and periodically threatened to write another series showing how to go the other way i.e. extend Python by calling your own C/C++ code[1]Typically because you want better performance, or because you want to run it multi-threaded, which Python is known to not handle very well..

Well, I've finally made good on that promise and written some tutorials on how to write a Python extension module:

A minimal setup

How to create a minimal Python extension module.

Calling functions

How to define a C/C++ function and call it from Python.

Handling errors

How to pass errors back to the calling Python code.

Callbacks

How to call back into Python from the C/C++ code.

Managing the GIL

How to manage the GIL, for better multi-threaded performance.

Never let it be said that I don't keep my promises! :roll:

   [ + ]

1. Typically because you want better performance, or because you want to run it multi-threaded, which Python is known to not handle very well.
Wednesday 30th October 2019 2:26 PM [General,Tutorial]

Many moons ago, I wrote a tutorial on how to set up an internet gateway on a Banana Pi, complete with DHCP, DNS, VPN, firewall and ad-blocking. It works well, I still use one today, and have even taken it with me on a few long backpacking trips. However, I worry about it being a bit fragile, and fear the day when an over-zealous customs officer decides it looks like something that could trigger a bomb :| , so I was overjoyed when I finally found my holy grail: something that does all of the above, in the form factor of a USB thumb drive.

GL-iNet's GL-USB-150 costs around USD 30, and comes with almost everything I need to get online when I'm on the road. This tutorial will be much shorter than the previous one, because nearly everything is already set up and ready to go :clap:

Getting started

Plug it in, give it 30 or 40 seconds to start up, then open a browser and go to http://192.168.8.1. To login, the default password is goodlife; once you're in, change this under More Settings/Admin Password.

It runs a DHCP server, and your computer will have already been assigned an IP address in the 192.168.8.xxx range.

Go to the Internet page, click on the Scan link, then connect to a WiFi network.

Open another browser window, and confirm that you're online.

Configuring the VPN

Go to the Management tab in the VPN/OpenVPN Client page, and upload your VPN configurations. This will typically be a ZIP of a bunch of .ovpn files, but if you have them, you will also need to include the .crt and/or .pem files.

Unfortunately, the stock firmware has a bug that prevents the ZIP file from being processed correctly, so you will need to upgrade the firmware first. Get the latest version from here[1]Version 3.026 worked for me., then install it via the Upgrade page.

Once the VPN configurations have been installed, you will be able to select which one to use from the VPN/OpenVPN Client page. Check your IP address to confirm that you are going through the VPN.

Installing software

To install additional software, go to the More Settings/Advanced page, and in the new browser window that opens, go to System/Software and update the package lists[2]This doesn't seem to persist after a reboot, so you have to remember to do this every time :-| .

I installed the following packages:

  • bash
  • tmux
  • openssh-sftp-server (so that I can scp files in and out)
  • openssh-client (for a version of ssh that allows forwarding)
  • coreutils (for GNU tools)
  • bind-dig (for dig)
  • mtr (a handy network monitoring tool)

To change your default shell to bash, update /etc/passwd.

The only down-side to this device: while you can just about install a minimal version of Python, the disk is so small, there won't be any room for anything else :-(

Ad-blocking

The only thing missing from this device is an ad-blocker. Since it uses dnsmasq for DNS, rather than bind as the Banana Pi does, the process is slightly different, but not much. Here's the script that I use:

# This script downloads blacklisted ad servers and updates dnsmasq to block them.
#
# The following line needs to be added to /etc/dnsmasq.conf:
#   conf-file=/root/dns-blacklist

BLACKLIST_URL="http://pgl.yoyo.org/as/serverlist.php?hostformat=dnsmasq&mimetype=plaintext"
BLACKLIST_FNAME=/root/dns-blacklist

echo "Downloading the DNS blacklist..."
TMP_FNAME=/tmp/dns-blacklist
wget -O "$TMP_FNAME" "$BLACKLIST_URL"
if [ $? -ne 0 ] ; then exit 1 ; fi
echo

# fixup the entries so that they return "NX Domain"
echo "Updating the DNS blacklist..."
sed -i 's/address/server/g;s/127.0.0.1//g' "$TMP_FNAME"
echo

# install the new DNS blacklist
echo "Installing the DNS blacklist..."
echo "  $TMP_FNAME => $BLACKLIST_FNAME"
mv "$TMP_FNAME" "$BLACKLIST_FNAME"
echo "Restarting dnsmasq..."
/etc/init.d/dnsmasq restart
echo

echo "All done."

The DNS blacklist is downloaded to a temp file, fixed up and then transferred to /root/dns-blacklist. You will need to tell dnsmasq to load this file by adding the following line to /etc/dnsmasq.conf:

conf-file=/root/dns-blacklist

This script can be configured to run periodically, or just run it manually every now and then.

Shutting down

There doesn't seem to be any way to shut down the device cleanly. I'm guessing it's been designed so that people can just pull the thing out of the USB port, but this really irks the sysadmin in me :roll: , so to shutdown cleanly, type the following in the console:

halt

The green LED light stays on, but the device will shutdown.

   [ + ]

1. Version 3.026 worked for me.
2. This doesn't seem to persist after a reboot, so you have to remember to do this every time :-|
Thursday 26th April 2018 6:22 PM [Awasu News]

Another year, another Awasu Day, and it's with great pleasure I can release the latest and greatest in the long line of Awasu releases: Awasu v3.2.

This release is faster, slicker and just damn prettier than ever before, with numerous improvements, in particular, to the search engine[1]Many thanks to Jacek for prodding me into action on this..

Check it out, not least of all because there will be, yet again, no price increase :clap: , and if you purchased in the last 2 years, this release will be included as part of that i.e. completely free.

And once you're up and running, don't forget about one of the key features of Awasu, it's extensibility, with many free plugins here (e.g. skins, Office integration, geo-location and translation tools), as well as a bunch of paid ones e.g. monitoring:

Have fun with this one, and I'm definitely off to the pub... :jig:

   [ + ]

1. Many thanks to Jacek for prodding me into action on this.
Sunday 18th March 2018 9:02 AM [Awasu News]

One thing that was quietly introduced in the recent 3.2.rc1 release was support for running on Linux, under Wine. This is something that I've been looking at for quite a while already, and Awasu has run reasonably well like this, but I've finally bit the bullet and made some changes to the code to help with the process.

Most of the issues relate to the embedded browser. They seem to have included a version of Gecko, tricked out to look like Internet Explorer, which mostly works, but Awasu has some sophisticated interactions with its embedded browser, which may or may not work. All the issues I came across have been documented in the wiki, along with work-arounds and other notes.

I'm one of those people who have become increasingly disillusioned with the direction Windows has taken recently, and while I've used Linux on the server for many years[1]And UNIX before that :eek: , it's never been much fun on the desktop. However, Gnome on Fedora is not bad these days, and given that my recent foray into the iUniverse has left me scarred for life, it looks like Linux might it, and so this version of Awasu will be getting a lot more dog-fooding in the near future... :-)

   [ + ]

1. And UNIX before that :eek:
Saturday 10th March 2018 9:38 AM [Awasu News]

Awasu 3.2.rc1 has been released here. This is a release candidate for 3.2, so it comes as an installer, and all you need to do is run it and it will auto-magically upgrade your installation[1]You must be running Awasu 3.0 or later..

This release contains numerous optimizations and performance improvements to speed up all operations, and... nah, just kidding :-) , all that work's been done already. This release contains a few bug fixes and UI tweaks, but there is one change of note: the Personal Edition now allows SSL feeds i.e. feeds that use the more secure https:// URL's (instead of http://).

There's been a big push over the past year or two to get people to use SSL, in particular with the major browsers slowly starting to flag web sites that don't use it as potentially insecure. SSL feeds had previously been available only in paid versions of Awasu, but can now be used even in the free Personal Edition, as well. It's a small thing, in the grand scheme of things, but it's our contribution to helping make the web a safer place ::-):

Have fun, and if all goes well, this will become the final 3.2 release.

   [ + ]

1. You must be running Awasu 3.0 or later.
Thursday 18th January 2018 6:34 PM [Awasu News]

Awasu 3.1.1 has been released here.

Lots of Awasu goodness in this release, with numerous changes to speed up the search engine, and also to optimize the shutdown process, which is much quicker and snappier now.

This release cycle will be shorter than the others, so get yourself set up with this one; the 3.2 release candidate is just around the corner... :jig:

Monday 11th December 2017 7:49 PM [Awasu News]

Just a quick alpha release to tie up some loose ends from the rather extensive work done in the previous release. Nothing earth-shattering to report, just more of the Awasu goodness we all know and love :-) , and the 3.1.1 beta will be hot on its heels in the new year...

Thursday 30th November 2017 5:11 PM [General]

OMG, this is just so freaking cool!

I can't remember exactly how, but I stumbled on these a few days ago, and I simply had to order a bundle, and they just arrived today :jig:

If you don't understand what it means, vi is a text editor for developers that is, while extremely powerful, notoriously difficult to learn and use, and :wq is what you have to type in to save your file and exit - ":" puts you into "command" mode, "w" writes the file out, and then "q" to quit.

This is, of course, not particularly intuitive and it's a rite of passage for Linux newbies to get insanely frustrated trying to use Ctrl-C to exit the program, to no effect, to the point where the developer of a popular clone felt it would be worthwhile to show a hint about how to exit if the user presses Ctrl-C[1]Although even doing this doesn't always help :-D .

The story of how these stickers came to be can be found here, and if you feel like getting a few for yourself, they're only a buck each! Even shipping to Australia was an extremely reasonable $2.20, so you really have no excuse, no matter where you are in the world.

I'm doing a lot of work right now for a client where I'm stuck on a back-end server, writing Python in vim[2]Lots of print() statements :cry: I got a license for PyCharm, but that turned out to be a complete waste of time :wall: , and while I'm not one for putting stickers on my laptop, one these these will brighten up my day enormously.

Almost enough to overcome the ignominy of having to use a Mac[3]Almost. :-(

   [ + ]

1. Although even doing this doesn't always help :-D
2. Lots of print() statements :cry: I got a license for PyCharm, but that turned out to be a complete waste of time :wall:
3. Almost.
Saturday 23rd September 2017 1:25 PM [Awasu News]

And the Awasu juggernaut just keeps on a-rollin' on... :-)

If we were ever to adopt the WordPress practice of naming releases after people, the 3.1.1.alpha1 release would surely be called "Jacek". He's a data analyst in Europe who has been pushing Awasu into territories far beyond where it had gone before, particularly wrt its search engine. To be honest, I was impressed that Awasu held up as well as it did, but even I would have to admit that it was a bit sluggish and crashy when under load, and so we've been working closely together over the past few months, sorting all these issues out.

A lot of the improvements relate to search, both in how the search index gets updated, as well as how search results are generated. In one case, I managed to speed up the code by 50 times, which made preparing search results significantly faster ::-): It's rare to get this kind of a win, so yes, I was definitely jumping around the room, pumping my fist in the air :jig: And since that bit of code is used elsewhere in Awasu, those operations will be noticeably snappier as well.

The other area that received attention was shutting down Awasu, which was prone to stalling or crashing. You might think that this is a funny thing to be focusing on, but startup and shutdown are often the fiddliest parts of a program. Large programs are split up into separate modules that do different things, and so while the program is starting and module A is coming up, it might need a service provided by module B, so you need to bring that up before module A. Except that module B needs a service provided by module A :wall: , so you need to partially bring up module A, just enough so that module B can start, then go back and finish bringing up module A. Sigh...

At least during startup, you're in a known state i.e. nothing. During shutdown, literally anything could be happening at that time e.g. the part of Awasu that updates channels finishes updating a feed at just the wrong time, and it wants to notify the database module that it needs to store some new content, except that the database module has already shutdown, or is half-way through shutdown, and Awasu gets confused. It's very easy to have problems with this kind of thing going on, and it's very difficult to debug since it's so dependent on what's happening at that exact moment :(

But things are much better now - I haven't received a crash report for, oh, at least a few days now (joking!) - and Awasu shuts down much quicker and much more reliably now.

Not that you would ever want to shut Awasu down, of course... :-D

Tuesday 18th July 2017 7:45 PM [Awasu News]

The first translation of Awasu 3.1 has been baking for a while, and is now ready to step out. Cláudio Mantovani Vieira was kind enough to do the translation, and it looks great :clap: There's just something about seeing Awasu running in a different language that is way cool; you shoulda seen the Arabic one, that was seriously freaky :-)

If you'd like to translate Awasu into your favorite language, drop us a line, and if you happen to speak German or Russian, since we have translations for Awasu 3.0, the job's already half-done!