A few years ago, I wrote a long series of tutorials showing how to embed Python into a C/C++ program, and periodically threatened to write another series showing how to go the other way i.e. extend Python by calling your own C/C++ code[1]Typically because you want better performance, or because you want to run it multi-threaded, which Python is known to not handle very well..
Well, I've finally made good on that promise and written some tutorials on how to write a Python extension module:
Many moons ago, I wrote a tutorial on how to set up an internet gateway on a Banana Pi, complete with DHCP, DNS, VPN, firewall and ad-blocking. It works well, I still use one today, and have even taken it with me on a few long backpacking trips. However, I worry about it being a bit fragile, and fear the day when an over-zealous customs officer decides it looks like something that could trigger a bomb , so I was overjoyed when I finally found my holy grail: something that does all of the above, in the form factor of a USB thumb drive.
GL-iNet's GL-USB-150 costs around USD 30, and comes with almost everything I need to get online when I'm on the road. This tutorial will be much shorter than the previous one, because nearly everything is already set up and ready to go
Getting started
Plug it in, give it 30 or 40 seconds to start up, then open a browser and go to http://192.168.8.1. To login, the default password is goodlife; once you're in, change this under More Settings/Admin Password.
It runs a DHCP server, and your computer will have already been assigned an IP address in the 192.168.8.xxx range.
Go to the Internet page, click on the Scan link, then connect to a WiFi network.
Open another browser window, and confirm that you're online.
Configuring the VPN
Go to the Management tab in the VPN/OpenVPN Client page, and upload your VPN configurations. This will typically be a ZIP of a bunch of .ovpn files, but if you have them, you will also need to include the .crt and/or .pem files.
Unfortunately, the stock firmware has a bug that prevents the ZIP file from being processed correctly, so you will need to upgrade the firmware first. Get the latest version from here[1]Version 3.026 worked for me., then install it via the Upgrade page.
Once the VPN configurations have been installed, you will be able to select which one to use from the VPN/OpenVPN Client page. Check your IP address to confirm that you are going through the VPN.
Installing software
To install additional software, go to the More Settings/Advanced page, and in the new browser window that opens, go to System/Software and update the package lists[2]This doesn't seem to persist after a reboot, so you have to remember to do this every time .
I installed the following packages:
bash
tmux
openssh-sftp-server (so that I can scp files in and out)
openssh-client (for a version of ssh that allows forwarding)
coreutils (for GNU tools)
bind-dig (for dig)
mtr (a handy network monitoring tool)
To change your default shell to bash, update /etc/passwd.
The only down-side to this device: while you can just about install a minimal version of Python, the disk is so small, there won't be any room for anything else
Ad-blocking
The only thing missing from this device is an ad-blocker. Since it uses dnsmasq for DNS, rather than bind as the Banana Pi does, the process is slightly different, but not much. Here's the script that I use:
# This script downloads blacklisted ad servers and updates dnsmasq to block them.
#
# The following line needs to be added to /etc/dnsmasq.conf:
# conf-file=/root/dns-blacklist
BLACKLIST_URL="http://pgl.yoyo.org/as/serverlist.php?hostformat=dnsmasq&mimetype=plaintext"
BLACKLIST_FNAME=/root/dns-blacklist
echo "Downloading the DNS blacklist..."
TMP_FNAME=/tmp/dns-blacklist
wget -O "$TMP_FNAME" "$BLACKLIST_URL"
if [ $? -ne 0 ] ; then exit 1 ; fi
echo
# fixup the entries so that they return "NX Domain"
echo "Updating the DNS blacklist..."
sed -i 's/address/server/g;s/127.0.0.1//g' "$TMP_FNAME"
echo
# install the new DNS blacklist
echo "Installing the DNS blacklist..."
echo " $TMP_FNAME => $BLACKLIST_FNAME"
mv "$TMP_FNAME" "$BLACKLIST_FNAME"
echo "Restarting dnsmasq..."
/etc/init.d/dnsmasq restart
echo
echo "All done."
The DNS blacklist is downloaded to a temp file, fixed up and then transferred to /root/dns-blacklist. You will need to tell dnsmasq to load this file by adding the following line to /etc/dnsmasq.conf:
conf-file=/root/dns-blacklist
This script can be configured to run periodically, or just run it manually every now and then.
Shutting down
There doesn't seem to be any way to shut down the device cleanly. I'm guessing it's been designed so that people can just pull the thing out of the USB port, but this really irks the sysadmin in me , so to shutdown cleanly, type the following in the console:
halt
The green LED light stays on, but the device will shutdown.
An Awasu user recently asked for some help in getting their Python code to work with Awasu, and while the problem turned out to be related to text encoding (which is not, strictly speaking, anything to do with Awasu), since this is such a common issue, I thought I'd write up some notes on how all it works.
Note that while this tutorial has separate sections on Python 2 and Python 3, even if you're only using one version of Python, you should read both sections if you want to really understand how things work.
This stuff is tricky to get your head around at first, but once you figure it out, it's actually not too bad. The problem is that even when you've got your code right, you start receiving content from elsewhere that is wrong, which breaks your code, so you change it to handle that content, but then your code breaks when you receive content from somewhere else that is doing things correctly[2]Or also doing things incorrectly, but in a different way , and you get stuck in a cycle where your code never works properly Hopefully, these notes will help you know when your code is right, and you can stick to your guns and start yelling at the other guy to fix their code...
A gateway lets you isolate computers in your home network from the internet. To reach the internet, a computer has to through the gateway, which means that if you put a firewall or virus checker or ad-blocker here, all your computers will benefit from them.
As before, this series of tutorials will walk you through the whole process of setting up a gateway, including a lot of not-essential-but-nice-to-have stuff. We start off by setting up a bare-bones gateway:
These days, a firewall is pretty much a necessity, and it's quite eye-opening to watch the logs and see the constant stream of attacks, as people try to break into your computer. And even if you run an ad-blocker like uBlock or AdBlock, a DNS-based ad-blocker can be run along-side it, without affecting browser performance at all[1]Browser plugins tend to slow the browser down noticeably, and can use huge amounts of memory..
Just a quick follow up on my recent epic tome on setting up a Banana Pi as a file server. I mentioned that I configured my disks to use the ext3 file system, and while it's generally fine, it does have one weakness: it is very slow deleting large files[1]Since I use my NAS for back ups, some of my files are well over 100GB.. Even worse, it locks up the file system, impacting other activity and causing stalls, which kinda sucks if you're watching a movie at the time
depesz took a very detailed look at the problem and some possible solutions, the TL;DR being that it's better to progressively shrink the file until it's all gone rather than asking the operating system to delete it as a file.
For the benefit of anyone having problems with this, here's a script that I wrote that implements this idea:
#!/bin/bash
# parse the command-line arguments
if [ $# -lt 1 ] ; then
echo "$(basename $0) file1 file2 file3 ..."
echo " Delete file(s) slowly."
exit 1
fi
chunk_size=100000000
for fname in "$@"; do
# check if we were given a directory
if [ -d "$fname" ]; then
# yup - process each file, then remove the directory
#echo "Deleting directory: $fname"
find "$fname" -type f -exec $(readlink -e "$0") \{\} \;
rm -rf "$fname"
continue
fi
# check if we were given a file
if [ ! -f "$fname" ]; then
# nope - ignore it
continue ;
fi
# yup - delete the file slowly
#echo "Deleting file: $fname"
while true; do
# get the current size of the file
fsize=$(ls -l "$fname" | cut -d' ' -f 5)
if [ $fsize -lt $chunk_size ] ; then
# the file is small enough to just delete
#echo "- Deleting file."
rm -f "$fname"
break
fi
# truncate the file, then loop back
#echo "- Truncating file: $fsize"
truncate -c -s -$chunk_size "$fname" || exit
sleep 0.25
done
done
Pass in a list of files and/or directories and it will slow-delete them. It will take longer to run, but will have far less impact on the rest of the system.
Nothing to do with Awasu, but hopefully someone out there in Internet-land will find it useful...
I've been a big fan of NAS's for many years, that is, a small file server that sits on my network and serves up music, movies, provides space for backups, etc. In the past, I've had Synology and QNAP units, and while they were both nice, they were both were relatively expensive, loaded with features I never used. They also both only lasted a few years, and rebuilding a NAS with 5-6 TB of data is a painfully long process
So for the next one, the plan was to grab an old laptop, load it up with FreeNAS, and then just hang a few disks off it. If and when the laptop dies, I can just set up a new one and the external disks, with all the data on them, should just plug straight in.
However, this is a bit of clunky solution, so when the Raspberry Pi came out, I got very interested in the idea of using that. Unfortunately, the rPi has one big drawback that makes it unsuitable for use as a file server: it only has 10/100 Mbps ethernet. All the computers on my network have gigabit ethernet, and since I'm moving 100's of GB's of data every night for backups, my file server also needs to have gigabit ethernet.
Enter the Banana Pi. Released in late 2014 by LeMaker in China, it's slightly more expensive but significantly more powerful, notably with gigabit ethernet and a SATA port. Add in a case, and I'll be able to build my own future-proof NAS for well under a hundred bucks, plus the cost of the disks.
There are quite a few tutorials floating around that explain how to set up a Banana Pi as a NAS, but they invariably only talk about how to set up the factory image of Open Media Vault[1]This is the successor to FreeNAS, written by one of the FreeNAS guys, that runs on Linux instead of FreeBSD. (which is relatively easy to do), but this series of tutorials will also talk about the many things you need to do after that to get a usable system.
A while back, I posted a tutorial that showed how easy it is to extend Awasu through the use of plugins and channel hooks, and continuing on from that, here's another series that shows how you can control your Awasu via its API.
Whether you just want to find out what state your channels or reports are in, or if you want to programmatically create, update and delete them, the Python and PHP libraries available make it a breeze.
Have a play with them, hope you find them useful and, as always, feel free to ask questions in the forums.
As promised, the tutorial on writing Awasu plugin channels and channel hooks is now up here.
Part 1 talks about how to get set up with the new awasu_tools library, and how to generate a basic feed, then Part 2 shows how to convert a basic script into something that can be called by Awasu.
Part 3 explores some more of the features offered by the awasu_tools library, while Part 4 rounds things off by explaining how to compile your extension, ready for distribution.
Writing Awasu extensions has always been relatively easy, but there was a bit of a learning curve, mostly spent looking at the samples supplied with Awasu. Hopefully, this tutorial will make things a bit easier, and the new awasu_tools library really makes things a breeze
As an aside, I've been programming computers for around 30 years (professionally for 25), yet publishing my first bit of code on PyPI and GitHub somehow makes me suddenly feel like a Real Programmer
So, one of our long-time users has been asking about how RSS tracks authors, which reminded me that I've been meaning to talk about this insanely cool feature of Awasu that I'm sure not many people know about.
RSS feed items typically come with metadata associated with them, like who the author was, what time they were published, etc. There are also non-standard things like licensing details, geocodes, information about who's commented or linked to the blog post. Not only is Awasu able to slurp up all this yummy data, it doesn't even have to be a standard type of metadata. You can even embed your own custom metadata in a feed and Awasu will be able to extract it - awesome if you're working in an enterprise environment.
One of the things you can do with this metadata is include it in the item pane, the list of feed items that appears in the top (by default) of a channel's window. Here's a screenshot of a channel I have using the <foreshadowing> new and improved plugin that monitors email accounts </foreshadowing>, with the email's sender and timestamp in the item pane.
Setting this up is pretty easy:
Open the channel's Properties dialog and on the first tab (Channel), click on the Item pane button.
Click on the green plus icon, to add a new column.
Enter the metadata value you want to show in the column. The most commonly-used ones are provided in the droplist, but you can enter anything you want.
Then enter the name of the column.
And ta da! Awasu will now show the relevant metadata in the item pane.
This is a pretty nice feature, and Awasu is not limited to just showing these values in the item pane, they can be included in the browser pane as well, which means they will show up in reports, or emails. Post in the forums if you need a hand sending any of this stuff up.
Those of you who know me know that I'm a big fan of Python. It's a really powerful and flexible scripting language that sits nicely in that space between shell scripts that have gotten too big (i.e. more than 10 lines ), and large-scale applications written in C++ or Java. I remember trying to print out the manual for 1.5.2[1]IIRC, this was the iconic version of Python, much like 3.1 was for Windows for a long time. way back in the late-90's[2]Yes kids, we often used printed manuals, back in the day :-O, and I'd already been using it for a few years then, so I guess that means I've been using it for nearly 20 years now
Awasu has had an embedded Python interpreter for some time, and one of the things that has been lurking in the murky depths of my to-do list for many years now is a write-up on how to actually do this, since the documentation is not particularly great. Inexplicably, I found myself with a bit of time recently and have finally managed to cross this one off my list
In which we look at what you need to distribute with your program for Python to work.
Hopefully, I'll be able to follow this series up with another one, that explains how to go the other way - extending Python with your own custom features. Just give me a couple of years...
Awasu and the stylized Japanese character in the orange box are trademarks of Awasu Pty. Ltd. Other brands and product names are trademarks of their respective owners. Awasu Pty. Ltd. believes the information in this publication is accurate as of its publication date. Such information is subject to change without notice. Awasu Pty. Ltd. is not responsible for inadvertent errors.
A few years ago, I wrote a long series of tutorials showing how to embed Python into a C/C++ program, and periodically threatened to write another series showing how to go the other way i.e. extend Python by calling your own C/C++ code[1]Typically because you want better performance, or because you want to run it multi-threaded, which Python is known to not handle very well..
Many moons ago, I wrote a 
, so I was overjoyed when I finally found my holy grail: something that does all of the above, in the form factor of a USB thumb drive.
This stuff is tricky to get your head around at first, but once you figure it out, it's actually not too bad. The problem is that even when you've got your code right, you start receiving content from elsewhere that is wrong, which breaks your code, so you change it to handle that content, but then your code breaks when you receive content from somewhere else that is doing things correctly[2]
A while back, I posted a massive tutorial on 
I am a 
