The Dawnclock – back from the dead

I’d been waiting for this moment. I’m not a electronics or coding expert and so, sooner or later, I expect all of my projects to die.

But the Dawnclock just kept on living – right up to the point when it didn’t.

I first blogged about this project in January 2013, which means that it’s been operating for eight years. It has been gently waking us every morning, playing birdsong while incrementally switching on LEDs. Then, at alarm time, it plays a random MP3 file from a folder before switching to streaming BBC Radio. It also chimes every 15 minutes for the next hour as a ‘snooze’ feature.

The LCD panel shows the time, date, temperature and barometric pressure (and whether the temp and pressure are rising or falling). It tweets which song we’ve just listened to. And it sends out messages over MQTT so that I can, if I wish, synchonise other IoT devices with the various stages of waking up. It communicates the temperature to a central IoT server via a REST API. And it runs a web server, which is how we set the alarm time and other configuration options.

The fact that this has run for eight years, pretty much non-stop, is impressive to me. But when you think that this was my first major electronics project and that’s it’s based on the very first version of the Raspberry Pi, the fact it has taken this long to die is little short of miraculous.

The Raspberry Pi is the original Model B. I had to hack about a case to mount it in the project, because that version of the Pi didn’t have mounting holes.

The software uses hundreds of lines of Python, which has been hacked over the years to add functionality like tweeting, sending out MQTT messages, reporting to an IoT server and more.

So what happened?

Not surprisingly, it was the SD card. For some reason, the 16GB card was set up with three partitions – the FAT32 boot partition, an 8GB EXT4 partition for Linux and most of the data, web server etc, and another 8GB partition that held the music files. I can’t remember why I did it that way. There probably wasn’t a good reason.

Anyway, the main Linux partition had become corrupted. These things happen to SD cards.The Dawnclock has had its power cut many times, due to storms and accidental unplugging – something that SD cards don’t like.

And I had no backup.

Open ‘er up

Getting to the card meant dismantling the whole thing. It revealed just how much I was making things up as I went along in those days. (I still do that.) Apart from feeding one signal through a 10-port ADC chip, the ‘electronics’ in this consisting mostly of connecting up a bunch of modules, several of them via I2C. Oh, and using port extenders to drive the LEDs.

The Pi was connected to a USB DAC for audio output. But that couldn’t be connected straight to the speakers, so I needed a small amplifier board from Adafruit. For some reason, this wasn’t mounted to anything, but was just hanging in the breeze. So was a small bit of perfboard that I’d used to cobble together speaker wire connections. Why did I do that?

And then there was a piece of stripboard that had the four-channel, I2C-safe level shifter from Adafruit so that the Pi (3.3V) could talk to the LCD panel, the port extenders on the LED boards and the temperature & pressure sensor (all 5V). This stripboard also acted as a 5V power distribution board for all these devices. Plus there was the 10-channel ADC chip and an area where I’d tried to use an amplifier that was no good and had removed (although many supporting components remained in place).

It was a mess, but it worked. So my first approach was not to fiddle with it too much. I’d take the opportunity to upgrade to Raspberry Pi OS Lite but keep the existing hardware. The only change I’d make would be to install an SD extender cable, so that I could mount the SD card in a way that was accessible from the outside. That would mean I could do occasional backups.

Option B

I’d installed that extender cable/socket – with the help of a lot of hot glue – into an existing hole (the existence of which is a puzzle to me) when I changed my mind.

Instead of an original Pi, I decided to go with a Raspberry Pi 3B+ with a DAC hat – the IQaudIO DAC+ – to make things a bit neater.

That meant the extender cable had to come out because it was designed for full-size SD cards. I needed one that terminated in a micro-SD plug. It turns out that hot glue is very good at its job. It took me around half an hour to hack away enough of it to get the old socket out.

Setting up

Anyway, I installed Raspberry Pi OS Lite on a micro SD card, set up Apache, PHP and MySQL and copied over all my home-made Python libraries. (I have a script I put on machines that pulls the latest version from a central server.)

No code was lost when the original SD became corrupted because all my Pi code in on my Macs. I don’t develop on the Pi – I code on the Mac and push the software to the appropriate Pi across the network.

I had to reverse engineer the MySQL tables from looking at the code to see what was being written or read. I’m not sure I got it quite right because I had to make a couple of subtle changes to the main code, but it seems to be working.

One more major change was to do with I2C. At the time I wrote the original code, the most popular I2C Python library was Quick2Wire. I’d written a wrapper class for I2C devices that give me methods that worked the way I wanted. But the Quick2Wire library was soon abandoned. I had to rewrite my class to work with smbus2. (As it turns out, having written a wrapper class was fortunate. I’d also written libraries, based on this class, sub-classing for specific devices. These library files didn’t need to be touched because all the necessary changes took place in the main I2C class.) While I was doing all this, I took the opportunity to clean up, rename and improve a lot of my Python libraries. This added to the workload but has left me in a better position for future projects.

All the music files, including the ambient sounds of birds in our garden and the alarm chimes, were backed up on the Mac. But, in any case, the partition on the original SD card that held the music wasn’t corrupted, so no problems there.

You may notice in the picture above that there’s an Ethernet cable plugged into the Pi. I tried using wifi, with a USB dongle, in the original machine, but connection was patchy. This new version has no problem with the Pi’s built-in wifi, so no more running CAT 5 to the Dawnclock. Yay!

Soundless

The major snag – as I knew it would be – was audio. I really struggled with this when I originally built the Dawnclock. Setting up the IQaudIO DAC+ (which, by the way, I’ve had sitting around for years waiting for a project) was a doddle – just a couple of tweaks to /boot/config.txt. But finding the right invocations for mplayer and amixer was a different matter.

Mplayer and alsa-utils seemed to be included in the OS, but it took a while for it to dawn on me that I might have to install amixer. I added the pi user to the audio group, but there was still nothing coming out of the speakers.

To be honest, I can’t remember all the options and steps I tried to get sound working. But there were a couple of things that made a big difference.

First, the original USB DAC I’d used used the PCM control to adjust volume. But amixer was barfing at any reference to PCM. It didn’t complain about references to ‘Master’, but still had no effect on volume.

In theory, the changes I made to /boot/config.txt resulted in the IQaudIO board being the only sound device. But when I ran alsamixer I found the default ‘device’ was something unnamed and unnumbered. The IQaudIO was device 0. I ensured that amixer refered to this by including the -c0 option. And looking at the IQaudIO controls in alsamixer I could see that the control I actually needed to change volume was called ‘Digital’. (The unnamed ‘device’ had a control called ‘Master’.)

In calls to mplayer, I needed to include: -nolirc -ao alsa:device=hw=0,0.

Tidying up

I’d already put hours into this fix so it made sense to try to do at least a half-decent job of the makeover. Instead of being scattered at random points around the box, I mounted the three main units onto a thick piece of plasticard. These consisted of the Raspberry Pi with its DAC hat, the Adafruit amplifier and a much smaller piece of stripboard for the level converter and power distribution. I arranged the SDA, SCL, GND and +5V strips to be alongside each other. This meant that the I2C devices could be plugged in with a simple four-way header.

The new SD extender cable was installed with plenty of hot glue and I was done.

Except that now I could hear some strange noises coming from the speakers. It quickly dawned on me that placing the amplifier hard up against the Raspberry Pi was bound to lead to some interference. I moved the amp to the top of the board and the weird ‘twistly’ noises went away.

It has been a lot of work getting back to a working device. But there are improvements – the externally accessible SD card and wifi, mostly. I also removed a couple of things. There was a light level sensor that I was planning to do something clever with, but never did. That’s why I had the 10-channel ADC chip which is no longer needed. The Dawnclock also has a motion sensor. I actually tried doing something clever with this – tried quite hard, in fact – but it never worked. So I’ve removed the wiring for it.

Let’s see if this version lasts eight years.

[EDIT 23/03/2021] It didn’t last one night. That chirpy, twistly noise – something vaguely similar to a Sinclair ZX Spectrum loading a game – is still there, whining away every few seconds. I thought it might be interference from the Pi’s radios so tried switching those off with rfkill. But that didn’t help. And it turns out the sound occurs only when my Python program is running. I’m stumped.

[EDIT 05/04/2021] Well it’s alive again. I couldn’t get rid of that noise, and couldn’t live with it. It’s the sort of thing that would drive you mad in the middle of the night.

I’m convinced it’s some sort of interference on the I2C bus. I wrote a program with two loops – one did some simple calculating, the other talked to the LCD via the I2C bus. The noise occurred only during the latter.

So I went back to the original USB DAC. I found a nice, short USB A to USB B cable, with right-angle plugs, and the whole thing fitted rather nicely.

No more annoying noise. It’s not quite as neat as the hat solution, but no-one’s going to see it.

While I was in there, I took the opportunity to clean out most of the dead bugs. And talking of which…

I also thought this would be a good time to clean up the code. It was my first big program written in Python, so the fact that it, too, has run almost continuously for eight years is astonishing.

Okay, so the code hasn’t run for eight years. It crashes on a semi-regular basis – maybe once every few days. I have a Python script, run as a cronjob, that checks every 5 minutes to see if the code is running and, if not, starts it up again. (All of this is logged to allow me to face my shame.)

In cleaning up the code, including adding lots of try/except/else structures to deal with those crashes (and just log non-critical problems instead), I fixed the software to the point where it no longer worked. Getting back from that point took a little time, but I think I’m there now. The device has been working fine for a few days with no crashes.

Here’s keeping my fingers crossed…