Repair and deployment of a UPS on R-Pi

Repair and deployment of a UPS on R-Pi

2020, Jul 22    

In this post I explain how I repaired a malfunctioning watchdog+UPS board for the Raspberry-Pi and show some sample scripts and cronjobs.

Introduction

For years I have been running a Domoticz server in my home to measure and control some appliances in my house. The server is running on a Raspberry-Pi 3+, after it was getting too slow for the old Raspberry-Pi where it started on. My domoticz installation does the following:

  • it shows temperature and humidity sensors in various rooms, using home built ESP8266 boards with ESP-Easy firmware
  • controls some lights and appliances via Sonoff boxes, with Espurna and Tasmota firmware
  • controls the KaKu (CoCo) 433MHz lights and appliances
  • controls the Hue lights
  • controls some Tasmota based LED lights directly
  • measures the power output of washing machine and dryer, and sends a text message when they’re finished
  • shows the weather outside from my weather station
  • controls the floor-heating of the bathroom floor when it’s cold outside
  • keeps stats on my internet connection

What I like about Domoticz is that it is stable, runs on your own server and connects with various manufacturer’s as well as home-brew devices. This in contrast to e.g. Apple’s Homekit which requires hefty manufacturer approval, or Alexa which throws away your privacy.

My Domoticz installation has reached a point where it becomes a nuisance when it doesn’t work. Still I prefer the standalone setup and did not want to move it to a big intel based server.

Uninterruptable Power Supply (UPS)

I was pleasantly surprised when I discovered a Kickstarter project for a low-cost watchdog and UPS specifically for the Raspberry-Pi, including drivers. It is open source and was made by a manufacturer with a proven track record, Sequent Microsystems, which is important when you pledge on Kickstarter.

the Super Watchdog and UPS

When the card arrived I did not have any 18650 batteries to test it with. But I did not want to wait for delivery, so I opened up an old Dell laptop powerpack and removed some ancient 18650 batteries.

Unfortunately, they did not get charged properly. So I fully charged them first with my bench supply. Two of them could not hold their charge, but two were still fairly good. When I tested my watchdog it stated that the charge was okay: 3.7 V.

I played around with the provided scripts and tools with the watchdog/ups. I quickly noticed something was wrong:

  • When I plug the power source in the R-Pi and the watchdog with battery installed, the UPS function works. But the wdt tool reports there is no battery, but it has a charge of 3.689V!

wdt -get c → 0 (charging off)

wdt -get vb → 3689 (mV)

  • Then, when I pull the USB +5V from the main R-Pi board, the UPS kicks in and the Pi keeps running: great!

HOWEVER: When I supply the +5V to the UPS board, which is the recommended way, the UPS does not kick in when I pull the plug. Instead, the Pi reboots. And keeps rebooting constantly (because it gets power)

Weird.

I report this to the manufacturer, but because these batteries are over 15 years old, I blame the battery and order some new ones.

Unfortunately, the same anomaly happens with brand new Sanyo batteries. So I report this to the manufacturer. They had done some research as well and found a slight adjustment in the circuit board layout right before the final release. (…) Never think “this fix is so easy, we can totally do this!”. The road to hell is paved with good intentions.

The problem was that the MT3608 step-up converter needs a large capacitor close by and not somewhere else on the circuit board. The schematic was the same, but the board layout was different. Hence the circuit did not function properly. The fix is to solder a 22µF ceramic capacitor over the step-up converter. Unfortunately the soldering needs to be done on a 0403 SMD resistor, which is about 1 mm small.

repair

I did not have such a cap, so opted instead for two parallel 10µF caps.

my repair

After soldering this extra capacitor the circuit performed really well and I could write some scripts and cron-jobs to test its functionality.

Software

raspi-config

The watchdog+ups communicates with the Pi over the I2C bus. You have to enable this with the raspi-config program as explained in the manual.

library

The base software for the watchdog/ups is available on GitHub. It has a command line tool wdt and a python library. My experience with python however is a mix of amazing! and cursing at apt-get + pip + easy-install and python versions.

Instead I opted for the over forty years old stable shell script language to test the ups.

uptime with a UPS

First I wanted to test how long the Pi would keep working with a full battery and mains power gone. For this I created a cron job which logs the battery charge every 5 minutes. I want to reset the Pi when the battery is below 3 volts.

EMAIL=your_email_here@example.com

# location where you installed the wdt utility from SequentMicrosystems
WDT=/usr/local/bin/wdt

# location where ssmtp is installed
SSMTP=/usr/sbin/ssmtp

VIN=`$WDT -g v`
VBAT=`$WDT -g vb`
VTHRESHOLD=3000

# For testing
# VIN=0
# VBAT=0

TODAY=`/bin/date +"%Y%m%d"`
LOG="/var/log/ups_$TODAY.log"
touch $LOG

HOSTNAME=`/bin/hostname`
NOW=`/bin/date +"%Y%m%dT%T"`

if [ "$VBAT" = "" ] ; then
    echo "$NOW UPS safe_shutdown, an error occurred reading the UPS." >> $LOG
    exit 1
fi

if [ $VIN -lt 3000 ] ; then
    
    if [ $VBAT -lt $VTHRESHOLD ] ; then
        echo "$NOW UPS Power issue, battery depleted: ($VBAT mV), Shutdown!" >> $LOG
        echo -e "Subject: $HOSTNAME Shutdown!\n\nBattery=${VBAT} mV\n\n${HOSTNAME} shutdown on $NOW" | $SSMTP $EMAIL
    else
        echo "$NOW UPS Power issue, running from battery: ($VBAT mV)" >> $LOG
        echo -e "Subject: $HOSTNAME UPS Power issue\n\n${HOSTNAME} UPS Power issue on $NOW\nBattery=${VBAT} mV\n" | $SSMTP $EMAIL
    fi
fi
exit 0

The final script available here: safeshutdown

The output of the script is as follows:

20200614T17:00:01 UPS Power issue, running from battery: (4047 mV)
20200614T21:30:01 UPS Power issue, running from battery: (3077 mV)
    ”
    ”
20200614T21:35:01 UPS Power issue, battery depleted: (2994 mV)

Times are in ISO-8901 format. Notice there is a 4½ hours gap between pulling the power and the imminent system failure. Hopefully real power glitches are shorter than that!

battery drain

The script has since been adapted to shutdown the Pi in a safe way before the power is completely gone, emailing cries for help before that!

watchdog

So, what is a watchdog, you might ask? Watchdogs detect system lock-ups and try to reboot the system. Sometimes this is all that’s needed to keep a system running. Say you have an application which overloads the Pi sometimes but not enough to warrant a complete rewrite or exchange of the application. Or a server process goes haywire and locks the system. A watchdog reboots the system unscrupulously when it is not fed with an ‘I’m alive’ message every minute or so. It is the poor man’s alternative of a fully-redundant server-park with automatic fail-over between servers.

Your application would ‘ping’ the watchdog periodically, say every 20 seconds, and if the watchdog did not get at least one ping within a minute’s time, it will reset the host computer.

The watchdog needs to be set to watchdog modus using the wdt -r command. When this is set, you need to ping the watchdog within the interval time, otherwise your server is reset! You ping it using the reload command wdt -r For my tests I used the following simple cron job, which executes the reload command every two minutes. When I stopped the cronjob, the pi was indeed reset.

crontab -l :

*/2 * * * * /usr/local/bin/wdt -r

caveats

Because there is now a watchdog available, a simple shutdown or reboot is now tricky! When you shut down the Pi, the watchdog does not get the pings it expects and will restart your Pi after the timeout time. To properly shutdown you can temporarily disable the watchdog like this: wdt -off 7200 which disables the watchdog for 7200 seconds, or two hours. You can of course also take out the battery…

Scripts

The safeshutdown script as well as a utility script you can download from my GitLab

Availability

Twitter
# hardware # repair # raspberry-pi