Automating hardware testing with a Teensy, Firmata, and OpenHTF
OpenHTF ("Open-Source Hardware Testing Framework") is a Python test framework open-sourced by Google at the 2016 Google Test Automation Conference with a big bold disclaimer: "This is not an official Google product", so hopefully OpenHTF cannot be discontinued.
OpenHTF is not that different from other task runners, but has some handy abstractions for interacting with hardware and taking physical measurements. However, OpenHTF does not embrace a "batteries included" philosophy and the documentation is minimal - there are a handful of examples but no real tutorial or coherent API guide. At the time of writing, the best way to learn the framework is to read the source code (which has several TODOs). Despite the lacking documentation, the core features of OpenHTF work well.
One of the fastest way to get up to speed with OpenHTF is to use it as a test suite for automating an Arduino or a Teensy. To speed up development, Firmata will be used as the communication layer between the MCUs and OpenHTF. Firmata abstracts away the need to write a serial protocol communicating pin state between the MCU and the host PC. The Arduino IDE ships with a default Firmata example and plenty of firmata implementations for Python (and many other languages) are available.
OpenHTF Basics
A barebones OpenHTF program can be written in a quick 10 lines. The snippet below defines a measurement 'P4_digital', assigns a docstring of "Pin 4 Digital Measurement", and sets the passing value for the measurement to "True". This code does not interact with hardware, yet, but demonstrates several important OpenHTF concepts.
import openhtf as htf
@htf.measures(htf.Measurement("P4_digital")
.doc("Pin 4 Digital Measurement")
.equals(True))
def digital_read(test):
test.measurements.P4_digital = False
test = htf.Test(digital_read)
test.execute()
OpenHTF outputs minimal information by default. Running the code results in the very spartan:
============= test: openhtf_test outcome: FAIL ============
The test is failing because the validator attached to the P4_digital measurement (.equals) is expecting a True value and the test result is set to False. Replacing False with True on Line 7 results in a passing output:
============= test: openhtf_test outcome: PASS ============
Adding two more measurements is straightforward:
@htf.measures(htf.Measurement("P4_digital")
.doc("Pin 4 Digital Measurement")
.equals(True))
@htf.measures(htf.Measurement("P5_digital")
.doc("Pin 5 Digital Measurement")
.equals(True))
@htf.measures(htf.Measurement("P6_digital")
.doc("Pin 6 Digital Measurement")
.equals(False))
def digital_read(test):
test.measurements.P4_digital = True
test.measurements.P5_digital = True
test.measurements.P6_digital = True
The output from OpenHTF is the same as above, even though we added more tests:
============= test: openhtf_test outcome: FAIL ============
By default OpenHTF does not provide any output outside of the single PASS or FAIL line. OpenHTF provides a callback mechanism to allow logging to files, the web, or the console. Out of the box, OpenHTF provides three example output callbacks: one to serialize the output to JSON, one to print a short console summary, and one to upload data to what appears to be an internal Google service mfg-inspector.com.
Adding the JSON and console summary callbacks to the test is simple:
...
from openhtf.output.callbacks import console_summary
from openhtf.output.callbacks import json_factory
...
test = htf.Test(digital_read)
test.add_output_callbacks(json_factory.OutputToJSON(
'./SampleJSON.{start_time_millis}.json', indent=4))
test.add_output_callbacks(console_summary.ConsoleSummary())
test.execute()
Now when the test is ran a JSON file is created in the current working directory and now the failing tests are printed to the console:
:FAIL
failed phase: digital_read [ran for 0.00 sec]
failed_item: P6_digital (Outcome.FAIL)
measured_value: True
validators:
validator: x == False
============= test: openhtf_test outcome: FAIL ============
The ConsoleSummary that ships with OpenHTF keeps the UI clutter-free by only showing failed test cases, but the output doesn't display the docstring. Fortunately, it is very straightforward to create a custom output callback. All it takes is a class that implements a __call__ function which consumes a TestRecord. A TestRecord contains all of the information OpenHTF has about a test, but one of the more interesting fields is log_records which contains a list of the LogRecord entries generated during the test. Creating an output callback to print these logs to the console is straightforward:
from datetime import datetime
...
class ConsoleLogs():
def __call__(self, record):
for log_record in record.log_records:
# Convert time stamp
timestamp = datetime.fromtimestamp(log_record.timestamp_millis / 1000.0)
timestamp_str = timestamp.strftime('%m/%d/%Y %H:%M:%S')
print(f"{timestamp_str}\t{log_record.level}\t{log_record.message}")
...
test.add_output_callbacks(ConsoleLogs())
Now the OpenHTF logs are printed to the console after the tests have completed
03/28/2020 22:40:55 10 Handling phase digital_read
03/28/2020 22:40:55 10 Executing phase digital_read
03/28/2020 22:40:55 10 Thread finished: <PhaseExecutorThread: (digital_read)>
03/28/2020 22:40:55 10 Phase digital_read finished with result PhaseResult.CONTINUE
03/28/2020 22:40:55 10 Tearing down all plugs.
03/28/2020 22:40:55 10 Finishing test execution normally with outcome FAIL.
03/28/2020 22:40:55 10 Thread finished: TestExecutorThread
03/28/2020 22:40:55 30 DUT ID is still not set; using default.
03/28/2020 22:40:55 10 Test completed for openhtf_test, outputting now.
User messages can be added to the logs by calling test.logger.info(), for example:
test.logger.info('Successfully Connected!')
One thing is missing from the logs: Whether the individual measurements passed or not!
The measurements are available in the TestRecord under .phases. As outlined in the OpenHTF README, tests are broken into phases which are collections of measurements. Phases will be demonstrated once actual hardware is attached, but for now the following callback is all that is needed to display the outputs of the measurements:
class ConsoleMeasurements():
def __call__(self, record):
for phase in record.phases:
print(f"{phase.name}:")
for name, measurement in phase.measurements.items():
print(f"\t{measurement.docstring}\t{measurement.outcome}")
...
test.add_output_callbacks(ConsoleMeasurements())
Now the following is printed to the console when the test runs:
digital_read:
Pin 6 Digital Measurement Outcome.FAIL
Pin 5 Digital Measurement Outcome.PASS
Pin 4 Digital Measurement Outcome.PASS
Note that the measurements are presented in reverse order of how they were decorated on the digital_read function; the callback functions for reporting test state are called after all tests are completed.
Interacting with hardware: Teensy & Firmata
Firmata ships with the default Arduino IDE and can be flashed onto a Teensy (or any Arduino-compatible development board) from the Examples menu.
There are several Firmata implementations for Python. PyMata is easy to use and is in PyPi:
python3 -m pip install PyMata
OpenHTF abstracts hardware behind plugs. A plug is simply a class that inherits plugs.BasePlug and optionally implements a tearDown function. OpenHTF provides a device wrapper to wrap existing Python objects (which pymata is) but for illustrative purposes a "vanilla" plug will be demonstrated first.
The code for the class is fairly straightforward outside of the conf.declare and @conf.inject_positional_args calls for interacting with the OpenHTF configuration. For now, note the conf is used to pass the serial port to Firmata. The rest of the functions are boilerplate to wrap the Firmata instance.
from PyMata.pymata import PyMata
from openhtf.util import conf
...
conf.declare('firmata_com_port', default_value='/dev/ttyACM2',
description='COM Port of Firmata Board')
class FirmataPlug(htf.plugs.BasePlug):
@conf.inject_positional_args
def __init__(self, firmata_com_port):
super().__init__()
self._board = PyMata(firmata_com_port, verbose=False)
def set_digital_input(self, pin):
self._board.set_pin_mode(pin, self._board.INPUT, self._board.DIGITAL)
time.sleep(1)
def digital_read(self, pin):
return self._board.digital_read(pin)
def tearDown(self):
self._board.close()
The plug can be "attached" to the test function using the @htf.plug decorator and adding an argument to the test function. The plug is then available within the test function.
@htf.plug(board=FirmataPlug)
def digital_read(test, board):
board.set_digital_input(4)
board.set_digital_input(5)
board.set_digital_input(6)
test.measurements.P4_digital = board.digital_read(4)
test.measurements.P5_digital = board.digital_read(5)
test.measurements.P6_digital = board.digital_read(6)
If Pins 4 and 5 are pulled high and if pin 6 is low, then the tests will pass!
To fully test the XOR gate output pins need toggled. One method would be to extend to the FirmataPlug defined above to continue to wrap PyMata functions. A fully-featured Plug would need to redefine and wrap nearly every existing PyMata function. Fortunately, OpenHTF provides a device_wrapper to prevent writing boilerplate code. Using the DeviceWrappingPlug, a new, fully-featured Firmata plug is quite compact:
from openhtf.plugs.device_wrapping import DeviceWrappingPlug
...
class WrappedFirmataPlug(DeviceWrappingPlug):
@conf.inject_positional_args
def __init__(self, firmata_com_port):
super().__init__(PyMata(firmata_com_port, verbose=False))
def tearDown(self):
self.transport.stop()
self.transport.close()
...
@htf.plug(board=WrappedFirmataPlug)
def digital_read(test, board):
board.set_pin_mode(4, board.INPUT, board.DIGITAL)
board.set_pin_mode(5, board.INPUT, board.DIGITAL)
board.set_pin_mode(6, board.INPUT, board.DIGITAL)
test.measurements.P4_digital = board.digital_read(4)
test.measurements.P5_digital = board.digital_read(5)
test.measurements.P6_digital = board.digital_read(6)
It does not make much sense to perform the pin mode setup every time the test function is called. As mentioned earlier, OpenHTF groups tests into phases, so the initialization, execution, and teardown of a test can be separated.
Instead of declaring a single function instead of the htf.Test creator, a PhaseGroup can be passed that contains lists of tests passed as setup, main, and teardown arguments.
@htf.TestPhase(name="Setup")
@htf.plug(board=WrappedFirmataPlug)
def setup(test, board):
board.set_pin_mode(4, board.INPUT, board.DIGITAL)
board.set_pin_mode(5, board.INPUT, board.DIGITAL)
board.set_pin_mode(6, board.INPUT, board.DIGITAL)
@htf.TestPhase(name="Digital Outs")
@htf.measures(htf.Measurement("P4_digital")
.doc("Pin 4 Digital Measurement")
.equals(True))
@htf.measures(htf.Measurement("P5_digital")
.doc("Pin 5 Digital Measurement")
.equals(True))
@htf.measures(htf.Measurement("P6_digital")
.doc("Pin 6 Digital Measurement")
.equals(False))
@htf.plug(board=WrappedFirmataPlug)
def digital_read(test, board):
test.measurements.P4_digital = board.digital_read(4)
test.measurements.P5_digital = board.digital_read(5)
test.measurements.P6_digital = board.digital_read(6)
test = htf.Test(
htf.PhaseGroup(setup=[setup], main=[digital_read])
)
Now, multiple tests can be put in the main iterator and will all run after the setup phase which establishes the pin modes.
In Summary
OpenHTF does not include many batteries. Any useful output callbacks or plugs must be written by hand since there is not a large open source community around OpenHTF producing code. However, the core OpenHTF framework is very flexible and performs as advertised. Coupled with a dev board running Firmata, OpenHTF provides a feature-rich springboard for developing sophisticated hardware tests.