Automated Python Test Frameworks for Hardware Verification and Validation

Editor's Notes

• #3: Not web developers
• #4: Fatigue TestSimulates flight service lifeOver 4000 channels totalOverall test to run for 165000 cyclesAverage cycle takes about 5 minutes (this includes pressurization and de-pressurization)Expected to take almost 3 years of continuous testing
• #5: TODO: Pass around strain gauge & TCs for visual aidBefore I get started, I’m going to give some definitions since I’m pretty most of you aren’t familiar with the test & measurement industry.Strain gauge – A sensor for measuring the strain of an object. basically a wire (foil) looped into a pattern of parallel lines and attached to an object. As the object is deformed or bent, the strain gauge is deformed and its resistance changes. (stretching causes higher resistance, compression reduces resistance) (Excitation voltage is applied, voltage is read from the output leads, eqn to convert to strain.)Thermocouple – A sensor consisting of two different types of wire which works a lot like a thermostat – it produces a voltage that’s proportional to the temperature difference between the two ends. The voltage change is mostly linear (within a certain range), but more precise measurements require approximation with a polynomial (up to 13 terms, coefficients depend on the metals in the TC & can get nasty). There are several different types using different metals for range and accuracy. Type K is common & cheap, range -200 to 1350 C.Channel – one input (one sensor).Sample rate – samples (measurements) per second, per channel. Our strain instrument supports 1 Sa/s to 10 kSa/s (or 25 kSa/s in limited conditions). Thermocouple instrument supports up to 1000 Sa/s.Waveform – an electronic signal (voltage), something that you’d observe on an oscilloscopeWaveform generator – a device that produces a waveformTODO: Note that “waveform generator” & “function generator” & “signal generator” are all equivalent, definitely not the yield statement
• #6: “No sense blowing up more than one at a time”Same reasons as testing softwareIn Hardware, as in Software, there’s often a published specification of what your device will do. If you don’t meet it, you get angry customers. To keep those customers happy and buying your product, you’ve got to make surethat you meet your published specifications.Often, these published specs are based on what the hardware can do, such as ‘a noise floor of 0.01 Decibels’, or 0.000010 arc seconds (2.78 x10^-9 degrees) of phase alignment between channels. For those of you who build web frameworks [raucous laughter], a noise floor means that you can detect any signal above that strength, and a phase alignment between channels means that if you put an identical signal into channel 1 and channel 2, you will see in the resulting data that the waveforms were processed at the same time.You also want to make sure that you’ve actually built your hardware correctly. Sometimes the production floor gets confused and they install a diode backwards. Sometimes they install a 10 ohm resistor instead of a 10,000 Ohm resistor. Whatever the reason, sometimes the failure isn’t enough to let the magic smoke out, but still gives your customers the wrong answer. You want your tests to show that your device works even once the design is shown to be good.
• #7: With unit tests, the point is to pick small components or subsections of your code and test those pieces, to test functionality at the smallest unit. In some pieces of hardware, there are discrete pieces like this that you can test. If that’s the case with your hardware, by all means make use of it and test those pieces of functionality separately. However, in many cases you will have two pieces of hardware which are inextricably linked, like an analog-to-digital converter connected through an analog filter. In this case the final data is your only method of data gathering, and you need to test the filter and ADC as a cohesive unit.In hardware testing, some unit-like testing is still possible (Isolated HW pieces exist), but in large part the HW is a big black box that all hangs together. If you contracted out the design, you may not even know what’s in it, just that it has specs that it has to meet.Mocks are the practice of faking out classes with stubs that appear to perform all the correct behavior, or give all the appropriate responses without doing all of the work. When testing hardware, the whole point is the stuff down at the very bottom of the stack - the chips and circuits. Those aren’t possible to mock, because then you’d be faking exactly the data that you need. (This can be very useful, but not for testing.)The only piece of the system you’re testing is the hardware - and unless you have a 1:1 hardware simulator (which is expensive to construct and harder to guarantee correctness - it ends up looking like the HW itself), you can’t mock out a piece of hardware.TDD is the practice of writing the test before, or as a guide to, writing the software. But when making hardware, it’s not really possible to design your hardware based on the design of your test - hardware’s driven by other considerations like PCB space, component cost, and specification goals. We can build a prototype back in the lab, but it won’t match the final PCB design. What we can do is work with that final piece, the known specifications. Since the hardware must meet some specification, and this and the circuit designs are a known quantity before it goes off for fabrication, we can design and write our tests based on these hardware designs and goals. This isn’t exactly TDD... but it is Spec-Based Testing.
• #8: So if you can’t mock it and you can’t break it down into unit tests... [pause] How DO you test hardware? I just mentioned that hardware has specifications it has to meet, specifications like Total Harmonic Distortion or Noise Floor. We can work with our hardware developers to put certain inputs into the product that should create certain types of outputs. We then use more equipment to sense those outputs, and verify that the responses we get are the responses the hardware engineers expect. The great news is, if you know the input signal, the type of hardware you’ll need to generate the input, the equipment you’ll need to measure the output, and the valid range for the outputs... you can write the entire test, and even construct the test harness, before the board comes back from the fabricator.
• #9: Sometimes you may get 50 or 100 boards back from the fabricator at once. Sometimes you’re doing engineering verification and proving one board works correctly is enough... other times you’re doing build validation and you need to test all of them. Testing them by hand would be a waste of our time and awesome coding skills, so we can build test harnesses or test rigs. In the case where you need a lot of test points on a piece of hardware to test supply voltages or chip power, and you are testing a lot of boards, it may be economical to build a bed-of-nails tester which connects directly to the PCB, like the one shown here. Then you just need some switching and measuring and you can test many boards very quickly by automating the test procedure. Other times, you are testing a small number of units, as we most often do, and you just need a Cat-5 cable with the ends stripped off and a waveform generator (point to test rig and/or photo of test rig) to test the product right.
• #10: If you need to test many things without changing cabling, a switching solution could come in handy. (possible picture of DIO switch test rig or Comparator test diagram) No, not a network switch. This lets you feed different signals into your hardware or send the outputs to different types of monitors, while only making one cable rig at the start of the test. Getting the switching paths correct can sometimes be tricky, but is a lot less tedious than doing it by hand.And your test won’t stop at 3am, waiting for you to change equipment connections.
• #11: Once you have your test stand set up, test progressions should advance automatically too. This is where these frameworks have the most in common with SW frameworks, as long as we can easily and reliably load the external equipment and math portions. We do it with python.We don’t use any of the common Python test frameworks, since they’re all geared for software testing. Since we’re not writing unit tests and we can’t use mocks, we had to roll our own.In our framework, each test module has “add” function, each test has “verify” function.The major part of a test is the calls list. This is just a list of calls to make in the future, when the test actually runs. The dirty little secret of the framework is that all the work done here just generates a list of calls and functions to call later, when the framework actually runs the test. It then uses the verify function to determine if the test passed.
• #12: For an example, our strain gauge instrument has a test for phase alignment. This diagram is the same test rig that was pictured earlier, with the waveform generator connected to two instruments.To guarantee the published phase alignment, the test spec defines the input as a sine wave at a particular voltage and frequency and then we run a Discrete Fourier Transform on the acquired data. These nasty equations are how we calculate the amplitude and phase.atan2 is the two-argument form of arctan
• #14: Agilent 33120Awaveform generator & Agilent E5810A LAN/GPIB gatewayControlling external equipment can be extremely easy, since we have a plethora of industry standards for instrument communication.We use a waveform generator that supports VXI-11, a protocol for instrument communication and control over a network. Does anyone use xml-rpc or json-rpc? VXI-11 is similar, but it’s based on the original ONC RPC implementation.IEEE-488 – GPIB (general purpose interface bus), introduced 1975, 488.2 defines instrument control commands/messages - *IDN?, *RST, *OPC? SCPI – standard commands for programmable instruments, defined common syntax, command structure, and data formats. Includes generic commands & instrument classes. Doesn’t specify the communication link.VXI-11 – spec published 1995, RPC (remote procedure calls) over IP, uses ASCII messages (inc IEEE-488)
• #15: *RST – reset*OPC? – operation complete query, returns an ASCII "+1" when all pending overlapped operations have been completedDEF – default
• #18: The test begins by setting the waveform generator to produce a sine wave.As the comment in the verify function notes, it only checks the return values of the calls. The actual verification is a little complicated, so that’s handled in a separate function that’s part of the calls list.
• #19: This is the main test sequence. It starts by setting all of the devices to acquire a thousand samples, using a sample rate of a thousand samples per second, from all 48 channels. It also configures the devices in a master/slave configuration. The master device will send sync and trigger signals on particular hardware lines and the slaves are set to listen for those signals.The inner loop is part of the reason the full test takes several hours to run. Each acquisition takes data for 1 second, but the test uses 2000 acquisitions for each of several trigger configurations.The analyze data method is where almost all of the work actually happens. It reads the data from each device and calculates the magnitude and phase of the data from each analog board in each device.There are several conditions that could generate a failure here - not enough data received, the data has an error code set, the magnitude indicates a weak analog signal, there's timestamp or phase skew between the analog boards within a device, or there's timestamp or phase skew between the master and slaves.
• #20: The analyze data method is where almost all of the work actually happens. It reads the data from each device and calculates the magnitude and phase of the data from each analog board in each device.There are several conditions that could generate a failure here - not enough data received, the data has an error code set, the magnitude indicates a weak analog signal, there's timestamp or phase skew between the analog boards within a device, or there's timestamp or phase skew between the master and slaves.
• #22: Summary/questions