Metaclass programming and testing

In recent months, I’ve decided to really dig into Python and learn the language. This is unusual in that I spend my day generally coding in C for embedded microcontrollers or device drivers. For those that care, yes, I can write C++, Java, Fortran, and a host of other languages too, I just don’t do it on quite the regular basis that I do C, because it is exceptional and well-supported at system level code. However, just about everything I do means writing object-oriented code in a language that isn’t particularly well-suited to describing object-oriented relationships. Python, on the other hand, is at the opposite end of the spectrum: it’s very good at describing those relationships, and in fact, frees you from the entanglements found in statically typed languages such as C++ and Java. So, when it came time to work on a network stack to facilitate testing of some of our devices, I finally made the plunge to learn some of the finer details of Python. This post is about some of the finer details of metaclass programming that I’ve learned.

If I had to describe metaclasses in a word, it would probably be: magic. Tim Peters said it best when it comes to metaclasses:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why).

I don’t agree entirely with that statement–you can’t know you need a tool unless you understand the existence of the tool, and how to use it. A hammer is of no use to anyone who doesn’t have the basic skills to swing it. In my case, I was interested in removing some of the drudgery from constructing new RPC level constructs from our network stack. Remember that my work tends to focus on embedded processing, and often times these processors don’t have much horse to begin with. As a result, our implementations tend to be rather thin: there is no automatic determination of parameters as you have available in XML-RPC, RPCs tend to be small and focused, and there generally isn’t a lot of data coming back. So why does the stack implementation cause so much trouble? In a word: quantity. Our devices have a lot of specialized commands to for a wide range of activities. In the past, we had developed a custom tool to manually send particular RPCs, and this was great for testing on a onsey-twosey basis. However, as we’ve become much more mature, we’ve recognized that aspects–like true long-term testing–are rather important, especially since it turns up problems that don’t exist in that finite amount of time that the device sits on the bench. In the end, the sheer quantity to test and maintain the different RPC mechanisms has become more difficult.

There are two problems with the current toolset: it’s manual, and more importantly, hard to add something new. The original tool was designed with the engineer in mind, meaning it was designed to help you set up the device, and then hit with a few different types of packets to test code paths and make sure you’re generating the correct results. It’s a cobbled together piece of MFC code that’s completely asynchronous, which makes it hard to add any real tests into it. It’s most troubling feature is that you can’t easily automate it. We’ve hacked in–and I do mean hacked–the core code to help perform some of our tests, but you don’t get any real long-term test results, and certainly none of the test data contains information that would aid in tracking down the defect over a period of time other than to show it’s existence.

Engineers have a hard time expanding on the tool precisely because its asynchronous nature makes it hard to do so. On the other hand, we can get some real benefit from the asynchronous approach by flooding the device with requests. So we took a third approach: offered both asynchronous and synchronous behaviors, but provided synchronous behavior by default.

The other aspect of expanding the tool is more of a maintenance problem: overcoming repetition, and providing mechanisms that make it convenient to work with. The difficult part about writing a network stack for testing is that you need to be able to inspect the packets at every single layer. If you don’t, then you aren’t doing nearly enough testing of the stack. The problem is incorporating that ability–and making it accessible to test cases–while still maintaining the stack’s ease of use for doing the higher-level RPC testing. Those goals are very conflicting as packet inspection forces you to expose more of the inner-workings, while higher-level facilities often work best when you insulate the user from the details.

One technique that I decided to try was using metaclasses to help describe the individual protocol layers, and some of the protocol data. I borrowed the idea from Django’s Model class, and tried implementing a Layer class in which you describe the fields using other Field classes. For example:

class MyLayer(Layer):
    field1 = IntegerField(bits=8)
    field2 = IntegerField(bits=16, endian='little')

The interesting aspect here is that since the metaclass is involved in creating the class object, we can do some interesting things. For instance, we can let each field know what their name is. We can also easily build a description of the layer, and have most everything else that we need Just Happen. However, the–severe–downside to this approach is that there is an incredible amount of magic involved (the same result that the Django guys discovered). Also, for this technique to be effective, you really need to have the fields reside at the class level, then as part of the metaclass, shove them off into a defaults variable, and then create copies of the defaults at __init__() time. It was also complicated in trying to accommodate some of the complexities of the protocol. For instance, the CRC field be a CRC-16 over the payload and some header data. Describing that all at the class level using class variables become cumbersome. That kind of logic is better left to methods, which is what we did in the end. However, I did borrow something that is commonly used in metaclass programming, and that is creating methods in the layers dynamically. I used the nested function ability in Python to dynamically create helper functions to aid in sending particular packets up and down the stack. Engineers simply write code to describe the particular RPC, and they get code that automatically creates the packet, and sends it down the stack for free. This small feature help cut the amount of coding needed dramatically. In the end, we have a stack that we can inspect packets at every level (and even modify, for out-of-bounds testing), but still retain a nice high-level interface for doing more functional testing of the device. All while being much easier to maintain than our previous tool. Definitely a win-win situation.