Testing

(post dated to reflect when material was discussed in class) Testing is hard... Reading the description of testing the space shuttle engines in Chapter 11 reminded me a great deal of how large software projects are debugged when one doesn't have much of and understanding of the system. You attach a debugger to the running program, try to reproduce the bug, maybe you have a break point set on an entry point that's likely a source of the bug and you follow potentially suspect data down to the point where it gets mangled incorrectly.

If you're lucky you have enough of an understanding of why the program decided to mangle the data that way and you can fix the bug. Otherwise you're left inserting probes and breakpoints in other suspect locations to give you a quick view of data as it moves around the system.

From my own experience top down debugging is tedious and time consuming. Once I've fixed a bug via a top down approach it's really nice if I can pinpoint the source of the problem well enough to test the fix "bottom up". If I can, putting a unit test in place to ensure the bug stays fix solidifies a bottom up test that identifies a failure point and ensures that future changes don't reintroduce the defect -- or, at the least, it identifies a place in the code that my need to be revisited if future changes make a bug fix obsolete.

Re: No Silver Bullet

When I initially started reading No Silver Bullet, by Frederick P Brooks, Jr I was unaware of its first publish date, and first few pages have little temporal evidence. But the farther I got in to the article a couple things show up ... such as referencing time-sharing systems, "high level" languages and unix. All of which have been around for the outset of my programming life and had already reached a more "mature" state long before I came in to the software world.

I suppose the relative agelessness of the first few pages wouldn't have been apparent if I had read the bits of text that dated the article beforehand. I think the agelessness of the text is important. Because it speaks to the fact that the problems Brooks initially defined haven't really changed all the that much in software engineering. I work in a totally different computing world than what was around when the article was written yet the problems he describes don't sound foreign at all to me.

Brooks introduces several things that promised to "solve" the software problem. Yet 20 years later, none of these solutions have broken through to the main stream. People are still getting things done with imperfect software and still making money. Granted, certain methodologies that may've fallen out of the utopia of "perfect software" are very applicable to certain domains.

For most people, software that's "good enough" is all that's needed. For me the most important point Brooks makes is that software is hard, and nothing is going to make software easy, hence the title No Silver Bullet ... pursuing infallible software is fine for academia but it is probably an expensive loosing battle for 80% of software projects. Anecdotally, I've experienced this in my own professional experience through writing my own fair share bugs. The anecdote follows the following premise: write a bug, ship it to customers for period of time, record the period of time that the bug was in wild until it was fixed and released again. If you look at that time period and you look at how much revenue the software generated in that time period you can come up with a rough estimate of the "cost index" of that bug. If you make 150k over a 2 month period selling the software with the imperfection it seems to show that the software was still delivering value, even though it may have an (innumerable) amount of imperfections.

Project Infrastructure Usage Requirements

I kind of see this class as "playing house" for doing more serious project management on bigger projects in a professional setting. One difference is that most of us won't have to set up all the project management goo, we'll just have to use it. Using the term "playing house" is a little demeaning but it communicates some of the vibes I'm sensing from my group members (which might not actually be there, my group members have lots of other classwork to worry about, and this is the only class I'm taking).

One of these "vibes" is that it seems a little artificial to build up all this project management stuff and then actually use it, because we haven't had use for this stuff previously are we aren't forced to use it in this class. We'd probably be better initiated in to using this stuff if we had a standard set of tools that the whole class could play with, along with assignments like Using a Ticket System, or Using a Revision Control System. In the meantime, I don't know how usage the project infrastructure will see since their don't seem to be many hard requirements in the class to do so.

Mailing Lists and Project Management Documentation

Our new mailing list has been working swimmingly. At least until the whole engineering wing of UNM apparently lost power. Hopefully we'll be able to use our list again soon.

I didn't manage to make it to any classes this week, but I should have plenty of time to catch up on what I missed over spring break. Hopefully we'll also be able to hammer out more of the design stuff for our project so we can get started writing code.

While I wasn't going to class I contributed to the Project Managment document that we needed to write (which was due today at 5p), which outlined the basic project tasks, as well as (the part that I wrote) an outline of all the machinery we've set up the manage the project and how we intend to use it.

Google Groups, Loosing E-Mail, GMail Marking Google Groups (DomainKey signed) Email as Spam

We decided today to 86 my choice of Google Groups as our group communication mechanism. Our group lead (Ben) posted several emails to the list about meeting times for this Saturday, none of which went through. So the meeting ended up being rather last minute (since no one knew where it was). I had problems with my GMail account thinking that the Google Groups email was spam.

Whenever I would send email from the wrong account (since the default email gmail sends mail from is not the one that's subscribed to our list) I wouldn't get any feedback from the list that said I couldn't post to list (since that email address is not a member). I happened to peruse my spam folder and found all the "you can't post to this list" emails laying there. GMail's spam classifier apparently didn't think the chain of valid domain keys (from one of Google's own domains even) was enough to prevent the mail from being spam. Needless to say Google Groups is out, hopefully a mailman list provided by the CS department's System Support Group is in.

Xen and the Art of Not Functioning

For the most part I enjoy doing system administrator work ... like setting up the software for our group to use. However, the real meat of sysadmin's job is not dealing with software that works well, it's dealing with, and working around software that's broken (most of the time with out the ability to change code). 

Such was the case with an Ubuntu bug I ran in to that caused Apache and other processes to hang (sometimes) when calling readdir_r or readdir64_r (which is a thread safe version of the readdir call). The bug initially manifested when I first installed the Ubuntu O/S image for my new Virtual Private Server (VPS) -- I've had had a VPS with another provider, but this was the first time using Ubuntu Gusty (version 7.10). My VPS is provided by VPSLink (subsidiary of Spry), I found the following report on the VPSLink forums of problems with the particular O/S image I had installed. I followed the advice there and installed a different (still Ubuntu) O/S image and set up all the software packages I needed by hand (mainly just Apache).

Everything was good for a while until I ran in to a problem with attempting to do an SVN move (which is implemented as a delete and a copy from the previous revision). The WebDAV "COPY" command was failing because the Apache server was misconfigured (this document describes several misconfiguration scenarios including mine). I fixed the configuration problem and attempted to restart Apache ... low and behold Apache went in to what appeared to be a busy wait loop just pegging the CPU. What's better is that the process didn't appear to be any code that I could see with user land tools. The system call tracer strace would sit silent when attaching to the process. When I started Apache under strace I could see that the process was stopping after calling getdents. The library call tracer ltrace showed that execution stopped inside of apr_read_dir which coincides with what strace said (since it's likely that apr_read_dir would call getdents).

APR is the Apache portable runtime which is supposed to provide a cross platform shim for C applications. Out of desparation (since I'd killed Apache in the middle of trying to get my group members set-up to use SVN) I tried installing an Apache build with a different concurrency model (the non-threaded model) even though this didn't really seem like a likely cause of the problem, since it was clear from the strace output that Apache wasn't even getting past reading it's configuration file. To my chagrine this did not help ... but discovering that readdir64_r was the last thing on the stack (by attaching gdb to the running process, which I had never done with gdb before), led me to ask the Oracle (aka Google) about bugs involving readdir64_r, Ubuntu, and Xen and I found this bug report which describes an almost identical problem.

The only exception is that they were seeing a segfault instead of the process grinding the CPU. At the bottom of the above bug report they suggest installing the Xen variant of the C library. This solution fixed everything. Clearly my VPS provider does not do adequate testing of their O/S images.

Subversion, Trac, Mailing Lists

I've been working on getting the management infrastructure going for our project. So fact I've successfully set-up Trac for our project management along with Subversion. I've also imported Ben's prior work on Hex Chess. I plan on writing up at least some brief documentation describing how we're going to use these tools. Hopefully this information can become part of our project management document. The biggest problems I had with setting up Subversion and Trac was figuring how to navigate Apache configuration files. Apache documentation is great and examples are plentiful but I had trouble getting useful feed back from apache when it didn't like something (the log messages emitted weren't very helpful). Perhaps I just need to enable more verbose logging.

Our Trac and Subversion URLs are as follows:

https://trac.bgn.deadbeefbabe.org
https://svn.bgn.deadbeefbabe.org

The SSL certificates served are self-signed so you'll have to navigate past your browser's warnings that the site might be lying about who it is (even though communication will still be encrypted). Trac should be set-up so that only authenticated users can modify it (that is, members of our group). Obvious Subversion can only be modified by members of our group (hopefully, if I set everything up correctly). I plan on only using https for communicating with Subversion so hopefully the Eclipse Subversion plug-in fully supports https access to Subversion.

As for a mailing list, the easiest solution was to use Google Groups (http://googlegroups.com). As long as we don't mind Google reading about our CS 460 project it provides a really plainless way to get a mailing list set-up (and we can set the group so that only we can communicate on it / read archives). Google groups provides a cohesive conjoining of the "newsgroup" and the "mailing list" since they are both pretty much the same product with a different implementation this works really well. Obviously we'll be using the Google Group as a mailing list.

More Interfaces...

Recently on projects were I really wanted to test everything I've been creating interfaces more frequently. With object oriented languages interfaces are usually described pretty easily in some fashion, despite the fact that some languages may not directly support them.

This practice of "creating interfaces for everything" means that if I'm interacting with a class that runs processes the only way to test code that depends on this class is to either actually run test processes on the system or create something that pretends to run processes. There are many situations in which running processes is bad, in my case it was the fact the processes in question could take a long time to complete, in other cases you may not want unit tests to depend on the state of the system. So that leaves creating something that pretends to run processes.

Inheritance in languages like Java and C# can be a messy business, in many cases I don't need the complexity imposed on me by API level classes, there's usually only a few methods of an API class that I need and the usage is fairly simple. Because of this implementing a subclass of an API level classes can leave wholes open to the underlying code that I don't necessarily want to expose. In some cases API level classes can't be subclasses (in the case of sealed/final classes in C#/Java). Therefore, we add more indirection, and create an interface.

What good is creating an interface? It can describe exactly how I'm using the API level class and it only exposes the bits of functionality that I need to test. With an interface I can implement a real class that actually runs processes and a fake class that lies about running process and pretends to fail in certain cases in order to test how the code reacts to failure cases. The downside is that creating interfaces everywhere can increase the complexity of the code, more types just means more names and ideas that can contribute to confusion about the overall design of the code.

In the alternate case of subclassing the process class for testing, the code remains simple and the need to create 3 new types is avoided (1 for the interface, 1 for the real implementer, and 1 for the test implementer). However, I think masses of types can still be managed and using interfaces frequently when using hard to test API classes can make code more reliable and easily tested. Interfaces around complex API classes can also improve the descriptiveness of the code, in that the interface can be used to describe how API classes are being used.