When you generate a test case, you get a test class (which is a subclass of SenTestCase). Just like any other class, you can declare members and methods, which are used to perform helper tasks and carry state.

A very basic codebase only requires test methods. OCUnit will load any method prefixed by test. These methods must return void and take no parameters. After setting up the test in the method, you can use the ST... macros to actually test the results. Here’s a contrived example:

1
2
3
4
5

-(void)testAdder
{
    int result = add(1, 1); // should return 2
    STAssertTrue(result == 2, @"1+1 should be 2!");
}

There are a number of test macros, which are listed in Appendix B of the Xcode Unit Testing Guide.

A real example taken from FlexArgs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

- (void)test_init_with_NSArray
{
    NSArray *testArgs = [NSArray arrayWithObjects:
                         @"foo=bar",
                         @"baz=1",
                         @"quux=-2.5",
                         @"spam=footastic",
                         @"eggs=false", nil];
    NSDictionary *expected = [NSDictionary dictionaryWithObjectsAndKeys:
                              @"bar", @"foo",
                              [NSNumber numberWithLongLong:1], @"baz",
                              [NSNumber numberWithDouble:-2.5], @"quux",
                              @"footastic", @"spam",
                              [NSNumber numberWithBool:false], @"eggs", nil];
    DNFlexArgs *flexArgs = [[DNFlexArgs alloc] initParserWithNSArray:testArgs];
    NSDictionary *parsed = [flexArgs retrieveArgs];

    STAssertTrue([parsed isEqualToDictionary:expected],
                 @"initParserWithNSArray failed to return the expected NSDictionary!");
}

This test is set up to verify that the dictionary returned by DNFlexArgs is what we’d expect it to be. [flexargs initParserWithNSArray:] expects an NSArray of NSStrings to be passed to it, I’ve initialised an NSArray of NSStrings. Because DNFlexArgs returns an NSDictionary, I’ve set up one with the expected results. Then I setup the instance, pass it the initial test arguments, and retrieve the results. The last step is to assert that the returned NSDictionary matches the expected one. I’ve made sure to hit one of each kind of possible argument that can be passed in.

In the last post, I listed some questions to help develop tests:

• Right now, I’m only testing what I expect possible input could be. What if someone passes in just arg or arg= - do I know that my code will handle that gracefully?
• What happens if I pass in an overflowed numeric value? I’ve tried to prepare for this by using long long values, but how do I know my code is doing the right thing?
• What if I craft a special string that does something shifty, like embedding a null byte? What happens then? What if the string has more than one = in it?
• What happens if I feed the parser random data?

If you look in the source, you’ll see I’ve covered all but the last. That’s because I haven’t yet found a good fuzzing library for Objective-C. However, the new tests allowed me to make a few improvements to and verifications of FlexArgs:

• I’ve verified that having an argument with no =value yields a boolean value (i.e. arg results in an arg = 1;)
• I’ve verified that having an argument with an empty value (i.e. arg=) yields an empty string value (i.e. arg = @"";)
• I was able to add support for multiple = in the class. Previously, only the part of the value after the first = and before any other =’s was captured. For example, passing foo=bar=baz resulted in foo = @"bar"; and operator=== resulted in operator = @"";. Now, I get foo = @"bar=baz"; and operator = @"==";.
• I was able to verify that passing a null byte in the middle of the string just cut the string off at the null byte instead of causing problems.

You can see the new code coverage output (as detailed in the last post) here.

More advanced test cases with setUp, tearDown, and class members

OCUnit gives us more control over our test cases. Just like any other class, we can include our own members in the class by putting their definitions and any @property declarations in the interface. For example, if we’re testing network code, we might want to create a socket.

If your members need to be set up for every test, or if certain preparation needs to be done before each test (like clearing out a temporary directory), you can reduce code duplication by putting the code in the setUp and tearDown methods. The setUp method is called before each test method, and the tearDown method is called after each. If you’re calling the same code before each test, you might consider moving them. If most of your tests are calling the same code and a few aren’t, consider creating a new test case for the ones that don’t, and moving the duplicate code into these methods and members.

guard-xcode

In the last post, I mentioned guard. What is guard?

Guard is a command line tool to easily handle events on file system modifications.

We can use this to trigger a build every time a source file is changed. Unfortunately, I couldn’t find any good Guards (the term for specific tasks to be done on a changed-file event) to handle running configurable builds. To address this, I wrote a Guard called guard-xcode, which kicks off an Xcode based on the options you configure it with. The source is, of course, on Github and it’s on RubyGems.org, so it can be installed via gem or bundle install. The [README](https://github.com/kisom/guard-xcode/blob/master/README.md) explains how to get started.

For FlexArgs, the setup is fairly straightforward. I already have growl-notify installed, so all I have to do is create my Gemfile:

1
2
3
4
5
6

source :rubygems

gem "guard-xcode"
gem "guard"
gem "growl"
gem "rb-readline"   # better interface for MRI

I’m using rvm, so I’ve already got the bundle gem installed. You can install it with gem install bundle if you need to. The next step is to run bundle install, and then guard init xcode. Of course, the Guardfile doesn’t know the name of your target, so you’ll need to open the Guardfile and edit it. Mine looks like this:

1
2
3
4
5
6
7

# template Xcode guard

notification :growl

guard :xcode, :target => 'FlexArgs', :quiet => true, :clean => true do
  watch(/^.+\.[hmc]$/)
end

Now all you need to do is run guard in your project root. Once files change, guard will kick off a build. I’ve set :quiet => true, so I only get Growl notifications if the build has warnings or errors.

Some final ideas for a test-based development cycle

There’s a few options you can set in your project build settings that I’ve found quite useful:

1. Setting Test After Build to Yes runs tests anytime a build is done.
2. Setting Treat Warnings as Errors to Yes adds more emphasis to writing good code.
3. Adding the Test product to the main build makes testing easier as well, and means you can set your build target to the main target, and still test.

Conclusion

This concludes the two-part series on Unit Testing in Xcode. I’ve tried to document what I learned trying to get testing set up, and hopefully other people will find it helpful as well.

References

• XCode Unit Testing Guide
• Code Coverage with Xcode 4.2
• FlexArgs tag for this project
• guard-xcode on Github and RubyGems
• Guard on Github
• growl-notify
• rvm

Through version 0.9.0, which was a functional version of the code missing a few other pieces, I was using the autotools suite to manage the build. (Why? I know autotools, and can set up the build environment quickly, whereas I don’t use Xcode enough to be terribly good at it.) Unfortunately, I couldn’t find any good libraries to do it. One of the goals of the project was to learn Objective-C better, so I buckled down and imported the project into Xcode. I subscribe to the idea that it’s a good idea to start writing tests before you write code, so writing tests after the bulk of the code was written felt a little janky. I digress.

First things you’ll need to do is add a new target (File->New->Target) and name it (@"%@Tests", ProjectName) (ex. MyClassTests), and save it. Because this is a library I want to give other people, it’s set up as a CLI application with a minimal main that gives an example of how to use code, and so we don’t really need coverage for that. However, the test suite should be exercising large portions of our code, so setting up code coverage for that is a good way to make sure your tests are exercising your code fully.

I’d like to point out here that while code coverage is great for making sure all of your code is being touched, it’s not a replacement for well-thought-out tests. It’s a useful tool in the tool box while you’re developing code, and great for profiling code to determine bottlenecks (but not prematurely!), but you still need to make sure you’re testing your code fully. I try to take the time to consider edge cases, places where the code might act inappropriately, and try to implement some fuzzing to throw unexpected things at the code.

You can set up your main target to run tests as well. You’ll need to edit the scheme:

Once there, select Test, and click the + to add the test case bundle.

Now, we can set up code coverage. Under the project settings, select the test target. Under the build settings, change the Generate Test Coverage Files and Instrument Program Flow options to Yes. Now the fun part is adding in libprofile_rt.dylib. I found it under /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib in OS X 10.8. Under the Build Phases tab, you’ll need to add this to the Link Binary with Libraries section. Once you start running tests, you’ll start getting coverage data.

I lied about there being only one fun part. The other fun part is getting that data, which you will find in the Projects tab in the Organiser. There is a line called Derived Data with an arrow next to it to open that folder in Finder. You’ll want to open that folder in your terminal emulator of choice. For FlexArgs, I had to navigate to DERIVED_DATA/Intermediates/FlexArgs.build/Debug/FlexArgsTest.build/Objects-normal/x86_64/. There, you should see some files named .gcno, .gcda, and so forth.

At this point you’ll want to install lcov which is, most fortuitously, in HomeBrew. lcov gives us pretty HTML output of our code coverage (via the included genhtml program). You’ll also want to stick this small script in your path:

1
2
3
4
5
6
7
8
9

#!/bin/sh
# ccovhtml.sh
# usage:
#   ccovhtml.sh ClassName OutputDirectory
# example:
#   ccovhtml.sh FlexArgsTests ~/tmp/FlexArgsCoverage

lcov --base-directory . --directory . -c -o $1.info
genhtml -o $2 -t "$1 code coverage" --num-spaces 4 $1.info

The first argument is the name of output file to generate, and is also used to generate the title for the HTML output. The second is the output directory to store the files in. Once you’ve run your tests, run that script and check out the results. For my code, it built this page. Note that this is with two tests that don’t yet test the full functionality (specifically +(id)parserWithNSArray:(NSArray *)inargv and -(id)initParser:(char **)inargv nargs:(int)nargs), so you can see that the output highlights those lines. If you take a look at the FlexArgs source at commit bec86374f3 or the helpful tag blog-post (you can grab a zipfile snapshot) you’ll see I only have two tests, and they both touch basic functionality.

Intelligent and well-planned and executed tests offer many benefits:

1. Validate the program’s logic
2. Drive development by letting you see what’s not implemented yet
3. Perform regression tests to let you know if changes broke your code, or if they’ve broken other parts of the codebase
4. Test edge cases to make sure your code doesn’t do anything unexpected
5. Assist in identifying possible security issues.

So given this code, what kinds of tests could I write to improve the functionality?

• Right now, I’m only testing what I expect possible input could be. What if someone passes in just arg or arg= - do I know that my code will handle that gracefully?
• What happens if I pass in an overflowed numeric value? I’ve tried to prepare for this by using long long values, but how do I know my code is doing the right thing?
• What if I craft a special string that does something shifty, like embedding a null byte? What happens then? What if the string has more than one = in it?
• What happens if I feed the parser random data?

As you can see, there’s a lot to think about. While writing tests ahead of time to verify basic functionality is a great idea (I wrote about README-Generated Test-Drive Development previously), your tests need to go further to fully verify your code. Just by looking at the questions above and thinking about the tests, I can already see that my code needs some work to address some of those questions. I can write the tests to validate the changes I’ll need to make.

I mentioned at the beginning that I like to test models using command line test drivers. What this means is that I write a small command line target that I can call from something like make tests or even python testrunner.py so I can constantly run my tests. This way, I don’t need to worry about the view or the controllers to develop the model. This follows my ideal of developing model first, and letting the controller and view follow from that. In Xcode, we can do this from the commandline inside the project using xcodebuild -target FlexArgsTest -configuration Debug clean build. Before you run this, set Test After Build to Yes in the Build Settings to ensure the tests will run after building. (At some point, I’ll write a Guardfile to automate testing.)

I hope you find this useful. Now, if you’ll excuse me - I have more tests to write…

Update: I’ve written a part 2, which covers a bit more and includes black magic where the dark lord has destroyed my soul and brought death, destruction, and chaos upon the world a Ruby gem I wrote to assist in testing.

References

• XCode Unit Testing Guide
• Code Coverage with Xcode 4.2
• Regression Testing
• Fuzzing
• README-Generated Test-Driven Development

(Obligatory xkcd…)

This unit was all Monte Carlo localisation, which generates probability distributions as to where the robot is in the world. This, of course, is all very simplified but it is still quite a fascinating learning experience.

The final homework question has us determining a probability distribution (i.e. a matrix of probabilities guessing where the robot is in the world) based on sensor readings and movements in a world of red / green cells. For example, one of the first the worlds we’re given in the examples looks like this:

You can see each cell in the world has a colour, and the robot’s worldview consists of a 2D matrix of probabilities.

Given a list of sensor readings, i.e. ['green', 'red'] and a list of corresponding motions in the form [y, x] such that [1, 0] is a movement downwards and [0, 1] is a movement right, the robot should be able to figure out where in the world it is. All the example code in the class is done as simple functions operating in the '__main__' namespace, but I used a Robot class to simulate everything. Testing was painful, because I had to hand-type in most of the reference probability distributions. Then, early Sunday morning (actually late Saturday night), I discovered the colorama module, which faciliates pretty terminal outputs (i.e. colors, bolding, etc…) I was able to write a method to print out the map (aka the .showmap() method in the screenshots) that made testing where the robot was a lot more fun.

The red/green world felt very contrived. I had a pretty good unit testing framework set up, so I decided to test my code using a road test: I built a world with three lanes that used black to represent the asphalt, yellow to indicate lane markers, and white to indicate the shoulder marker. I simulated having the robot start at the shoulder, drive forward and then switch to the middle lane. I expected the robot should believe itself to be somewhere in the middle lane.

The world looks something like this:

With that in mind, I wrote the road test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import localisation         # the module with the robot code

def road_test():
        world = [['black', 'yellow', 'black', 'yellow', 'black', 'white'],
                 ['black', 'black', 'black', 'black', 'black', 'white'],
                 ['black', 'yellow', 'black', 'yellow', 'black', 'white'],
                 ['black', 'black', 'black', 'black', 'black', 'white'],
                 ['black', 'yellow', 'black', 'yellow', 'black', 'white'],
                 ['black', 'black', 'black', 'black', 'black', 'white']]
        sensor = localisation.Sensor(0.8)
        robot = localisation.Robot2D(world, sensor, 0.6)
        print 'robot initalised.'

        motion = [[0, 0], [-1, 0], [0, -1], [0, -1], [0, -1], [-1, 0]]
        measurements = ['white', 'white', 'black', 'yellow', 'black', 'black']
        print 'driving!'
        robot.localise(motion, measurements)
        print translate_drive(motion, measurements)

        guess = robot.locate()['guess']
        q = [(i, j) for p, j, i in guess]
      q = [i == 2 for i, j in q]
      return robot

The translate_drive function maps the motions and measurements into a pretty output form to make it more intuitive as to what’s being simulated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

def translate_drive(motion_seq, readings):
    directions = {
                    '[0, 0]':   'nowhere',
                    '[0, 1]':   'right',
                    '[0, -1]':  'left',
                    '[1, 0]':   'down',
                    '[-1, 0]':  'up'
    }

    drive_str = ''
    for i in range(len(motion_seq)):
        motion = motion_seq[i]
        reading = readings[i]
        drive_str += '\tgo ' + directions[str(motion)]
        drive_str += ', saw ' + reading + '\n'

    return drive_str

Here’s an example of the road test in action:

As expected, the robot thinks it is somewhere in the middle lane.

Next week, we start learning particle filters and kalman filters.

I’m actively working to develop a physical AGV (autonomous ground vehicle). The objective is to develop a platform I can use to build later, more practical robots on; the AGV platform will be focused on navigation. As part of this task, I’m also working to translate the Python code to C++ (suitable for the Arduino, for example).

Stay tuned!

References:

• Robust Monte Carlo Localization for Mobile Robots
• Python versions of the repo) (private until the 28th)
• C++ version

For a long time, I used mostly git and far less mercurial. I wrote a bunch of aliases in my .zprofile that looked something like:

1
2
3

alias commit='git commit'
alias checkout='git checkout'
...

For mercurial, I just entered everything normally. However, as I started to use mercurial more, I wanted to use those aliases for both systems. I ended up writing a bunch of shell functions to do this. They are all strict POSIX compatible, so they work under at least zsh, ksh, and bash. I haven’t tested any others, so your mileage may vary. The latest version of this is available at my dotconf github repo, you can view it here

The core of the code is the pair of functions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

get_repo_type () {
    git status 2>/dev/null 1>/dev/null
    if [ 0 -eq $? ]; then
        echo 1
        return 1
    else
        hg status 2>/dev/null 1>/dev/null
        if [ 0 -eq $? ]; then
            echo 2
            return 2
        else
            echo 0
            return 0
        fi
    fi
}

not_a_repo () {
    echo "not a git or mercurial repo!"
}

get_repo_type does exactly what it says it does: it outputs a number that identifies what type of source control the repo uses. The not-a_repo simple provides a shortcut for displaying the error message. All of the commands use these two functions. The commands are implemented in a similar style, so let’s take a look at the first defined function, pull:

1
2
3
4
5
6
7
8
9
10

pull () {
    repo_type=$(get_repo_type)
    if [ "1" = "$repo_type" ]; then
        git pull $@
    elif [ "2" = "$repo_type" ]; then
        hg pull $@
    else
        not_a_repo
    fi
}

Unfortunately, shell scripting isn’t a terribly advanced programming language, so there’s a lot of redundancy in the code; in fact all of the commands use the same basic template of

1
2
3
4
5
6
7
8

repo_type=$(get_repo_type)
if [ "1" = "$repo_type" ]; then
   # git commands go here
elif [ "2" = "$repo_type" ]; then
   # hg commands go here
else
   not_a_repo
fi

I thought of some other ways to do this, but they all ended up being far more complex and time-consuming than just knocking it out like this. This style is also POSIX-compatible, meaning it can be used with really any shell.

Another feature of note is that I’ve ensured to pass through the shell variable $@, which means any arguments are passed directly to the command; this lets you still enable the full use of the specialised commands without having to mentally switch context between just typing the shortened command and the full one.

So, let’s look at what commands are supported (use vcshelp to list them):

• commit
• add
• pull
• push
• checkout
• fetch
• clog
• which_dvcs
• vcdiff

For the most part, they are wrappers around the $scm version of the command, passing through any arguments as before. The last three aren’t (but do pass through any command line options as appropriate):

• clog is a shortcut for ‘commit log’, and shows the $scm log. For mercurial, it will pipe it to less (by default, hg doesn’t).
• which_dvcs is a wrapper around get_repo_type to print the name of the SCM instead of the numeric value used in the functions.
• vcdiff is a version control diff; like clog, it will pipe hg diff to less.

There are a few commands that aren’t documented in vcshelp: * co is an alias for checkout * st is a variant of status that shows only tracked files

I’ve found this system to work out pretty well for me, mostly because it requires less mental power to handle the basic SCM workflow. It also satisfies my coder’s itch to remove unnecessary code (i.e. always having to prefix git or hg to source control commands) by making the shell “aware” of which SCM I’m using at the time.

There is, of course the caveat that Josh Rickmar pointed out. I’ve come to grow used to a lot of the specifics of working with the different SCMs. Two common idioms I use a lot with this setup are commit -a in a git repo and pull -u in a mercurial repo. If you are using an SCM, you should definitely get to know it before using it for serious work. Of course, you can also take my code and tweak it so that it behaves differently. The code is yours.

Thanks go to Chris LePetit, who suggested I write the article. Samuel Goodwin and Wally Jones proofread it for me.

In order to improve this situation, I took two steps:

1. I enabled Google Analytics: while I’m not a particular fan of feeding the privacy black hole with even more data, it appears to be the only viable option at this time. I have been eyeing Mint, but I am abstaining from purchasing anything new until I leave the Netherlands (in a little over a week) just to stay on the safe side of my bank account. (30 USD may not sound like much, but that’s about two days worth of döner or shawarma for dinner.)
2. I set up the site on CloudFlare.

What I didn’t realise is that several of my older posts actually rank high on Google’s search results; however, the link on Google points to the old url from when I was using blazeblogger. Because the content is actually useful documentation, I was able to set up redirects so that the page is back online and people can use the information now.

The App Store as a sole source of software

I’ve heard a lot of complaints about this. The fact of the matter is that for the average user, this makes sense. It limits the exposure due to malware. For users who need expanded privileges, it is extremely easy to do. This follows a principle of secure by default and requiring a decision on the user’s part to open up their system. Does Apple benefit? Surely. However, in this case the decision also benefits users.

Messages

We are surrounded by a variety of chat systems. On mobile phones, SMS reigns supreme; while on the desktop, there are more chat protocols than you can shake a stick at. Bringing Messages onto the desktop starts to unify the two systems. While Jabber and AIM / OSCAR are two very common protocols, it would be good to see Twitter and perhaps even IRC. (While I would love to see SILC supported, I don’t expect to see that anytime soon). One other thing missing is OTR support. While the iMessage compatibility is interesting, lack of Twitter and OTR support make it not compelling enough for me to switch over as my chat client*.

XCode

Apple is now providing a much smaller download with just the compiler and commandline tools needed to use Homebrew or other package management systems.

Conclusion

We’re starting to see some of the benefits of tablet revolution folded back into the desktop and laptop realm. For normal users, these changes create a more streamlined experience and improve the overall security of the system. For developers, there are some changes to simplify non-iOS / Cocoa developers’ setups. I think this is a step in the right direction.

Except that it’s called “Mountain Lion.”

Footnotes

1. I currently use Adium for my chat and twitter client.

I assume Aquamacs has been downloaded and leiningen is installed. First, in a terminal, you’ll need to install swank-clojure. As of today, the current version is 1.4.0, but I strongly recommend you check the README to see if there’s a new version out. In the shell, lein plugin install swank-clojure "1.4.0".

I use Marmalade for package management, so the first thing to do is to add Marmalade to Aquamacs. Open up "~/Library/Preferences/Aquamacs\ Emacs/Preferences.el" in your editor of choice (I used MacVim), and add the folowing:

1
2
3
4
5

;; Marmalade
(require 'package)
(add-to-list 'package-archives
             '("marmalade" . "http://marmalade-repo.org/packages/"))
(package-initialize)

I’m assuming you don’t have package.el installed yet, so make sure to

1

`curl "http://repo.or.cz/w/emacs.git/blob_plain/1a0a666f941c99882093d7bd08ced15033bc3f0c:/lisp/emacs-lisp/package.el" > ~/Library/Preferences/Aquamacs\ Emacs/package.el`

Now fire up Aquamacs (or evaluate the additions to Preferences.el with C-x C-e. clojure-mode needs to be installed, either via M-x package-list-packages, and marking clojure-mode for installation (with i) and installing (with x), or with M-x package-refresh-contents followed by M-x package-install clojure-mode. I also like paredit but you might not, it takes some getting used to.

Now, open up a file in your lein’d project and use M-x clojure-jack-in. You might see some errors pop up in your *Compile-Log* buffer, but you should be very shortly greeted with a REPL.

Happy hacking!

The End Result

Here’s a screenshot of how it turned out (click to view it full-size):

I usually run aquamacs full-screen with two panes, left-side for editing source code and right-size for SLIME.

References

I patched together my knowledge from a couple of pages:

• Incanter’s article Setting up Clojure, Incanter, Emacs, Slime, Swank, and Paredit
• The Doctor What’s article Aquamacs 2.3a and Marmalade
• Phil Hagelberg’s swank-clojure README

One of the issues with practically using the code in the last post is that the initial subsets were defined arbitrarily and not derived from the superset. In this post, all the examples are derived from the superset. We’ll use a couple techniques for doing this illustrate some of the various ways to do it.

In Python, we’ll use an object-oriented approach, creating a few classes and working on Book objects. In Clojure, we’ll use records. Though we’ll approach language a little differently, I hope they still bring clarity to the subject.

Foundation: A Collection of Books

The first thing we need to do in a useful system is determine what we mean by book. The last post represented each book as a string denoting the title; while that worked for a brief introduction, in practise it gives us very limited options for building subsets. What we need to do is identify more information, called attributes or fields, that give us the information we need to build our subsets.

Python

In Python, we’ll approach this using a class. I’ve saved them in library.py in the Python example code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# used to validate the list of formats passed to a book
SUPPORTED_FORMATS = [ 'epub', 'mobi', 'pdf' ]


class Book:
    """
    Represents a book, with title, author, and summary text fields. A book
    should be given a list of formats supported as a dictionary in the form
    {fmt: True}, and optionally a list of tags.
    """
    title = None
    author = None
    summary = None
    formats = None
    tags = None

    def __init__(self, title, author, summary, formats):
        """Initalise a new book. The format shoud be a dictiontary in
        the form { 'epub': True } where each key is a format that we
        have the book in."""

        self.title = title
        self.author = author
        self.summary = summary

        assert(not False in [fmt in SUPPORTED_FORMATS for fmt in formats])
        self.formats = formats

    def __str__(self):
        """
        Return string representation of a book.
        """
        out = "%s\n\tby %s\n\t%s\n\tformats: %s"
        out = out % (self.title, ', '.join(self.author),
                     self.summary, ', '.join(self.formats))
        return out

We’ll also want a BookCollection class to store a set of books and provide some utility methods for dealing with the collection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

class BookCollection:
    """Representation of a collection of books. Internally, they are stored
    as a set. It's main utility is in its helper methods that make accessing
    the books easier."""

    def __init__(self, books, book_filter=None):
        """Instantiate a collection of books. It expects a collection of
        books, e.g. a list or set, and optionally takes a filter to
        only put some of the books into the collection."""

        if book_filter:
            self.books = set([book for book in books if book_filter(book)])
        else:
            self.books = set(books)

    def __len__(self):
        return len(self.books)

    def show_titles(self, description=None):
        """Print a list of titles in the collection. If the description
        argument is supplied, it is printed first and all the books are
        printed with a preceding tab."""
        if description:
            print description
            fmt = '\t%s'
        else:
            fmt = '%s'

        for book in self.books:
            print fmt % (book.title, )

    def get_titles(self):
        """Return a list of titles in the collection."""
        return [book.title for book in self.books]

These two classes are very short (and we’ll extend them later to make them more useful) but provide a solid foundation to begin building on. You’ll want to load the books in the class.

To load an example book, you would do use code similar to this:

1
2
3
4
5
6
7
8
9
10

books = set([
    Book("Natural Language Processing with Python",
         ['Steven Bird', 'Ewan Klein', 'Edward Loper'],
         'A highly accessible introduction to natural language processing.',
         ['mobi', ]),
    Book('Learning OpenCV', ['Gary Bradski', 'Adrian Kaehler'],
         'Puts you in the middle of the rapidly expanding field of ' +
         'computer vision.',
         ['pdf',])
    ])

Manually entering all these details is tedious. Fortunately for you, I put up with the tedium to create a sample dataset in sample_library.py. You use the function get_library() from the file to use it.

Clojure

In Clojure, we’ll use a record to define a book:

1
2
3
4
5

;; define a book record
(defrecord Book
  #^{ :doc "Representation of a book. title is a string, authors a vector, 
summary is text, and formats is a vector." }
  [title authors summary formats])

We’re not using objects, so we don’t need a record to store a collection. (If we wanted to validate formats, we could do it using a Ref and a :validator argument - that’s left as an exercise for the reader). I have, however, defined a few helper functions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

(defn in?
  "Check whether val is in coll."
  [coll val]
  (if (map? coll)
    (val coll)
    (not= -1 (.indexOf coll val))))

;; format validation
(def valid-format?
  "Check a record or object with a :formats key to ensure it fits the list
of valid formats."
  #(or (in? (:formats %) :epub)
       (in? (:formats %) :mobi)
       (in? (:formats %) :pdf)))

(defn list-titles
  "Print a list of titles of a book."
  [books & description]
  (let [titles  (map :title books)]
    (if description
      (do
        (println description)
        (doseq [title titles]
          (println "\t" title)))
      (doseq [title titles]
        (println title)))))

(defn get-titles
  "Get a list of titles of a book collection."
  [books]
  (map :title books))

(defn book-str
  "Return a book as a string."
  [book]
  (format "%s\n(by %s\n\t%s\n\tformats: %s\n"
          (str (:title book))
          (join ", " (:authors book))
          (str (:summary book))
          (join ", " (map #'name (:format book)))))

Adding books is a simple affair:

1
2
3
4
5
6
7
8
9

(set 
   [(Book. "Natural Language Processing with Python"
           ["Steven Bird" "Ewan Klein" "Edward Loper" ]
           "A highly accessible introduction to natural language processing."
           [ :mobi ])
    (Book. "Learning OpenCV" ["Gary Bradski" "Adrian Kaehler"]
           (str "Puts you in the middle of the rapidly expanding field of "
                "computer vision")
      [ :pdf ])])

I’ve loaded a sample dataset into the sample_library.clj source file, available from the Clojure example code.

Building Subsets

Now that we have a way to represent a book (with more useful information than simply the title), we can start to build some subsets. Let’s start by looking at set notation (aka how to write a set both mathematically and in code), and then continue on to recreate the two subsets in the previous article, epub and mobi.

Set Notation

In set notation, we denote a set by writing:

A = { x | x ∈ N, x < 10 }

which means the set of numbers that are members of (∈ means ‘element of’) the set of positive integers and are less than 10. You might generalise this as such:

given the universal set S, which defines all the elements under consideration, and some predicate P which is a function that returns either true if the element satisfies the predicate (and thus should be included in the set):
{ x | x ∈ S, P(x) }

We would express this set as:

A = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }

In Python, this is easily expressed with a list comprehension (see also the Python documentation:

1
2
3
4
5
6
7
8

# a Python list comprehension isn't aware that once N is above 10, it should
# terminate, so we cheat and create a list of integers from 1 to 100.

# define N
N = range(100)

# build the set
A = [ x for x in N if x < 10 ]

And in Clojure, we could use something similar:

1
2
3
4
5
6

;; define N
(def N (iterate inc 0))
(def N #^{:doc "Representation of the set of positive integers."} N)

;; build the set
(filter #(< % 10) N)

Building the Subsets

As mentioned earlier, I have already built sample datasets for both Python and Clojure, so be sure to use those and save yourself from having to build your own just yet!

Python

In Python, we can use the built-in filter function to build a list. It will serve as our predicate function.

1
2
3
4
5
6
7
8
9
10
11
12
13

import library
import sample_library

# my_library is our superset
MY_LIBRARY = sample_library.get_library()

# build our filters
IS_EPUB = lambda book: 'epub' in book.formats
IS_MOBI = lambda book: 'mobi' in book.formats

# build the subsets
EPUB = library.BookCollection(MY_LIBRARY.books, IS_EPUB)
MOBI = library.BookCollection(MY_LIBRARY.books, IS_MOBI)

This gives me the output:

1
2
3
4
5
6
7
8
9
10
11
12
13

In [7]: formats.EPUB.show_titles("books in epub format:")
books in epub format:
  Code Complete
  The Joy of Clojure
  Mining the Social Web

In [8]: formats.MOBI.show_titles("books in mobi format:")
books in mobi format:
  Introduction to Information Retrieval
  Code Complete
  Natural Language Processing with Python

In [9]:

If you recall the definition of BookCollection, the filter method is called as filter(predicate, collection). In the case of the mobi subset, it filters out anything that fails the test 'mobi' in book.formats. We might write this as

{ book | book ∈ my_library, is_mobi(book) }

in set notation. I’ve predefined some filters in the file formats.py which is again in the example code.

Clojure

Likewise, Clojure has a built-in filter function, in the form (filter pred coll). We’ll use two anonymous functions to do our filtering:

1
2
3
4
5
6
7
8
9
10
11
12
13

(use 'using_set_theory.library)
(use 'using_set_theory.sample_library)

(def epub? #(in? (:formats %) :epub))

(def mobi? #(in? (:formats %) :mobi))

(def my-library (get-library))
(def epub (filter epub? my-library))
(def mobi (filter mobi? my-library))

(list-titles epub "list of books in epub format:")
(list-titles mobi "list of books in mobi format:")

In the repl, this gives me:

1
2
3
4
5
6
7
8
9
10
11
12
13

using_set_theory.core=> (list-titles epub "list of books in epub format:")
(list of books in epub format:)
   The Joy of Clojure
   Mining the Social Web
   Code Complete
nil
using_set_theory.core=> (list-titles mobi "list of books in mobi format:")
(list of books in mobi format:)
   Introduction to Information Retrieval
   Natural Language Processing with Python
   Code Complete
nil
using_set_theory.core=>

I’ve put these filters in the filters.clj source file, along with definitions for epub-books and mobi-books:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

(ns using_set_theory.filters
  (:use [using_set_theory.library]
        [using_set_theory.sample_library]))

(def epub?
  #^{:doc "Filter a collection of books by those supporting the epub format."}
  (fn [book] (in? (:formats book) :epub)))

(def mobi?
  #^{:doc "Filter a collection of books by those supporting the mobi format."}
  (fn [book] (in? (:formats book) :mobi)))

(defn- get-epub
  "Takes a collection of books and returns the list of books in epub format."
  [books]
  (filter epub? books))

(defn- get-mobi
  "Takes a collection of books and returns the list of book in mobi format."
  [books]
  (filter mobi? books))

(def epub-books (set (get-epub (get-library))))
(def mobi-books (set (get-mobi (get-library))))

Parallels with SQL

This introduction of filters might remind you of SQL, and for good reason. Edgar Codd designed SQL with set theory in mind. You can think of tables as sets (provided, of course, proper data preparation is done to ensure there are no duplicates in the database), and operations like SELECT return subsets. For example, if we were storing the books in a library, we would write something like

1

SELECT * FROM books WHERE has_epub = TRUE;

Moving On

Now that we have a programmatic way to build subsets, we can automate the entire set of sequences in the last post:

Python

1
2
3
4
5
6
7
8
9
10

import formats
import library

either_format = set.union(formats.EPUB.books, formats.MOBI.books)
either_format = library.BookCollection(either_format)
either_format.show_titles("books in either format:")

both_formats = set.intersection(formats.EPUB.books, formats.MOBI.books)
both_formats = library.BookCollection(both_formats)
both_formats.show_titles("books in both formats:")

which gives me the results:

1
2
3
4
5
6
7
8
9
10
11
12

In [31]: either_format.show_titles("books in either format:")
Out[31]:
books in either format:
  Code Complete
  Mining the Social Web
  Natural Language Processing with Python
  Introduction to Information Retrieval
  The Joy of Clojure
In [32]: both_formats.show_titles("books in both formats:")
Out[32]:
books in both formats:
        Code Complete

Clojure

1
2
3
4
5
6
7
8
9
10
11

(require 'clojure.set)
(use 'using_set_theory.filters)
(use 'using_set_theory.library)

(def either-format
    (clojure.set/union epub-books mobi-books))
(def both-formats
    (clojure.set/intersection epub-books mobi-books))

(show-titles either-format "books in either format:")
(show-titles both-formats "books in both formats:")

In the Clojure REPL, I get the following output:

1
2
3
4
5
6
7
8
9
10
11
12
13

using_set_theory.core=> (show-titles either-format "books in either format:")
(books in either format:)
   Introduction to Information Retrieval
   The Joy of Clojure
   Natural Language Processing with Python
   Code Complete
   Mining the Social Web
nil
using_set_theory.core=> (show-titles both-formats "books in both formats:")
(books in both formats:)
        Code Complete
nil
using_set_theory.core=>

Sets v. Lists

Remember that one of the key attributes of a set is that each member is distinct. Let’s compare a set with a list; we’ll do this with an intersection.

Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

import library
import formats
from sample_library import get_library

epub_list = [book for book in get_library().books
             if 'epub' in book.formats]
mobi_list = [book for book in get_library().books
             if 'mobi' in book.formats]

both_formats = []
both_formats.extend(list(epub_list))
both_formats.extend(list(mobi_list))
print 'books in both formats:'
for book in both_formats:
    print '\t%s' % book.title
  

The result:

1
2
3
4
5
6
7

books in both formats:
  Mining the Social Web
  Code Complete
  The Joy of Clojure
  Code Complete
  Natural Language Processing with Python
  Introduction to Information Retrieval

Clojure

In Clojure, we’ll use the vector type, which is like a list but the first element isn’t evaluated:

1
2
3
4
5
6
7
8

(use 'using_set_theory.library)
(use 'using_set_theory.sample_library)
(use 'using_set_theory.filters)
(use '[clojure.contrib.seq-utils :only [includes?]])

(def epub-list (vec (map :title epub-books)))
(def mobi-list (vec (map :title mobi-books)))
(def both-list (concat epub-list mobi-list))

Which yields:

1
2
3
4
5
6
7
8

clojure.core=> (doseq [title (union epub-list mobi-list)] (println title))
The Joy of Clojure
Code Complete
Mining the Social Web
Introduction to Information Retrieval
Natural Language Processing with Python
nil
clojure.core=>

So what?

You’ll notice “Code Complete” shows up twice in the list. The advantage of sets here is that only unique items are returned. A union is actually the list of elements in both sets, minus the list of items that are in both sets.

A Second Stab: Python

Implementing the set operations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

in_both = lambda x, a, b: x in a and x in b

def intersect(seta, setb):
    both_list = []
    both_list.extend(seta)
    both_list.extend(setb)
    intersect_list = []
    temp_list = both_list[:]

    while not temp_list == []:
        element = temp_list.pop()
        if not element in intersect_list:
            if in_both(element, seta, setb):
                intersect_list.append(element)

    return intersect_list

def union(seta, setb):
    both_list = []
    both_list.extend(seta)
    both_list.extend(setb)
    intersect_list = intersect(seta, setb)

    while not intersect_list == []:
        element = intersect_list.pop()
        while both_list.count(element) > 1:
            both_list.remove(element)

    return both_list

Applying this to our lists:

1
2
3
4
5
6
7
8
9

In [30]: union(epub_list, mobi_list)
Out[30]:
['Mining the Social Web',
 'The Joy of Clojure',
 'Code Complete',
 'Introduction to Information Retrieval',
 'Natural Language Processing with Python']

In [31]:

A Second Stab: Clojure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

(ns myset
  (:use [clojure.contrib.seq-utils :only [includes?]]))

(defn unique? [el ulst]
  (= 0 (count (filter #(= % el) ulst))))

(defn get-intersect [ilist seta setb both-list]
  (if (empty? both-list)
    ilist
    (let [element (first both-list)]
      (if (and (includes? seta element)
               (includes? setb element)
               (not (includes? ilist element)))
        (get-intersect (conj ilist element) seta setb (rest both-list))
        (get-intersect ilist seta setb (rest both-list))))))


(defn check-unique [ilist both ulist]
  (if (empty? ilist)
   ulist
   (let [element (first ilist)]
     (if (includes? ulist element)
       (check-unique (rest ilist) both ulist)
       (check-unique (rest ilist) both (conj ulist element))))))

(defn intersect [seta setb]
  (get-intersect [] seta setb (concat seta setb)))


(defn union [seta setb]
  (let [both-sets (concat seta setb)
        intersection (intersect seta setb)]
        (unique-element intersection both-sets)))

Applying this:

1
2
3
4
5
6
7
8

clojure.core=> (doseq [title (union epub-list mobi-list)] (println title))
The Joy of Clojure
Code Complete
Mining the Social Web
Introduction to Information Retrieval
Natural Language Processing with Python
nil
clojure.core=>

Applications

This has been just a quick introduction to the topic, but hopefully you can see the relevance to areas like data mining. Coincidentally, datasets tend to conform to the mathematical idea of sets, and typically with some data massaging (i.e. to filter out duplicates), those that don’t can be made more like mathemtical sets. Once appropriately represented in the computer, they can be acted upon with the basic set operations.

I’ve created an additional example: a web service providing a rest API to the book collection. As with the code in this post, there is an example in Python and in Clojure. The README in either example explains what dependencies are required. You can also view the Bitbucket repo for the Python example, or the GitHub repo for the Clojure example.

Acknowledgements

Stephen Olsen reviewed many iterations of this article and helped me to properly articulate the important points (like illustrating that unions require the subtraction of the intersection). I originally wrote the bulk of this article on the 25th, but it took me until the 28th to finish writing the API example code, until the 31st to add in the additional union explanation, and until the 1st to polish it up.

I wrote a quick REST API server as an illustration for a blog post, but I wrote both a Python and a Clojure version. I wrote a test suite to cover the entire API (of course - you do write tests too, right?), and I figured while I was at it, I might as well benchmark the two. Here are the results of 1,000 runs:

• noir: average run time: 0:00:00.171184 (0.171184 seconds)
• flask: average run time: 0:00:00.147073 (0.147073 seconds)

Notes:

• the time to start the noir server is much longer, about 5-10 seconds on my 2011 Macbook Air (1.6 GHz Intel Core i5 with 4G of RAM and a 64G SSD)
• both servers were running on the same machine at the same time, obviously just listening on different ports
• I tested this with the Python test suite

Source Code:

• clojure / noir example
• Python / flask example

References:

• Noir
• Flask

Basic Properties of Sets

The first thing we have to do is to explain what is meant by a set -

definition: set
A set is any collection of items where each item is unique and the order of items in the collection is not important.

The uniqueness property is very important to sets: there are no duplicates in a set.

So what does a set look like? In my database, I have a list of all the books I have electronic copies of. Each book comes in at least one of three formats: PDF, epub, or mobi. We’ll call the superset (the universal set of all the items under consideration) the list of all the books in the library. We’ll call this set ‘L’ (for Library). Part of the set might look like:

1
2
3
4
5

L = { 'Natural Language Processing with Python', 'Learning OpenCV', 
      'Code Complete', 'Mastering Algorithms with C', 
      'The Joy of Clojure', 'Mining the Social Web', 
      'Algorithms In A Nutshell', 'Introduction to Information Retrieval', 
      ... }

We use '{}' to denote the members of a set. The order of books in the library doesn’t matter here, and it doesn’t make sense to have more than one entry for a book in the library.

Building a set in Python is very easy:

1
2
3
4
5
6

library = set(['Natural Language Processing with Python', 'Learning OpenCV',
               'Code Complete', 'Mastering Algorithms with C',
               'The Joy of Clojure', 'Mining the Social Web',
               'Algorithms In A Nutshell',
               'Introduction to Information Retrieval',
               'Network Security With OpenSSL', 'RADIUS'])

Clojure has set notation built in using the #{ } syntax, and any collection can be turned into a set with (set coll):

1
2
3
4
5
6
7
8
9
10

(def library #{ "Natural Language Processing with Python",
                "Learning OpenCV",
                "Code Complete",
                "Mastering Algorithms with C",
                "The Joy of Clojure",
                "Mining the Social Web",
                "Algorithms In A Nutshell",
                "Introduction to Information Retrieval",
                "Network Security With OpenSSL",
                "RADIUS"})

So now we need to build some subsets.

definition: subset
A subset is some part of a set.

definition: proper subset
A proper subset is some part of a set, but is not the whole set.

For example, we’ll create a subset of books P that are on or in Python. We’ll also create a subset of books E that are in the English language. For my library, because not all of my books are in or about Python, the number of members of P is smaller than the number of elements in L. However, all of my books are in English, so the number of elements in E is the same as the number of elements in L. Therefore P is a proper subset, while E is not.

The Basic Set Operations

Now let’s consider two proper subsets of the library to explain some of the basic set operations: M is the subset of ebooks that I have in mobi format, and we’ll redefine E to be the list of ebooks in epub format. For the sake of the rest of this article, let’s note the following:

1
2
3

M = { 'Natural Language Processing with Python', 'Code Complete',
      'Introduction to Information Retrieval' }
E = { 'The Joy of Clojure', 'Mining the Social Web', 'Code Complete' }

In practical terms, this means in my library I have copies of:

• “Natural Language Processing with Python,” “Introduction to Information Retrieval,” and “Code Complete” in mobi format
• “The Joy of Clojure,” “Mining the Social Web,” and “Code Complete” in epub format.

In Python:

1
2
3

mobi = set(['Natural Language Processing with Python', 'Code Complete',
           'Introduction to Information Retrieval'])
epub = set(['The Joy of Clojure', 'Mining the Social Web', 'Code Complete'])

In Clojure:

1
2
3

(def mobi #{"Natural Language Processing with Python", "Code Complete",
           "Introduction to Information Retrieval"})
(def epub #{"The Joy of Clojure", "Mining the Social Web", "Code Complete"})

Union

A union is the set of members that appear in either set - if it’s in at least one of the sets, it will appear in a union of the two sets. So we could define a subset of L that contains all the books I have in a mobile format, which for our purposes means copies exist in epub or mobi format. In Python, you can use the set.union method, and in Clojure you can use the functions in the clojure.set namespace.

In Python:

1
2
3

mobile = set.union(mobi, epub)
for book in mobile:
    print book

which yields the output:

1
2
3
4
5

Natural Language Processing in Python
Code Complete
Introduction to Information Retrieval
The Joy of Clojure
Mining the Social Web

Remember that one of the properties of sets is that order is irrelevant, so you might get the books in a different order (this applies to Clojure as well).

The same thing, in Clojure:

1
2

(def mobile (clojure.set/union mobi epub))
(doseq [book mobile] (println (str book)))

You would see a similar output to the Python example.

Again, the practical result of this is a set of all the books I have in my library in a mobile format.

Intersection

The intersection of two sets is a list of all the members that only appear in both sets. In the library example, taking the intersection of the mobi and epub sets gives me a set of my books that I have in both epub and mobi format. The intersection function gives me this result.

The Python example:

1
2
3

both_formats = set.intersection(mobi, epub)
for book in both_formats:
    print book

And in Clojure:

1
2

(def both-formats (clojure.set/intersection mobi epub))
(doseq [book both-formats] (println (str book)))

For either example, the output should be just one book, given the sample sets:

1

Code Complete

I could use this result to know which books I can use on any mobile device.

Difference

The difference of one set from another is a list of all the members in the first set that are not in the second set. This operation is a bit different from the first two; the first two operations are commutative, but the result of a difference is dependent on the order of the sets. I’ll illustrate this with some code examples:

In Python:

1
2
3
4
5
6
7
8
9

only_mobi = set.difference(mobi, epub)
print 'books only in mobi format:'
for book in only_mobi:
    print '\t' + book

only_epub = set.difference(epub, mobi)
print 'books only in epub format:'
for book in only_epub:
    print '\t' + book

In Clojure:

1
2
3
4
5
6
7
8
9

(println "books only in mobi format:")
(def only-mobi (clojure.set/difference mobi epub))
(doseq [book only-mobi]
    (println "\t" book))

(println "books only in epub format:")
(def only-epub (clojure.set/difference epub mobi))
(doseq [book only-epub]
    (println "\t" book))

As the output messages show, this gives us the set of books that are only in mobi and the set of books that are only in epub. The output should look something like:

1
2
3
4
5
6

books only in mobi format:
  Introduction to Information Retrieval
  Natural Language Processing with Python
books only in epub format:
  The Joy of Clojure
  Mining the Social Web

Complements

When discussing complements, we do so when considering a subset and it’s superset. The complement of a subset is the difference of subset from the superset; i.e., the set of all members in the superset that are not in the subset. For example, if I wanted to check my library for all ebooks I have that are not in mobi format, I would use the superset library and take the difference of mobi from library:

In Python:

1
2
3
4

not_mobi = set.difference(library, mobi)
print 'books not in mobi format, using the library superset:'
for book in not_mobi:
    print '\t' + book

and in Clojure:

1
2
3
4

(println 'books not in mobi format, using the library superset:')
(def not-mobi (clojure.set/difference library mobi))
(doseq [book not-mobi]
    (println "\t" book))

This gives us the output:

1
2
3
4
5
6
7
8

books not in mobi format, using the library superset:
   Mining the Social Web
   Algorithms In A Nutshell
   Mastering Algorithms with C
   RADIUS
   The Joy of Clojure
   Network Security With OpenSSL
   Learning OpenCV

Conclusion

This has been a very basic look at set theory and what it means in practise. There is a lot more to set theory (see the references) but this should help get you started. There are a lot of applications for set theory, such as in data mining and natural language processing; it is a powerful tool that is worth spending some time to get to know.

Stay tuned for the next post, which will be on how to use sets in your code. We’ll develop the library idea a bit more.

UPDATE: The next post is up!

References

• I’ve been reading Alfred Aho’s The Theory of Parsing, Translating, and Compiling (Volume I: Parsing) (Amazon link)
• There is, of course, a good wikipedia article.

Reviewers

I’d like to thank the following people for reviewing this:

• Wally Jones
• Stephen Olsen
• Aaron Bieber
• Jason Barbier
• Shawn Meier
• Matt Sowers

Code Samples

The complete python source code, which you can save to a file and run directly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# author: kyle isom <coder@kyleisom.net>
# date: 2012-01-23
# license: ISC / public domain (brokenlcd.net/license.txt)
#
"""
Python illustrations for blog article "Basic Set Theory"
    (see http://kisom.github.com/blog/2012/01/23/basic-set-theory/)

Note that this is slightly tweaked from the examples in the article:
    1. PEP8 dictates that all globals be in all caps; as all the variables
    in this illustration are globals, they have been modified to be all caps.
    2. There is a little extra output to explain what is going on; namely,
    tabs are added before printing books and there is an output line showing
    which example the book set is associated with.
"""

# variables are in all caps because they are globals, and PEP8 dictates
# that globals be in caps.

# the superset
LIBRARY = set(['Natural Language Processing with Python', 'Learning OpenCV',
               'Code Complete', 'Mastering Algorithms with C',
               'The Joy of Clojure', 'Mining the Social Web',
               'Algorithms In A Nutshell',
               'Introduction to Information Retrieval',
               'Network Security With OpenSSL', 'RADIUS'])

# the subsets
MOBI = set(['Natural Language Processing with Python', 'Code Complete',
           'Introduction to Information Retrieval'])
EPUB = set(['The Joy of Clojure', 'Mining the Social Web',
            'Code Complete'])


print '[+] list all the books in a mobile format (union example):'
MOBILE = set.union(MOBI, EPUB)
for book in MOBILE:
    print '\t' + book

print '[+] list all the books in both mobile formats (intersection example)'
BOTH_FORMATS = set.intersection(MOBI, EPUB)
for book in BOTH_FORMATS:
    print '\t' + book

ONLY_MOBI = set.difference(MOBI, EPUB)
print '[+] books only in mobi format:'
for book in ONLY_MOBI:
    print '\t' + book

ONLY_EPUB = set.difference(EPUB, MOBI)
print '[+] books only in epub format:'
for book in ONLY_EPUB:
    print '\t' + book

You’ll want to run this with python set_theory.py (or whatever you choose to name the file, obviously).

The complete Clojure source code, which you can likewise save and run:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

;; set-theory.clj
;; author: kyle isom <coder@kyleisom.net>
;; date: 2012-01-23
;; license: ISC / public domain (brokenlcd.net/license.txt)
;;
;; code examples for blog post "Basic Set Theory"
;;     http://kisom.github.com/blog/2012/01/23/basic-set-theory/
;;

(require 'clojure.set)

; the superset
(def library #{ "Natural Language Processing with Python",
                "Learning OpenCV",
                "Code Complete",
                "Mastering Algorithms with C",
                "The Joy of Clojure",
                "Mining the Social Web",
                "Algorithms In A Nutshell",
                "Introduction to Information Retrieval"
                "Network Security With OpenSSL",
                "RADIUS" })

; the subsets
(def mobi #{"Natural Language Processing with Python", "Code Complete",
           "Introduction to Information Retrieval"})
(def epub #{"The Joy of Clojure", "Mining the Social Web", "Code Complete"})

; union illustration
(println "union illustration (books in either mobile format)")
(def mobile (clojure.set/union mobi epub))
(doseq [book mobile]
    (println "\t" book))

; intersection illustration
(println "intersection illustration (books in both mobile formats)")
(def both-formats (clojure.set/intersection mobi epub))
(doseq [book both-formats]
    (println "\t" book))


(println "books only in mobi format:")
(def only-mobi (clojure.set/difference mobi epub))
(doseq [book only-mobi]
    (println "\t" book))

(println "books only in epub format:")
(def only-epub (clojure.set/difference epub mobi))
(doseq [book only-epub]
    (println "\t" book))

; complement illustration
(println "books not in mobi format, using the library superset:")
(def not-mobi (clojure.set/difference library mobi))
(doseq [book not-mobi]
    (println "\t" book))

You’ll want to run this with clj set-theory.py - I’ve deliberately chosen not to make this a lein project in order to make it easier to share, but I did upload a lein project. You should be able to just run lein deps, test, run.

Of course I’m talking about PIPA and SOPA. The story I told isn’t entirely isomorphic to the current situation, but it gets the idea across. The politicians enacting this legislation have admittedly no technical knowledge, despite the fact that the vast majority of the people working in the tech industry have decried this as a universally uneducated and ineffective decision that will do more to hurt the innocent than to accomplish its stated objectives.

This is the exact reason why I would not start a tech business in the US anymore. Starting a business is hard enough with everything else; worrying about the legal environment is too much and there are plenty of places where law enforcement does its job instead of placing the burden on you.

Unfortunately, just leaving the country isn’t going to let me just dodge the effects. What happens when you lose a large part of your market share (i.e. the US market)? Ars Technica has a good writeup on how SOPA/PIPA affect foreign users.

So what could the government do instead?

• work on improving the technical skills of their workforce (not driving away the people with these skills with the government’s current overwhelming ineptitude would be a good start)
• being far more transparent about the process of taking down a site

Unfortunately, I don’t have enough money to pay the government to listen to me.

Further reading:

1. The EFF has a good Takedown Wall of Shame if you don’t believe the government would possibly abuse or misuse their takedown powers.
2. The EFF also has a good writeup on SOPA.
3. Stop American Censorship
4. Fight For the Future has a good video

Thanks to Samuel Goodwin, Beau Holton, Jason Barbier, Aaron Bieber, and Wally Jones for reviewing this.

1
2
3
4
5

commit e358120dd3760e64436f5652895c751b39148ebd
   Author: Kyle Isom <coder@kyleisom.net>
   Date:   Wed Dec 28 19:22:59 2011 +0300
   
    initial commit

A brief stint playing with clojure made me miss common lisp, so I’m working through Paul Graham’s ANSI Common Lisp with a copy of On Lisp. My last foray, I learned from David Touretzky’s A Gentle Introduction to Symbolic Computation, so this time I’m trying PG’s book. So far I’ve done more useful things, mostly by actually reading a bit more of the sbcl user manual (from which I learned some useful things such as sb-ext:*posix-argv* and sb-ext:save-lisp-and-die) and by the immensely useful site Rosetta Code, from which I learned about the DRAKMA HTTP client library. I’ve also been aided quite a bit by Zach Beane’s quicklisp; in fact, one of the things I’ve done is to write a short script to build an sbcl image with quicklisp and my most commonly used libraries built-in.

One of the things I love about functional programming is the idea that instead of relying on a lot of variables, you use functions as sort of “organic variables” that provide immutable data based on some input. The ability to build what feels more organic, less static. I think Steve Yegge’s blog post Execution in the Kingdom of Nouns is spot on.

I anticipate this to be the year of Lisp for me, as I delve into Common Lisp, Scheme, and Clojure.

I’m not too worried though; everything in there is either out of date (i.e. that credit card already expired and I have a new one now), my password was used only on that site (but was a 22 character phrase with punctuation), and that email address was only used for stratfor. My address was also already published due to my domain names (most of which are now privately registered, but I couldn’t always afford that).

Now that I have a doc drop that actually affects me (the mtgox break-in gathered a similarly difficult and unique password, and besides some spamming from idiots trying to exploit the situation with new bitcoin services there was no fallout), I can talk about what I think of the spate of LulzSec-style attacks this year. And that is - getting mad at Anonymous, LulzSec, AntiSec, or whatever the nom du jour is, is an exercise in futility and ignorance. It’s like getting mad at your five year old for getting into the cookie jar you left on the kitchen table. Your antiquated and ineffective security mechanism (the kid couldn’t possibly get on the table) should have been replaced by something more effective (maybe locking it in the pantry, overkill for a cookie jar though).

The issue is it’s 2011 (almost 2012) and we’ve been doing this for a while now. The rampant incompetence of people setting up these sites should be made a crime. That would be a far better use of legislative effort than the brain dead attempts at anti-piracy we’re seeing now. The script kiddies aren’t exhibiting any serious talent, the security (I use the term loosely here) people are setting up is juvenile.

Afterthoughts:

As gyardley pointed out on hackerne.ws, when I compare the LulzSec-type script kiddies to five-year-olds, I don’t mean to create the impression that they shouldn’t be held legally liable. I do fear, however, that focusing on legal actions against those responsible will cause us to lose focus on the bigger problem.

Today’s git commit occurred while I was working on getting a web development test VM / environment working. The goal was to update a CGI script when I pushed to the dev vm. The commit log:

1
2
3
4
5

commit 2de6f8444c68b0dd5ad31dd815d71a5590c5120e
   Author: Kyle Isom <coder@kyleisom.net>
   Date:   Sat Dec 3 00:24:34 2011 +0300
   
       suddenly enlightenment

It took a while for me to grok what was happening with the hook, but finally it clicked. I did a lot of reading online, and was greatly helped by the O’Reilly book Version Control with Git and the githooks(5) man page.

My remote repository was a bare git repo (one initialised with git init --bare that I pushed my local changes to. I created a staging directory (${HOME}/stage/cgitest) and created the following hook:

kyle@www-dev:~/code/cgitest/hooks$ cat post-update
#!/bin/sh
export GIT_DIR=/home/kyle/code/zipcgi
export GIT_WORK_TREE=/home/kyle/stage/zipcgi
git reset --hard
git checkout -f
cp ${GIT_WORK_TREE}/zipcgi.py ~/bin/cgi/

As a side note, make sure the script is chmod +x‘d.

The reason why we have to specify the git dir is that by default, because this is in the bare repository, git will assume the git directory is the repository directory. The problem is, that directory doesn’t have a working tree. A working tree is required to checkout the repository - i.e. so we have a named file to work with. To work around this, I explicitly specify a working tree . Then I copy the CGI script to my CGI directory.

Why not just symlink the file? Well, symlinks work on inodes. This allows multiple names to refer to the same file, but it does mean that even though the file is in the same directory and shares the same name, it is not guaranteed the same inode number. The git checkout can, in essence, unlink the old file and create a completely new file. The end result is that your symlink will likely be broken, pointing to a now non-existent inode. The safest method is just to copy the new version on top of the old one.

Why do we have to manipulate the environment variables GIT_DIR (which points to the directory containing the actual git repository, more on that in a second) and GIT_WORK_TREE, which represents the working tree. To really understand this, you need to understand the difference between the working tree and the repository. You could take the long route and read the excellent book I mentioned above and wade through man pages (which are pretty well written, but there is a lot of information to keep track of). An alternative is to buckle in and keep reading for my crash course.

Still here? Buckled in? Let’s do this. A git repository is basically a filesystem-based database that uses hashes for identification and great success. If you poke around in your git repository (which in a standard local repository is in ${PROJECT}/.git), particularly under objects, you will see what I mean. Everything is stored as a hash object. Git uses SHA-1, and under .git/objects you will see a list of subdirectorys. These subdirectories (with the exception of pack and info) named after the first byte of the SHA-1 hash (which is two bytes when stored as a semi-human readable hex digest). Under these subdirectories, git stores the objects as the remaining 19 bytes (again, 38 bytes when stored as a hex digest) of the hash. The file is zlib-compressed. Don’t believe me? If you clone my woofs project and look up

.git/objects/bf/2f7383ca7343f85f1308fc6dc3c34dbd047d90.

Try the following python code:

import zlib
print zlib.decompress(open('2f7383ca7343f85f1308fc6dc3c34dbd047d90').read())

You should see a working version of the script (and the latest version as of this writing). This is how git sees everything. (If you want to see what git sees a file as, use git hash-object <FILE>.)

The working directory is where you, the developer or end user, interact with the contents of the database. This is where things can be staged to be committed, and in a bare repo (typically found on remote repos), there won’t be a working directory because you aren’t working directly on that copy of the repo. Try this:

mkdir -p ~/tmp/stage/woofs_working
export GIT_DIR=~/Code/woofs/.git 
export GIT_WORK_TREE=~/tmp/stage/woofs_working
cd ~
git reset --hard
ls ~/tmp/stage/woofs_working

Voilà! You should see the contents of the repo there. (I’d recommend either closing out that terminal session or running

unset GIT_DIR GIT_WORK_TREE

to prevent problems later on. Also, while I’m using a repo I chose at random from my ~/Code directory, you could (and should) be trying with a repo of your own.

It should be clear now why I had to explicitly specify the two. The next two commands just reset the working directory to the lastest commit (i.e. the one that was just pushed) and check out a fresh copy, to make sure everything that should be present is present.

This turned out to be a longer post than I had expected, but my hope is that it helps other people quickly get their hooks operational. The cool thing about hooks is they are just executable shells scripts, which means:

1. the script’s ${PWD} is the hooks directory in the git repo.
2. the ${GIT_DIR} is by default ‘.’ and is the repo directory. for example, if we had a bare woofs repo, it would be something like /home/kyle/code/woofs, while in a local repo it would be /home/kyle/code/woofs/.git.
3. because it’s just a shell script, you can use any language you can use a shebang for.

Git hooks are a powerful tool and can greatly boost your productivity, automatically deploy code, and help us fight SkyNet. You should consider using them in your next project.

 commit 40bbc533313a43192506b682fe546304d8603d11 
 Author: Kyle Isom <coder@kyleisom.net>
 Date:   Mon Nov 28 17:34:30 2011 +0300

    add .emacs.d/init.el

I’ve started using emacs, which is an act of such great blasphemy for a red-blooded stalwart vim-wielding hacker such as myself that I find it difficult to come to grips with sometimes. But there is a method to my madness, and it isn’t just that my morals are so comprised right now in this nadir of my life that I’ve even started learning javascript (a running joke).

Due to my current work situation, and the prospect of traveling to and spending several months in a region with little to no network connectivity, I’ve purchased a new 11” Macbook Air. I chose the 11” model solely for price reasons; I would much prefer a larger laptop. I’ve previously owned two EeePCs (the 7” Eee PC 701 and the 11” Eee PC 1101HAB), so I’m familiar with the smaller form-factor, and not a huge fan to be honest.

Enter emacs - I can do all my work on emacs, with an integrated python development environment incorporating a shell, pdb (with a pane that shows the current line of the file being executed as you’re stepping through code). I’ll still have the OS X desktop, iterm2, tmux, and macvim. But for getting things done, I think that emacs is going to help out a lot.

Plus it’s backed by a Lisp flavour.

TL;DR One more combat veteran is disgusted and appalled by the actions of a few police officers during the course of the Occupy movement.

I just got done watching this video, and the outrage I feel towards those peace officers who disrespect the freedoms that I, and many better man than I, have ostensibly fought to protect, welled up in my chest like a Midwest thunderstorm. While we were told that our actions in Iraq were in the name of freedom, while many people back home who supported us with letters and care packages were told we were defending freedom, the basic rights of American citizens are being trampled on by a few officers and police departments.

Right up front: I don’t have anything against police officers. Yes, they do unpopular things. I understand that. When I came home, there were times when anti-war protesters took their protests to a personal level. As a 19 or 20 year old to return home from war, having seen friends and brothers killed, having experienced the trauma of being having your vehicle hit by an IED, and having had to make decision whether to take another man’s life or not (despite not being considered old enough to be responsible enough to consume alcohol), to have people tell you that you are a baby killer is a fairly difficult and emotionally trying experience. I took several of my best years out of my life to serve my country and gave up quite a number of opportunities in order to do so. I understand many of the feelings that I am sure many police officers feel when ordinary citizens harass and rail on them for actions they take that are unpopular. I get it. I know that as despicable as resorting to violence is, the fact of the matter is that humanity forces the need. Would that we were all enlightened, instead of being little more than clever apes with Facebook, but I understand that is not the case. Trust me.

But as a fighting man in an actual war zone - and despite the so-called “threat of terror” that supposedly “looms” here, America is not a war zone - had I fired even non-lethal rounds at unarmed civilians unprovoked because I felt scared due to the loud noises they were making, I would have faced military punishments. Many soldiers were punished for less. As a professional soldier, I am trained and expected to make reliable snap shoot/no-shoot judgements. During the surge in Iraq, when I was in a very kinetic (military word for an area of active fighting) environment, we still were held accountable for our actions. If use of force was justified (and it was on many occasions), there is no issue.

So to see American police officers engaging American citizens who are not wielding weapons is aggravating in no small way. As I mentioned in the opening paragraph, rage is a much better term for the emotion. It isn’t all the police officers, but just as a few soldiers cast shame over the entire US military, a few idiot police officers are casting shame over their departments and the governments they represent. If those officers were in my platoon, in addition to military judgement they would be taken out back and beaten.

I don’t want to rant, I don’t want to say anything stupid, and I don’t want to end up resorting to ad-hominem attacks. Suffice it to say that many of us in the military, regardless of our agreement (or lack thereof) with the protesters’ cause, stood up and defended the right of those protesters to have their voice heard. It is a sad day when the populace needs protection from the very police officers whose job is allegedly to protect that self-same populace. Let us not forget who pays the salary of those police officers.

For those police officers who maintain their calm and professional bearing, I applaud you. I understand the desire to smash stupid people in the face for saying and thinking stupid things. Unfortunately, as a police officer, you are bound to protect those people. I have been in similar situations in the military, and I understand the frustration. If you feel so strongly about it, do it the old-fashioned way - go on your own, out of uniform, and throw down. If it’s a departmental policy to carry out those types of actions, then the government no longer serves its people and no longer upholds the Constitution, and it’s time for you to be replaced.

I recently was explaining to someone that as a coder, I do (or should do) a lot more than just code. I figured since I hadn’t written anything here in a while, I’d put my thoughts down here.

i found myself needing to generate a patchfile today from a git repo. it turns out to be a very easy task.

• first, commit to a clean working directory. i’ll asume you are on the ‘master’ (git) or ‘tip’ (hg) branch, but s/master/$branch/ as appropriate.

• if you have only one commit between you and the commit you need to diff against:

1

git format-patch master^ --stdout > my.patch`

or

1

hg export tip > my.patch

• otherwise, substitute in the appropriate commit

• to apply the patch, it’s

1

git apply --stat my.patch

or

1

hg patch my.patch

I did say it was a very easy task… You’ll notice mercurial makes this easier (or at least I think so) than git.

the tl;dr

Coders code. That much is obvious from the title, but there is much more that can and should be involved for anyone writing real code, at least for UNIX coders.

intro

So you’ve spent the last couple weeks / months / years writing some really brilliant bit of software that you think would benefit a lot of people. Or maybe, just a few, but you still are of the mindset that since you did the work to solve this problem, other people might have the same problem and if they had the solution, they could concentrate on other problems. Regardless of the quality of code and the development process you followed, which endless books have been written on the subject, there is still a lot more work to be done if you intend to make your software both useful and accessible to other people. You still need to make sure you have a reasonable portable (for the scope of the usefulness of your code) build system, good documentation, an easily accessible online place for people to get your code, and proper follow-through. Let’s talk through these bits.

the build system

No matter how wizard your code is, if it’s more work for other people to build it than it’s worth, it won’t be used. That’s a simple fact. By now, users have come to expect the proverbial ./configure && make && make install (or perhaps scons or waf or jam or one of the other solutions). Regardless, the build process should not require much work for end users, except in cases where the code is a very purposeful bit of code that requires careful configuration. I personally have begun making use of the autotools suite (my personal stance on the GPL notwithstanding, a rant for another day but the curious can take a look at the license for most of my code on my github page). This comprises autoconf and automake primarily. You will easily spend many hours just writing out the configuration files on your end to properly support and build the software, determining what needs to be checked on the user’s system so that they can be sure the code will run on their node. Once this is set up and functioning, for the most part and in theory, users will be able to just do the typical configure-and-make pattern they have come to know and love. The autotools are really designed for C and C++. For python, there’s always the Python setuptools, and of course for Perl there’s CPAN.

Of course, these tools are quite often in a different language than your code is. For example, the autotools use POSIX shell, M4, and POSIX Makefiles to generate the configure script and Makefiles for distribution. This takes time to learn, especially given some of the nuances involved. There is of course some debate (see (“Stop the autoconf insanity! Why we need a new build system”)[http://freshmeat.net/articles/stop-the-autoconf-insanity-why-we-need-a-new-build-system]) as to how useful these are, but for the most part the reward is worth the work. For the autotools suite, take a look at the No Starch Press book Autotools:A Practitioner’s Guide to GNU Autoconf, Automake, and Libtool. I found this book indispensable in learning the tool suite.

documentation

Documentation extends much further (or should) than the typical README and INSTALL files found in many distributions. Many developers learn the basics of TeX or LaTeX typesetting to produce aesthetically pleasing manuals; Texinfo is also quite common. Markdown is becoming popular as well and with the advent of tools like pandoc, even easier to convert from Markdown to other formats (pandoc supports html and LaTeX). Besides just the technical side of writing documentation and learning the typesetting language used, there’s the art of technical writing as well. Many companies have full-time technical writers whose sole purpose is writing documentation. This is because of another simple fact: your software is of no use if the users can’t figure out how to use it. While many users may be technically saavy enough to read the code to figure out how to use it, for your code to be truly useful, they should not have to resort to this. This is what I see as the Apple factor: many developers use Apple’s hardware and operating system because not only do things Just Work, but there is also excellent documentation available. Another operating system leading the way in documentation is my beloved OpenBSD. Users should have a clear set of instructions of not only how to use the software, but ways to extend it, what things it can do that they may not realise, and how to solve problems that may crop up. So a truly good coder is both at least a proficient typesetter but also a proficient writer of whatever human language the software is in (or aimed at).

Some projects go further and include a full copy of the license the software is released under (which you should do for the safety / peace of mind / convenience of your users - it took lteo constantly reminding me of this for many of my projects before I started doing it out of habit) which is most often in a file called LICENSE or COPYING; a copy of the ChangeLog, which could also be gotten from source control such as git log; an AUTHORS file to list contributors; a README and INSTALL file to give a quick usage and overview as well as installation instructions; and perhaps a HACKING document to explain how to modify the code to be useful.

The README file is still rather useful; in fact, many times I will write the README first as part of my development process.

No matter how you approach it or what you use to write and format your user manuals, you should still have them included.

distribution

Today, distribution is one of the easiest aspects of coding. Numerous websites exist for the sole purpose of distributing your software, such as github, bitbucket, sourceforge, freshmeat, among others. Typically, such sites will also host a remote version of your version control system (you are version controlling, right?) in addition to supporting release downloads. A well-setup build system offers the ability to build a distribution release, often in tarball or tarred bzip2 format as well. Some sites still offer just a release tarball (for a while, this is how I released my libdaemon project, via my devio.us homepage. In fact, this is rapidly becoming one of the easiest pieces of the project lifecycle. If you haven’t already, take a look at one of the sites that works as a remote repo for whatever source control you are using. You will probably see that besides distribution, these sites are extremely useful for the last important additional part of coding I want to talk about.

support and maintenance

Once the user has a copy of your software and knows how to use it, they will inevitably encounter bugs or find that while they would really like to see a feature in the software, they don’t have the technical skills to implement it themselves (or perhaps the courage to look through your code…) Still other users might fix the bugs or add new features themselves, and would like to offer you those changes so you can incorporate them into the software. So the last important additional part of being a coder is support and maintenance.

Many of the sites that offer to host releases of your code provide additional tools, like wikis, bug reporting (aka trouble tickets), and feature requests. Users may also provide patchfiles or a git pull request to give you their contribition (and accordingly, you credit them in the documentation as well). A good coder needs to be able to support and maintain the software - users are more apt to use software if it gets patched or updated with new features (or if it just works and they don’t need new features or bugs patched, which is less likely but still possible).

conclusion

As I’ve explained, being a good coder and providing useful software encompasses so much more than just good technical skills or great development processes. There’s the administrative side (i.e. the build system, feature request and bug tracking) and the human side (i.e. documentation and responding to support requests). While it may not be as much fun as the actual coding, it is still integral to the development process.

update (2012-03-25)

One of the things I’ve completely neglected to talk about in this discussion is the use of tests. Functional tests, unit tests, regression testing, continuous integration, basically – TEST ALL THE THINGS. Why? First - it helps you write better code, and to ensure that changes don’t break everything (or if they, the breakage is the expected breakage). Second, it’s a form of literate coding where users can see how to use your code in practise (if it’s a library) or can get a warm fuzzy knowing you cared enough to validate and test your code as you went. You might think, well - this is a binary for end users. They won’t know or won’t care about unit tests and so forth. Maybe that’s true. However, part of the craft of writing good code is paying attention to detail. Any open source project that wants to be open to contributions should have tests so quality is enforced (i.e. don’t bother submitting a patch or pull request if your changes don’t pass the tests) and so they can see how you are using your code. Yes, you should be writing your code so that it’s obvious from reading it what it does. If there’s a lot of it, and a developer wants to make some quick changes to fix a bug, tests provide a good way for them to see where things happen and how they happen. I assert that good coders write good test code. (Testing joke!)

RGTDD has made a difference in how I develop code and stay productive; furthermore, above just using a specific development methodology, I’ve found certain tools to assist me in being productive.

So, RGTDD. It stands for README-generated Test-Driven Development. I took Tom Preston-Werner’s README-driven development and adapted it to my own use. Without rehashing his post, READMEs factor in like this: once the project is started, the first task to do is write the README. Once the README is written, you lock it in and should not be changed except to fix typos and spelling errors. This has several advantages:

1. First, you end up having a single introductory piece of documentation. This file contains the justification for you program, a quick introduction to its features, and usage information.

2. Second, locking in the README means you prevent feature creep. You contract yourself to the end user (yourself, future end users, the client, etc…) to implement specifically the features in the README. As a rule of thumb, each iteration of the README should have no more than five features - each feature should be a concrete task. If the tasks are particularly complex, I generally avoid implementing more than three. I have found for myself that trying to implement much more than that results in code and projects that stagnate or quickly become spaghetti code.

3. Third, you know what you need to code and what the code should specifically do. I’ve found that a lot of times I have an idea for project X, but only have a vague of idea of what it will do. It is part of good engineering to have a well-laid out design and path for the project to direct development. I have found this helps me to keep my code from getting overgrown.

The README at this point now specifies an interface for users to interact with the code. From here, you begin writing tests that cover the features. As you begin writing these tests, you will likely figure out other components to the code. You start writing tests from these. At this point, using the README as a guiding document, I switch to a test-driven mode. Once the tests perform as they should (including tests that are expected to fail), I consider this a release. (I’ll write a post later on about what I’m doing for releases).

That’s a quick introduction to the project management methodology I use. I’ll cover some specific tools to help out with various languages, my version control and release methods, things I wasn’t taught in school, and what I’ve found to help me keep my life organised.

in the past, i found simon budig’s woof script but i wanted an SSL-secured version. i finally got around to writing an SSL-secured version. i’d started one in december, but i was still fairly new to python, but i finally pulled it off. the repo can be found here.

interestingly enough, if you look at the git commit logs there are three activity clusters: when i started the project in december, a brief period in may when i started the major rewrite to include my own http server, and a flurry of activity today when i added in ssl support.

so what does it do? as the name implies, it serves a file via https and by default serves it only once. it’s designed to allow quick filesharing between two systems; the transfer is protected by SSL. i won’t rewrite the documentation here, so be sure to check the documentation to take a look at usage. perhaps it will be useful to you as well.