Ultimate Prep

Friday May 30, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 73 F, Sunny
Field: dry
Perfect conditions for Ultimate.

It was nice weather for yesterday's pickup game in preparation for tomorrow's end of season BUDA spring hat league tournament.

Ultimate Statistics (since January 2008):
Total Games Played: 48
Total Hours Played: 55

Ultimate Energy

Wednesday May 28, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 60 F, Sunny, Breezy
Field: dry
Perfect conditions for Ultimate.

Felt good to be back out there playing yesterday after missing a Monday game due to the long weekend.

Ultimate Statistics (since January 2008):
Total Games Played: 47
Total Hours Played: 54

What is a Darkstar test?

As a new member of the Project Darkstar team, one of the initial tasks that I am facing is to come up with a method of automating distributed Darkstar tests. As the codebase is beginning to stabilize, much of our time is being spent evaluating performance. However, this is a very difficult and time consuming task when we don't have a mechanism to easily reproduce our test scenarios. For example, we have several sets of tests to run against the Project Wonderland application. In rough terms, execution of these tests requires:

1. Launching the Wonderland server application by hand.
2. Tapping into the Darkstar server profiling data (if enabled, Darkstar will spit profiling data out via a specific port)
3. Launching sets of simulated clients by hand.
4. Visually monitoring the Darkstar profiling data (using a set of human eyes)
5. Making a decision on whether or not the system is overloaded based on the profiling data.
6. Adding more clients into the system by hand if the system is not overloaded.
7. Rinse and repeat until satisfied that you have determined the maximum load on the system.

This might be fun to do once (or not), but if I'm trying to do some performance tuning, the mechanics of setting up the tests, running the tests, and determining the results can quickly become frustrating. Not only that, but running against a multi-node configuration is even more difficult, error-prone, and complicated to monitor. Maybe I would script a few pieces of the process, but it would likely just be an ad-hoc rig, incapable of supporting other tests.

Clearly we have a problem. We need a test harness that can execute tests such as the one described above, and also collect, record, and report results in a way that is easy for consumers to interpret. It should also be abstract enough to plug in any Darkstar application test, such as the Wonderland example above. Before we can build that, though, we need to answer a more basic question: What is a Darkstar test? Here's my stab at it:

A Darkstar test consists of four main components:

1. A Darkstar server distribution (i.e. the server package downloaded from projectdarkstar.com)
2. A Darkstar server application
3. A set of Darkstar client application simulators
4. A set of system probes

A Darkstar test must be run on a set of resources (presumably pieces of hardware):

1. A set of systems to run the server application (more than one if multi-node).
2. A set of systems to run the client application simulators.

A Darkstar test reports results based on what type of test it is:

1. Functional test: The client application simulators are responsible for running tests and reporting pass/fail decisions as results.
2. Load test: A preconfigured number of client application simulators are introduced into the system. The system probes are responsible for monitoring conditions of the system. If the specific condition that a probe is monitoring violates a preconfigured threshold, a fail decision is reported as a result.
3. Capacity test: The client application simulators are incrementally introduced into the system until one or more of the system probes reports a threshold violation. The number of clients that the system can support without violating the thresholds is reported as a result.

I'm starting to design an automated test harness around the above definition (details, of which, I'll leave to a later post). There is one thing, though, that is obviously apparent about such a system: it should be valuable for testing both Darkstar itself, and applications built on top of Darkstar. This would include Project Wonderland, any example applications that we have already developed, and also applications developed in the community. With that said, I will be treating this project as though it will be consumable by community members outside of the core Darkstar team, and hope to engage others in my efforts through this blog.
Stay tuned...

Ultimate Breeze

Friday May 23, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 70 F, Windy
Field: dry
Perfect conditions for Ultimate.

It was really breezy for Friday's pickup game.

Ultimate Statistics (since January 2008):
Total Games Played: 46
Total Hours Played: 53

Ultimate Warm

Wednesday May 21, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 67 F, Partly Sunny
Field: dry
Perfect conditions for Ultimate.

Looks like we're finally starting to get into a stretch of consistently warm weather.

Ultimate Statistics (since January 2008):
Total Games Played: 45
Total Hours Played: 52

Ultimate Dust

Monday May 19, 2008
12PM Pick-up game
Rahanis Field, Burlington, MA
Weather: 60 F, Cloudy, Windy
Field: dusty
Perfect conditions for Ultimate.

I'm going to start writing about Darkstar soon, I promise.

Ultimate Statistics (since January 2008):
Total Games Played: 44
Total Hours Played: 51

Ultimate Callahan

Sunday May 18, 2008
10AM BUDA Spring Hat League Game
Fernald School, Waltham, MA
Weather: 66 F, Partly Sunny
Field: dry
Perfect conditions for Ultimate.

We came full circle in this morning's BUDA league game, winning just like we did in the first game of the season. It was a decisive 15-3 victory to cap off the regular season (and I brought in the final point with a callahan to end it). The tournament is June 1.

Ultimate Statistics (since January 2008):
Total Games Played: 43
Total Hours Played: 50

Ultimate Arm

Friday May 16, 2008
12PM Pick-up game
TRW Field, Burlington, MA
Weather: 62 F, Cloudy
Field: dry
Perfect conditions for Ultimate.

Good game today but I have a little bit of a tired arm from all the playing and throwing this week. It's the four game weeks that always do me in!

Ultimate Statistics (since January 2008):
Total Games Played: 42
Total Hours Played: 48

Ultimate Nice

Wednesday May 14, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 70 F, Sunny
Field: dry
Perfect conditions for Ultimate.

Nice day out there today.

Ultimate Statistics (since January 2008):
Total Games Played: 41
Total Hours Played: 47

Ultimate Sore

Monday May 12, 2008
12PM Pick-up game
TRW Field, Burlington, MA
Weather: 51 F, Windy
Field: dry grass
Perfect conditions for Ultimate.

I was a little bit sore yesterday from Sunday's BUDA game but still made it out to play a windy pickup game.

Ultimate Statistics (since January 2008):
Total Games Played: 40
Total Hours Played: 46

Ultimate Hat

Sunday May 11, 2008
10AM BUDA Spring Hat League Game
Fernald School, Waltham, MA
Weather: 60 F, Sunny
Field: damp
Perfect conditions for Ultimate.

After a slow start, we played probably our best BUDA game for this year's spring league since the first game of the season. Unfortunately, we still came up short. Hopefully we'll come full circle next week heading into the tournament and win a game like we did in our first one.

Ultimate Statistics (since January 2008):
Total Games Played: 39
Total Hours Played: 45

Ultimate Showers

Friday May 9, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 52 F, Showers
Field: damp
Perfect conditions for Ultimate.

The rain held off for the most part for today's game (not that that matters).

Ultimate Statistics (since January 2008):
Total Games Played: 38
Total Hours Played: 43

Ultimate Layout

Wednesday May 7, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 72 F, Sunny
Field: dry
Perfect conditions for Ultimate.

It was a crowd at today's lunchtime pickup game clearly due to the perfect weather. A total of 19 came out to play. Also, Keith, one of my new co-workers in the Project Darkstar group decided to come out as he used to play ultimate "many years ago." He clearly has retained some skills but he just stopped by my office claiming that "it didn't even take until tomorrow for the legs to stiffen up." :)

Ultimate Statistics (since January 2008):
Total Games Played: 37
Total Hours Played: 42

Ultimate Mud

Sunday May 4, 2008
10AM BUDA Spring Hat League Game
Fernald School, Waltham, MA
Weather: 48 F, Showers
Field: muddy
Perfect conditions for Ultimate.

Unfortunately we weren't able to field a full team for today's game. I guess people are afraid they're going to melt? In any case we still played five on five but came up short on a muddy morning.

Ultimate Statistics (since January 2008):
Total Games Played: 36
Total Hours Played: 41

Ultimate Friday

Friday May 2, 2008
12PM Pick-up game
TRW Field, Burlington, MA
Weather: 50 F, Partly Sunny
Field: dry grass
Perfect conditions for Ultimate.

Almost forgot to post this game.

Ultimate Statistics (since January 2008):
Total Games Played: 35
Total Hours Played: 39

Ultimate Chilly

Wednesday April 30, 2008
12PM Pick-up game
Mitre Fields, Burlington, MA
Weather: 48 F, Partly Sunny
Field: damp
Perfect conditions for Ultimate.

It was only a few weeks ago that playing a game when it's in the upper 40's felt balmy.

Ultimate Statistics (since January 2008):
Total Games Played: 34
Total Hours Played: 38