Showing posts with label Project Darkstar. Show all posts
Showing posts with label Project Darkstar. Show all posts

February 4, 2010

RedDwarf

As I mentioned previously, I do intend to continue working on Project Darkstar part-time at least while I search for new employment opportunities. Not only will it allow me and whoever else is interested push along the original goals of the project, but it will also give me a perfect place to keep my skills sharp while I'm out of work. Selfless and selfish reasons, molded into one. However, it does not seem appropriate to continue working using what are now Oracle's resources and contributing code to Oracle's official repository. Not only are there questions about outside contributions, but there are also no guarantees about how long Oracle's official repository will remain active. So I am going with an equally reasonable and open-source-y alternative.

I have established what I hope will become an official community fork of Project Darkstar. This fork will go by the name RedDwarf and is hosted as a sourceforge project. I cannot take credit for the new name, as it was dreamt up by the original instigator of Project Darkstar at Sun Labs (Jeff Kesselman), but I do hope that it will become an even more well known name than Project Darkstar ever was in the games industry. So without further adieu, here are the new community guidelines:
  • The sourceforge project is used to host what were once three separate projects on java.net. The reddwarf-server, reddwarf-shared, and reddwarf-java-client all live in the same subversion repository, but under completely separate sub-trees, each with their own trunk, branches, and tags.
  • The development process should remain largely unchanged from Project Darkstar. All commits to any trunk repository must undergo a thorough review by at least one other committer, and commit privileges are earned. Review requests should be sent to the mailing list: reddwarf-develop on sourceforge.
  • All design, support, and informational documents should go in the Trac instance hosted at sourceforge.
  • All issues and bugs should be filed as a ticket in Trac.
  • Forum discussions should continue as usual on the sourceforge hosted forums.
  • Releases will be done periodically as appropriate. All releases will be published to the central Maven repository rather than the java.net Maven repositories. (This is my first task and may take some time). They will also be published as files for download on the sourceforge site.
Progress will be much slower than before, but I think it's very important to maintain the structure and code quality standards that we had established as a fully funded project. Will anything come of this effort? I'm not sure. But I think it's the best chance for coordinated progress to continue with Project Darkstar.

February 3, 2010

Shocking News

It's been a while since I posted anything on this blog, and I wish my return was the result of better circumstances but no such luck: my position at Sun Microsystems has been eliminated, and as a result I have been let go. It is ironic that the European Union's lengthy delay in approving the Oracle-Sun acquisition gave the Project Darkstar team and community such a long, uninterrupted stretch of time to make some unbelievable progress towards our goals. However, once the deal did finally close, the decision had already been made that Oracle will discontinue investing resources in the project, and so here I am: newly unemployed.

I must say that the shock and disbelief of learning that you've been laid off is a predictably emotional time. Of course I didn't think it would happen to me, but it did: proof that job security is all but an illusion. They say losing your job is like dealing with any other type of loss, which is absolutely true. Knowing this doesn't make it any easier though. Despite this difficult situation, however, I have received nothing but support from my family, friends, colleagues and even people who were previously just casual acquaintances. Thank you to everyone who has been there so far; I know for a fact I'll come out the other side of this a strong person.

As for my future, and the future of Project Darkstar? Well both are uncertain. I have already started ramping up and preparing for a full scale job search in the hope that I will find something even better than the best job I've ever had. In terms of Project Darkstar, a core group of former members of the team have already started exploring alternative ways to keep the project going. This includes both potential for-profit and volunteer efforts to carry out the original mission objectives laid out years ago. At the very least, after getting myself organized, I personally intend to continue working on the project on a part-time basis during my job search and hopefully beyond. More details to come...

November 12, 2009

Rule 1 of Programming: It's Always Your Fault (Almost)

Over the past week or so, I've been working on putting together some micro benchmarks for Project Darkstar. There has been a significant uptick in forum activity lately relating to stress testing and performance issues. In particular, we've seen many questions along the lines: "I can only connect X users to my darkstar server, what's wrong?" First of all, this is great news. It means that people are making significant progress with their darkstar based games/applications and are working to push the limits of the technology. However, I also think that this is the completely wrong question to ask. As I've demonstrated before, Project Darkstar has a pretty high ceiling for raw capacity in terms of number of users. A properly tuned app with a light load can easily handle tens of thousands of users per node. However, connecting mostly idle users to a mostly idle server is not very interesting. These capacity numbers naturally decrease as the number of messages between the clients and server and the amount of processing per message increases. This seems obvious, but people still ask the capacity question as though all games developed with darkstar are going to have identical limitations. This is simply not the case.

With this said, though, we can still strive to identify upper bounds on Project Darkstar's performance at a more fine-grained level. Project Darkstar is an event driven transactional system, so all operations are not without cost. With these micro benchmarks, I'm hoping that I can establish a relative cost to each of the operations using the DataManager, the ChannelManager, and the TaskManager. For example, how expensive is it to retrieve an object using DataManager.getBinding() vs ManagedReference.get(). How much overhead is involved with each transaction? How expensive is it to create a Channel or send a message on a Channel? With more or less users? While the cost of retrieving data from Darkstar's data store should be an order of magnitude faster than using J2EE and a RDBMS, it is also likely an order of magnitude slower than retrieving data from a data structure that is already in memory and using no synchronization. This is information that users really need to be aware of and be able to take into perspective when designing their game, structuring their tasks, and establishing their own expectations of what the performance should be like.

So over the past couple of days, I've been debugging a problem in these benchmarks. In one particular test, I was attempting to measure the raw execution time per call to DataManager.getBinding() from the Project Darkstar API. The test was pretty simple, I just set a large number of bindings in a single set of setup transactions. Then I would time the execution of another set of transactions that would make some subset of calls to getBinding() on the names that I had just setup. Taking into account previously measured transaction overhead I could then come up with a reasonable estimate of the cost per operation. Seems easy right? Well it turns out that I hit a snag. In running this test, I was repeatedly getting a situation where a seemingly random name binding was not being set properly during setup. Most of the calls to getBinding() would work fine, but a couple were throwing NameNotBoundException. What? This didn't make much sense. I went back and looked over my code many times, I tried a myriad of variations, logging output, and print lines, but still no luck. I was still getting NameNotBoundException for what seemed like a random name in the sequence. Hmmph.

At this point, I went through a whole series of exercises, all centered around one assumption, that my code was right. I tested the native edition vs. the Java edition of BDB, suspecting maybe there was a weird bug in one of them: same result. I tried longer transactions, more operations, larger serialized data objects: same result. I tried running my benchmarks in different orders: same result. I even started writing test cases for DataManager.setBinding() that simulated transaction rollback and retry, large numbers of consecutive calls to setBinding() and binding and rebinding of the same name. I thought I was going to uncover some weird corner case bug. But those tests were passing! I was at a loss. After probably two days of sporadic attempts at debugging this, I finally went back and looked really hard at my own test code. And... I found a bug (doh!). It turns out that I was being too cute with my setup transactions, and was modifying a non-local counter variable inside of my anonymous nested transaction class. In random situations, this class would abort and retry (a normal darkstar operation), but since it was modifying a variable that lived outside of the task itself, this value was not being rolled back. The result was that a name binding would be skipped periodically (exactly the behavior that I was seeing).

So here's my question. Why did I assume that code that I wrote in less than a day was more likely to be correct than Berkeley DB itself, a project that's been developed and tested for a couple of decades? Why did I assume that code that I wrote in less than a day was more likely to be correct than Project Darkstar's Data Service code which has been developed and tested for years? I mean, I knew better than to think that Tim's code is the likely culprit, but I still started writing test cases thinking I was going to heroically find some obscure bug. This, my friends, is a violation of the number 1 rule of programming: If you're having problems, It's Always Your Fault (almost). I mean, don't get me wrong, I've found (and reported) bugs in well established open source projects before, but those situations are actually few and far between. I also don't mean to suggest that Project Darkstar is bug free. I do think, though, that sometimes it's too tempting to conclude that there's a bug in that library you're using, or there's a performance limitation in that technology that is fundamentally impossible to overcome. Maybe that's true, but 99% of the time, it's your fault.

And with regard to those micro benchmarks, I'm hoping to publish some results soon (assuming I don't get hung up with any more boneheaded mistakes!)

September 25, 2009

Austin GDC 2009

agdcLast week, a number of us from the Project Darkstar team were in Austin for the Austin Game Developer's Conference. Like last year, we had a large booth on the expo floor. However, while last year we were largely focused on demonstrating Project Darkstar's capabilities to scale and distribute load across multiple cores and processors on a single node, this year our focus was on showcasing the team's progress on multi-node capabilities. Here's a recap of the week's events:

If you've been following along with Project Darkstar's progress over the years, you know that transparent multi-node scaling capabilities are one of its main attractions as a platform. On the other hand, you should also know that with Project Darkstar being a still maturing research project in Sun Labs, these features are not yet done. In fact, we have still not proven if what we are attempting can even be done. Our fearless lead architect, Jim Waldo, has put together an excellent series of posts on his blog outlining why and how we hope to achieve this multi-node scaling. He calls it the "Four Miracles."

Our goal for Austin was to put together some measure of a compelling visual demonstration of our current progress towards achieving these "Four Miracles." Specifically, we hoped to show how Ann's miracle of transparently moving clients from one node to another in a cluster worked in tandem with Keith's miracle of monitoring a node's health and intelligently making decisions about which clients to relocate and when. Also, while Jane's miracle of detecting and organizing clients into groups of affinity groups based on social networking algorithms is not complete, we also hoped to portray how it should work by rigging up an app that formed affinity groups based on a players location in the game (specifically which chat room it was in).

In the weeks leading up to the conference, Keith and I came up with two demos that for the most part seemed to hit the mark. Using the JMX facilities already built into Darkstar, I hacked up a monitoring GUI that tapped into a running Darkstar cluster and displayed each of the nodes as a vertical bar. The height of the bar represented the total number of clients connected to that node, and the color optionally represented either the node health, or fragments of colors represented the affinity group of each connected client. For the first demo, we had Project Snowman running in a multi-node cluster and showed how the new node health monitoring features of darkstar caused client traffic to spill over and be intelligently distributed between the nodes. Here's a quick video of Keith talking through it on expo floor in Austin:

The second demo had Darkchat running on a multi-node cluster and was designed to show how clients would relocate between nodes depending on which rooms they were connected to in the application. Here's another quick video given by yours truly on the floor at Austin:

Both of these demos we had running throughout the week on the floor and the response from people who came by was generally positive. I think one of the main differences that I noticed between this year and last year was the amount of quality traffic that we had come through the booth. I talked to a lot of people at Austin last year, but a large percentage of them had never heard of Project Darkstar or were just vaguely familiar with it. This year many people came by who were already committed to a project using Darkstar, or were very interested in our progress, or were familiar with the technology and had a strong desire to learn more. I think the best piece of anecdotal evidence was one person from the Intel booth who came by and said "Oh, so this is for real now?" Referring, of course, to the significant headway that we are finally making and showing on the multi-node scaling capabilities of the Project Darkstar platform.

In a tough year for everyone in just about every industry, I think going to this conference and putting together these demos have injected some energy into both the Project Darkstar community and the team. A few more personal observations:
  • The expo floor seemed smaller this year and overall attendance appeared to be down. Not too surprising but hopefully a sign of things past and not things to come.
  • A little tidbit on some of the pains of getting the demos setup. Leading up to the conference, we had everything running just fine in Burlington and the demos packaged up and ready to go. After a few hours of pulling cables, moving pods around, and booting up systems, we fired up the node health demo on Tuesday, the day before the expo floor opened. Except, it didn't work. At least not like it did in Burlington. When simulating an overloaded node, instead of offloading clients onto the other node right away, there was some seemingly random and arbitrary delay of 20 - 30 seconds before any clients would be moved. Huh?
  • After some hours of debugging, and pulling Seth into the mix for help, we finally tracked it down. The (still unfinished) node health code offloads identities from a node when it gets overloaded. However, it doesn't just move client identities, it also moves identities of robots and other NPC's in the system. Since each snowman game has a number of robots, it was choosing to move the robot identities before moving the client identities. The question, of course, is why didn't we ever see this behavior in Burlington? Well it turns out that the order in which identities are chosen to be moved is deterministic and seemingly alphabetical. While in Burlington, our client simulated players were generating identity names according to the hostname of the client machine (dstar1, dstar2, dstarX, ...). The hostnames of the machines used in Austin? x2250-1, x2250-2, etc. So in our Burlington deployment, the client identities were always getting chosen before the robot identities since they started with a d; in our Austin deployment, the client identities were always getting chosen after the robot identities since they started with an x. Unbelievable.
  • Keith gave a talk during a one hour session which was awesome. He went through a number of obstacles and challenges he faced when building Darkchat for JavaOne and I think it came across as very real and genuine.
  • One final note, it appears as though Chuck Norris still doesn't need scalable server technology. All of his CPU's run faster to get away from him. Also, any code that he writes cannot be optimized. For anyone else, though, Project Darkstar could be a solution.

July 7, 2009

JavaOne Podcast

communitycornerAbout a month ago I spent a week in San Francisco for JavaOne 2009. One of the activities that I participated in there was the java.net Community Corner where I recorded a podcast on Project Darkstar. Well, the podcast is now up on the java.net site for download and it turned out really nice! It's available for download here.

June 10, 2009

Hello Project Darkstar! in Netbeans and Eclipse

With Project Darkstar's new distribution structure (introduced in version 0.9.8 and explained here), people ask from time to time what the best way is to setup a project in Netbeans or Eclipse. I think the typical answer that you see the most fits the acronym "TMTOWTDI" (There's More Than One Way To Do It). This is true but not necessarily very helpful. In the hopes of eliminating some pain I'm here to offer a way to setup a project in these IDE's (but this should by no means be considered the way).

First, here is a quick description of the project and the tools that I will be using:
  • This will be a purely server side project that will print "Hello Project Darkstar!" when the server is booted.
  • I will be using Maven as the build tool for the project.
  • The project will leverage the Project Darkstar Maven Plugin to aid in deploying, booting, and shutting down the server according to the new distribution format for the official Project Darkstar server package.
  • It should be possible to build, run, and test the raw project outside of an IDE, within Netbeans, and also within Eclipse (with a few tweaks for each respective IDE).
There are three source files that are of interest in this project. The first is the actual Java source file that represents the main class for the server. This class should look something like this:
package my.pkg.hello;

import java.io.Serializable;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;
import com.sun.sgs.app.AppListener;
import com.sun.sgs.app.ClientSession;
import com.sun.sgs.app.ClientSessionListener;

public class HelloProjectDarkstar implements AppListener, Serializable {
  private static final long serialVersionUID = 1L;
  private static final Logger logger = 
      Logger.getLogger(HelloProjectDarkstar.class.getName());

  public void initialize(Properties props) {
    logger.info("Hello Project Darkstar!");
  }

  public ClientSessionListener loggedIn(ClientSession session) {
    return null;
  }
}

Second, it is strongly recommended for a Project Darkstar application that you include an app.properties file in the META-INF directory of your resulting JAR file.  This file will be used as the base set of configuration properties for the server and is a good place to include default values for application specific configuration properties.  In our simple example, we'll include just two required properties, the name of the server and the listener class:

com.sun.sgs.app.name=HelloProjectDarkstar
com.sun.sgs.app.listener=my.pkg.hello.HelloProjectDarkstar

Third, you need a Maven POM file.  If you're unaware of how Maven works, you can read up on it at the Maven website.  Essentially, the POM file is used to declaratively describe your project's resources and dependencies.  Maven then uses this information to correctly build it.  Our project has only one dependency (the sgs-server-api) so our POM file should look something like this:

  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/maven-v4_0_0.xsd">
  4.0.0

  my.package.hello
  hello-projectdarkstar
  0.1-SNAPSHOT
  Hello Project Darkstar
  jar

  
    
      com.projectdarkstar.server
      sgs-server-api
      ${sgs-server.version}
    
  

  
    
      
      
 org.apache.maven.plugins
 maven-compiler-plugin
 
   1.6
   1.6
 
      
    
  

  
    0.9.9
  

  
    
      java.net
      java.net Maven2 Repository
      http://download.java.net/maven/2/
      default

These three files should be organized in a directory structure that looks something like this:
- hello-projectdarkstar
  - pom.xml
  - src
    - main
      - java
        - my
          - pkg
            - hello
              - HelloProjectDarkstar.java
    - resources
      - META-INF
        - app.properties

At this point you should be able to build a JAR file that is deployable in a Project Darkstar server. Since this is a Maven project, this should be possible:
  • From outside an IDE using the command mvn package
  • From within Netbeans (ensure that you have the Maven plugin installed, then try "Open Project" and Netbeans should automatically detect that the root folder is a Maven project)
  • Or from within Eclipse (ensure that you have the Maven plugin installed, then choose "Import" from the File menu and import as a "Maven project").
This is all a good first step, but in order to boot up our server, we still need to manually take the JAR file that is built by Maven, deploy it into a Project Darkstar server, and then fire up the server. This is tedious, and we'd rather be able to do this using one command (or one click from the IDE). Enter the Project Darkstar Maven plugin. The plugin is pretty well documented on its website, but essentially what it does is automate the process of taking your built JAR file, deploying it into a Project Darkstar container, configuring the container with the appropriate configuration files, and then booting up the server. It also includes the capability to shutdown a running server. Using this, therefore, we can automate the process of running our Project Darkstar application by simply adding a configuration entry for this plugin into our POM in a separate Maven profile:
  
      run-server
      
 
          
            com.projectdarkstar.maven.plugin
            sgs-maven-plugin
            1.0-alpha-3
            
              
                sgs-run
                pre-integration-test
                
                  install
                  configure
                  deploy
                  boot
                
              
            
            
              ${sgs-server.version}
              
                ${project.build.directory}/sgs-server-dist-${sgs-server.version}
              
              ${basedir}/conf/hello.boot
              ${basedir}/conf/hello.properties
              
                ${project.artifact.file}

This configuration combines the install, configure, deploy, and boot goals of the sgs-maven-plugin into one. Also notice that there are two additional configuration files that are included in your source tree under the conf directory, hello.boot and hello.properties. These files are respectively used to replace the sgs-boot.properties and sgs-server.properties configuration files in the main Project Darkstar distribution. We can keep these files simple for now, but any set of valid configuration properties can be included:

# hello.boot
# =======
JAVA_OPTS = -server -Xmx768M

# hello.properties
# =======
com.sun.sgs.app.root=data

So we now have a complete project that looks something like this:
- hello-projectdarkstar
  - conf
    - hello.boot
    - hello.properties
  - pom.xml
  - src
    - main
      - java
        - my
          - pkg
            - hello
              - HelloProjectDarkstar.java
    - resources
      - META-INF
        - app.properties
The project can be built inside or outside of Netbeans or Eclipse using the same mechanisms as described above. However, with this additional configuration, we can also automatically build, deploy, and boot the application in a Project Darkstar container by simply activating the run-server Maven profile. When this is done, Maven will build the project JAR file, download and install the Project Darkstar distribution into the project's target directory, deploy the built JAR file into the Project Darkstar installation, copy the hello.boot and hello.properties configuration files into the Project Darkstar installation conf directory, and then boot up the server using the sgs-boot.jar from the Project Darkstar installation. This is completely automated and requires doing just the following for each of our respective environments:
  • From outside any IDE: just run the command mvn verify -Prun-server. Shutting down the server requires simply a Ctrl-C. Also, in order to clean out the Project Darkstar datastore in between executions, it is necessary to run mvn clean.
  • From Netbeans: You should configure the properties of the project by right clicking the project and selecting the "Properties" option. Configure a new Action (or one of the existing actions like the "Run" action) to run the verify goal with the run-server profile activated. Then you should be able to boot up the server by executing that specific action for your project (Right click the project and choose "Run" or whatever action you created under the "Custom" menu). In order to shutdown the server, you should also configure an additional action (i.e. Stop) that runs the sgs:stop goal with the run-server profile activated. Prefer using this action to shutdown the server rather than just killing the process (since multiple JVMs are started when Project Darkstar boots).
  • From Eclipse: Similar to Netbeans, you need to configure two "Run Configurations", one that runs the verify goal with the run-server profile activated, and one that runs the sgs:stop goal with the run-server profile activated. These two configurations should be respectively used to bootup and shutdown the server.
There are a lot more sophisticated ways that you can setup your project with the Project Darkstar Maven plugin that allows you to customize the runtime environment from the command line. You can also tweak it to support booting multi-node configurations by modifying the boot configuration file, or use filterable properties to customize these values at runtime. I won't get into these here, but I would suggest looking at Project Snowman's server POM for some ideas. Hopefully, though, this post should give you enough to get the ball rolling in setting up your development environment for a Project Darkstar application.

June 5, 2009

JavaOne: Signing Off

javaoneI just went to two more sessions here on the last day at JavaOne and they were actually both quite good. The first was given by William Pugh who is one of the creators of the FindBugs static analysis tool. We use FindBugs extensively in our build process with Project Darkstar and it has proven to be quite useful. It was interesting to see that a few of the bug examples given during this session actually had some overlap with Josh Bloch's Effective Java talk that I attended earlier in the conference. In addition to the FindBugs talk I also went to a talk on the ins and outs of the Java Virtual Machine. I am certainly no JVM expert so this was an interesting talk for me to see some examples of how the JVM does speculative optimization, garbage collection, and some other neat tricks to speed up your code.
I have one more session on my schedule for today but this will likely be my final blog post on this year's JavaOne conference. A few closing thoughts:
  • On the subject of Project Darkstar, I would say it was a generally positive event. Darkstar is a technology that clearly many have heard of, are interested in, and have a sincere desire to apply it for their own needs. With DarkChat, the CommunityOne and JavaOne sessions, and the hands-on lab, this conference seemed to do a decent job of generating some additional buzz.
  • On a more general note, there's obviously a bit of uncertainty surrounding the future of Java, JavaOne, and how Oracle will handle its future direction. I think that showed a bit at this conference, as there was no clear air of excitement and energy around anything in particular. There weren't really any groundbreaking announcements, no killer apps, no prototypes that can truly be considered game changers. I did see many examples of fine engineering and innovation but nothing worthy of eliciting that "Wow" factor from a general audience.
  • Finally, on a personal note, I enjoyed coming to this conference and felt like I was able to connect a little more to the community than at the two GDC conferences I've been to this year. While Project Darkstar is a technology targeted specifically at games, I wouldn't say I consider myself a "game developer" at all. I'm a Java developer, and more generally a software developer, and while it was a long and exhausting week, I was glad I got the opportunity to be here and expand my horizons a bit.
So that's it for my week at JavaOne 2009. I hope you enjoyed following along. Next week I'll be back to my regularly scheduled Ultimate-playing ways.

JavaOne: Days 3 and 4

javanet_logoI'm here at the final day of the JavaOne conference and things are starting to wind down. The pavilion floor is closed, there are only a few sessions left on the schedule, and people are starting to wrap things up. Karl and Keith actually caught flights home this morning but this being my first JavaOne (and last?), I wanted to stay for the whole thing. What JavaOne will look like next year is anyone's guess.

In any case, yesterday was a light day for me at the conference. I attended a session which was essentially a survey of a grab-bag of small test tools and gizmos. An interesting takeaway from that session was that testing Java code is not always best done using Java. Often times certain necessary and useful features of the language (like private methods and security features) make things difficult to test. Technologies like Groovy and JRuby, though, contain hooks that allow you to circumvent some of these features for the purposes of testing, without making your test code ugly, brittle, and difficult to maintain. I think the case that was made was good in the case of some tools, but others just traded one piece of complicated (but recognizable) syntax, to another piece of complicated (and unrecognizable) syntax instead. There were some things worth investigating though.

After that session, as mentioned I also recorded a quick podcast on Project Darkstar at the java.net CommunityCorner. The podcast was in the form of a conversational type interview with one of the editors at java.net and he told me that he'll be posting it and a blog of the event somewhere on java.net within the next couple of weeks. Stay tuned for that.

I didn't do a lot Thursday afternoon. I wandered the pavilion floor, talked with a few people, and handed out the rest of the hundred or so Project Darkstar pens that I brought with me to the conference. I then met up with Keith, Karl, and Mike. Mike and his wife offered their generous hospitality with a low-key, very nice get together at their house in Oakland Hills.

I attended James Gosling's general session "Toy Show" this morning and there were some neat things showcased. One of the cool innovations that was demoed was the mashup of a Wii remote, a JavaFX application being projected onto a piece of transparent, frosted glass, and a glove with infrared sensors on the finger tips. The result was a makeshift touch screen that gave a user interface similar to what they compared to as something out of the movie Minority Report. Pretty slick. There was also a more sobering presentation on how one company is using Java to do image recognition and analysis to identify images, including sharpening the ability to more quickly, easily, and accurately diagnose and identify cancer from biopsy images. Other highlights included a Java powered, energy efficient, hybrid Lincoln Continental, and an Audi capable of driving itself.
I'm getting ready to attend a few more sessions before the conference draws to a close, and I'll have some closing thoughts a little later.

June 4, 2009

JavaOne: CommunityCorner

sunrayI'm writing this post from one of the hundreds of Sun Ray stations scattered throughout the Moscone Center at JavaOne. If you don't know what a Sun Ray is, it's essentially a stateless thin client that, when I stick my JavaOne badge into, will pull up a virtual desktop associated with my JavaOne id from a centralized server. In the case of these SunRays, JavaOne attendees have the option of using an Ubuntu, OpenSolaris, or Windows Vista desktop. When I pull out my card, the desktop will be saved in its exact state, and I can pull it back up again from any other Sun Ray at the conference. In my opinion, Sun Rays are only useful in a limited number of use cases, but a conference like this definitely qualifies as one of those use cases. Instead of cracking open my laptop and trying to hook into wifi and not drain too much battery, I can just pop into one of these stations to check my email, register for sessions, or write a blog post!

Today at 11:30AM I'm going to be at the java.net CommunityCorner located at booth 101 in the Pavilion. A few other members of the Project Darkstar team should be joining me as well, and I'll be doing a brief podcast on Project Darkstar. If you're here at the conference, please stop by to chat (and take a free pen!).

June 3, 2009

JavaOne: Day 2

darkstar_logoI'm back for another recap, this time of day two at JavaOne. I managed to get up early again this morning and attend the morning keynote. To be honest, it was a bit of a yawn today. It was basically a marketing pitch for Sony Ericsson's platform and how they are making it easier to bring Java applications to their mobile platform. I could tell it was a marketing pitch because Christopher David, who drove the session said at least four or five times during the hour: "... and this is not just a marketing pitch". It is true that mobile apps are just going to grow, though, so it is a good market to move into. The session didn't do it for me though.

After the keynote, I sat in on Kohsuke Kawaguchi's session on bringing continuous integration in Hudson to the cloud. This seemed intriguing and relevant since I maintain a Hudson instance internally for Project Darkstar builds and am looking for ways for us to expand its usefulness. I was also interested to see what exactly he meant by using "the cloud". The term "cloud computing" is such an overloaded term these days and in fact I would suggest that a very small percentage of things that are named "cloud computing" actually qualify as such. What Kohsuke has done for this session though, is one of those things. Essentially, he's built a Hudson plugin that will automatically provision, configure and use an Amazon EC2 instance as a Hudson build slave dynamically and on demand based on load. So, if you have a Hudson build cluster, and your build machines are getting swamped with work, Hudson will spin up a new virtual build slave automatically and use it to help handle the additional demand. When the build queue dies down, after a period of time the EC2 instances will shut themselves down and disappear. Really slick.

After taking a break for lunch, as I mentioned yesterday I wanted to make sure I went to Josh Bloch's session on Effective Java. He talked about several topics including Java generics, enums, varargs, and serialization tricks. It was all really useful stuff and it turns out that I clearly have never really learned how to properly use wildcards with Java generics. In other words when to use List<? extends MyType>, List<? super MyType>, or just List<MyType>. This "PECS" rule, as Josh called it, is so simple that it's seems almost embarrassing that I didn't know it. Further incentive for me to read the rest of his book and overall a very worthwhile session.

In the late afternoon, I gave my newly tweaked Project Darkstar talk. I was actually really encouraged by this talk as we had I would say around 100 attendees in the room. Additionally, after the talk (which seemed to go pretty well), there were quite a few good honest questions and several people came by afterwards who were genuinely interested in learning more and doing more with the technology. In fact, there were more questions asked at the end of this talk then any other talk that I've attended so far at both CommunityOne and JavaOne this week. It's interesting too, that even though Project Darkstar is a technology specifically designed for games and game developers, I feel like it has had a much greater presence and interest at JavaOne, a Java developers conference, then at GDC, a game developers conference. That could also simply be because of the quantity and prominence of the sessions and events going on that are related to it though (including DarkChat).
Overall, it was a good day but also pretty busy and exhausting. I met up with Karl, Keith, Mike, and John for a good dinner and am now pretty much ready to crash.

June 2, 2009

JavaOne: Day 1

snowmanlabWhew. Technically it's still only day one of the JavaOne conference and I already feel like I've put in a week's worth of energy. I just returned from running the Project Snowman hands-on lab this afternoon. It was moderately successful with a few painful and time consuming hiccups - more on that in a minute. First though, another quick rundown of my day:

The keynote was great. Chris, of course, kicked off the event and introduced the DarkChat application that will be running throughout the conference. It's basically a simple social networking tool that shows off JavaFX. The other thing about it, though, is that the backend is built on Project Darkstar, and Keith is the one who had been spending a lot of his time recently getting that put together. Chris also snuck in a plug for the Project Darkstar sessions during that intro (the Project Snowman lab was completely full this afternoon).

After that, obviously, he yielded to CEO Jonathan Schwartz who ran most of the show. There were several announcements, including preview releases of JDK 7 and J2EE 6 and updates to JavaFX. Perhaps the most interesting, though, is the launch of the Java Store beta program. The Java Store is essentially a distribution mechanism that taps into the huge install base of Java to deliver Java applications to consumers in a consistent manner. Think iPhone's app store except replace iPhone users with Java users. Now, the concept behind the Java Store is not a new idea, but packaging and distributing Java applications has always been sort of a funny animal. I actually think this is a great initiative and hope it takes off.

After the keynote, I attended two sessions. The first was on using Java on game handhelds. This session caught my eye because the description claims that they were able to connect a Sony PSP to a Nintendo DSi using a Project Darkstar server as the game backend. Wow that sounds cool right? Well, it is cool, but not quite as cool as you think. What they actually did is hack the PSP and DSi and put their own firmware on the devices which supports JavaME. They then wrote a game client using the JavaME darkstar client and hooked them both up to the same darkstar server. After some finagling they did eventually get a PSP and two DSi clients logged into the same game, but they were not using anything native to the two devices. A ways off from what we'd like to see but still pretty neat.

The second session that I went to caught my attention because Josh Bloch was one of the presenters. For a while now, I've been meaning to get my hands on and read all the way through his book Effective Java. This session further solidified that desire. During the session, he and Neal Gafter analyzed five or six Java code snippets that look correct but actually have tricky and subtle bugs. Not only was the content interesting, but the back and forth banter style presentation format was entertaining and engaging. I plan to attend another of Josh's talks scheduled for tomorrow.
After a break, as I mentioned above, I proctored and ran the Project Snowman hands-on lab in the late afternoon (with the help of Dan, Keith, and a few other proctors as well). Overall, I would say it ran ok. There was one large and unfortunate problem at the beginning of the lab that reeked havoc throughout the remainder of the session. Essentially, one of the first steps in the lab is to install and configure the Maven plugin for Netbeans. Unfortunately, when all 100 people in the room tried to do that, we overwhelmed something on the network (whether it was the outgoing pipe in the room, the Netbeans server, or what who knows). Getting Maven up and configured for everyone took a while, and once it was up, network connectivity issues continued to plague the attendees as Maven would occasionally search for artifacts on the network causing it to pause or hang during builds. The result is that running the session was not very smooth as I and the other proctors (special thanks to them btw) spent most of the time running around the room debugging these annoying Maven issues rather than Project Darkstar issues. Regardless, though, I think many people did enjoy it and they were given all of the materials that they need to try to complete it on their own.

So that was my day. There's another session tonight on the Java Collections Framework that I'm currently signed up to attend. But first, dinner.

June 1, 2009

CommunityOne

communityone_logoGetting to JavaOne early to test out the snowman lab actually gave me a chance to attend some of the sessions and events going on today at CommunityOne. This is a free conference that is geared towards users and developers of open source technologies and software. Here's a rundown of what I did/saw:
  • The keynote this morning had two main focuses: cloud computing and OpenSolaris. There was some good stuff in there; Solaris has long had a bad reputation for being very unfriendly, making many oblivious to its core quality. Solaris is consistently the best performer in many of the Project Darkstar benchmarks that we've run, for example. I think OpenSolaris is slowly, but surely, making Solaris more accessible to the masses.
  • I attended two morning talks, one on the features in the latest Subversion release and one on a Hudson use case. Most of the material I was already aware of but looking at some of the things they were doing with Hudson, I think we should really investigate expanding our Hudson deployment. Currently we have a Hudson server doing continuous integration builds on each commit to the Project Darkstar trunk. However, it's only deployed on a single Solaris machine so our continuous test platform is limited to that. In the presentation, these guys were using Hudson to fire up multiple virtual machines representing various platforms on demand to do cross-platform testing of Netbeans. This could also be an option for us.
  • In the afternoon, I went to Chris Melissinos's talk on Project Darkstar. Chris is actually the Master of Ceremonies for the entire JavaOne conference and is one of the original promoters of the Project Darkstar project. If you've ever met him, you know that he's one of the most dynamic and outspoken people there is. Which makes this next fact even more surprising in that I actually ended up giving part of Chris's talk. About five minutes before the session was about to start, I bumped into Chris and chatted with him for a few minutes. Then he said something like "Oh good I'm glad you're here, do you mind coming up and sitting on stage to answer any technical questions people might have?" Um, sure? The next thing I know some guy is putting a microphone on me and I'm sitting in front of these blinding lights. Then, about halfway through the talk, Chris stumbles across some technical slides and says "Owen do you want to give this part of the presentation? Better you then me butchering it." Ummmm, ok? I spent the next fifteen minutes or so ad-libbing a sequence of presentation slides, having no idea what was coming next. Thanks Chris! The truth is, the presentation actually did turn out pretty well. Chris was his usual dynamic and entertaining self and I managed to bring just enough technical depth to the show. Karl suggested I stay away from Chris before he gives the JavaOne keynote tomorrow morning though!
  • During the keynote this morning I briefly heard them mention something about the OpenSolaris Juicer. The Juicer is a service which allows anyone to submit a piece of software for OpenSolaris for test and review to be included in the official "contrib" repository of OpenSolaris packages. Software packaging and distribution is interesting to me so I took a peak at their late afternoon session on the Juicer. It's good to see that OpenSolaris is finally approaching something close to the apt system that has been around for years on Debian and now Ubuntu.
That's about it for my day. I met up with a few others for dinner and am now getting ready to go to sleep. Big day tomorrow with the Project Snowman lab scheduled in the afternoon.

May 31, 2009

JavaOne Prep

javaone_logoI'm back in San Francisco this week in the familiar downtown area near the Moscone Center where JavaOne is being held. It wasn't that long ago when I was here at the same spot for GDC, and now I'm back for another first time experience. I've never been to JavaOne before, and what better way to gain my first exposure to it than as a first class citizen (speaker).

If you haven't already seen it, John already gave an excellent run down of the Project Darkstar related activities in the Project Darkstar team blog. At the risk of sounding repetitive, I'll be participating in three of those events this week:
  • On Tuesday, I'll be running a hands on lab on Project Darkstar. The lab will essentially be a step-by-step tutorial on coding up the Project Snowman game. I just zipped through the actual lab on the actual lab machines earlier this afternoon and I think it's going to turn out to be a pretty fun lab.
  • On Wednesday, I'll be giving a newly tweaked version of my Project Darkstar technical talk. Taking a queue from Brian (award winning high school history teacher), I've worked in some verbal sci-fi references to go along with the visuals. This should make it awesome.
  • On Thursday, I've been tagged to do a quickie podcast in the JavaOne Community Corner. I think it will be a pretty informal chat about Project Darkstar.
So those are my responsibilities this week. Other than that, I've scoped out and signed up for some other sessions that peaked my interest, including among others one on Hudson, a few on unit testing, and a couple being given by the Java veteran Josh Bloch. Then there are the keynotes, general sessions, Java Pavilion, and probably too many other events going on for me to keep track of. And of course I'll try (and most certainly fail) at keeping up with the host of parties and bashes going on in the evenings.

It will be a busy week ahead but I'm pretty excited about it. To be honest, a part of me was dreading this event. Mostly because I'm not a very outspoken person, and often like to keep to the shadows. Being put front and center for these sessions is essentially way outside of my comfort zone. I was also talking to Katy last night about how travelling can be draining for me. Between Austin, GDC, and now JavaOne, this is the third time I've travelled in a year's time. Small potatoes for some, but the most ever for me. Now that I'm here, though, I'm ready for the challenge and am looking forward to a successful (and fun!) week at JavaOne. Maybe I'll see you there...

May 12, 2009

Capacity Testing

There's one question that we get asked a lot about Project Darkstar: "How many users can you connect to one server?" This is a difficult question to answer, mainly because it's extremely sensitive to the context. The game type, game behavior, and hardware specifications all can have an extremely large effect.

Today I decided to see if I can establish an upper bound for this question. My goal was to put together an ad-hoc test to see how many idle clients I can log into a server. I used Tim's request app which is basically a little performance testing widget that accepts commands from clients (such as "JOIN_CHANNEL", "LEAVE_CHANNEL", etc.). It doesn't do anything when a client logs in, though, and will happily sit idly if the client doesn't send any commands. This makes it a perfect candidate for this test. I wrote a simple client that does nothing but login a configurable number of users. Here's what I found:

Machine configurations (1 used as server, 4 as clients):
Sunblade 6220
2 dual core AMD 2200 2.8Ghz
16GB RAM
Solaris 10u6

Maximum connected clients:
32bit JVM, 128MB max heap : ~800
32bit JVM, 1GB max heap : ~6000
32bit JVM, 2GB max heap : ~12000

I noticed when a limit was reached because each time, the server would throw an exception that looked something like this:

[INFO] SEVERE: acceptor error on 0.0.0.0/0.0.0.0:11469
[INFO] java.lang.OutOfMemoryError: Direct buffer memory
[INFO]  at java.nio.Bits.reserveMemory(Bits.java:633)
[INFO]  at java.nio.DirectByteBuffer.(DirectByteBuffer.java:95)
[INFO]  at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
[INFO]  at com.sun.sgs.impl.protocol.simple.AsynchronousMessageChannel.(AsynchronousMessageChannel.java:86)
[INFO]  at com.sun.sgs.impl.protocol.simple.SimpleSgsProtocolImpl.(SimpleSgsProtocolImpl.java:167)
[INFO]  at com.sun.sgs.impl.protocol.simple.SimpleSgsProtocolImpl.(SimpleSgsProtocolImpl.java:139)
[INFO]  at com.sun.sgs.impl.protocol.simple.SimpleSgsProtocolAcceptor$ConnectionHandlerImpl.newConnection(SimpleSgsProtocolAcceptor.java:316)
[INFO]  at com.sun.sgs.impl.transport.tcp.TcpTransport$AcceptorListener.completed(TcpTransport.java:331)
[INFO]  at com.sun.sgs.impl.nio.AsyncGroupImpl$CompletionRunner.run(AsyncGroupImpl.java:161)
[INFO]  at com.sun.sgs.impl.nio.Reactor$ReactiveAsyncKey.runCompletion(Reactor.java:858)
[INFO]  at com.sun.sgs.impl.nio.Reactor$PendingOperation$1.done(Reactor.java:630)
[INFO]  at java.util.concurrent.FutureTask$Sync.innerSet(FutureTask.java:251)
[INFO]  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[INFO]  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[INFO]  at com.sun.sgs.impl.nio.Reactor$PendingOperation.selected(Reactor.java:563)
[INFO]  at com.sun.sgs.impl.nio.Reactor$ReactiveAsyncKey.selected(Reactor.java:803)
[INFO]  at com.sun.sgs.impl.nio.Reactor.performWork(Reactor.java:323)
[INFO]  at com.sun.sgs.impl.nio.ReactiveChannelGroup$Worker.run(ReactiveChannelGroup.java:268)
[INFO]  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[INFO]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[INFO]  at java.lang.Thread.run(Thread.java:619)

From these numbers and the exception above, it looks like the maximum capacity of the server closely correlates with the configured maximum heap size, which makes sense. However, there are a few things that are odd:
  • Why are the numbers so low? The clients aren't doing anything once they login and yet they are eating up memory seemingly very quickly.
  • Even though increasing the heap size helps, connecting to the server JVM using JConsole shows that the memory usage never comes close to the max heap limit.
After digging through the stack trace as well as the darkstar I/O code, I discovered that the culprit lies in our use of DirectByteBuffers. First, for each client that connects, a DirectByteBuffer of length 128K is allocated to serve as buffer space for incoming packets. Second, memory allocated for DirectByteBuffers is not recorded as used in the Java heap space (even though the heap limit seemingly does have an effect) so it is confusing to monitor the JVM.

Fortunately, there are a couple of things I can do with this information to help improve my numbers. First, Project Darkstar provides a configuration property (com.sun.sgs.impl.protocol.simple.read.buffer.size) where you can change the read buffer size. Instead of 128K, I switched it to 8K, it's specified minimum. In most games, packet sizes should be very small, much much smaller than 128K, so changing this limit may be an acceptable solution in many cases. Second, and more of a big hammer approach is to switch to using a 64bit JVM. This would allow us to configure a heap limit greater than 2GB. Here's what I observed with these changes:

Maximum connected clients:
32bit JVM, 2GB max heap, com.sun.sgs.impl.protocol.simple.read.buffer.size=8192 : ~64000
64bit JVM, 16GB max heap, com.sun.sgs.impl.protocol.simple.read.buffer.size=131072 : ~64000
In both of these cases, the limit was tripped up by throwing a different exception this time:
[INFO] SEVERE: acceptor error on 0.0.0.0/0.0.0.0:11469
[INFO] java.io.IOException: Too many open files
[INFO]  at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[INFO]  at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
[INFO]  at com.sun.sgs.impl.nio.AsyncServerSocketChannelImpl$1.call(AsyncServerSocketChannelImpl.java:254)
[INFO]  at com.sun.sgs.impl.nio.AsyncServerSocketChannelImpl$1.call(AsyncServerSocketChannelImpl.java:251)
[INFO]  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[INFO]  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[INFO]  at com.sun.sgs.impl.nio.Reactor$PendingOperation.selected(Reactor.java:563)
[INFO]  at com.sun.sgs.impl.nio.Reactor$ReactiveAsyncKey.selected(Reactor.java:803)
[INFO]  at com.sun.sgs.impl.nio.Reactor.performWork(Reactor.java:323)
[INFO]  at com.sun.sgs.impl.nio.ReactiveChannelGroup$Worker.run(ReactiveChannelGroup.java:268)
[INFO]  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[INFO]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[INFO]  at java.lang.Thread.run(Thread.java:619)
This is a much better looking number, and our exception also suggests that we're now running into a different problem, most likely the max file descriptors limitation of the OS. This is likely configurable as well, but I haven't tried to increase it. A few closing thoughts:
  • The current, default implementation of the server has a perhaps overly conservative, fixed buffer size allocated for each connected client. This can be tweaked with the com.sun.sgs.impl.protocol.simple.read.buffer.size to reduce memory usage.
  • Properly tweaking this property gives us an upper bound of approximately 64000 connected clients (on Solaris 10, without making an effort to increase the max file descriptor setting for the OS).
  • It should be noted that the server handled login storms with minimal effort. In the final tests I bombarded the server with about 20000 logins at a time. Under these circumstances each client averaged a round trip (login initiation to login completion) of anywhere from 500 milliseconds to 15 seconds.
  • The clients were overloaded long before the server. I was unable to spin up more than 2000 clients per JVM before hitting out of memory errors and was forced to manage 20 to 30 client JVM's spread across 4 machines in these tests. A bit of a pain, and suggests that both the client could/should be optimized, and that some automation would be helpful.

March 30, 2009

Recap: GDC 2009

gdcI spent most of last week in San Francisco attending the annual Game Developer's Conference - a pure geek fest with all of the big names exhibiting like Nintendo and Sony, as well as many many smaller companies, independent developers, and students floating around. Having only previously attended GDC Austin last September, this experience was quite different without a Project Darkstar booth on the expo floor. On the one hand, I was able to attend more sessions, see more of the other booths and expos, and speak with more of the other attendees. On the other hand, though, there was a distinct lack of presence that Project Darkstar clearly had down in Austin. Here are some thoughts on the conference, in no particular order:
  • It was really great to finally meet Esko, Jonathan, and Andres - three of the four contest winners from the Project Darkstar Developer Challenge.
  • The aforementioned contest winners were interviewed on Chris Melissinos's radio show SHIFT Radio on Friday morning. I found some time to listen to it after I returned, and honestly, it's a great display of the internal gears of the Project Darkstar open source community at work. Here we have three different people, from three different countries, all actively contributing to the Project Darkstar effort for three different reasons. Esko, a student, is coming at it with academic curiosity and has independently built a feature into the core that many in the community have been clamoring for. Jonathon, a game developer, has taken an active interest in Project Darkstar from a practical standpoint and put together an ActionScript client that inter-operates with a Project Darkstar server. While Andres, an independent game developer, has built a real game with Project Darkstar. If you're interested in Project Darkstar at all, have a listen. On a side note, see if you can count how many times Chris says some form of the words ridiculous and awesome.
  • Probably the biggest splash that was made by a new player on the expo floor was the completely over-the-top OnLive booth. In twenty five words or less, OnLive is trying to make a grab for some of the console games market with a subscription based, games on demand model. Rather than buying a console and then paying for shrink-wrapped games, consumers get a small, network enabled device that gives them access to the complete library of games that you play over the network for a monthly fee. In terms of the business model, I think they have an approach that could take off (others seemed to disagree with me though). However, I have my doubts that they'll be able to iron out all of the technical issues with such a setup. Essentially, the only thing happening on the client side is that it is sending controller actions in real time back to the server, the server is processing those actions, and then sending just the video feed back to the client. The guys at the booth were talking mostly about their awesome video compression technology (and in fact I have no doubts that it is, in fact, awesome). However, I don't see that being the real problem. Because the client and server are in complete, synchronous lock-step, it doesn't matter how fast you can compress the video, the long tail by far is the network latency. You can improve your chances of low latency by sprinkling OnLive servers all over the globe, but the internet is still prone to blips, spikes, and just general unpredictable latencies. Add to that the fact that during peak demand, the OnLive servers could potentially have millions of users asking for not just processing power, but high performance graphics accelerator type processing power, and I just don't see how it's going to scale.
  • What's the deal with so many countries and provinces having extravagant booths? I saw booths for Canada, many of its provinces (Alberta, Newfoundland, etc), Korea, Germany, Scotland, Argentina... I suppose many places see a real value in reaching out to this industry as it can serve as a boon for the economy, it can generate jobs, and it can just generally stir up interest in ways that maybe other industries just can't do.
  • I did get a chance to make it to several sessions and lectures during the conference. Most of the talks that I went to were mostly related to game design and less so on the technical programming problems that we focus on for Project Darkstar. I still found them valuable, though, to get a general sense of the issues that game developers tend to think about.
  • Despite its simplicity, I found Hideo Kojima's Thursday Keynote both really intriguing and really entertaining. He basically chronicled the development lifecycle of the entire Metal Gear series of games, from the original Metal Gear developed for the MSX2 platform, to Metal Gear Solid 4 developed for the Playstation 3. I think what's most interesting is that many of the game design choices and game play choices were heavily influenced by the capabilities (or lack thereof) of the hardware and software being used.
Overall, GDC 2009 was a great experience. I hope that we get some positive fallout from it.

February 10, 2009

Project Darkstar Maven Plugin

As I mentioned in a previous post, one of the main goals I had in mind for version 0.9.8 of Project Darkstar was establishing an application deployment mechanism that was both simple and consistent. A cornerstone feature of any application container, at least from an IT and system administrator's standpoint, is a clearcut and highly predictable structure for application deliverables. This streamlines maintenance, keeping the costs down and contributing to just one of the value-adds of the technology. Personally, I think that we accomplished this with this release.

There is a flip side to this coin, however. In addition to administrator and deploy-time efficiencies, there are also developer use-cases that require simple, consistent, and automated bring up of a deployed Project Darkstar application. Whether it's for testing, debugging, or one-off temporary deployments, the need for quickly and automatically booting up a Project Darkstar server application during development is obvious. The question is, can we leverage the new distribution structure of 0.9.8, which accomplishes our first goal, to also establish a "best practice blueprint" for our second goal? I think the answer is yes.

The Project Darkstar Maven Plugin does exactly this. Using the standardized structure established in version 0.9.8, the plugin provides the capability to control a Project Darkstar server installation, including deployment, bootup, and shutdown. Is it a silver bullet? Probably not. But I've already ported the wanderer example and Project Snowman to use it with good success. If you're developing a Project Darkstar application, and especially if you're using Maven, I would encourage you to check it out.

January 6, 2009

Project Darkstar 0.9.8

Project Darkstar is, in the simplest description, an application server for games. What J2EE has done for business applications, we want Project Darkstar to do for MMOPGs, virtual worlds, and the like. If you've used a J2EE server such as Glassfish or JBoss, then you know that deployment and maintenance of applications into these containers is simple, and more importantly consistent. However, until just recently, this simplicity and consistency has been sorely lacking with Project Darkstar deployments.

Just before the holidays, we managed to roll a new release, version 0.9.8, of Project Darkstar. Officially announced today, this release features a major packaging improvement that greatly simplifies deployment, startup, and shutdown of Darkstar applications. Let's take a look at how this works.

At the heart of a deployment, is the Project Darkstar container itself. Unlike past releases, installing the server package is as simple as unzipping the distribution zip file. No additional steps are required. The extracted archive will leave a directory structure as follows:

- sgs-server-dist-0.9.8
  - API-CHANGES
  - CHANGELOG
  - NOTICE.txt
  - README
  - bin
  - conf
  - deploy
  - doc
  - lib
  - license
  - src
  - tutorial

The README file has a decent introductory explanation of this structure, but the main directories that you want to concentrate on are bin, conf, deploy, and lib:
  • bin - The bin directory contains two executable JAR files, sgs-boot.jar and sgs-stop.jar which are respectively used to startup and shutdown the container. Executing the command "java -jar /path/to/sgs-server-dist-0.9.8/bin/sgs-boot.jar" will startup the container and initialize the deployed application according to the configuration in the directories below.
  • conf - The conf directory contains three configuration files used for the runtime configuration of the system. The sgs-boot.properties file is used to configure the system environment and JVM configuration, the sgs-server.properties file represents the application configuration file which is fed to the Darkstar Kernel, and the sgs-logging.properties file is used as the java.util.logging.config.file during runtime.
  • deploy - The deploy directory contains all application-specific JAR files. When the container is booted up, these JAR files are automatically included in the JVM's classpath. Additionally, a single one of these JAR files can optionally include an embedded META-INF/app.properties file which is combined with the sgs-server.properties file from the conf directory to make up the set of properties used by the Darkstar Kernel. Out of the box, the deploy directory is empty and so booting up the container will immediately fail since there is no application to run.
  • lib - The lib directory suitably contains all of the JAR files required for any Project Darkstar application. These are automatically included in the JVM's classpath upon bootup of the container.
So we see above that the Project Darkstar container offers a rigid, well-defined structure, while at the same time being customizable and highly flexible. This brings us to the second component of a deployment, the Project Darkstar application. As stated previously, we want simplicity and consistency when developing and deploying an application into a container:
  • An additional feature of the 0.9.8 release of Project Darkstar is the explicit decoupling of the API interfaces from the implementation classes. The only compile-time dependency required when developing an app now is the sgs-server-api JAR file (found in the lib directory as well as the Maven repository). Simple.
  • Application specific Darkstar properties (e.g. com.sun.sgs.app.listener, etc.) can be bundled directly with the application by embedding a META-INF/app.properties Darkstar configuration file directly into your application JAR file. Simple. Consistent.
  • A Project Darkstar application is a JAR file or set of JAR files, one of which optionally contains a META-INF/app.properties Darkstar configuration file. Consistent.
  • A Project Darkstar application can be deployed into any Project Darkstar container by simply dropping the application JAR files into the deploy directory of the container and suitably tweaking the configuration files in the conf directory for your needs. Simple. Consistent.
There are other, subtle details about the flexibility and configuration of the container (including, among many things, how to use native libraries, how to use custom BDB libraries, how to use a different deploy directory, etc.) but I'll leave those for a separate discussion. In a future post, I'm hoping to case-study a real deployment using this new distribution.

October 9, 2008

Project Darkstar and Unit Tests

In my previous Project Snowman post, I mentioned that when developing games using Project Darkstar, there is a strong tendency to make gratuitous use of the AppContext class throughout all corners of your game server. Why is this a problem? Let me give you an example. Consider the following code:

public class Monster implements ManagedObject, Serializable {
  private int health;
  ...
  public void attack(int damage) {
    AppContext.getDataManager().markForUpdate(this);
    health -= damage;
  }
  ...
  public int getHealth() {
    return health;
  }
}

Now this is a pretty simple, stripped down class.  A Monster object has a health rating and this rating can be affected by inflicting damage via the attack(..) method.  So let's write a test for this method.

Since this is a unit test we don't want to fire up the whole Project Darkstar kernel; instead we want to just stub or mock out any dependencies this class may have. At first glance, though, it doesn't look like there are any, so this should be pretty easy right? Let's try:

public class TestMonster {
  @Test
  public void testAttack() {
    Monster testMonster = new Monster();
    int damage = 10;
    int finalHealth = testMonster.getHealth() - damage;

    testMonster.attack(damage);
    Assert.assertEquals(testMonster.getHealth(), finalHealth);
  }
}

All that we are doing here is instantiating a test Monster, and verifying that the attack method appropriately modifies the health of the Monster.  Does this test work though?  No.  The problem lies in the fact that there is a hidden dependency on the static AppContext method getDataManager().  When run outside of the context of a running Project Darkstar kernel, the behavior of this method is not defined (in fact it will throw a NullPointerException).  What's worse, since this method is static, it is impossible to mock its behavior to return a dummy implementation of a DataManager.  So what can we do?

For the Project Snowman effort, I devised a mechanism to wrap the AppContext static method calls in an interface called SnowmanAppContext. In fact, the strategy I used was very similar to the pattern described in this Google blog post which I just stumbled across today. This isolated the calls to the static methods in AppContext to within a single implementation of SnowmanAppContext. Unit testing became easy as I could simply replace the usages of SnowmanAppContext with a mocked out implementation.

Should this be done for all Project Darkstar games though? It seems like a built-in generalized solution should be possible. Fundamentally what is required for me to complete my unit test above is the ability to swap in a different context which is used by the AppContext underneath. Currently, the AppContext implementation is tied to the Project Darkstar kernel. If the kernel hasn't booted up, the behavior of AppContext is invalid. Since the AppContext is static, I can't stub it out. I am proposing a slight modification to the Project Darkstar API that would allow users to modify the behavior of AppContext to use whatever implementation that they would like. Further, this would completely decouple the Project Darkstar API classes from the current implementation. Here is my proposal:

First, add an additional interface to the core set of API classes:

public interface ManagerLocator {
  public ChannelManager getChannelManager();
  public DataManager getDataManager();
  public TaskManager getTaskManager();
  public  T getManager(Class type);
}

As you can see, this interface mirrors the set of methods that are exposed through the AppContext. The idea is that the AppContext will use a ManagerLocator underneath in order to retrieve each of the managers provided by the system. In order to set this, a static setter must also be added to the AppContext:

public final class AppContext {
  ...
  public static synchronized void setManagerLocator(ManagerLocator managerLocator) {
    ...
  }
}

Any implementation of the Project Darkstar API now must provide a class that implements the ManagerLocator interface, and set the context using the new AppContext setter method.  If these two minor changes are made, I can now complete the unit test that I originally set out to do using mock objects:

public class TestMonster {
  @Before
  public void swapContext() {
    ManagerLocator dummyLocator = EasyMock.createMock(ManagerLocator.class);
    DataManager dummyDataManager = EasyMock.createNiceMock(DataManager.class);

    EasyMock.expect(dummyLocator.getDataManager()).andReturn(dummyDataManager);
    AppContext.setManagerLocator(dummyLocator);
  }

  @Test
  public void testAttack() {
    Monster testMonster = new Monster();
    int damage = 10;
    int finalHealth = testMonster.getHealth() - damage;

    testMonster.attack(damage);
    Assert.assertEquals(testMonster.getHealth(), finalHealth);
  }
}

Now, this may not be quite how you'd set up your test with a real piece of code, but it demonstrates the point.  If you're not familiar with EasyMock, the code above essentially sets up a dummy DataManager that will be returned by the AppContext by putting it in a dummy ManagerLocator.  When the test is run, the call to AppContext.getDataManager().markForUpdate(this) will simply do nothing, and the test will pass.

I've coded up the necessary changes to the Project Darkstar server codebase for this API modification in a branch. If you're interested in having a look, the branch is located here. Feedback welcome!

September 23, 2008

Project Snowman: Lessons Learned

Project SnowmanIt was about two months ago that I was pulled into the Project Snowman effort for Austin GDC. The goal? Put together a complete, playable 3D action demo game that we can showcase on the expo floor at the conference as a demonstration of Project Darkstar's capabilities. The good news? We pulled it off. Thanks to the herculean effort by Keith, Josh, and Yi (plus the work that I did), we found ourselves with a networked, multiplayer "first person snowballer" as David described it to most of the people he encountered on the floor in Austin. [It's actually a third person snowballer with the main objective being to capture the flag but those are just details :)]. It really did work out well as the game itself drew a lot of people into the booth, and it made for a compelling demo with thousands of clients hammering away against a single server with it barely even breaking a sweat.

Now that the fun is over, though, it's time to take a step back and make some observations. What did we notice when developing this game? What did we do right? What did we do wrong? What advice can we give to help others trying to build a game with Project Darkstar? Here's my best shot:
  • HOLY CRAP! Developing a game with Project Darkstar is easy! Ok, well, maybe that's a little bit over the top. The truth is, though, Project Darkstar does a lot of things under the hood that normally a developer would have to think about and implement on her own (see the slides from my presentation at Austin GDC). Before I joined the effort in late July/early August, most of the work up to that point had been done by Yi on the client side. Animations, graphics, and building the client around JMonkeyEngine is where he had spent most of his time since starting as an intern at the beginning of the summer. Very little work had been done on the server side. A couple weeks later? We had a nearly feature complete and mostly unit-tested server side Project Snowman application. This is no hyperbole or exaggeration either, that's really how the timeline worked out.
  • Chuck Norris doesn't depend on external libraries, external libraries depend on him. This is actually true.
  • You should develop a client simulator early on. From the beginning, this project was meant to serve as a demo for Austin GDC. Therefore, we knew right off the bat that we would need some way to simulate a large amount of load on the server. We built a headless client that simulated typical client actions and not only did it prove valuable for testing, but it's also something that should be written for basically any Project Darkstar based game.
  • Chuck Norris can simulate maximum load on a server using only his fists and without boiling his blood. I've seen it done.
  • Plan to spend a lot of time tracking down performance bottlenecks and contention problems. While we were able to get a working game running in relatively short order, we quickly began noticing scalability problems and almost all of these problems were related to contention in some way. For example, one of the first contention issues we ran into was related to the AI snowmen that we introduced into the game. When the game started, the first thing that one of these snowmen did was loop through all of the game information (including all of the other snowmen and the flags) to determine who it should attack and where its opponent's flag is. On the surface, this sounds reasonable as it's just acquiring all of this information for reading. However, since the other AI snowmen were doing the same thing at the same time, each of them were attempting to acquire write locks on themselves. When the number of snowmen in a game was increased significantly, we were seeing pathological deadlock scenarios during game startup.
  • Code written by Chuck Norris never has any performance bottlenecks. Little known fact.
  • Built-in Project Darkstar profiling tools can prove to be extremely valuable. One of the most difficult things when working with a complex system like this is establishing clear ways to quantify performance. Fortunately, Project Darkstar has built in profiling capabilities that give you real-time feedback in terms of what the system is doing and how well it is handling the load. Seth has written a good blog post which can help you get started working with these profilers. In our experience, the most useful numbers were given by the SnapshotProfileOpListener which periodically output the number of attempted tasks, the number of successful tasks, and the average task queue size over 10 second intervals. This gave us a simple metric to be able to quickly determine whether the system is keeping up (the queue size remains small), and whether there is a lot of contention in the system (a high task failure rate is indicative of high contention). Another useful tool was the SnapshotTaskListener. Using it we could quickly determine which tasks were failing giving us better insight with regards to where contention is happening in the system.
  • Chuck Norris doesn't need profiling tools. He stares down the server until the profiling data comes to him. There's a rumor that he took on ten servers in a multi-node deployment simultaneously.
  • There is clearly a need for some type of Project Darkstar application test rig. Despite the fact that we were able to track down a lot of performance problems and contention issues using the built-in profilers, it was clear that a lot of the work required to setup and run these tests was highly mechanical and error-prone. Not only that, but without very careful record keeping, it was often difficult to keep track of what results came out of what conditions and whether or not certain changes helped or hindered the performance of the system. Most of our tests were setup in a very ad-hoc way and a framework that could consistently and automatically repeat our tests and give definitive results would prove monumentally useful. (Fortunately, this is on my to-do list).
  • Chuck Norris doesn't test his code. It always works because he tells it to. This would also make things easier for us.
  • Scalable data structures will likely be useful in almost any Project Darkstar game/application. Another problem that we faced was an issue with logins. In order to introduce a considerable load into the system, we needed to login a large number of clients in a short amount of time. However, this quickly became a problem. Our original implementation to handle logins simply added incoming players to a waiting queue to be asynchronously processed and matched into a game later. However, since there was a single queue, simultaneously adding a large number of players to the back while also removing players off the front created a massive amount of contention on the one queue object. How did we solve this? With David's ScalableDeque available in the com.sun.sgs.app.util package. The ScalableDeque allows for concurrent modification by allowing simultaneously writers on both the front and the back of the deque. We provided virtual support for multiple writers on each end by using an array of ScalableDeques. See the code for more insight on what we actually did. (Clearly this is something that could be generalized as a standalone utility. Add one to the to-do list.)
  • Chuck Norris can concurrently modify any data structure with no contention. Convenient.
  • Be careful of the AppContext temptation. If you're familiar with the Project Darkstar API, you know that access to the core Project Darkstar services is given through Manager objects. These Managers are acquired directly from the Project Darkstar stack by making static method calls against the com.sun.sgs.app.AppContext class. Why is this important? Well in my experience with code written against this API, I've noticed that there is a strong tendency to litter your application with direct calls to AppContext.getDataManager() or AppContext.getTaskManager(), etc. Why is this a problem? It tightly couples just about all of your classes with the static, unchangeable AppContext class of the core Project Darkstar API. This makes it extremely difficult to isolate your individual classes from the rest of the system for unit testing purposes. Now this can be worked around by making judicious use of the AppContext method calls and by being explicit in defining each class's dependencies. However, I would like to see this taken one step further and provide an alternative means of acquiring Managers from the Project Darkstar stack without relying on so many static method calls (another one for the to-do list).
  • Chuck Norris doesn't need Project Darkstar, he can roundhouse kick a piece of Java code into a complete MMORPG in 2.4 seconds. Just wait until version 1.0 though. By then even Chuck Norris will be using Project Darkstar.
That's about all the insight I can offer for one blog post. Don't forget that Project Snowman is an open source project itself. We do hope that members of the community take an interest in helping move its development forward.