Sunday, January 30, 2011

Berkeley DB

Berkeley DB is a Java based non-relational database that is scalable and offers multiple thread and process support.  This is ideal for use in the Solar Decathlon home management system, now renamed to iHale.  The main idea is shown in the diagram below:

Client-server-database communication

On the client side, Wicket provides the front end user interface via a webpage.  Through this webpage, a user may request to perform a GET, PUT, or DELETE operation.  The client communicates to the server via HTTP over the Restlet framework, where the request is interpreted and executed by the server.  Information in the database is persisted through a file saved on the disk.

To get familiar with Berkeley DB, I worked on two code katas that dealt with the storage and retrieval of contacts.  These were modified versions of the katas I completed on Restlet in the last two blog posts.  More details can be seen here.

Kata 1: Timestamp as a secondary index
The primary index for a contact is their unique ID.  This means that entries can be added or searched in the database via a unique ID.  The task was to add a timestamp as a secondary index.  The timestamp is appended to a contact when it is created.  The most time-consuming part of Kata 1 was writing unit tests and learning Berkeley DB's API to query the database.  It was fairly straightforward and took about 3 hours to complete.

Command line usage of all the available operators: put, get, get-timestamp, get-range.

Kata 2: Wicket, REST, and Berkeley DB: A match made in heaven
This kata is a combination of all the technologies I have learned so far which includes Wicket, REST, and Berkeley DB.  It combines the use of Wicket in Restlet Part I, concepts of persistency from Restlet Part II, and Berkeley DB from Kata 1 above.  The difference is that contact information is now persisted in a file on the disk via Berkeley DB instead of in-memory.

Webpage for adding entries and querying the database.

The overall layout of Wicket, REST, and Berkeley DB is readily understood, but getting it all working properly together is a whole different story.  So far, this kata has been a debug-fest.  It took about 5 hours to code the core Wicket, resource server, database, and ant build files, but debugging has taken over 7 hours.

Some of the problems I encountered were trivial, such as forgetting to attach the timestamp to a contact XML document.  These resulted in exceptions when trying to get or put a contact into the database.  Other problems were due to communication errors between Wicket and the resource server, which resulted in odd exceptions such as Internal resource error 500.  I was able to resolve these by desk checking my code several times.

The lessons that I learned from this exercise is that one must be extra careful when copying and pasting code from working modules in past projects.  This method is extremely error-prone.  Also, it is ideal to import small portions of code at a time and perform frequent testing instead of importing huge blocks of code all at once and testing everything afterwards.

The final distribution of my katas can be downloaded here.

Saturday, January 22, 2011

Restlet Part II

In continuation from Restlet Part I, Part II includes three new katas that exercised the use of GET, PUT, and DELETE operators.  It was many times more difficult to implement because it required six different components to interact with each other:  a client, record of a client, in-memory persistent database, server for accessing the database, server for handling individual requests for a contact, and a server for handling requests for all contacts in the database.  More details can be found here.

Kata 5: Contacts resource
The task was to add a new resource that returns an XML representation of all the contacts in the database.  In other words, the contacts were stored in a Java collection and the goal was to transform it into an XML document.  At first I felt overwhelmed, but I identified the steps required to get the job done in about 4 hours:
  1. Create an XML document.
  2. Iterate through each contact in the collection, then attach it to the document.
  3. Return the XML document when the servers asks for this resource.
XML representation of contact resources in the database

Kata 6: Command line client manipulation of Contacts resource
The task was to extend the command line client functionality to include additional operators "get-all" and "delete-all". The get-all operation would retrieve all contacts in the database, retrieve their respective URLs, then print out contact information.  The steps required to do this was similar to Kata 5 but in reverse; an XML representation of a contact had to be transformed into its Java representation.  This kata took about 5 hours to complete because I had to look online and learn how to play with XML nodes.

Printout of contact information after calling the get-all operator

Kata 7: Add a telephone number to the Contact resource
The task was to extend the Contact resource to include a person's phone number.  In addition, the server would check to see if it was in the correct format ###-####.  If not, it would throw an error.  This kata was particularly easy and took about 1 hour to complete.

Error code 417 when the phone number does not adhere to the xxx-xxxx format

By completing these katas, I learned how to implement a RESTful client and server system that uses an in-memory persistent database.  I also improved my skills in performing meaningful unit testing to meet quality assurance standards.

A distribution of my work can be downloaded here.

Tuesday, January 18, 2011

Restlet Part I

Representational State Transfer (REST) is a software architecture designed for use on the web.  Whereas the web is mainly used for interaction between a client and a server, REST allows for interaction between servers themselves.  It relies on an architectural usage of HTTP URLS which includes resources and verbs such as GET, PUT, DELETE, and POST.  For example, would have the resource ( and the verb GET (news).  Its action is to retrieve news from Hawaii.

A system like REST can be implemented with a Java framework called Restlet, which is the topic for today's blog post.  To introduce myself to this new framework, I practiced on four Restlet Code Katas.  Additional information on these katas can be found here.

Kata 1: Time resources
The task was to add three new resources to return the current hour, minute, and second.  I completed it in about 10 minutes.  Since it was the first introductory kata, the most difficult part was figuring out the actual resource and verb, which was localhost:8111/dateservice/[hour/minute/second].

The current month is January so the output corresponds to 0.
Kata 2: Logging
The task was to log Restlet's messages into a log file instead of having it display in the console.  The program was to read configuration settings from a properties file and use a handler to record all messages into a text file.  I completed this in about 2 hours.  The bulk of the time was spent trying to understand the whole process and just figuring out how to get it done.  A really handy explanation of logging is explained here.

Console output from Restlet is recorded into this log file.

Kata 3: Authentication
The task was to authenticate a user by asking for a username and password and verifying it against locally stored credentials.  Similar to Kata 2, this took me about 2 hours to complete.  Nearly all of the time was spent on research.  The solution can be found here.  Most of the problems I had were finding only outdated articles.  One article recommended using the class Guard, which was dated back to Java 1.4.

[Update (Jan. 24): I discovered that this feature is buggy.  The application does not always challenge the user with a username and password even after the server has been restarted.  I will update this section with an appropriate screenshot and reupload a new .zip distribution file once I figure out the problem.]

Kata 4: Wicket
The task was to create a client that could access the server via a webpage instead of the command line.  This required the use of the Wicket web framework.  I was able to complete this kata with about 6 hours of effort.  Several hours were wasted in debugging a client-to-server connection issue.  At first, I thought it might have been due to authentication but it turned out to be trivial problem.  The issue was due to a simple mismatch in ports; the server was being hosted on port 8111 but the client was trying to connect to 8112.

Basic webpage with buttons for requesting the date.

By completing these four katas, I learned the concepts of the Restlet framework.  I also learned the basics of networking in setting up a client and server side system.

A distribution of my files are available here.