Posts Tagged derby

The cache design

The biggest challange in my GSoC project is the design and implementation of a cache system. Why a cache system is needed? For two reasons:

  1. To show the revision graph of a file I need a lot of information. If the file is at revision 123 I need to ask the repository for its history from the base revision to the 123th revision. Next time, suppose the file is at revision 234, so I’ll need to ask the repository for its history from the base revision to the 234th revision. But, I’ve asked twice for the information from the base revision to the 123th revision. This is, the information I asked will never change. I should only ask for updates.
  2. The second reason is the most important one. There is no way to ask SVN for the whole revision history of a file. I mean, its history and the history of its branches. If you ask for /trunk/Foo.java, SVN won’t tell you anything about its branches. If you ask for /branches/2.x/Foo.java SVN won’t tell you anything about other branches or changes in /trunk/Foo.java since the 2.x branch was created.

So what to do? The solution appears to be building a cache system. The cache will store information previously asked and also will store the information about branches in order to query easily for the whole history of a given file.

My cache design is based in an embedded database (Apache Derby) with the following structure.

The “files” table

I’ve decided to give a unique ID to each file. The same ID is shared with all its branches. So /trunk/Foo.java and /branches/2.x/Foo.java have the same ID. This table tells you for example:

/trunk/Foo.java was added in revision xxx, deleted in revision yyy and has the ID 123

With this table you can ask for exmple

Which ID has the file /foo/bar.c in revision 123?

And when you get the ID of a file you can ask for all its branches and when they were created.

The other tables

There are other two tables: revisions and change_paths. These tables just store all the log information: author, revision number, date, action,…

The algorithm

When someone wants to see the graph of a file. I follow these steps.

  1. Fetch all the log information from the repository. Or only the updates from the last revision that was stored in the cache.
  2. Store the fetched information in the cache. Calculating IDs and branches.
  3. Query the cache for the *whole* history of a file. This is a simple query: Give me all the log messages of the file with ID xxx.

The problem is that the first two steps can be high time comsumption tasks. But in my opinion the big goal is to make the third step as fast as possible. As I learnt in database design the queries are the most important. However I’ve thought about possible changes to improve performance.

Possible alternatives

To only use the “files” table. Storing just the branching information will make the cache smaller. With the information of this table you know when files were added, deleted and branched. The information about authors, comments, dates and M (modify) actions will be fetched on demand. This approach could have two steps. First building a simple graph with the branches and then the information about authors, comments,… could be loaded asynchronously.

To only store the log messages of selected files. This is, not to store all log messages but only the log messages of those files that are required to show the graph. This is the same than the previous approach but storing the log messeges once they have been fetched from the repository.

These approaches make the cache smaller. But they need to connect to the repository several times. I will only change my main approach to one of these if queries are slow.

My focus now is to resolve some bugs and improve the graphical part. This is, I want to integrate the graph into an eclipse view / editor and make it work with small but real-life repositories. I also want to show a progress bar while the cache is fetching information from the repository or while is calculating the IDs and branches.

Leave a Comment

First screenshot

This is the first screenshot about my progress in the Google Summer of Code.

Of course there is a lot of work to do. This is just the first visual thing I can show you. I have been working with the Draw2d API and I have tried to do something similar to the RevtreePlugin. Each column is a branch and each change is ordered by revision number from the top to the bottom. At the top you can see the names of the different branches.

My project at Google Summer of Code can be divided into two tasks.

  • The first task is the “cache”. Since I need a lot of information from the repository I need a cache system. I am developing a cache using an embedded database writen in pure Java: Apache Derby. By the moment the cache works, but at the moment is very simple. I will need to do some perfomance enhancements to be able to work with real-life repositories. I will talk about this in a future post.
  • The other task is the graph. Once we have taken the information about the revisions of a given file I need to show it. This is what I’m showing you. I have spent some time reading this article and playing with the Draw2d API and that is the first result. You can see it like the first version of the Wikipedia (something that will be good in the future, but not at this moment 😉 ). As you can see the graph is shown in a separated window. However it should open inside an Eclipse View. So this is work in progress :P.

Leave a Comment