Posts Tagged cache

Cache implementation tests

I have just committed the test suite I’ve been using in order to test the cache implementation. You can find it at:

I have created a basic plain text format to simulate repositories and perform queries to the cache. This is an example:

A	/trunk
A	/branches
A	/tags
A	/trunk/project
A	/trunk/project/foo.txt
M	/trunk/project/foo.txt
A	/trunk/project/bar.txt
M	/trunk/project/foo.txt
M	/trunk/project/bar.txt
D	/trunk/project/foo.txt
test	/trunk/project/foo.txt	3
3	A
4	M
6	M
7	D

The first line clears the cache. The second line says that the cache is going to be loaded with data. The following lines contain the log messages that will be saved into the cache. The lines with a number start a revision. That number is the revision number. The other lines contain the changed paths information. For example A /trunk indicates that the /trunk folder is created (A). The author, date and log messages are not indicated. The action (A) and the path (/trunk) are separated with a tab. Every argument in each line is separated with a tab. To finish loading data into the cache the update command is used. Some change paths require more information. For example:

A	/branches/1.0	/trunk	2

That changed path indicates that the folder “/branches/1.0” is copied from “/trunk” at revision “2”.

To test the queries a against the database the test command is used. The test keyword is followed by a path and a revision number that identify the file. Then the cache creates a graph of the specified file in memory. To test verify the graph is ok all branch paths are listed followed by their nodes. This is:

3	A
4	M
6	M
7	D

Those lines tell that the branch /trunk/project/foo.txt should have the following nodes: A at revision 3, M at revision 4 and 6 and D at revision 7.

The test files are stored in a folder called “testfiles” and to run them you just need to run test.CacheTest using JUnit.

Note: the code depends on the following projects:

  • org.tigris.subversion.clientadapter
  • org.tigris.subversion.subclipse.core
  • org.tigris.subversion.subclipse.graph

Leave a Comment

Cache implementation reloaded

It’s several days since I don’t talk about my task on Google Summer of Code. I’ve been having a great discussion in the development mailing list with Stefan Fuhrmann. Stefan has developed the revision graph for TortoiseSVN. Thanks to him I noticed my implementation didn’t scale. The algorithm wasn’t lineal, it had and exponential performance. These were very bad news.

The problem was with deleting or copying directories. When I got a “D /trunk/subfolder” I was marking all subfolders and subfiles as deleted too in that revision. Clearly the performance degrades when the number of files grows. And obviously this is not acceptable since copying a directory is very common: is what is done when you create a branch!

During this week I’ve rewritten most of the code. Now the cache follows these steps:

  1. Store all log messages without processing them and using batch updates. Since it doesn’t process the messages (i.e. mark any calculation about subfiles or subfolders) and it uses batch updates this step has a lineal perfomance and is faster than in the older implementation.
  2. Given a path and revision number it finds the first revision and path of that file (i.e. the root branch). It finds the first “A” action that is not a copy.
  3. From the root branch it iterates over the next revisions finding branches.

Now the performance meets my expectations. Steps 2 and 3 take about 8 seconds in a repository with 4,000 revisions. Both steps just make two queries against the database. They don’t execute any write statement so they are very fast. All the graph is calculated in step 3. The graph information is holded in memory and is serializable so it gives me an opportunity to save it in a file for a future use. That’s currently not implemented but I’m discussing it in the development list. Also I’m thinking a way to avoid the use of the embedded database since the current queries and insertions are very simple and probably I could use just files to store the information. I would like to remove the database dependency because it takes about 10 seconds to be initalizated (i.e. making the getConnection() call).

Leave a Comment

The cache design

The biggest challange in my GSoC project is the design and implementation of a cache system. Why a cache system is needed? For two reasons:

  1. To show the revision graph of a file I need a lot of information. If the file is at revision 123 I need to ask the repository for its history from the base revision to the 123th revision. Next time, suppose the file is at revision 234, so I’ll need to ask the repository for its history from the base revision to the 234th revision. But, I’ve asked twice for the information from the base revision to the 123th revision. This is, the information I asked will never change. I should only ask for updates.
  2. The second reason is the most important one. There is no way to ask SVN for the whole revision history of a file. I mean, its history and the history of its branches. If you ask for /trunk/, SVN won’t tell you anything about its branches. If you ask for /branches/2.x/ SVN won’t tell you anything about other branches or changes in /trunk/ since the 2.x branch was created.

So what to do? The solution appears to be building a cache system. The cache will store information previously asked and also will store the information about branches in order to query easily for the whole history of a given file.

My cache design is based in an embedded database (Apache Derby) with the following structure.

The “files” table

I’ve decided to give a unique ID to each file. The same ID is shared with all its branches. So /trunk/ and /branches/2.x/ have the same ID. This table tells you for example:

/trunk/ was added in revision xxx, deleted in revision yyy and has the ID 123

With this table you can ask for exmple

Which ID has the file /foo/bar.c in revision 123?

And when you get the ID of a file you can ask for all its branches and when they were created.

The other tables

There are other two tables: revisions and change_paths. These tables just store all the log information: author, revision number, date, action,…

The algorithm

When someone wants to see the graph of a file. I follow these steps.

  1. Fetch all the log information from the repository. Or only the updates from the last revision that was stored in the cache.
  2. Store the fetched information in the cache. Calculating IDs and branches.
  3. Query the cache for the *whole* history of a file. This is a simple query: Give me all the log messages of the file with ID xxx.

The problem is that the first two steps can be high time comsumption tasks. But in my opinion the big goal is to make the third step as fast as possible. As I learnt in database design the queries are the most important. However I’ve thought about possible changes to improve performance.

Possible alternatives

To only use the “files” table. Storing just the branching information will make the cache smaller. With the information of this table you know when files were added, deleted and branched. The information about authors, comments, dates and M (modify) actions will be fetched on demand. This approach could have two steps. First building a simple graph with the branches and then the information about authors, comments,… could be loaded asynchronously.

To only store the log messages of selected files. This is, not to store all log messages but only the log messages of those files that are required to show the graph. This is the same than the previous approach but storing the log messeges once they have been fetched from the repository.

These approaches make the cache smaller. But they need to connect to the repository several times. I will only change my main approach to one of these if queries are slow.

My focus now is to resolve some bugs and improve the graphical part. This is, I want to integrate the graph into an eclipse view / editor and make it work with small but real-life repositories. I also want to show a progress bar while the cache is fetching information from the repository or while is calculating the IDs and branches.

Leave a Comment