What is the “read length line” error?

This particular error manifests itself because of a particular kind of corruption that has occurred in the repository. Malcolm Rowe provided a great analysis and description of the problem here: http://svn.haxx.se/dev/archive-2006-02/0473.shtml. To put it in a nutshell, the crux of the issue is that two handles are eventually opened to the transaction by mod_dav_svn, and one of them is freed later which ends up writing the remaining portion of it’s cache. The end result is that the representation is now malformed, and this extra data has invalidated a number of the offset specified later in the file. The “read length line” error is a result of the the backend trying to read from the wrong offset.

What can you do about it?

At the moment, not much. The root cause of the problem is very elusive, making it extremely difficult to track, and more difficult to produce. If for some reason, you’ve come up with a reproduction recipe though, please email dev@subversion.tigris.org, as we’d very interested in having such a recipe. However, I feel it’s mostly an issue with configuration, load, and some decision process that Apache2 makes when deciding to fork a new process to handle a request, coupled with some other error, making this a very difficult bug to reproduce.

Update: I’ve found that most corruptions happen on systems configured to use a threading model whether via svnserve or mod_dav_svn. If you have a system that supports forking, use either Apache’s mpm_prefork or make sure svnserve is not running in threaded-mode. That substantially reduces the chance of corruption.

What is the Subversion team doing about it?

The team is aware the problem exists, and is eager to get it resolved. There are on-going discussions of how to fix this issue, but without knowing where the problem actually resides it’s difficult to fix this issue properly. We may just end up masking the problem if we aren’t careful. That said, there are discussions right now about limiting the write handle to a transaction to just one process. That comes with some tradeoffs, mainly in terms of commit speed, as we may not have the option to pipeline the commit operation.

Update: There have been several commits (see below) that have addressed several bugs that are related to some of the corruptions. Some have been backported to 1.4.x. I believe a few more fixes will be in 1.5. Here is a sampling of commits that address several types of corruption found in some FSFS repositories:

• 21738
• 29091

How do I fix my repository?

Fsfsverify to the rescue! Attached to this page is a script that can—hopefully— fix the corruption for you. To use it, simply type:

fsfsverify.py -f /path/to/repos/db/revs/corrupted-rev-file

Note that this script was designed “on-the-fly” to help diagnose repositories suffering from the read length line error when I didn’t have access to their repository or rev files. It was never meant to be a repair facility for FSFS (as you can tell from the name). It’s not a very well-written piece of code either—the effects of writing code at 2AM are definitely present. Finally, I’m not actively maintaining this script either. The correct fix for this problem is to prevent the corruption in the first place, which means we need to handle the situation better in mod_dav_svn, or put in place some extra restrictions. In the meantime, feel free to use the attached script. And please, if you make changes, push them back at me, and I’ll do my best to get them up here to share with everyone.

Update: fsfsverify now exists in Subversion’s contrib area (contrib/server-side/fsfsverify.py). Hopefully that means it’ll receive a little more attention. Also, I completely rewrote the script months ago to gear it more towards fixing various corruptions.

Can you (jszakmeister) fix my repository?

If you ask nicely, I can try. Either post the affected revision somewhere that I can reach, or attach it to an email (if it’s less than 1MB). Shoot me an email at john@szakmeister.net, and I’ll get back to you as soon as I can. Please, keep in mind that I’m not paid for this. I have a full-time job, two kids and many other interests. My “free” time is pretty limited (just an half an hour or so in the wee hours of the morning) and I generally like to spend it doing something else.

I’ve had a few folks get belligerent because I wasn’t spending as much time on their problem as they would like. If you want immediate service, call CollabNet and see if they can help. I will not tolerate such an attitude. I’ve got better things to do than deal with idiots.

Recently, I’ve had a rash of people with issues contact me to help fix their repository. Unfortunately, I’m going to start turning some folks away–especially those who appear to have a series of issues and won’t let me have access to the complete repository. It’s just to much work and far too time consuming to keep iterating over email one revision at a time. For those corporate folks, no, I will not sign an NDA. I don’t want any restrictions placed on me that may damage my current or future employment.

Can you (jszakmeister) teach me how to fix corrupt revisions?

Short answer: no, but you can teach yourself. I learned this on my own from the documentation in the Subversion repository. The two most important documents to read are: the fsfs structure, and svndiff. Between those two documents, you should have everything you need to understand the format of the FSFS backend, and thusly, fix your own corruptions. At some point in the future, I may write a few things on how to do some of this, but right now, I’m simply too busy.

Updates

July 24, 2006

I’ve recently updated fsfsverify to try and handle errors in the svndiff stream itself, instead of just errors that occur at the start of the stream. It hasn’t received much testing, but works against several revisions that were submitted to me by a Subversion user. As I don’t want this tool to take on a life of it’s own–since it becomes easier to cope with the bug, and doesn’t motivate us to fix it–I’m not going to implement version numbers for the tool and such. However, I will try and keep this page updated with any new fixes that are made to fsfsverify.

February 22, 2007

I’ve had several requests for support for the svndiff1 format that the new Subversion tools can create. Fortunately, the nature of the corruption is that it occurs on a window boundary (when it’s within the svndiff stream), so I didn’t have to write my own de-compressor, and fsfsverify will still be able to recover the rev file (provided there was no actual data loss). With that, here’s the new version! A fair amount of it has been re-written, mainly to support compressed streams, but also to just clean things up. It should be far easier to hack, if you need to do so. If you find a problem, shoot me an email and let me know. I’ll try my best to fix it, if time permits.

A special thanks to Chris Thomas for providing a patch to add svndiff1 to fsfsverify. Even though I didn’t use it, I very much appreciate the gesture.

April 24, 2007

There was a regression in fsfsverify that has been fixed along with a few other minor details. Enjoy!

April 27, 2007

Just a minor change that prevents getting a traceback for exceptions that I expect to occur on corrupted repositories. I’ve had a few users email me thinking that fsfsverify was broken, so this should hopefully make it more clear that we’ve identified a problem, and that you should run with -f to fix it.

July 25, 2007

Added the ability to detect corruption in the window header, and fsfsverify will no longer traceback if directory entries are stored in another revision. Also made a couple of other minor cleanups, with the goal of getting this into the Subversion tree.

September 23, 2007

The latest version of fsfsverify has been checked into Subversion for inclusion in 1.5.x. Hopefully, that means it’ll be there when people will need it. Thanks to C. Michael Pilato for prodding me!

February 20, 2008

Made a few modifications related to handling the NULL digest and copyfrom parsing.

June 9, 2011

Updated the links to svndiff and the fsfs structure.

Download

Use at your own risk! By downloading fsfsverify, you agree that I, nor the company I work for, are liable for any damages caused, either directly or indirectly, by using fsfsverify. You should always backup your repository before using fsfsverify.

The Subversion tree has the latest version of fsfsverify.