[cvsnt] CVSNT server tries to write non UTF-8 characters infileattr.xml

Erik Larsson catacombae at gmail.com
Mon Dec 29 06:24:00 GMT 2008

Hi Arthur,

Arthur Barrett wrote:
> Try and keep the thread on the newsgroup  - just send the attachments only to support at march-hare.com since the newsgroup will reject them.

Thunderbird was playing tricks on me, not sorting that mail in the 
correct folder, so it appeared to be a private mail. My apologies.
All the previous attachments along with two new ones (trace logs) have 
been sent to support at march-hare.com .

>> I have used two different clients while getting this error, SmartCVS 
>> 7.0.4 and TortoiseCVS 1.10.8/CVSNT
>> I don't think the clients are the problem here though.
> There are two parts to the 'problem' you have reported, and they both need investigating:
> 1) fileattr filenames are not UTF8 when CVSNT Server is not running in UTF8 mode
> 2) some CVS clients get corrupt filenames when server running in UTF8 mode

I would expect problem 2 to be something that the particular CVS client 
vendor should address, and problem 1 to be the server's issue.

> SmartCVS is a GUI - what I need is the CVS version.  WinCVS, TortoiseCVS and MacCvsX all include CVSNT - I imagine that SmartCVS is including some very very broken version of CVS that is claiming to support UTF8 filenames but isn't.  The CVS client verion will either be some embeded java thing, the 'standard' one in /usr/bin or embedded in the .app directory for SmartCVS (eg: SmartCVS.app), ...

SmartCVS seems to be using the Java based CVS client library that comes 
with NetBeans (so I would expect NetBeans to behave badly with UTF-8 
CVSNT servers as well, but I have not tested that).
I can't tell which particular version is being used... only that the 
library org.netbeans.lib.cvsclient is included with the distribution.

> If possible just replace the CVS version on the Mac client with the latest CVSNT and you should be able to run the CVSNT server in Unicode mode, this would be a good 'workaround' for your fileattr problem.  If SmartCVS is using some embedded Java client then you'll need to contact SmartCVS for a fix, though we may be able to 'workaround' the bug if we can identify the client from the connection string on the server.  In fact I've just had a quick look and this does appear to be the case.  I've no idea why so many programmers think it's a good idea to re-invent the wheel ... it inevitably causes these compatibility problems.

As stated above, it's not possible to simply replace some binary.
About reinventing the wheel... it's not really proper to use native 
libraries or binaries in a cross-platform java app (it breaks the whole 
concept of Java), so in this particular case it's inevitable. :) (at 
least until someone releases a usable C-to-JVM compiler)

>> I don't think any automagic conversion takes place with clients that 
>> don't support UTF-8. At least when using SmartCVS on Mac OS 
>> X, switching 
>> the server to Unicode mode and then doing a CVS update corrupted all 
>> filenames with non-ASCII characters (all instances of 'å', 
>> 'ä', 'ö' were 
>> replaced by two completely incorrect characters, so you can tell that 
>> it's a matter of incorrect UTF-8 handling).
>> TortoiseCVS handles the Unicode mode fine though.
> In SmartCVS 7 under project->Settings on the Text File Encoding tab there are some options you can try.  If you cannot find any setting that works with the server in unicode mode then I can use the SmartCVS client identifier to automatically adjust the CVSNT Server to convert the encoding (bug5479). 

I have tried setting the text file encoding to UTF-8 (not proper, as the 
files themselves more often are stored in iso-8859-1, but it actually 
varies), and that didn't have any effect on the filename encoding 
(filenames are still garbled when the server is in Unicode mode), which 
is what one would expect.

> More questions:
> A) which client do you think added ETT_FALL_OCH_EN_LÖSNING.txt - was it Mac or Windows (CVS  or CVSNT)?

In my testing I tried adding the file from both Mac (SmartCVS 7) and 
Windows (TortoiseCVS/CVSNT 2.5.03). In both cases cvs add completed 
successfully with a proper file name (upon inspection, it seems that the 
add operation doesn't write any entries in fileattr.xml). It's when 
doing cvs commit on the added file that the error message appears. It 
does not seem to matter which client I use, I have tried both.

> B) What 'connection string' is the Mac using/what protocol? Eg: :pserver:host:/blah

:sserver:my.cvs.server.dns:/ROOT (not really my.cvs.server.dns, but you 
get the picture)

> C) If you add a new test file with an Ö character in the filename from the windows 2.5.05 client with the server running NOT utf8 is the fileattr corrupted?  If so is it possible to enable 'allow clients to trace server' on the server and trace the add and commit (cvs -ttt add; cvs -ttt commit -m "") and send the entire trace to support at march-hare.com?

Yes, the same kind of corruption occurs when committing files with the client to a server in non-unicode mode also running
The trace logs you requested have been sent to support at march-hare.com .

> D) Try downloading the 2.5.05 mac client and installing it and trying to use it from the command line with the windows server running in UTF8 mode.  Checkout the same file and if the filename becomes corrupted then enable 'allow clients to trace server' on the server and trace the checkout (cvs -ttt co path/to/ETT_FALL_OCH_EN_LÖSNING.txt) and send the entire trace to support at march-hare.com

As I said above, SmartCVS isn't using CVSNT. I expected CVSNT for mac to 
work just like on Windows. When testing I also found that this was the 
case, CVSNT handles the Unicode server fine in Mac OS X too.

> E) what is the LANG of your Mac client?  (echo $LANG)


> F) what is the iconv version of your Mac client? (iconv --version)

GNU libiconv 1.11 (SmartCVS doesn't use libiconv, or CVSNT, so I 
wouldn't think that it matters)

> G) what is the MacOS X version of your Mac client, eg: ppc 10.3.7 or intel 10.5.1?

Mac OS X 10.5.6 / Intel


- Erik ( http://hem.bredband.net/catacombae )

More information about the cvsnt mailing list