[Svnmerge] [RFC] Pipes as iterators / multiple pipes

[Giovanni Bajo]

Hello,

it's been a while I have been thinking of modifying launchsvn() so that it
returns an opened pipe, instead of running the program and catching the whole
output at once. This can have a couple of benefits:

- It provides a way to parallelize operations and thus gaining speed benefits
when the SVN connection time is the bottleneck. For instance, the attached
patch constructs the commit log message by running numerous parallel "svn log"
processes to get the error message, and the gain in speed is enormous.
- It gives a little more feedback to the user. For instance, the actual svn
merge output is shown on the screen while the merge is being done, instead of
all at once at the end.
- I received a bugreport of pipe overflow, which I can't wrap my head against.
Basically, on the GCC repository, when doing merges of old branches, the
initial "svn log --quiet HEAD" call (which is used to find available revisions)
sometimes hit some internal limits (of either Python or kernel, I don't know),
and aborts with an overflow error. Don't ask, but I reproduced it once under
Linux. The suggested fix I was presented changed the os.popen() call with
subprocess (the new Python 2.4 module), and that managed to work around the
problem. I believe that if we read from the pipe incrementally (by also making
good use of the XML format of svn log and thus avoiding parsing errors) the
problem should also be fixed, without having to use the subprocess module.

I have attached a preliminar patch to this mail. Right now, I have just added a
new launchsvn_pipe and used it in a few places. If there's agreement that this
is the way to go, I'll complete the patch.

Comments are welcome!

Giovanni Bajo
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: svnmerge_pipe.patch
Url: /pipermail/svnmerge/attachments/20060517/7a434c08/attachment.txt