Tag Archives: svn

Version control console using ReactJS

Created a web app to generate a report from the version control repository, Apache Subversion™. Similar approach is possible targeting a different repository, like Git.

TL;DR
Someone said a process we follow could not be automated. I took that as a challenge and created a proof of concept (POC) tool.

The final GUI using ReactJS is relatively complex: five data tables that hide/show/expand/collapse. Searches on those tables, sorting, navigation links, help page, Ajax requests to access Subversion repo data, export to CSV, report generation, and client side error handling. It started as just a GUI to a report, but since it was easy, added more features: Zawinski’s law.

To top it off, the app had to automatically invoke the default workflow or no one would use it.

Result?
1. It is a complex disaster that works. And works surprisingly fast. Using ReactJS and Flux made it into a fast elegant (?) disaster that works … kind of.
2. The app served as an example of a SPA in the our dev group. But, mostly to try out the ReactiveJS approach.
3. My gut feel is that there are some fundamental problems in the client side MV* approach which leads to control flow spaghetti (a future blog post).

Since the time I wrote that app I have noticed a mild push back on React that point out the hidden complexities. There are now new ideas and frameworks, like Redux or Cycle.js. Most recently, to tackle the action-coordination, there is much digital ink written on Redux Sagas, for example: “Managing Side Effects In React + Redux Using Sagas“.

Note, though there are critiques of the ReactJS approach or implementation, this does not imply that React was not a real breakthrough in front end development.

Report generation
Creating simple reports from a version control repository can be accomplished with command line tools or querying XML output from SVN log commands. In this case generating the criteria to generate the report was the hard part. Details are not relevant here: this web app would remove a lot of manual bookkeeping tasks that our group currently must follow due to a lot of branch merging and use of reports for error tracking, verification, and traceability. Yup, long ago legacy Standard Operating Procedures (SOP) of an IT shop.

Architecture
Server
A simple Java web app was created and deployed to a Tomcat server. A Java Servlet was used at the server to receive and send JSON data to the browser based client. This server communicates with the version control repository server.

Client
The browser is the client container with ReactJS as the View layer and Flux (implemented in the McFly library) as the framework client implementation. Dojo was used as the JavaScript tools library. Dojo supplied the Promise and other cross-browser capabilities. Why Dojo? That is already in use here. If we were using JQuery, that is what I would use.

Local application service
Performance note: Since the repo query and processing occurs at the server, multiple developers accessing the service would have a performance impact. A future effort is to deploy this as an runnable Jar application (Spring Boot?) that starts an embedded app server, like Tomcat or Jetty, at the developer’s workstation. The browser would still be used as the client.

Repository Query
Some options to generate SVN reports:

1. Use a high level library to access SVN information.
2. Export SVN info to a database, SQL or NoSQL.
3. Use an OS or commercial SVN report generator.
4. Use command line XML output option to create a navigational document object model (DOM)
5. Use SVN command line to capture log output, and apply a pipeline of Linux utilities.

This was a ‘skunkworks’ project to determine if some automation of a manual process could be done and most importantly, if doable, would the resulting tool be used? The first option, seemed easiest, and was chosen. The repo was accessed with the SvnKit Java library. (For Java access to a Git repo, JGit is available).

The process approach was to generate and traverse a Log collection. A simple rule engine was executed (just a series of nested conditionals) to determine what to add from the associated Revision objects.

This seemed like a workable idea until a requirement was requested after the POC was completed: instead of listing a particular source file once per report, show multiple times per each developer who made a commit to it. An easy change if this were implemented as an SVN log query sent to a pipe of scripts. However, with the design this required going into the nuts and bolts of the “rule engine” to add support for filtering, and further changes to the model.

Yup, a POC solution can be a big ball of mud, and unfortunately can be around a long time. Incidentally, this happened with Jenkins CI; where I …

Very recently a flaw in the design will force a revisit of the algorithm again. Instead of making the ‘rule engine’ more powerful, an alternative approach is to start from a Diff collection. The diff result would be used to navigate the Log collection. A similar approach is shown here: http://www.wandisco.com/svnforum/forum/opensource-subversion-forums/general-setup-and-troubleshooting/6238-svn-log-without-mergeinfo-changes?p=36934#post36934

But, already a problem was found with diff output. There is no command line or Java library support for pruning of deleted folders. For example, if a/b/c is a hierarchy, and you delete b, c is also deleted. Now if you generate a diff, the output would contain delete entries for: a/b and a/b/c. What was needed was just a/b. Simple, you say. Sure, but this information is a OOP object graph, so can be complicated.

I solved it: a diff list of 1800 folders was reduced to just 8 folders! I’m surprised a solution or something similar was not found in a web search. I wrote about this in “Can conditional state be used in RxJava Observable streams?

Perhaps revisit the alternative approaches, like export to database? Not sure if this really would have simplified things, but instead just change where the complexity was located. Is the complexity of a software solution a constant?

Other systems take this export approach. One system I saw years ago, exports the version control history (it was CVS) into an external SQL database and then used queries to provide required views.

Client Single-Page Application
What to use as the browser client technology? From past experience, I did not want go down the path of using event handlers all over the place and complex imperative DOM updates.

Anyway, React seemed interesting and had a shorter learning curve. I looked at Angular, but it seemed to be the epitome of embedding the developer into the product (future blog post on the application development self-deception).

A few ReactJS components were created:

  • BranchSelect
  • CommentLines
  • ControlPanel
  • DiffTable
  • ErrorPanel
  • ExclusionRow
  • ExclusionTable
  • FilesRow
  • FilesTable
  • ManifestRow
  • ManifestTable
  • ProgramInfo
  • ProjectPanel
  • RevisionRow
  • RevisionTable
  • ViewController

Lessons Learned
This project progressed very quickly. React seemed very easy. But, that was only temporary. Until you understand a library or a paradigm, start with a smaller application. Really understand it. Of course, these too can fool you. For example, when this app first loads, I had to invoke the most likely use-case. There was a endless challenge of chicken/egg model flow disasters. Solved it, but can’t understand how I did it. Somehow I tamed the React flow callbacks. Or this is just a lull and will blow up in as a result of an unforeseen user interaction.

All the new cutting edge JavaScript front-end frameworks are very very complex for us average developers. Just the tooling is daunting if your coming from a vanilla JavaScript shop. Node dis! See this: ‘I’m a web developer and I’ve been stuck with the simplest app for the last 10 days

Next SPA project?
My next app will probably use Redux as the Flux framework. Or may leave Reactjs and go directly with Cycle.js which is looking very good in terms of ‘principles’ or conceptual flow and is truly Reactive, based on a ReactiveX library: RxJS.

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Prune deleted folders of repo diff report using Java

Sometimes the contents of deleted folders are not relevant. Just showing the top level deleted folder or dir is enough. For example, in a branch diff against trunk, folder moves are shown as adds and deletes. But in a diff report, the whole sub node of the deleted folder is output.

In the particular report I was creating everything under the deleted directories was just noise. How to prune that?

BTW, I searched the SVN docs and found no support for this kind of pruning. I also looked at the Git docs to see if it was different, but also found no mention of this use-case.

Example scenario:

d1 (modified)
├ d2 (deleted)
   ┝ d3 (deleted)

If we create a deleted prune report of the above, it would contain, d1 modified, and d1/d2 deleted. It would not contain d1/d2/d3.

The algorithm in psuedocode

Jan 8, 2015: I found a use-case where this algorithm does not work when used with an actual Subversion repository being accessed by SvnKit. Not due to SVNKit of course. Will update when fixed.

 

boolean deleting := false
String deletePath = “”

for each path{
    if deleting then 
            if path starts with deletePath then
                skip path
            else
                deleting = false
                deletePath = “”
                result.add(path)
            endif       
    else
        if path is directory and it is deleted then
            deleting = true
            deletePath = path
        endif
        
        result.add(path)
        
    endif
}

This can be implemented at the script level, for example, using Groovy or some other language.

SvnKit Scenario

In an Subversion Console application I wrote using ReactJS, I used SvnKit at the server side to interface to the Subversion server. Future blog post will discuss this single-page app. One of the React components was a Diff table.

Using the SvnKit library I generated an svn diff summary list. This is a list of SvnDiffStatus objects: List. Among other things, these object contain a path string of the respective resource.

To prune the excess deleted folders I first attempted to create an Object tree structure, then I gave up on that (too slow), finally a co-worker suggested I just sort the paths and do a simple contains comparison. Wow, how did I miss that? I guess I was sidetracked by “Objects” and didn’t use simple scripting techniques.

So to prune the folders we sort and prune the results of an SvnDiffSummary object, svnDiff::
List list = pruneDeletedFolders(sortDiffList(((Receiver)svnDiff.getReceiver()).getEntries()));

Where Receiver is an implementation of ISvnObjectReceiver.

(The SvnDiff entries may already be sorted, but the SvnKit API docs do not specify this).

Sort the diff list using Guava

	private List<SvnDiffStatus> sortDiffList(List<SvnDiffStatus> list) {
		return new Ordering<SvnDiffStatus>() {
			@Override
			public int compare(SvnDiffStatus left, SvnDiffStatus right) {
				return left.getPath().compareTo(right.getPath());
			}
		}.sortedCopy(list);
	}

prune the list

	private List<SvnDiffStatus> pruneDeletedFolders(List<SvnDiffStatus> list){
		String deletedPath = "";
		boolean isDeleting = false;
		
		List<SvnDiffStatus> prunedList = new ArrayList<>();

		for (Iterator<SvnDiffStatus> iterator = list.iterator(); iterator.hasNext();) {
			SvnDiffStatus diff = iterator.next();
			String path = diff.getPath();
			if (isDeleting) {
				if (path.startsWith(deletedPath)) {
					// skip this diff
				} else {
					isDeleting = false;
					deletedPath = "";
					prunedList.add(diff);
				}
			} else {
				if (isDirectory(diff) && isStatusDeleted(diff)) {
					isDeleting = true;
					deletedPath = path;
				}

				prunedList.add(diff);
			}
		}

		return prunedList;

	}

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Getting “svn info” using Groovy

In a Subversion utility I had to the get the revision number of the HEAD of the current working copy. An answer found gave a Linux command line solution. Here is my Groovy version.

Groovy allows the ability to execute a String. This returns a Process instance, we wait for its termination, and then we query the process for the return code, the output to stderr, and to stdout.

Below I take the output and stick it into a Properties object, then dump the entries. Of course, in practice I will be using the Properties in the solution (should have used a Map).

def command = ['svn','info','-rHEAD']
def proc = command.execute()
proc.waitFor()

int code = proc.exitValue()
if(code != 0){
    println "code: '${code}'"
    println "stderr: ${proc.err.text}"
    return
}

def props = new Properties()

proc.in.eachLine{
    def m = (it =~ /^(.*?):(.*)$/)
    if(m){
        def matches = m[0]
        def key = matches[1].trim()
        def value = matches[2].trim()
        props.put(key,value)
    }else{
        println 'Could not parse the output of svn info'
    }
}

println 'Properties ...'
props.each{
    println "[${it.key}]=[${it.value}]"
}

Example output
Note: The brackets were added to provide a visual test of output, they are not stored. Also, the content was manually changed.

Properties ...
[Last Changed Date]=[201.............pr 2014)]
[Last Changed Rev]=[737]
[Path]=[Widget]
[Repository UUID]=[ec14......b668719305]
[Revision]=[744]
[Relative URL]=[^/widget/branches/skunk]
[Repository Root]=[https://widget/svn/widget]
[Last Changed Author]=[alfred e. newman]
[URL]=[https://widget/svn/widget/branches/skunk]
[Node Kind]=[directory]

Example error output
On error the output would be something like:

code: '1'
stderr: svn: E155007: 'C:\temp' is not a working copy

Via SvnAnt?
An alternative is to use the svnant Ant task. This task has a subcommand ‘WcVersion’. It didn’t work for me.

Ant example is shown below, but you could of use the AntBuilder in Ant.

<target name="svn-info" depends="" description="- svn project info">
	<svn username="${svn.username}" password="${svn.password}">
		<wcversion path=".." prefix="svn" />
	</svn>		
</target>

Update
SvnAnt’s info command can be used to get this information. See http://subclipse.tigris.org/svnant/svntask.html#info

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

New Subversion version will centralize working copy metadata

Subversion version 1.7 removes that gruesome use of those very numerous .svn folders in your working copy. CVS also used that approach, just named .cvs.

This is more like how many ‘newer’ version control systems (VCS), like Git or Mercurial, lay out a repository.

More info: Working Copy Metadata Storage Improvements (client)

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.