Category Archives: Configuration Management

Version control console using ReactJS

Created a web app to generate a report from the version control repository, Apache Subversion™. Similar approach is possible targeting a different repository, like Git.

Someone said a process we follow could not be automated. I took that as a challenge and created a proof of concept (POC) tool.

The final GUI using ReactJS is relatively complex: five data tables that hide/show/expand/collapse. Searches on those tables, sorting, navigation links, help page, Ajax requests to access Subversion repo data, export to CSV, report generation, and client side error handling. It started as just a GUI to a report, but since it was easy, added more features: Zawinski’s law.

To top it off, the app had to automatically invoke the default workflow or no one would use it.

1. It is a complex disaster that works. And works surprisingly fast. Using ReactJS and Flux made it into a fast elegant (?) disaster that works … kind of.
2. The app served as an example of a SPA in the our dev group. But, mostly to try out the ReactiveJS approach.
3. My gut feel is that there are some fundamental problems in the client side MV* approach which leads to control flow spaghetti (a future blog post).

Since the time I wrote that app I have noticed a mild push back on React that point out the hidden complexities. There are now new ideas and frameworks, like Redux or Cycle.js. Most recently, to tackle the action-coordination, there is much digital ink written on Redux Sagas, for example: “Managing Side Effects In React + Redux Using Sagas“.

Note, though there are critiques of the ReactJS approach or implementation, this does not imply that React was not a real breakthrough in front end development.

Report generation
Creating simple reports from a version control repository can be accomplished with command line tools or querying XML output from SVN log commands. In this case generating the criteria to generate the report was the hard part. Details are not relevant here: this web app would remove a lot of manual bookkeeping tasks that our group currently must follow due to a lot of branch merging and use of reports for error tracking, verification, and traceability. Yup, long ago legacy Standard Operating Procedures (SOP) of an IT shop.

A simple Java web app was created and deployed to a Tomcat server. A Java Servlet was used at the server to receive and send JSON data to the browser based client. This server communicates with the version control repository server.

The browser is the client container with ReactJS as the View layer and Flux (implemented in the McFly library) as the framework client implementation. Dojo was used as the JavaScript tools library. Dojo supplied the Promise and other cross-browser capabilities. Why Dojo? That is already in use here. If we were using JQuery, that is what I would use.

Local application service
Performance note: Since the repo query and processing occurs at the server, multiple developers accessing the service would have a performance impact. A future effort is to deploy this as an runnable Jar application (Spring Boot?) that starts an embedded app server, like Tomcat or Jetty, at the developer’s workstation. The browser would still be used as the client.

Repository Query
Some options to generate SVN reports:

1. Use a high level library to access SVN information.
2. Export SVN info to a database, SQL or NoSQL.
3. Use an OS or commercial SVN report generator.
4. Use command line XML output option to create a navigational document object model (DOM)
5. Use SVN command line to capture log output, and apply a pipeline of Linux utilities.

This was a ‘skunkworks’ project to determine if some automation of a manual process could be done and most importantly, if doable, would the resulting tool be used? The first option, seemed easiest, and was chosen. The repo was accessed with the SvnKit Java library. (For Java access to a Git repo, JGit is available).

The process approach was to generate and traverse a Log collection. A simple rule engine was executed (just a series of nested conditionals) to determine what to add from the associated Revision objects.

This seemed like a workable idea until a requirement was requested after the POC was completed: instead of listing a particular source file once per report, show multiple times per each developer who made a commit to it. An easy change if this were implemented as an SVN log query sent to a pipe of scripts. However, with the design this required going into the nuts and bolts of the “rule engine” to add support for filtering, and further changes to the model.

Yup, a POC solution can be a big ball of mud, and unfortunately can be around a long time. Incidentally, this happened with Jenkins CI; where I …

Very recently a flaw in the design will force a revisit of the algorithm again. Instead of making the ‘rule engine’ more powerful, an alternative approach is to start from a Diff collection. The diff result would be used to navigate the Log collection. A similar approach is shown here:

But, already a problem was found with diff output. There is no command line or Java library support for pruning of deleted folders. For example, if a/b/c is a hierarchy, and you delete b, c is also deleted. Now if you generate a diff, the output would contain delete entries for: a/b and a/b/c. What was needed was just a/b. Simple, you say. Sure, but this information is a OOP object graph, so can be complicated.

I solved it: a diff list of 1800 folders was reduced to just 8 folders! I’m surprised a solution or something similar was not found in a web search. I wrote about this in “Can conditional state be used in RxJava Observable streams?

Perhaps revisit the alternative approaches, like export to database? Not sure if this really would have simplified things, but instead just change where the complexity was located. Is the complexity of a software solution a constant?

Other systems take this export approach. One system I saw years ago, exports the version control history (it was CVS) into an external SQL database and then used queries to provide required views.

Client Single-Page Application
What to use as the browser client technology? From past experience, I did not want go down the path of using event handlers all over the place and complex imperative DOM updates.

Anyway, React seemed interesting and had a shorter learning curve. I looked at Angular, but it seemed to be the epitome of embedding the developer into the product (future blog post on the application development self-deception).

A few ReactJS components were created:

  • BranchSelect
  • CommentLines
  • ControlPanel
  • DiffTable
  • ErrorPanel
  • ExclusionRow
  • ExclusionTable
  • FilesRow
  • FilesTable
  • ManifestRow
  • ManifestTable
  • ProgramInfo
  • ProjectPanel
  • RevisionRow
  • RevisionTable
  • ViewController

Lessons Learned
This project progressed very quickly. React seemed very easy. But, that was only temporary. Until you understand a library or a paradigm, start with a smaller application. Really understand it. Of course, these too can fool you. For example, when this app first loads, I had to invoke the most likely use-case. There was a endless challenge of chicken/egg model flow disasters. Solved it, but can’t understand how I did it. Somehow I tamed the React flow callbacks. Or this is just a lull and will blow up in as a result of an unforeseen user interaction.

All the new cutting edge JavaScript front-end frameworks are very very complex for us average developers. Just the tooling is daunting if your coming from a vanilla JavaScript shop. Node dis! See this: ‘I’m a web developer and I’ve been stuck with the simplest app for the last 10 days

Next SPA project?
My next app will probably use Redux as the Flux framework. Or may leave Reactjs and go directly with Cycle.js which is looking very good in terms of ‘principles’ or conceptual flow and is truly Reactive, based on a ReactiveX library: RxJS.


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

“.ignore” files should not be stored in a project’s source

Many version control systems use ignore files in a folder to indicate resources that should not be tracked , the target ignored files. These ignore files could be in any folder in the project where something must be ignored, natch.

These ignore files are a separate concern, not germane to the purpose of the source code. Should we also have files that indicate how to compile? Compile and config concerns are localized, either in the build scripts or in other types of management system. Likewise, the version control metadata should also be separate from the source.

This is comparable to how in the past Subversion version control system had .svn files throughout a project’s source folders. These were used as part of their vcs “bookkeeping”. Currently the “bookkeeping” was centralized into one .svn file at the project’s root folder, like found in Git. (Not an SVN expert so this is just what I see when I use SVN).

A few that use ignore files all over the place are: Git, Mercurial, CVS, and Subversion. Yes, Subversion too, except that it hides these in this nebulous ‘properties’ objects that are not seen but can cause great pain.

To see how to use these, see for example Git’s: gitignore(5) Manual Page

Any issues?
Other than be a context leak, any issues? None directly. Yet, if you search the web, there are many pleas for making automation tools and apps work around the inclusion of these files. Extra steps must be taken to remove from views in IDE or in searches. And so forth.

What should be done?
Ignore files or properties should be handled just like everything else. They should be localized in the version control system’s metadata folder (db or workspace).

For example, with Git, the .git repository could have a folder called “ignores”:


There are probably many reasons for the current approach: performance, simplicity, etc. WTHDIK

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Jenkins CI Server is great

Finally got a Jenkins server installed. Had a host of system issues, like communicating to our source code repo.

Jenkins is a joy to use. Well, it is not perfect, what is? Like, I need to pass the user’s name that invoked a build via Jenkins to the target DOS script (yea, Windows) that eventually invokes the legacy Ant scripts. A quick Google search shows that this is asked in various ways, but no answers. For example, here or here. Hmmmm.

Anyway, now comes a trial use, to see if it is what we really need and can we manage it to do what we will want. With 400 plugins, I don’t see how it could lack. Plus, I’m sure I can use the Groovy plugin to cobble something up. Jenkins even includes a Groovy Console. Finally, there is a road map for possible migration of legacy Ant scripts to Gradle using the Gradle Plugin.

I take back my past snarky comment. Jenkins is not just a pretty face on Cron.

Off Topic
Was watching the Easyb intro video. BDD is interesting. Definitely “should” is a better then “test”. With so many great tools why are products still bug ridden?

Enterprise Level CI Server?
I was told that QuickBuild is better for corporate use.

Wikis and the home link
BTW, is there some Wiki law that says a wiki shall never ever have a link to the parent project? If you get tossed into a wiki by following a link, invariably you will click in agony at links that should go to the real home. Instead, you have to edit the URL in the address bar. Since I never curse, I can’t write “wtf”.

More stuff

  1. Jenkins home page
  2. QuickBuild

  3. Continuous integration Not a very good Wikipedia article
  4. Continuous Integration Much better
  5. Continuous Integration in Agile Software Development
  6. Hooking into the Jenkins(Hudson) API
  7. Five Cool Things You Can Do With Groovy Scripts
  8. Parameterized Builds in Jenkins – choosing subversion folders
  9. Groovy Console
  10. Groovy plugin
  11. Switching to Jenkins–Download and Install Artifact Script for Tester
  12. Gradle Plugin
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Virtual Machine Applicance for development environment

Configuration of a development environment can be very time consuming, error prone, or difficult. This is especially true when investigating or getting up to speed on a new technology or framework. In a corporate environment this is a also a drain on resources and existing developer staff who must take the time to prep a new developer.

One approach to mitigate this is to use a Virtual Appliance.

Virtual appliances are a subset of the broader class of software appliances. Installation of a software appliance on a virtual machine creates a virtual appliance. Like software appliances, virtual appliances are intended to eliminate the installation, configuration and maintenance costs associated with running complex stacks of software.

A virtual appliance is not a complete virtual machine platform, but rather a software image containing a software stack designed to run on a virtual machine platform which may be a Type 1 or Type 2 hypervisor. Like a physical computer, a hypervisor is merely a platform for running an operating system environment and does not provide application software itself. — Virtual Appliance

Creating a Virtual Machine Applicance
The available VM software such as Oracle VirtualBox and the VMware VM have facilities to generate appliances. Thus, when a functioning development environment is created by a lead tech or group, an appliance can be generated for the rest of the team. This appliance can even be provided using a Virtual Desktop Infrastructure (VDI).

Open Virtualization Format
While a VM system can be used to create individual VM instances that can be reused, a more recent technology (supported by some vendors) is the use of OVF:

… is an open standard for packaging and distributing virtual appliances or more generally software to be run in virtual machines.

The standard describes an “open, secure, portable, efficient and extensible format for the packaging and distribution of software to be run in virtual machines”. The OVF standard is not tied to any particular hypervisor or processor architecture. The unit of packaging and distribution is a so called OVF Package which may contain one or more virtual systems each of which can be deployed to a virtual machine.

An OVF package consists of several files, placed in one directory. A one-file alternative is the OVA package, which is a TAR file with the OVF directory inside. —

Using ready made appliances
Each VM vendor can/does make available an appliance marketplace. Thus, one can find ready-made LAMP based environments with a development software stack, for example.

Alternative 1, an installable virtual disk
Where resources are constrained, such as places where developers are still on 3GB of ram and ancient PCs, a Virtual Machine is just not going to cut it.

One easy alternative is to create a dev environment on an installable soft hard drive. TrueCrypt can be used for this purpose. One simply create a true crypt volume, which is just a single file. Then creates the desired dev env in that volume, and that file can now be copied to load into other dev’s workstations as a new hard drive.

TrueCrypt is really for security and privacy concerns, it encrypts data, so may not be ideal for this application. Since TrueCrypt is so useful as a virtual disk, it would be great if it had the option of not encrypting content. But, that would perhaps be outside of its feature space. For that the next alternative is available.

Alternative 2, use VHD files
An alternative is using something directly targeted at virtual disks such as the VHD file format. However, this does not seem to have easily useful public gui or command support (for the end user: developer).

On Windows following the instructions here and using these Send To scripts will allow one to seamlessly use vhd files as mountable hard disk volumes.

Note that Windows 8 will support native mounting of ISO and VHD files.

Further Reading


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Hudson/Jenkins CI Server, can’t edit a job?

I was looking at a possible use of a Continuous Integration Server to quickly set up a build process.

Downloaded the Jenkins war file, put into Tomcat and defined a simple Job that invoked an Ant file to echo “Hello World!”. Cool, that was easy. But, then I wanted to expand that job to do more. Could not find a modify or edit capability. Huh? What’s up with that?

I searched and found very little. There was even some mention of using SED to edit the Job configuration XML, yeeech! Edit using a text tool for a tree-based data structure?

Anyway, not impressed. Of course, this was a quick tryout. Or maybe Linux people are so perfect they never have to edit their work. 🙂

Is Jenkins/Hudson just a pretty face on *nix utilities?

I looked at a few other CI Servers. So far Pulse and Team City look interesting, but they are not free.

Mar 2, 2012: Used one of the latest Jenkins version. Much much better! Though I’m having issues getting Active Directory authentication going. Can log in ok, but then it uses the wrong user “name” that our PCs must use. You know how LDAP has all this distinguished this or that.


  1. Jenkins
  2. Active Directory Plugin
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

JSON configuration file format

JSON is a data interchange format. Should it also be used as a configuration file format, a JSON-CF?

Had to write yet another properties file for configuration info. Started to think that maybe there are better alternatives. Wondered about JSON for this.

What are requirements of a configuration file format?

  • Simple
  • Human readable
  • Cross platform
  • Multi-language support
  • Unicode support

Looks like JSON has all the right qualities.

If all you want to pass around are atomic values or lists or hashes of atomic values, JSON has many of the advantages of XML: it’s straightforwardly usable over the Internet, supports a wide variety of applications, it’s easy to write programs to process JSON, it has few optional features, it’s human-legible and reasonably clear, its design is formal and concise, JSON documents are easy to create, and it uses Unicode.
— Norman Walsh, Deprecating XML

JSON-CF Limitations

  • Instead of angle brackets as in XML, we have quotation marks everywhere.

What does it need?

  • Inline comments, see for example, json-comments
  • Interpolation (property expansion)
  • Namespaces
  • Inheritance
  • Includes
  • Date value
  • Schema
  • Cascading


            "modified":"1 April 2001",
            "dc:author": "John Doe"
             "name":"John Doe",
             "organization":"Acme Widgets Inc."
             "_comment_server":"use IP address in case network name resolution is not working",

Programmatic Access using Groovy

Now we can easily read this file in Java. Using Groovy is much easier, of course. Groovy version 1.8 has built-in JSON support, great blog post on this here.

import groovy.json.*;

def result = new JsonSlurper().
                          parseText(new File("config.json").text)

result.each{ section ->
	println "$sectionn"

>groovy readConfig.groovy
Resulting in Groovy style data structure, GRON, (look ma, no quotation marks):

logger_parser={qualname=compiler.parser, level=DEBUG, propagate=1, handlers=hand01}

owner={organization=Acme Widgets Inc., name=John Doe}

_HEADER={dc:author=John Doe, modified=1 April 2001}

database={port=143, file=payroll.dat, server=, _comment_server=use IP address in case network name resolution is not working}

logger_root={level=NOTSET, handlers=hand01}

In Groovy you can access the data with GPath expressions:
println “server: ” + result.database.server

You can also pretty print JSON, for example:
println JsonOutput.prettyPrint(new File(“config.json”).text)


Raised the question of the use of JSON as a configuration file format.

What I don’t like is the excess quotation marks. YAML is more attractive in this sense. But, the indentation as structure in YAML, similar to Python, may not be wise in configuration files.

Well, what is the answer, should there be a JSON-CF? I don’t know. A very perceptive caution is given by Dare Obasanjo commenting on use of new techniques in general:

So next time you’re evaluating a technology that is being much hyped by the web development blogosphere, take a look to see whether the fundamental assumptions that led to the creation of the technology actually generalize to your use case.


  • After writing and posting this I searched the topic and, of course, this is not a new question. Search. I updated the reading list below with some useful links.
  • JSON Activity Streams are an example of how JSON is used in new ways.
  • types and properties as RDFS in the JSON format:
  • Just learned that node.js uses the NPM package manager which uses a JSON config file format.
  • Jan 7, 2012: Java JSR 353: Java API for JSON Processing

Further Reading
Apache Avro 1.7.7 Specification Schemas
Analysis of JSON use cases compared to XML
JSON Validation Roundup
A very simple data file metaformat
HAL – Hypertext Application Language
NPM configuration file format
XML or YAML for configuration files
Using JSON for Language-independent Configuration Files
INI file
Comparison of data serialization formats
Data File Formats, in Art of Unix Programming, Eric Steven Raymond.
ConfigParser – Work with configuration files
Cascading Configuration Pattern
RFC 4627
XML-SW,a skunkworks project by Tim Bray. Brings a bunch of XML complex together into one spec.
ISO 8601
Learning from our Mistakes: The Failure of OpenID, AtomPub and XML on the Web
Groovy Object Notation (GrON) for Data Interchange
Groovy 1.8 Introduces Groovy to JSON
JSON-LD ‘JSON for Linking Data
JSON Schema


” Sacred Place ” , R.Towner / P. Fresu, live in Innsbruck , Part 4

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Easy stream parsing using Groovy, CVS example

You use every combination of options but that dam command won’t give you what you want?

I faced this last week at work. I had to get a list of my commits to CVS. I tried a bunch of stuff and also searched for a solution. None really worked well. An example of an approach is shown here: “how to search cvs comment history“.

The root problem is that the output of many tools are not always easily reusable. In this situation (and I’m sure in more modern tools like Subversion, Git, or Mercurial) the output resembles (I took out work related info):

RCS file: /cvs/A...
Working file: Java So..
head: 1.1
locks: strict
access list:
keyword substitution: kv
total revisions: 4;     selected revisions: 3
revision 1.1
date: 2011/03/  
filename: Produc...tsA
branches:  1....;
file Produ...
date: 201....
filename: ProductsA....;
date: 2011/0
filename: ProductsA....;
ExampleNightMare - ....

RCS file: /cvs/Am...
Working file: Java S..
head: 1.1
locks: strict
access list:
keyword substitution: kv
total revisions: 4;     selected revisions: 3
revision 1.1
date: 2011/03/  
filename: Pro...

This output goes on for thousands of lines! Sure if you use a tool often and dug into its idioms or have a guru near by, you could probably get what you want, but …. (of topic, but why don’t Man pages and other docs give examples for every option?).

There is no need to take out the dragon book and start writing a parser (is ‘parser’ the correct term in this context?), or even create a DSL. If your very familiar with real scripting languages like Python, Perl, or even pure shell utilities, this is easy. If your not, on Windows (and don’t use Powershell), or just as another approach, Groovy is easy to use.

The usual pattern I would imagine is to just just read the input and trigger on a start phrase to indicate a block of interest, then the data is captured when the including line is subsequently detected in the input stream. However, in my situation depicted above, I did the opposite, I got the data I needed, but only printed it out when I got a subsequent trigger phrase, the commit comment.

Sure you could generalize or find some tool that does this, but you’d probably spend more time learning the tool or creating a reusable system that only you need or understand.


// file: ParseCvsLog_1.groovy
// Author: jbetancourt

def inside = false
def workingFile

new BufferedReader(new InputStreamReader({ s ->
	if(s.startsWith("Working file:")){
		inside = true
		workingFile = s.split("Working file:")[1] // got what I want?
	// this indicates that it is.
	def found = s ==~ /.*ExampleNightMare.*/
		println(workingFile)   // send to next pipe
		inside = false

Probably not a good example of idiomatic Groovy code, but easy to follow. A Groovy expert could probably do it on one line (I don’t like those smarty one-liners; one week later, you don’t know what you did.).

This is used as (all one line):

cvs inscrutable bunch of gibberish that doesn't answer question | groovy ParseCvsLog_1.groovy > myChanges.txt

Nothing new in this post, of course. The value of any scripting approach is that it is infinitely adaptable. And, when the scripting language is easy to use, the results could even be reusable. Perl, Python, and Ruby, for example, have great facilities for sharing of snippets and modular code solutions. Groovy and other JVM based languages like Scala are beginning to add this capability to Java environments.


  • 20110323T1906-5: Cleaned up the sample code a little; don’t want to give the wrong impression.
  • 20110402T1702-5: While looking thru the book “Groovy In Action” noticed that section 13.5.3 Inspecting version control, deals with this subject.

Further Reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Cascading Configuration Pattern


Many existing systems load configuration in cascade to reduce the use of duplicate properties and allow fine grained customization. This post expresses this common usage into a design pattern.

Keywords:   SCM, CM, Properties, Groovy, Config, CSS

Content:  Context, Forces, Solution, Consequences, Implementation, Code Example, Related Patterns, Related Intellectual Property, Further Reading


A business system uses property (configuration) files to configure particular environments or subsystems.   Many environments share the same properties and values, however, some are different and crucial.  To avoid missing any properties, all the properties are duplicated in each file, and required differences are changed appropriately. 

For example, if there are 100 properties required and there are 23 possible environments, that is 2300 lines of source to manage.  If there are any duplicated properties that do not vary between environments, then there is an opportunity to simplify the configuration system.  In this case, if we make only a “root” file have the full set of properties,  the total size is given by: 

T = L+npL; 
    T is total size, 
    n is number of files, 
    p is percent of each file that is overridden.
    L is number of properties  

Here the value is 560=100+23*.2*100). The reduction over the duplicated data is 75%.


There are many issues with having the same properties in multiple locations.  One obvious disadvantage, is adding a new property would require changing multiple files.  Also the files are larger and not really cohesive.   Tracking errors is also complicated, especially run-time errors due to configuration settings.  Any solution should not add its own complexity.  Thus, we should not require new types of configuration files and requirements.

Even when configuration could be isolated or systems modularized, there may be a need to have duplicated properties or reassignment to satisfy software development life cycle (SDLC).  A system will be different in dev, test, and production environments.


A hierarchy of property sources and a system that can load each source and override properties at a prior level will provide a partial solution.  This is the approach already taken in many systems.  For example, in software applications and servers, the final properties used are composed of those taken from various standard locations in a particular Operating System host.  In Windows for example, the Environment Variables are composed of those found in System and User name space.  In software tools, such as Mercurial DVCS, there is a command to show the results of the final configuration after all configuration sources are loaded: showconfig   show combined config settings from all hgrc files. In Git, one executes: git config -l.

Many build systems such as Ant and Maven, allow and expect the cascading of property files.  Note that in Ant, the inverse policy is used, the first property assignment wins.

And, of course, this cascading is seen in the Cascading Style Sheet (CSS) technology.


There is now the requirement that the cascade order is well known, non-variable, and robust.  Any intervening file must be loaded, else the system may destabilize.  Adding a new property or changing a property will require extra care since an environment or subsystem may be effected if the file contents are not properly designed beforehand.

To provide management and debugging support the implementation should provide traceability of configuration.  Thus, during use, the system must report (log) what was overridden, not changed, what is missing, etc.


Using this pattern is very easy.  One just has to determine how the system handles reassignment in the configuration loader subsystem.  If it allows resetting then the file hierarchy is from global to specific.

Where complexity may come is when one wants to satisfy more sophisticated requirements.  For example, a property may need to be appended.  In OS paths, for instance, paths are combined to create the final path.  Thus, the operations on properties could be a combination of:

  • create: initial assignment of a property
  • update:  a new value is assigned
  • delete:  the property is removed
  • append: append to existing value
  • merge:  use a pattern to merge a new value
  • locked: allow no change
  • fail on replace:  if a value already assigned throw an exception.

Code Example

In Java the java.util.Properties class will overwrite an existing property with the same key value.  Thus, to cascade properties, one just reuses the same instantiated Properties Object via the various load(…) methods.

Note that in some frameworks, like Ant, the reverse is true, properties do not get overwritten. So, the “root” properties must be loaded last in a cascade sequence.

Shown in listing one is a simple implementation written in Groovy, a dynamic language on the JVM.  The PropCascade class extends Properties and adds methods to load a list of sources.  The first source in the list is the root of the “cascade”.  In addition, a few methods explicitly specify the root source. From a traceability point of view, reusing java.util.Properties class may not be optimal. It gives no access to low level capture of the actual “put” action. Even with the use of Aspect Oriented Programming, with for example the AspectJ language, there is no joinpoint available to add it.

listing 1

 * File: CascadedProperties.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt

import java.util.List;
import java.util.Properties;

 * An extension of Properties that adds Convenience
 * methods to load lists of sources.
 * @author jbetancourt
class CascadedProperties extends Properties {
	//private Properties rootProperties = new Properties();
	//private boolean firstWins = true;
	//private boolean failOnDuplicate = false;
	//private boolean isTrace = false;

	 * Load a list of properties sources.
	 * @param list
	public void load(List list) throws IOException, IllegalArgumentException {
		list.each {

	 * Explicit file path is specified.
	 * @param path
	public void load(String path) throws IOException, IllegalArgumentException {
		load(new File(path).newInputStream());

	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 * @param inStream
	 * @param list
	public void load(InputStream inStream, List list) throws IOException, IllegalArgumentException {
		load inStream
		load list

	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 * @param reader
	 * @param list
	public void load(Reader reader, List list) throws IOException, IllegalArgumentException {
		load reader
		load list

	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 * @param path
	 * @param list
	public void load(String path, List list) throws IOException, IllegalArgumentException {
		load path
		load list

} // end of CascadedProperties

In listing two, the JUnit test class is shown.
listing 2

 * File: CascadedPropertiesTest.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt

import java.util.List;

import org.junit.Before;
import org.junit.Test;

import groovy.util.GroovyTestCase;

 * Test the {@link CascadedProperties} class.
class CascadedPropertiesTest extends GroovyTestCase{
	private CascadedProperties cp;

	/** excuted before each test method run */
	public void setUp() throws Exception {
		cp = new CascadedProperties();

	public void testloadListPaths() throws Exception {
		List list = new ArrayList();
		list.add path1
		list.add path2



	public void testloadListReaders() throws Exception {
		List list = new ArrayList();
		list.add reader1
		list.add reader2



	public void testloadListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream1
		list.add iStream2



	public void testloadStreamAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3



	public void testloadPathAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3



	public void testloadReaderAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add reader2
		list.add reader3



	public void testPutAgain() {
		cp.put("k1", "v1")
		cp.put("k1", "v2")
		assertEquals(cp.get("k1"), "v2");

	public void testLoadOneFilePath() throws Exception {

	public void testLoadTwoFiles() throws Exception {


	// class fields
	String path1 = ""
	String path2 = ""
	String path3 = ""
	File file1 = new File(path1)
	File file2 = new File(path2)
	File file3 = new File(path3)

	InputStream iStream1 = file1.newInputStream()
	InputStream iStream2 = file2.newInputStream()
	InputStream iStream3 = file3.newInputStream()
	Reader reader1 = file1.newReader()
	Reader reader2 = file2.newReader()
	Reader reader3 = file3.newReader()


Property Files Analysis

It is very difficult to eyeball a set of property files to see if there are many duplicates. I guess one could do some magic one line script that concatenates, sorts, prunes, etc. Here is an alternative using the Groovy language.
listing 3

 * Script: AnalyizePropertyFiles.groovy
 * Author: J. Betancourt
import java.util.Hashtable;
import java.util.Map.Entry;

 * @author jbetancourt
class AnalyizePropertyFiles {
	// these could have been created dynamically by reading the target folder.  But, that may pick up non-used files.
	static fileNames = []

	// where are the files?
	static basePath = ""

	// run analysis....
	static main(args) {
		def propList = [] // each property file will be load into a cell which contains a properties object.

		// put each file into properties object in propList.
		fileNames.each {
			def p = new Properties()
			p.load(new File(basePath + "" + it).newInputStream())

		// put all properties in one allProps object.
		Properties allProps = new Properties();
		propList.each {
		}// each propList
		def result = []
		int totalCount = 0
		// get how many times each property is used in all properties object
		allProps.each { prop ->
			String key = prop.key
			String value = prop.value
			def values = []
			int count = 0
			propList.each { p ->
					def curValue = p.get(key)
			} // each property file
			StringBuilder vb = new StringBuilder()
			values.each{ s ->
				vb.append("[" + s + "], ")
			int numberValues = values.size()

			StringBuilder sb = new StringBuilder()			
			sb.append(count).append(",").append(values.size).append(", ").append(key).append(", ").append(vb.toString())
			totalCount += count				
		} // each allProps property
		println("Total number of times properties are repeated: " + totalCount)
	} // end main

} // end class

Related Patterns

Related Intellectual Property

Cascading configuration using one or more configuration trees“, U.S. Patent number 7760746, 30Nov2004,

Further Reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.