Tag Archives: groovy

Cascading Configuration Pattern

Synopsis

Many existing systems load configuration in cascade to reduce the use of duplicate properties and allow fine grained customization. This post expresses this common usage into a design pattern.

Keywords:   SCM, CM, Properties, Groovy, Config, CSS

Content:  Context, Forces, Solution, Consequences, Implementation, Code Example, Related Patterns, Related Intellectual Property, Further Reading

Context

A business system uses property (configuration) files to configure particular environments or subsystems.   Many environments share the same properties and values, however, some are different and crucial.  To avoid missing any properties, all the properties are duplicated in each file, and required differences are changed appropriately. 

For example, if there are 100 properties required and there are 23 possible environments, that is 2300 lines of source to manage.  If there are any duplicated properties that do not vary between environments, then there is an opportunity to simplify the configuration system.  In this case, if we make only a “root” file have the full set of properties,  the total size is given by: 

T = L+npL; 
where 
    T is total size, 
    n is number of files, 
    p is percent of each file that is overridden.
    L is number of properties  

Here the value is 560=100+23*.2*100). The reduction over the duplicated data is 75%.

Forces

There are many issues with having the same properties in multiple locations.  One obvious disadvantage, is adding a new property would require changing multiple files.  Also the files are larger and not really cohesive.   Tracking errors is also complicated, especially run-time errors due to configuration settings.  Any solution should not add its own complexity.  Thus, we should not require new types of configuration files and requirements.

Even when configuration could be isolated or systems modularized, there may be a need to have duplicated properties or reassignment to satisfy software development life cycle (SDLC).  A system will be different in dev, test, and production environments.

Solution

A hierarchy of property sources and a system that can load each source and override properties at a prior level will provide a partial solution.  This is the approach already taken in many systems.  For example, in software applications and servers, the final properties used are composed of those taken from various standard locations in a particular Operating System host.  In Windows for example, the Environment Variables are composed of those found in System and User name space.  In software tools, such as Mercurial DVCS, there is a command to show the results of the final configuration after all configuration sources are loaded: showconfig   show combined config settings from all hgrc files. In Git, one executes: git config -l.

Many build systems such as Ant and Maven, allow and expect the cascading of property files.  Note that in Ant, the inverse policy is used, the first property assignment wins.

And, of course, this cascading is seen in the Cascading Style Sheet (CSS) technology.

Consequences

There is now the requirement that the cascade order is well known, non-variable, and robust.  Any intervening file must be loaded, else the system may destabilize.  Adding a new property or changing a property will require extra care since an environment or subsystem may be effected if the file contents are not properly designed beforehand.

To provide management and debugging support the implementation should provide traceability of configuration.  Thus, during use, the system must report (log) what was overridden, not changed, what is missing, etc.

Implementation

Using this pattern is very easy.  One just has to determine how the system handles reassignment in the configuration loader subsystem.  If it allows resetting then the file hierarchy is from global to specific.

Where complexity may come is when one wants to satisfy more sophisticated requirements.  For example, a property may need to be appended.  In OS paths, for instance, paths are combined to create the final path.  Thus, the operations on properties could be a combination of:

  • create: initial assignment of a property
  • update:  a new value is assigned
  • delete:  the property is removed
  • append: append to existing value
  • merge:  use a pattern to merge a new value
  • locked: allow no change
  • fail on replace:  if a value already assigned throw an exception.

Code Example

In Java the java.util.Properties class will overwrite an existing property with the same key value.  Thus, to cascade properties, one just reuses the same instantiated Properties Object via the various load(…) methods.

Note that in some frameworks, like Ant, the reverse is true, properties do not get overwritten. So, the “root” properties must be loaded last in a cascade sequence.

Shown in listing one is a simple implementation written in Groovy, a dynamic language on the JVM.  The PropCascade class extends Properties and adds methods to load a list of sources.  The first source in the list is the root of the “cascade”.  In addition, a few methods explicitly specify the root source. From a traceability point of view, reusing java.util.Properties class may not be optimal. It gives no access to low level capture of the actual “put” action. Even with the use of Aspect Oriented Programming, with for example the AspectJ language, there is no joinpoint available to add it.

listing 1

/**
 * File: CascadedProperties.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt
 */

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.Properties;

/**
 * An extension of Properties that adds Convenience
 * methods to load lists of sources.
 *
 * @author jbetancourt
 */
class CascadedProperties extends Properties {
	//private Properties rootProperties = new Properties();
	//private boolean firstWins = true;
	//private boolean failOnDuplicate = false;
	//private boolean isTrace = false;

	/**
	 * Load a list of properties sources.
	 * @param list
	 */
	public void load(List list) throws IOException, IllegalArgumentException {
		list.each {
			load(it)
		}
	}

	/**
	 * Explicit file path is specified.
	 * @param path
	 */
	public void load(String path) throws IOException, IllegalArgumentException {
		load(new File(path).newInputStream());
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param inStream
	 * @param list
	 */
	public void load(InputStream inStream, List list) throws IOException, IllegalArgumentException {
		load inStream
		load list
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param reader
	 * @param list
	 */
	public void load(Reader reader, List list) throws IOException, IllegalArgumentException {
		load reader
		load list
	}

	/**
	 * A load method that explicitly specifies the "default" source in
	 * the cascade order.
	 *
	 * @param path
	 * @param list
	 */
	public void load(String path, List list) throws IOException, IllegalArgumentException {
		load path
		load list
	}

} // end of CascadedProperties

In listing two, the JUnit test class is shown.
listing 2

/**
 * File: CascadedPropertiesTest.groovy
 * Date: 23OCT10T18:13-05
 * Author: JBetancourt
 */

import java.io.File;
import java.io.FileInputStream;
import java.util.List;

import org.junit.Before;
import org.junit.Test;

import groovy.util.GroovyTestCase;

/**
 * Test the {@link CascadedProperties} class.
 */
class CascadedPropertiesTest extends GroovyTestCase{
	private CascadedProperties cp;

	/** excuted before each test method run */
	public void setUp() throws Exception {
		cp = new CascadedProperties();
	}

	public void testloadListPaths() throws Exception {
		List list = new ArrayList();
		list.add path1
		list.add path2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadListReaders() throws Exception {
		List list = new ArrayList();
		list.add reader1
		list.add reader2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream1
		list.add iStream2

		cp.load(list);

		assertEquals("v2",cp.get("k1"));
	}

	public void testloadStreamAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3

		cp.load(iStream1,list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testloadPathAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add iStream2
		list.add iStream3

		cp.load("datafile1.properties",list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testloadReaderAndListStreams() throws Exception {
		List list = new ArrayList();
		list.add reader2
		list.add reader3

		cp.load(reader1,list);

		assertEquals("v3",cp.get("k1"));
	}

	public void testPutAgain() {
		cp.put("k1", "v1")
		cp.put("k1", "v2")
		assertEquals(cp.get("k1"), "v2");
	}

	public void testLoadOneFilePath() throws Exception {
		cp.load("datafile1.properties");
		assertEquals("v1",cp.get("k1"));
	}

	public void testLoadTwoFiles() throws Exception {
		cp.load(iStream1)
		cp.load(iStream2)

		assertEquals("v2",cp.get("k1"));
	}

	//
	// class fields
	//
	String path1 = "datafile1.properties"
	String path2 = "datafile2.properties"
	String path3 = "datafile3.properties"
	File file1 = new File(path1)
	File file2 = new File(path2)
	File file3 = new File(path3)

	InputStream iStream1 = file1.newInputStream()
	InputStream iStream2 = file2.newInputStream()
	InputStream iStream3 = file3.newInputStream()
	Reader reader1 = file1.newReader()
	Reader reader2 = file2.newReader()
	Reader reader3 = file3.newReader()

}

Property Files Analysis

It is very difficult to eyeball a set of property files to see if there are many duplicates. I guess one could do some magic one line script that concatenates, sorts, prunes, etc. Here is an alternative using the Groovy language.
listing 3

/*
 * Script: AnalyizePropertyFiles.groovy
 * Author: J. Betancourt
 */
import java.util.Hashtable;
import java.util.Map.Entry;

/**
 * @author jbetancourt
 */
class AnalyizePropertyFiles {
	// these could have been created dynamically by reading the target folder.  But, that may pick up non-used files.
	static fileNames = []

	// where are the files?
	static basePath = ""

	// run analysis....
	static main(args) {
		def propList = [] // each property file will be load into a cell which contains a properties object.

		// put each file into properties object in propList.
		fileNames.each {
			def p = new Properties()
			p.load(new File(basePath + "" + it).newInputStream())
			propList.add(p)
		}

		// put all properties in one allProps object.
		Properties allProps = new Properties();
		propList.each {
			allProps.putAll(it)
		}// each propList
		
		def result = []
		
		int totalCount = 0
		
		// get how many times each property is used in all properties object
		allProps.each { prop ->
			String key = prop.key
			String value = prop.value
			def values = []
			int count = 0
			propList.each { p ->
				if(p.containsKey(key)){
					count++
					def curValue = p.get(key)
					if(!values.contains(curValue)){
						values.add(curValue)
					}								
				}
			} // each property file
				
			StringBuilder vb = new StringBuilder()
			values.each{ s ->
				vb.append("[" + s + "], ")
			}
			
			int numberValues = values.size()

			StringBuilder sb = new StringBuilder()			
			sb.append(count).append(",").append(values.size).append(", ").append(key).append(", ").append(vb.toString())
			result.add(sb.toString())	
			
			totalCount += count				
			
		} // each allProps property
		
		
		result.each{
			println(it)
		}
		
		println("Total number of times properties are repeated: " + totalCount)
		
	} // end main

} // end class

Related Patterns

Related Intellectual Property

Cascading configuration using one or more configuration trees“, U.S. Patent number 7760746, 30Nov2004, http://patft.uspto.gov/netacgi/nph-Parser?Sect2=PTO1&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&d=PALL&RefSrch=yes&Query=PN%2F7760746


Further Reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Why a Repository for Java Dev?

Excellent article on why a Repository Manager is crucial for software development process.  Makes the case that not using a Repository Manager is the cause of many anti-patterns. It could be that the Repository is the next essential besides the VCS in development best practices.

The article is using Maven as a case in point and also it is selling a product (nothing wrong with that) so perhaps one could be a little wary. However there are other dependency managers like Ivy which are used by other Build systems like Gradle available.

I have seen places that do not use a Software Configuration Managment (SCM) Version Control System (VCS).  And, then there are places that use a VCS incorrectly; as this article points out the VCS becomes an ad hoc file store for everything.   I remember one place where our partner gave us access to their SCM to download a project source, and we got everything!  They had application executables, utilities, documents, binaries and other things like their Office apps and other tool chains, which had nothing to do with the project we wanted the source of.  Someone must have accidentally  imported their whole PC into CVS, yikes.

Enter the Repository which, I believe, first became “popular” with the introduction of Maven.  When I tried to introduce use of an internal Repository into a former company I got push back:  “It’s very easy to just put one’s jars and dependent binaries into version control” or  “Who needs that Repository play toy stuff!”  Oh well. In that situation, it was probably the best decision, there is initial complexity in adopting any tool that aims to reduce complexity.

Is just using Mavan or Gradle with an internal Repository as proxy to external ones enough Repository Management (which Sonatype calls stage one: Proxying Remote Repositories) or does one have to use a full blown Repository Manager subsystem? Why does using the internal Repository for one’s own output destination require a Repository Manager (which Sonatype calls ‘stage two’)? The Maven site has this to say:

Aside from the benefits of mediating access to remote repositories, a repository manager also provides something essential to full adoption of Maven. Unless you expect every member of your organization to download and build every single internal project, you will want to provide a mechanism for developers and departments to share both SNAPSHOT and releases for internal project artifacts. A Maven repository manager provides your organization with such a deployment target. Once you install a Maven repository manager, you can start using Maven to deploy snapshots and releases to a custom repository managed by the repository manager. Over time, this central deployment point for internal projects becomes the fabric for collaboration between different development teams. — Repository Management with Maven Repository Managers.

If you don’t think this is important you probably have not been on a project where disasters like xxx.jar was sent to a customer and we don’t know what version it is and who made it. You know, using version numbers as part of binary files would defeat the purpose of using a VCS no?

On a side note: Why did the Java development community develop its own repository system when there were plenty out there such as the application-level package systems used by the Linux community?

Links

Maven Repository Managers for the Enterprise

Why Do I Need a Repository Manager?: link

Maven Repository Manager Feature Matrix: link

Archiva

Artifactory

Nexus

Gradle:  http://www.gradle.org/

Ivy:  http://ant.apache.org/ivy/

Manage dependencies with Ivy

Maven:  http://maven.apache.org/

Ant:  http://ant.apache.org/

Continuous Integration:  http://en.wikipedia.org/wiki/Continuous_integration


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Concurrent Code Using Groovy

 
Summary

Presented are a few ways of coding a concurrent program using Java threading and also using a Communicating Sequential Processes (CSP) library.  They are written in the Groovy language. Used as an example is a simple lottery number generation application.

[intro],  [problem],  [Listings],  [simple thread],  [object lock],  
[executor],  [Fork/Join],  [jCSP],  [Actor],  [Declaritive CSP using Spring],  [Links],   [bottom of page]

Intro

In a prior blog post (Java Plain Old Concurrent Object) I wrote about a possible need for a higher level concurrency support in Java.   Here I give the program code that I used in September 2007 to learn more then the rudimentary Java concurrency concepts.

I managed to use CSP and the usual Java concurrent library support, and recently was about to try Actors using the new GPars library.  It was a good way to get a feel for each approach.  Note that it was not enough to become a concurrency expert; that was not a goal; still studying Goetz’s book Java Concurrency In Practice.

My first use of multitasking code was in C++ code I did in 1992:

… technical hurdles with the product, so I put it aside until I could get back to it. One thing that I was proud of was learning a little bit of Fuzzy Logic and using it as the controller. I even wrote a graphical simulator in the C++ language; threading was fun stuff. Watching the fuzzy sets behave like analog surfaces or neural EEG waves gave me the idea for the biomimicry aspects. — An Adaptive Controller Using Fuzzy Memory

Problem

For code example I used lottery number play generator.  Powerball is a U.S. multi-state lottery game.  A Powerball play is composed of 5 white balls in range 1 – 55 and one red ball in the range of 1 – 42.  As you can imagine the odds are extremely high that you could guess the Jackpot set (1 in 195,249,054).  The jackpot gets very large and that is a great enticement to forgo any sanity and “you can’t win if you don’t play” you know. 

If you enjoy the frills, ambiance, and clientele, gambling is a great way to throw away money.   For more on Lottery see: Lottery Links
back to top

Source Code

To simulate real random number generation, the program will continuously generate plays, but only produce the requested numbers of generated sets when the user clicks the enter key.  I’m assuming a user waiting a certain amount of time adds a random element to the PRNG being used.  Does it?  I don’t think so, but that’s another topic.

A few points:

  • The problem could have been solved without using concurrent approach.
  • The code could be optimized and made more Groovy.
  • I used the Eclipse IDE and the Groovy Eclipse Plugin.
  • Example code use at your own risk.  No correctness or quality guaranteed.

A Thread monitor view using VisualVM doesn’t show much, just six threads with four daemon ones.

We use the following batch file, which simply invokes the Main.groovy script.

@rem File: run.cmd
@rem Created 28 Dec 2008 J. Betancourt
call setEnv.cmd
groovy ..\src\main\groovy\net\coxmembers\jbetancourt1\Main.groovy %*

The main script driver is shown in listing 1.  Two arguments are specified, how many dollars to lose and what type of concurrency approach to use.  Executing it with no arguments would give:

c:Users\jbetancourt\Documents\projects\dev\ConcurrentGroovy-1bin>run
usage: usage -t type -d dollars [-h]
 -d,--dollars    How many dollars to lose
 -h,--help       Usage information
 -t,--type       Type of threading

Options for '-t' are:
 "simple"    ----> "Using simple Threads"
 "lock"      ----> "Using lock"
 "executor"  ----> "Using Futures"
 "csp"       ----> "Using JCSP"
 "actor"     ----> "Using Actors"
Example: run -d 5 -t simple

And, here is a sample run:

c:Users\jbetancourt\Documents\projects\dev\ConcurrentGroovy-1bin>run -d 5 -t simple
Setting up games 5 using "simple"
The odds of winning jackpot are: 1 in 146,107,962.00
Enter return key

Number of games generated are: 5
 1:   7 11 14 23 34 PB:  9
 2:  21 22 27 28 55 PB: 21
 3:   2  9 30 37 46 PB: 30
 4:   7 49 50 51 53 PB: 30
 5:   9 18 24 40 42 PB: 32
Good luck!

Success?

Not really.   A funny thing about this code, if you invoke it and quickly hit the return key, it doesn’t work correctly.  Concurrent code can be hard to craft.
Example:

>run -d 5 -t executor
Setting up games 5 using "executor"
Press the Enter key on keyboard

Number of games generated are: 1
 1:  18 20 22 27 41 PB: 25
Good luck!

Listing 1 Main driver class:

package net.cox.members.jbetancourt1;

import org.apache.commons.collections.buffer.CircularFifoBuffer
import org.apache.commons.collections.Buffer
import java.security.SecureRandom
import org.apache.commons.collections.BufferUtils

/**
 * Generate Powerball plays.
 * Required libraries:
 * - Commons CLI
 * - Commons collection
 *
 * A Powerball play is composed of 5 white balls in range 1 - 55
 * and one red ball in the range of 1 - 42
 * This script gets how many games to play, then
 * continually generates this number of games
 * and adds them into a circular buffer.  This is
 * repeated until the user hits a key, which
 * interrupts the generation thread and dumps the results.
 *
 * A circular buffer is used since the results are
 * only needed when the user hits a key,
 * thereby making it "random".  In actuality,
 * since we are using a psuedo-random
 * generator, this is still not true random.
 * There is no way to improve the odds of
 * picking the correct numbers for a lottery game.
 */
 class Main{
 def apps = [
    "simple": [new GuessSimple(),"Using simple Threads"],
    "lock":   [new GuessWithLock(),"Using lock"],
    "executor": [new GuessWithExecutor(),"Using Futures"],
    "csp": [new GuessCSP(),"Using JCSP"],
    "actor" : [new GuessActor(),"Using Actors"]
 ]

 /** Main entry point */
 static main(args){
    def main = new Main()

    def options = main.parseArguments(args)
    if((options != null ) && options.d && options.t){
        println "Setting up game" + (options.d>0? "s": "") +
        " ${options.d} using "${options.t}""
        def dollars = Integer.parseInt(options.d)
        def games = BufferUtils.synchronizedBuffer(
           new CircularFifoBuffer(dollars))

           def threadType = ""
           if(options.t){
               threadType = options.t
           }

           def guess = main.apps[threadType][0]
                guess.generate(dollars,games)
    }

    System.exit(0)
 } // end main

 /****************************************************************************
 * Using CLI builder, parse the command line.
 */
 def OptionAccessor parseArguments(String[] args){
    def cli = new CliBuilder(usage: 'usage -t type -d dollars [-h]')
    cli.h(longOpt: 'help', 'Usage information')
    cli.t(longOpt: 'type', 'Type of threading',args:1)
    cli.d(longOpt: 'dollars','How many dollars to lose',args:1,required:false)
    def options = cli.parse(args)

    if(options == null || options.getOptions().size() == 0 || options.h){
         cli.usage()
         println "nOptions for '-t' are: "
                    apps.each{
         println("t"${it.key}"".
              padRight(12) + " ----> "${it.value[1]}"")
    }
    println "nExample: run -d 5 -t simple"
    return options
  }

  return options
 }
} // end class Main

Listing 2 Base class:

<pre>package net.cox.members.jbetancourt1;

import org.apache.commons.collections.buffer.CircularFifoBuffer
import org.apache.commons.collections.Buffer
import java.security.SecureRandom
import org.apache.commons.collections.BufferUtils

/**
 *
 */
class GuessBase{
 def whiteBalls = [0]
 def redBalls = [0]
 def random = new SecureRandom()
 //each element is an array of 0-4 white ball values and 5 is the red ball value.
 def games //= BufferUtils.synchronizedBuffer(new CircularFifoBuffer(dollars))

 def createBallSets(){
      def i = 0
      for(j in 1..55){
        whiteBalls[i++] = j
      }
      i = 0
      for(j in 1..42){
         redBalls[i++] = j
      }
 }

 /** Pick the set of white balls */
 def pickWhite(){
     def array = []
     array.addAll(whiteBalls)
     Collections.shuffle(array,random)
     return array[0..4]
 }

 /** swap two elements in a list */
 def swapElements(list,i,j){
      def temp = list[i]
      list[i] = list[j]
      list[j] = temp
 }

 /**  Pick the single red 'powerball'. */
 def pickRed(){
      int offset = random.nextInt(redBalls.size())
      return redBalls[offset]
 }

 def showGames(){
     def gamesSorted = games.sort(){a,b -> a[5] <=> b[5]}
     def rownum = 1
     println "Number of games generated are: ${games.size()}"
     for (loop in gamesSorted){
       print ((rownum++ + ": ").padLeft(8))
       for(x in (  loop[0..4]).sort()  ){
         print ((x.toString()).padLeft(3))
       }
       println " PB:" + ((loop[5]).toString()).padLeft(3)
     }
     println "Good luck!"
 }

} // end class GuessBase

Listing 3 Using simple Java Thread class:

package net.cox.members.jbetancourt1;

/**
 * Example that uses a simple 'inline' thread.
 */
class GuessSimple extends GuessBase{

 /** create d guesses and store into games array */
 def synchronized generate(d,games){
   this.games = games
   createBallSets()

   def worker = new Thread();
   worker.setDaemon(true);

   worker.start{ // Groovy allows closure here.
     while(!Thread.currentThread().isInterrupted()){
       def w = pickWhite()
       def r = pickRed()
       games.add((Object)(w+r))
       random.setSeed(System.currentTimeMillis());
     }
   }

  println "The odds of winning jackpot are: 1 in 146,107,962.00nEnter return key"
  new InputStreamReader(System.in).readLine()

  worker.interrupt()
  Thread.sleep(1000) // required!
  showGames()
 }
} // End of GuessSimple.groovy

Listing 4 Using Object lock:


package net.cox.members.jbetancourt1;

/**
 *
 */
class GuessWithLock extends GuessBase{
  def lock = new Object() // for thread synchronization
  /** create d guesses and store into games array */
  def generate(d,games){
  this.games = games
  createBallSets()

  def worker = new Thread(){
     void run(){
       while(!Thread.currentThread().isInterrupted()){
         def w = pickWhite()
         def r = pickRed()
         games.add((Object)(w+r))
         random.setSeed(System.currentTimeMillis());
       }
       synchronized(lock){
          lock.notify()
       }
     }
 }

 worker.start()

 println "The odds of winning jackpot are: 1 in 146,107,962.00"
 println "Enter any key<cr>"
 new InputStreamReader(System.in).readLine()
 worker.interrupt()
 showGames()
 }
}

Listing 5 Using Executor:

/**
* File: GuessWithExecutor.groovy
* Author: Josef Betancourt
* Date:  9/1/2007
*
*/
package net.cox.members.jbetancourt1;

import org.apache.commons.collections.*
import java.util.concurrent.*

/**
 * Generate powerball plays using Executor.
 */
 class GuessWithExecutor extends GuessBase {

 /** create d guesses and store into games array */
 def generate(d,games){
       def dollars = d
       try{
         def exec = Executors.newSingleThreadExecutor()
         def gen = new GameGenerator<Buffer>(dollars,games)
         gen.createBallSets()
         def future = exec.submit(gen)
         println "Press the Enter key on keyboard"
         new InputStreamReader(System.in).readLine()
         exec.shutdownNow()
         waitForTermination(exec,100,10) // 100ms, max of 10 tries

         try{
           def buffer = future.get()
           def exec2 = Executors.newSingleThreadExecutor()
           exec2.execute(new GenerateReport(buffer))
           exec2.shutdown()
         }catch(ExecutionException ignore){
            ignore.printStackTrace()
         }
       }catch(Exception ex){
           ex.printStackTrace()
       }
 }

 /**
 *
 * @param exec the Executor
 * @param time how long to wait
 * @param unit unit of time
 * @param tries how many times to wait
 */
 def waitForTermination(exec,time,tries) throws Exception {
    def count = 0
    while(!exec.awaitTermination(time, TimeUnit.MILLISECONDS)){
        if(count++ >= tries){
           break
        }
    }
 }

 } // end class GuessWithExecutor

 /**
 * A Callable that generates games until interrupted.
 */
 def class GameGenerator extends GuessBase implements Callable {
       def GameGenerator(dollars,games){
                this.games = games
       }

      /**
       * Create d guesses and store into games array.
       * @see java.util.concurrent.Callable#call()
       */
     public Object call() throws Exception {
         while(!Thread.currentThread().isInterrupted()){
             def w = pickWhite()
            def r = pickRed()
            games.add((Object)(w+r))
            random.setSeed(System.currentTimeMillis());
         }
         return games
 }

 }  // end class GameGenerator

 /**
 * Class to render the results
 */
 def class GenerateReport extends GuessBase implements Runnable {
      def GenerateReport(Buffer games){
              this.games = games
      }

      public void run(){
         showGames()
     }
} // end class GenerateReport

Listing 6 Using Fork/Join:


// to do:  approach will be to fork each required game set into its own thread, then join them to get the final result.  Crazy, but a way to learn about Fork/Join and have sample code.

CSP

Some experts are touting the benefits of CSP. Below I use the jCSP library to create a network of processes that collaborate to solve the same problem. Note that some of the people involved in jCSP are adding some CSP support to GPar.

Listing 7 Using jCSP:

/**
* File: CspGuess.groovy
* Author: Josef Betancourt
* Date:  9/1/2007
*
*
*/

package net.cox.members.jbetancourt1;

import jcsp.lang.*;

/**
 * Generate powerball plays using CSP.
 *
 * This version of the example uses the JCSP library.  Note
 * that unlike the other examples, there is an explicit
 * declaration of a 'network'.
 *
 * Required libraries:
 * - JCSP ver. 1.0-rc8 CSP library in Java:
 * <a href="http://www.cs.ukc.ac.uk/projects/ofa/jcsp"></a><br/>
 *
 * Uses PAR and ALT methods presented in:
 * <a href="http://www.soc.napier.ac.uk/publication/op/getpublication/publicationid/9097759">
 * "Groovy Parallel!  A Return to the Spirit of occam?"</a>
 * by Jon KERRIDGE, Ken BARCLAY, and John SAVAGE
 * The School of Computing, Napier University, Edinburgh EH10 5DT in
 * Communicating Process Architectures 2005 13
 * Jan Broenink, Herman Roebbers, Johan Sunter, Peter Welch, and David Wood (Eds.)
 * IOS Press, 2005
 *
 * Perhaps, a better approach is possible using actors?  See for example,
 * <a href="http://gpars.codehaus.org/">Groovy Parallel Systems</a>
 *
 */
class GuessCSP extends GuessBase {
 def generate(dollars,games){
       def suspendChannel = new One2OneChannel()
       def resultChannel = new One2OneChannel()

       // Create the CSP network and run it.
       new PAR(  [
            new GameProcess(suspendChannel,resultChannel,games,dollars),
            new UserInputProcess(suspendChannel),
            new ReportProcess(resultChannel)]
       ).run()
 }

}

/** process that generates games */
def class GameProcess extends GuessBase implements CSProcess{
    def out
    def suspend
    def dollars

    /**  Create the process instance */
    def GameProcess(done,out,games,dollars){
       this.dollars = dollars
       this.suspend = done
       this.out = out
       this.games = games
       createBallSets()
    }

    /** run the process network */
    public void run(){
       def alternative = new ALT([suspend, new Skipper()])
       def STOPPING=0, RUNNING=1
       println "Creating  ${dollars} game${(dollars>0? "s": "")}  "
       def suspended = false
       while (!suspended){
          switch (alternative.priSelect ())
             {
               case STOPPING:
                 suspend.read();
                 if(games.size()>0){
                      suspended = true;
                      out.write(games)
                 }
                 break;
               case RUNNING:
                 def w = pickWhite()
                 def r = pickRed()
                 games.add((Object)(w+r))
                 random.setSeed(System.currentTimeMillis());
                 break
             }
      }
 } // end run()

} // end GameProcess process

/** Process that gets the user input */
def class UserInputProcess implements CSProcess{
     def suspend

     def UserInputProcess(done){
            this.suspend = done
     }

     public void run(){
              println "Enter any key<cr>"
              new InputStreamReader(System.in).readLine()
              suspend.write(new Object())
     }
} // end UserInputProcess process

/**
 * Process to render the results to console
 */
def class ReportProcess extends GuessBase implements CSProcess{
 def input

 def ReportProcess(input){
     this.input = input
 }

 public void run(){
     games = input.read() // get result via channel only
     showGames()
 }
} // end of ReportProcess process

/**
 * PAR A Groovyish Parallel.
 */
def private class PAR extends Parallel {
       PAR(processList){
            super( processList.toArray(new CSProcess[0]))
       }
}

/**
 * ALT A Groovyish Alternative.
 */
def private class ALT extends Alternative {
       ALT (guardList) {
              super( guardList.toArray(new Guard[0]) )
       }
}
// end of CspGuess.groovy

Actor

I started to look into the Actor approach. Below is the start of code to solve the same problem. Even with the few lines of code, attempts to run it give an an exception, and classpath and other easy fixes do not solve it. On my todo list.

Setting up games 5 using "actor"
start
Caught: groovy.lang.MissingMethodException: No signature of method: static groovyx.gpars.actor.Actor.actor() is applicable for argument types: (net.cox.members.jbetancourt1.GuessActor$_generate_closure1) values: [net.cox.members.jbetancourt1.GuessActor$_generate_closure1@1fe571f]
Possible solutions: stop(), wait(), start(), any(), call(java.lang.Object), wait(long)
	at net.cox.members.jbetancourt1.GuessActor.generate(GuessActor.groovy:55)
	at net.cox.members.jbetancourt1.Main.main(Main.groovy:63)

Listing 8 Using Actors:

package net.cox.members.jbetancourt1
//import static groovyx.gpars.actor.Actors.actor
import groovyx.gpars.actor.*
import groovyx.gpars.actor.Actor.*
// @Grab(group='org.codehaus.gpars', module='gpars', version='0.9')
// @GrabResolver(name='jboss', root='http://repository.jboss.org/maven2/')

/**
 */
class GuessActor extends GuessBase  {
	def main = new GuessActor()
	        def dollars
	        def games
	        main.generate(dollars, games)
        }

	def generate(dollars,games){
		println "start"

		def gen = Actor.actor { index ->
			loop {
				react {message ->
					if (message instanceof String) reply "got it"
					else stop()
				}
			}
		}

} // end of class GuessActor

Declarative CSP Using Spring

While experimenting I had an idea that ordinary Java beans could be used in a CSP network with suitable object wrappers or weaving.  That is, an ordinary POJO could be wrapped to appear as a Process, just hook up the bean’s entry point for service into the CSP channel end points.  This would be similar to the way that ordinary beans can be annotated to behave as Active Objects in the GPars Groovy library.

Below is the Spring Framework configuration file that does this for this software example.  The code for the wrapper objects are not included here, too kludgey.

Off topic: The XML language config of Spring is looking long in tooth. Good thing Spring now supports annotations and Java config. Time for a Groovy builder approach too?.

Update:

Listing 9 bean configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<!--
 File: applicationContext.xml
 Spring Framework based bean definitions for JCSP example program.
 Author: Josef Betancourt
 Date:  9/19/2007
-->
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:lang="http://www.springframework.org/schema/lang"  xmlns:util="http://www.springframework.org/schema/util"
 xsi:schemaLocation="http://www.springframework.org/schema/beans
     http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
     http://www.springframework.org/schema/lang
     http://www.springframework.org/schema/lang/spring-lang-2.0.xsd
     http://www.springframework.org/schema/util
     http://www.springframework.org/schema/util/spring-util-2.0.xsd">

 <!-- ========================================================= -->
 <!-- CSP NETWORK -->
 <!-- ========================================================= -->
 <bean name="network" dependency-check="objects" class="jcsp.lang.Parallel">
     <constructor-arg>
         <list>
              <ref bean="gameProcess" />
              <ref bean="reportProcess" />
              <ref bean="userInputProcess" />
              <ref bean="progressProcess"/>
         </list>
     </constructor-arg>
 </bean>

 <!-- ========================================================= -->
 <!-- CHANNELS -->
 <!-- ========================================================= -->
 <bean name="suspendChannel" class="jcsp.lang.One2OneChannel"/>
 <bean name="outputChannel" class="jcsp.lang.One2OneChannel"/>
 <bean name="progressChannel" class="jcsp.lang.One2OneChannel"/>
 <bean name="skipperChannel" class="jcsp.lang.Skip"/>

 <!-- ========================================================= -->
 <!-- GUARDS  -->
 <!-- ========================================================= -->
 <bean name="alternative" class="jcsp.lang.Alternative" dependency-check="objects">
        <constructor-arg>
            <list>
               <ref bean="suspendChannel" />
               <ref bean="skipperChannel" />
            </list>
        </constructor-arg>
 </bean>

 <!-- ========================================================= -->
 <!-- POJOs -->
 <!-- ========================================================= -->
 <!--  Creates the game generator, but it is setup programmatically
 since it needs the user supplied number of games to play, see GameGenerator.setup(int).
 -->
 <bean name="gameGenerator" class="net.cox.members.jbetancourt.pb.game.GameGenerator">
      <property name="random">
         <bean class="java.security.SecureRandom"/></property>
         <property name="maxRed" value="42" />
         <property name="maxWhite" value="55" />
 </bean>

 <!-- ========================================================= -->
 <!-- PROCESSES -->
 <!-- ========================================================= -->
 <!-- Receives signal from user to accept current generated game sets.
 -->
 <bean name="userInputProcess" init-method="init" class="net.cox.members.jbetancourt.pb.process.JCspSingleShotProxy">
    <property name="outputChannel" ref="suspendChannel" />
    <property name="methodName" value="run"/>
    <property name="target">
            <bean class="net.cox.members.jbetancourt.pb.game.KeyListener" scope="prototype"/>
    </property>
 </bean>

 <!--  Game generator.  Communicates with input and report processes.
 -->
 <bean name="gameProcess" dependency-check="objects"
            class="net.cox.members.jbetancourt1.pb.process.GameProcess">
     <property name="alternative" ref="alternative" />
     <property name="suspendChannelIn" ref="suspendChannel" />
     <property name="reportChannelOut" ref="outputChannel" />
     <property name="progressChannelOut" ref="progressChannel"/>
     <property name="gameGenerator" ref="gameGenerator"/>
     <property name="showInterim" value="false"></property>
 </bean>

 <!-- Output results to console.
 -->
 <bean name="reportProcess" dependency-check="objects"
         class="net.cox.members.jbetancourt1.pb.process.ReportProcess" scope="prototype">
       <property name="input" ref="outputChannel" />
 </bean>

 <!-- Output interim results to console
 -->
 <bean name="progressProcess" dependency-check="objects"
    class="net.cox.members.jbetancourt1.pb.process.ReportProcess" scope="prototype">
      <property name="input" ref="progressChannel" />
 </bean>

</beans>

Listing 10 Java main to load the network:

package net.cox.members.jbetancourt1.pb;

import jcsp.lang.Parallel;
import net.cox.members.jbetancourt1.pb.game.GameGenerator;

import org.apache.commons.cli.*;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

/**
 * 
 * Main program.
 * 
 * @author JBetancourt
 * 
 */
public class Main {
    private static final String TITLE = 
    "====== Lottery Generator using JCSP and Spring ============";
    private static Parallel network;
    private static ApplicationContext context;

    /**
     * Entry point for running the Guess application. Parses 
     * command line for number of games to play, loads 
     * wired application using Spring Framework
     * based IoC, sets up game array, and then starts 
     * the network of processes. 
     * @param args
     */
    public static void main(String[] args) {
        System.out.println(TITLE);
        int dollars = parseCommandLine(args);
        if ( dollars == 0 ) {
            return;
        }

        context = new ClassPathXmlApplicationContext(
            new String[] { "applicationContext.xml"});
        
        GameGenerator gen = (GameGenerator) context.
        getBean("gameGenerator");
        gen.setup(dollars);

        network = (Parallel) context.getBean("network");
        network.run();
    }

    /**
     */
    private static int parseCommandLine(String[] args) {
        ... ellided ...
    }

}

Further Reading

Lottery Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Groovy Object Notation (GrON) for Data Interchange

Summary

Foregoing the use of JSON as a data interchange when Groovy language applications must interact internally or with other Groovy applications would be, well, groovy.

Introduction

JavaScript Object Notation (JSON) is a language-independent data interchange format based on a subset of the JavaScript (ECMA-262 3rd Edition) language. Many languages and libraries now support marshal to and from JSON using external libraries or extensions. This complicates applications since they must rely on more subsystems and there may be a performance penalty to parse or generate an external object notation.

If an application must only interact within a specific language or environment, such as the Java Virtual Machine (JVM), perhaps using the host language’s data structures and syntax will be a simpler approach. Since Groovy (a compiled dynamic language) has built-in script evaluation capabilities, high-level builders (for Domain Specific Language (DSL) creation) , and meta-programming capabilities, it should be possible to parse, create, transmit, or store data structures using the native Groovy data interchange format (GDIF), i.e., based on the native Groovy data structures.

Syntax example

Below is an example JSON data payload.

JSON (JavaScript) syntax:

{"menu": {
  "id": "file",
  "value": "File",
  "popup": {
    "menuitem": [
      {"value": "New", "onclick": "CreateNewDoc()"},
      {"value": "Open", "onclick": "OpenDoc()"},
      {"value": "Close", "onclick": "CloseDoc()"}
    ]
  }
}}

Below is the same data payload; this time using Groovy syntax. Note that there are not too many differences, the most striking is that maps are created using brackets instead of braces. It looks simpler too.

Groovy syntax:

[menu: [
	id: "file",
	value: "File",
	popup: [
	menuitem : [
	 [ value: "New", onclick: "CreateNewDoc()" ],
	 [ value: "Open", onclick: "OpenDoc()" ],
	 [ value: "Close", onclick: "CloseDoc()" ]
     ]
  ]
]]

Code Example

/**
 * File: GrON.groovy
 * Example class to show use of Groovy data interchange format.
 * This is just to show use of Groovy data structure.
 * Actual use of "evaluate()" can introduce a security risk.
 * @sample
 * @author Josef Betancourt
 * @run    groovy GrON.groovy
 *
 * Code below is sample only and is on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
 * or implied.
 * =================================================
*/
 */
class GrON {
    static def message =
    '''[menu:[id:"file", value:"File",
     popup:[menuitem:[[value:"New", onclick:"CreateNewDoc()"],
     [value:"Open", onclick:"OpenDoc()"], [value:"Close",
     onclick:"CloseDoc()"]]]]]'''

    /** script entry point   */
    static main(args) {
       def gron = new GrON()
       // dynamically create object using a String.
       def payload = gron.slurp(this, message)

        // manually create the same POGO.
        def obj = [menu:
	    [  id: "file",
	       value: "File",
                 popup: [
                   menuitem : [
                   [ value: "New", onclick: "CreateNewDoc()" ],
                   [ value: "Open", onclick: "OpenDoc()" ],
                   [ value: "Close", onclick: "CloseDoc()" ]
          	]
           ]
         ]]

         // they should have the same String representation.
         assert(gron.burp(payload) == obj.toString())
    }

/**
 *
 * @param object context
 * @param data payload
 * @return data object
 */
def slurp(object, data){
	def code = "{->${data}}"  // a closure
	def received = new GroovyShell().evaluate(code)
	if(object){
		received.delegate=object
	}
	return received()
}

/**
 *
 * @param data the payload
 * @return data object
 */
def slurp(data){
     def code = "{->${data}}"
     def received = new GroovyShell().evaluate(code)
     return received()
}

/**
 * @param an object
 * @return it's string rep
 */
def burp(data){
     return data ? data.toString() : ""
}

} // end class GrON

Possible IANA Considerations

MIME media type: application/gron.

Type name: application

Subtype name: gron

Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32

Additional information:

Magic number(s): n/a

File extension: gron.

Macintosh file type code(s): TEXT

API

To be determined.

Security

Would GrON be a security hole? Yes if it is implemented using a simple evaluation of the payload as if it were a script. The example shown above used evaluate() as an example of ingestion of a data object. Incidently, this is the same naive approach used in some JavaScript applications and general not recommended.

For real world use, some kind of parser and generator for object graphs would be needed. The advantage would accrue if the underlying language parser could be reused for this.

Now this begs the question, if Groovy must now support a data parser, why not just use JSON with the existing libraries, like JSON-lib? [May 7, 2014: the built in JSON support].

Is using the Java security system an alternative as one commenter mentioned?

Notes

The idea for GrON was formulated about a year ago. Delayed posting it since I wanted to create direct support for it. However, the task required more time and expertise then I have available at this time.

I was debating what to call it, if anything. Another name I considered was Groovy Data Interchange Format (GDIF), but I decided to follow the JSON name format by just changing the “J” to “G” and the “S” to “r” (emphasizing that Groovy is more then a Scripting language, its an extension of Java).

Updates

10Sept2011: See also this post: “JSON Configuration file format“.

9Feb2011: Looks like Groovy will get built in support for JSON: GEP 7 – JSON Support

I found (May 18, 2010, 11:53 PM) that I’m not the first to suggest this approach. See Groovy Interchange Format? by DeniseH.

Recently (Oct 3, 2010) found this blog post:
Groovy Object Notation ? GrON?

Mar 22, 2011: Groovy 1.8 will have JSON support built in.

Further Reading


Creative Commons License
Groovy Object Notation (GrON) for Data Interchange
by Josef Betancourt is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 Unported License.
Based on a work at wp.me.

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.