Category Archives: JSON

Groovy implementation of INIX file format, part 2

Continued experimentation with the INIX “ini” file format. I take the original post code and add an alias feature.

Background
In three prior posts I presented a very simple metadata file storage approach. Pretty much using the INI file format with the sections as “heredocs.” In original INI format, afaik, the data within the sections were properties only, x=y.

Updates

  • Dec 25, 2015: Just saw a mention of Tom’s Obvious, Minimal Language (TOML) This is not directly related to what I’m after with Inix, but interesting as an example of a simple markup language.
  • Alias
    Now I am working on adding the ability for a section of data to load data from another section. The sections to load are indicated by the ‘@’ symbol and then the section path. Multiple sections can be indicated. Though I’m using the term ‘alias’ for this, perhaps a better term is ‘importing’. So far, I can parse the alias from the tag string.

    I have not implemented the actual import. One complexity left to solve is recursion. If this section imports another section, what if that section imports others sections?

    Alias use case
    Since currently the Inix file format is being used for test data, aliasing allows reuse of data without duplication, i.e., DRY. This is problematic with hierarchical data like JSON or XML, but much easier with lists or maps. Further features like overriding and interpolation would be useful for Java Properties data. The goal would be to eventually support use of the Cascading Configuration Pattern.

    Example 1

    [>First]
    Data in first    
    [<]
    
    [>Second@First]
    Data in second
    [<]
    

    Now when the data in section “Second” is loaded, the data from the aliased section is prepended to the current section data:

    Data in first    
    Data in second
    

    Tag format
    The section tag format is now: [>path#fragment@aliases?querystring]. Note that unlike a URI, the fragment does not appear at the end of the string.

    The section ID is really the path#fragment. Thus, the end tag could be [<] or can use the section ID: [<path#fragment]. Example 2

    [>demo1/deploy#two@account897@policy253?enabled=true&owner=false]
    stuff here
    [<demo1/deploy#two]
    

    Grammar
    The start of a grammar follows, but has not been ‘checked’ by attempted use of a parser generator like Antlr.

    grammar Inix;
    section: start CRLF data end;
    start: '[>' path (fragment)?(alias)*('?' args)? ']';
    end: '[<' path? ']';
    path: NAME ('/' NAME)*;
    fragment: '#' NAME;
    alias: '@' NAME
    args: (NAME=NAME ('&' NAME=NAME)*)?;
    data: (ANYTHING CRLF)*;
    NAME: ('a'..'z' | 'A'..'Z')('a' .. 'z' | 'A'..'Z'|'0'..'9'|'_');
    
    TODO:

    1. Do the actual import of aliased section data.
    2. Allow multiple params per param key: ?measure=21,measure=34,measure=90. Or better yet, just allow array in arg string: measure=[21,34,90],color=red

     

    Implementation
    Source code available at Github: https://gist.github.com/josefbetancourt/7701645

    Listing 2, Implementation

    Test class
    Note that there are not enough tests and the implementation code has not been reviewed.

    Listing 3, Test class

    The test data is:

    Listing 4, data file

    Environment
    Groovy Version: 2.2.2 JVM: 1.7.0_25 Vendor: Oracle Corporation OS: Windows 7

    Further Reading

    1. The Evolution of Config Files from INI to TOML
    2. Groovy Object Notation using ConfigSlurper
    3. Configuration Files Are Just Another Form of Message Passing (or Maybe Vice Versa)
    4. INI file
    5. Data File Metaformats
    6. Here document
    7. JSON configuration file format
    8. Creating External DSLs using ANTLR and Java
    9. Groovy Object Notation (GrON) for Data
      Interchange
    10. Cloanto Implementation of INI File Format
    11. http://groovy.codehaus.org/Tutorial+5+-+Capturing+regex+groups
    12. URI
    13. Designing a simple file format
    14. The Universal Design Pattern
    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    Understanding JSON

    There seems to be a little bit of confusion among some developers as to what JSON is.

    Some think some that JSON is a JavaScript serialization, or that it is just a way to pass JavaScript data around. Well, there are good reasons why this may be assumed. One is that JSON is usually used in HTML based client UIs, and it does have JavaScript in its acronym, JavaScript Object Notation. There are also many sites and YouTube videos that present JSON in a very JavaScript and browser client centric way.

    It is not these things, though it is used like that (to some extent). The definition is plain enough:

    JSON … is a lightweight data-interchange format. … It is based on a subset of … JavaScript …. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. — http://json.org/

    The important points are:

    • data-interchange
    • text based
    • began as a subset of JavaScript’s collections
    • language independent

    Note that there is nothing about Objects in there. The Object, that comes later in the spec, is what would be called a Map or Table in other scenarios. Yes, “but, JavaScript has Object Literal Syntax”, no matter.

    I expect that the JSON word will lose its acroynm-ity in time and just become the name for a format. A similar thing happened to the OSGi trademark, it no longer means what it used to mean, and XML, means pointy brackets everywhere.

    Updates
    Aug 2, 2014: There is one scenario where thinking that JSON is just JavaScript object serialization is correct. When your working with JavaScript object serialization. Thus, when you send JSON from a browser client to a backend Java middleware, a library, such as GSON, will be used to deserialize the JSON to Java objects. In this case, contextualizing that in JSON “{ }” is an object, makes it clear that your process will need a Java class at that point.

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    Diagnostic Differential Logging with JavaScript Example

    Synopsis
    Reducing the quantity of information to what is essential is very important in maintaining complex software. Modern development environments have debuggers and other tools to assist in this effort: conditional breakpoints, watch expressions, logging, stop on change techniques, and so forth.
    As a supplementary technique an approach to debugging is shown which uses the dynamic differences in a running system to output a reduced level of information. A simple programmatic implementation is shown in a JavaScript scenario, but the technique is applicable using other languages and automated techniques.

    KEYWORDS: logging, diff, debugging, diagnostics, BIST, JavaScript, AOP, BIST

    Context
    Recently I was debugging a legacy JavaScript program. I used the outstanding FireBug plug-in in the FireFox browser. By the way, FireBug is truly indispensible, such a wealth of features and it is FOSS. Now combine that with Selenium which gives you the ability automate the browser and you have a powerful Integrated Development Environment (IDE). All that could be lacking is time and skill.
    Unfortunately, I was not finding the problem. Later at home I got an idea. What if I could simply log what changed? That way I could more clearly see how the program behaves. All programs are just obfuscated state machines. Through a use-case or scenario, state variables are being manipulated and non-state data is being changed. Correlating these two with the events that drive the system, is a prime aspect of debugging.

    Forces
    • You want to log what is happening in a system, but you don’t want to be inundated with non-useful data.
    • You or someone else want to be able to quickly resume this investigation in another situation or project.
    • By saving ‘interests’, a Built-in-self-test aid can be implemented.
    • During development hooks can be put in place to provide differential logging.
    • You want to capture the live system not a system which is controlled by a debugger.
    • You cannot install new systems, tools, or frameworks to help.
    • Breakpoints will break as the system is maintained.

    Solution
    My solution was to dynamically log the differences. I’m sure this is not new. However, it probably is not explicitly discussed. The usual approach is to log as much as possible, but since this is prohibitive, instead the logging is limited, level filters, and contexts are used to control the output. Breakpoints and watches while usually enough to do the job are in the long term brittle since the target program may change during the dev and debug process. Thus, dynamic difference reporting would be a better approach in this situation.

    Consequences
    Todo.

    Implementation
    There are a few implementation choices.

    API
    One is to declare the data of interest. Then the logging system will track these and only output anything when there is a change in any of the data. With this approach the logger must be triggered either manually or automated. Manually is very easy; at an event change or at enter and exit points of a module or function, the differential logger is invoked.

    Automated
    To avoid changing the source code (it may not even be available), AOP could be used to implement the above. For example, an Around advise can be used to differentially trace the changes in methods.

    Framework Plugin
    If the system under test is running as part of a debug or is instrumented, it may already have a facility to capture change events. For example, FireBug, the FireFox plugin has this and the FireBug plug-in, FireDiff, implements a change monitor and visualization output. There may be or should be a comparable plug-in for other systems like the Eclipse IDE.

    Code Example
    To illustrate this concept it is used to debug a web page’s DOM changes via JavaScript. The page has a button that when clicked will randomly change two input field’s content. Notice that when the change involves just an ordering of two adjacent characters in a field, it is not readily apparent. The Differential logging makes it explicit. The log output is just the result object converted to JSON. An example: {“event”:”run”,”record”:{“foo2”:[“”,”accd”],”foo4″:[“”,”ollrr”]}}. A real implementation would do otherwise.

    To specify which DOM nodes to monitor, a JavaScript object is used:

    {"foo2":{"get":"v"},"foo4":{"get":"v"}};
    

    The “get” field is used to specify how to get the value from the browser DOM. Another method would be “innerHTML” access. A better approach is to detect the appropriate ‘getter’ via using the node type.

    demo screen capture

    Listing 1, implementation

    
    /**
     * 
     * Differ constructor.
     * 
     * @param data which nodes to record
     * @param formatter how to show results
     * @returns
     */
    function Differ(data, formatter){
    	this.data = data;
    	this.formatter = formatter;
    	
    	if(!formatter){
    		this.formatter =this.format;
    	}
    };
    
    /**
     * Data and methods of class.
     * 
     */
    Differ.prototype = {
      MAXSIZE : 15,
    	
      /**
       * Detect changes and save latest.
       * 
       * @param event Where was diff created
       * @returns result object
       */
      diff : function(event){
        var results = {};
    		
        if(!data){
        	return results;
        }
    		
        if(event){
        	results = {"event":event};
        }
    		
        results["record"] = {};
    		
        var record = results["record"];
    		
        for(var key in data){			
          var ref = document.getElementById(key);			
          var domValue = "";			
    			
          switch (data[key].get) {
            case "v":
              domValue = ref ? this.shorten(ref.value) : "";
              break;
            }
    			
            var prior = data[key].value ? data[key].value : "";
    			
            if((domValue != prior)){
              record[key] = [prior, domValue];
              data[key].value = domValue;
            }
        } // end for	
    		
        return results;
    		
      }, 
    
      /**
       * Convert results to message.
       * 
       * @param results object
       * @returns JSON
       */
      toMessage : function(results){
        var obj = results;
        if(!results){
          obj ={"error":"results is undefined or null"};
        }		
    		
        return this.format(JSON.stringify(obj));
      },
    	
      /**
       * Get size of an object
       * @param obj
       * @returns {Number}
       */
      size : function(obj){
        var len = 0;
        for(var key in obj){
          if(obj.hasOwnProperty(key)){
            len++;
          }
        }
    		
        return len;
      },
    	
      /**
       * Limit size of value.
       * 
       * @param value
       * @returns
       */
      shorten : function(value){
        return (value.length > this.MAXSIZE) ? 
    	value.substring(0,this.MAXSIZE-1):value;		
      },
    	
      /**
       * Make message human readable.
       * 
       * @param message
       */
      format : function(str){
      	return str + '<br/>';
      }	
    	
    };	
    

    Listing 2, Demo script

    var data = {
      "foo2" : {
        "get" : "v"
       },
      "foo4" : {
        "get" : "v"
       }
    };
    
    var differ = new Differ(data);
    differ.diff(); // prime values
    
    function runIt() {
      var ref2 = document.getElementById('foo2');
      var ref4 = document.getElementById('foo4');
    	
      if(Math.floor(Math.random()*101) > 60){
        swapInputs(ref2,ref4);
      }else{
        scramble(ref2,ref4);
      }
    
      var msg = differ.toMessage(differ.diff("run"));
      document.getElementById('output').innerHTML += msg;
      document.getElementById('output').innerHTML += 
        Array(60).join("-") + "<br/>";
    }
    
    function scramble(ref2, ref4){
      var ch = ref2.value.charAt(1);
      var r2v = ref2.value;
      ref2.value = replaceChar(r2v, 2, ch);
      ref4.value = replaceChar(ref4.value, 2, ref4.value.
        charAt(1));
    }
    
    function swapInputs(ref2,ref4){
      var temp = ref2.value;
      ref2.value = ref4.value;
      ref4.value = temp;	
    }
    
    function replaceChar(str, ndx, v) {
      return str.substr(0, ndx) + v + 
        str.substr(ndx + v.length);
    }
    

    Listing 3, the demo web page

    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 
        "http://www.w3.org/TR/html4/loose.dtd">
    <html>
    <head>
      <meta http-equiv="Content-Type" content="text/html; 
        charset=iso-8859-1">
      <link rel="stylesheet" type="text/css" href="style.css" />	
      <script type="text/javascript" src="DebugUtils.js"></script>	
      <script type="text/javascript" src="DemoDiff.js"></script>	
    </head>
    <body>
      <span class="mytitle">Differential Logging Example</span>
      <div style="width:600px;">
    	<div class="box inputBox">		
    		1: <input type="text" id="foo1" value="xyyz"/>
    		2: <input type="text" id="foo2" value="acbd"/><br/>
    		3: <input type="text" id="foo3" value="wzlm"/>
    		4: <input type="text" id="foo4" value="olxrr"/>
    
              <span class="buttonClass">
    	    <input type="button" id="run" value="Test" 
                   onclick="runIt()"/>
    	  </span>		
    	</div>
    	Output
    	<div id="output" class="box inputBox"></div>
    	<span style="float:right;margin-right:10em;font-size:xx-small;">
               &copy; Josef Betancourt</span>		
    	</div>
    </body>
    </html>
    

    Listing 3, css

    body {
      background-color:#E6E6E6;margin:3em 8em 12em 3em;
    }	
    .box{
      border: 3px solid #D4D4D4;padding:8px;
      padding:2em 1em 2em;margin:10px 5px 20px;
    }
    .inputBox{
       background-color:Beige;margin-right:36px;
       width:475px;font-family:monospace;
    }
    .buttonClass{
        float:right;margin-left:1em;margin-right:3em;
    }
    .mytitle{
       text-decoration:underline;font-weight:bold;
    }
    

    Related Patterns
    None.

    Related Intellectual Property
    None.

    Further Reading

    Debuggers are for Losers

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    Archive files using Ant from Groovy

    In Ant The zip task takes a fileset but only creates one archive. How do you zip files into individual archives?

    Use case
    One use case for this is log files. Sooner rather then later, every system must deal with the growth of log files. One of the best utilities for handling this is logrotate. This is available on *nix systems. There are probably native versions that do similar tasks in Windows, but I did not locate any ‘free as in beer’ ones.

    Approach
    Since I’m working with Ant, on Windoze, and can’t install cygwin or other system, will use Groovy to get around Ant limitations. This will not be an attempt to recreate logrotate, of course.

    As in a previous post, we can use the Groovy AntBuilder and iterate a fileset to repeatably call an Ant task. In this case we’ll be invoking the zip task. Note that the same approach could be used to create a scriptdef to do this directly in an Ant build script. This would be more usable since we could declaratively define the fileset, etc.

    Example

    First the configuration. In listing 1, we use a JSON configuration file. No reason, I’m just tired of Property files.

    Listing 1, JSON configuration file

    { "about":
        {
        	"project":"archivelogs",
        	"author":"jbetancourt",
        	"date":"20110909T2156"    	
        },
      "config":
    	{
    		"basedir":"test/logs",
    		"archivedir":"test/logs/archive",
    		"includes":"*.log",
    		"propertyName":"found.files",
    		"when":"before",
    		"update":"true",
    		"days":"1",
    		"level":"9"
    	}
    }

    In listing 2, the main entry point calls the config loader and then the ‘archiveLogs’ method.

    The fileset created with the AntBuilder uses an Ant ‘date’ selector to find the files that are x amount of days old. Then the resulting fileset is scanned in a loop, invoking the zip task as we go and then deleting the just zipped file. It is assumed here that the target file name has descriptive information, like a date. And, we are not adding a “rotate” counter to the name for future logrotate like behavior.

    Listing 2, Groovy archive source

    package ant
    
    import org.apache.tools.ant.*
    import java.text.*
    import groovy.json.JsonSlurper
    
    /**
     * @author jbetancourt
     */
    class ArchiveLogs {
    	def format = "yyMMdd'T'kkmm"
    	def df = new SimpleDateFormat(format)
    
    	/** entry point */
    	static main(args) {
    		def als = new ArchiveLogs()	
    		
    		def p = (
    		  (args) ?
    		  args[0] : "test/conf/archivelogs.json")
    				
    		def map = als.loadConfiguration(p)
    		als.archiveLogs(map)
    	}
    
    	/**  */
    	def archiveLogs(conf) {
    		def base = conf.base
    		def targetDateString = df.format(
    			(new Date() - Integer.parseInt(conf.days))
    		)
    		
    		def ant = new AntBuilder()		
    		ant.fileset( 
    			dir:conf.basedir, 
    			includes:conf.includes, 
    			id:conf.propertyName){			
    				date( 
    					datetime:targetDateString,
    					when:conf.when,
    					pattern:format)
    		}	
    		
    		def ref = ant.project.getReference(
    			 conf.propertyName)
    		
    		ref.each{ fileResource ->
    			File file = fileResource.getFile()
    			def name = file.getName()
    			
    			ant.zip(
    				basedir:conf.basedir,
    				destfile:
    				 "${conf.archivedir}/${name}.zip",
    				update:conf.update,
    				level:conf.level,
    				includes:name
    			)
    			
    			ant.delete(verbose:"true",file:file.getPath())			
    		}		
    		
    	}	
    	
    	/** load json file */
    	def loadConfiguration(String path){
    		def slurper = new JsonSlurper()
    		def obj = slurper.parse(
    			 new File(path).newReader())
    		
    		if(obj.config.debug == 'true'){
    			obj.config.each{
    				println it
    			}		
    		}
    		
    		return obj.config
    		
    	}
    }
    

    Caveats
    This is, of course, a simplistic approach. For example, what if the file is being used when we attempt to delete it? Or we fail to zip the file, but continue to delete it? And, this source, being an example, does not have any error handling.

    Alternatives
    The ant-contribs project has looping support, foreach. Directly using the Java util packages for archiving, such as java.util.zip. Or, the libraries found in, for example, the Apache commons project.

    Further Reading


    “cafe”, egberto gismonti

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    JSON configuration file format

    JSON is a data interchange format. Should it also be used as a configuration file format, a JSON-CF?

    Overview
    Had to write yet another properties file for configuration info. Started to think that maybe there are better alternatives. Wondered about JSON for this.

    Requirements
    What are requirements of a configuration file format?

    • Simple
    • Human readable
    • Cross platform
    • Multi-language support
    • Unicode support

    Looks like JSON has all the right qualities.

    If all you want to pass around are atomic values or lists or hashes of atomic values, JSON has many of the advantages of XML: it’s straightforwardly usable over the Internet, supports a wide variety of applications, it’s easy to write programs to process JSON, it has few optional features, it’s human-legible and reasonably clear, its design is formal and concise, JSON documents are easy to create, and it uses Unicode.
    — Norman Walsh, Deprecating XML

    JSON-CF Limitations

    • Instead of angle brackets as in XML, we have quotation marks everywhere.

    What does it need?

    • Inline comments, see for example, json-comments
    • Interpolation (property expansion)
    • Namespaces
    • Inheritance
    • Includes
    • Date value
    • Schema
    • Cascading

    Example

    {
        "_HEADER":{
                "modified":"1 April 2001",
                "dc:author": "John Doe"
        },
        "logger_root":{
                "level":"NOTSET",
                "handlers":"hand01"
         },
        "logger_parser":{
                "level":"DEBUG",
                "handlers":"hand01",
                "propagate":"1",
                "qualname":"compiler.parser"
        },
        "owner":{
                 "name":"John Doe",
                 "organization":"Acme Widgets Inc."
         },
         "database":{
                 "server":"192.0.2.62",     
                 "_comment_server":"use IP address in case network name resolution is not working",
                 "port":"143",
                 "file":"payroll.dat"
          }
    }
    

    Programmatic Access using Groovy

    Now we can easily read this file in Java. Using Groovy is much easier, of course. Groovy version 1.8 has built-in JSON support, great blog post on this here.

    import groovy.json.*;
    
    def result = new JsonSlurper().
                              parseText(new File("config.json").text)
    
    result.each{ section ->
    	println "$sectionn"
    }
    

    >groovy readConfig.groovy
    Resulting in Groovy style data structure, GRON, (look ma, no quotation marks):

    logger_parser={qualname=compiler.parser, level=DEBUG, propagate=1, handlers=hand01}
    
    owner={organization=Acme Widgets Inc., name=John Doe}
    
    _HEADER={dc:author=John Doe, modified=1 April 2001}
    
    database={port=143, file=payroll.dat, server=192.0.2.62, _comment_server=use IP address in case network name resolution is not working}
    
    logger_root={level=NOTSET, handlers=hand01}
    

    In Groovy you can access the data with GPath expressions:
    println “server: ” + result.database.server

    You can also pretty print JSON, for example:
    println JsonOutput.prettyPrint(new File(“config.json”).text)

    Summary

    Raised the question of the use of JSON as a configuration file format.

    What I don’t like is the excess quotation marks. YAML is more attractive in this sense. But, the indentation as structure in YAML, similar to Python, may not be wise in configuration files.

    Well, what is the answer, should there be a JSON-CF? I don’t know. A very perceptive caution is given by Dare Obasanjo commenting on use of new techniques in general:

    So next time you’re evaluating a technology that is being much hyped by the web development blogosphere, take a look to see whether the fundamental assumptions that led to the creation of the technology actually generalize to your use case.

    Updates

    • After writing and posting this I searched the topic and, of course, this is not a new question. Search. I updated the reading list below with some useful links.
    • JSON Activity Streams are an example of how JSON is used in new ways.
    • schema.org types and properties as RDFS in the JSON format: schema.rdfs.org/all.json
    • Just learned that node.js uses the NPM package manager which uses a JSON config file format.
    • Jan 7, 2012: Java JSR 353: Java API for JSON Processing

    Further Reading
    Apache Avro 1.7.7 Specification Schemas
    Analysis of JSON use cases compared to XML
    json-schema.org
    JSON Validation Roundup
    A very simple data file metaformat
    GRON
    HAL – Hypertext Application Language
    NPM configuration file format
    XML or YAML for configuration files
    Using JSON for Language-independent Configuration Files
    INI file
    JSON+Comments
    Comparison of data serialization formats
    Data File Formats, in Art of Unix Programming, Eric Steven Raymond.
    ConfigParser – Work with configuration files
    JSON
    Cascading Configuration Pattern
    java.util.Properties
    RFC 4627
    XML-SW,a skunkworks project by Tim Bray. Brings a bunch of XML complex together into one spec.
    YAML
    ISO 8601
    Learning from our Mistakes: The Failure of OpenID, AtomPub and XML on the Web
    Groovy Object Notation (GrON) for Data Interchange
    Groovy 1.8 Introduces Groovy to JSON
    JSON-LD ‘JSON for Linking Data
    JSON Schema

     


    ” Sacred Place ” , R.Towner / P. Fresu, live in Innsbruck , Part 4

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.