Category Archives: AOP

Use AOP with RxJS to make functions Observable

There are no Observable create functions that will observe function invocations on an Object. As a proof of concept I used a JavaScript AOP library, Meld, to accomplish this “function to stream” capability.

Alternatives
This sounds like a job for a mixin, proxy, Object.observe, inheritance, or decorator pattern. There are hundreds of mixin libraries, the new JavaScript Proxy is not readily available, Object.observe was cancelled, etc.
Edit: Via a Dan Abramov tweet (what a busy guy!), I found out about MobX.

Aspect-Oriented Programming (AOP)
AOP is paradigm that allows the separation of cross-cutting concerns via applying “advices” to objects. There are many advice types. The Meld library supports several, for example, ‘before’, ‘around’, ‘on’, ‘after’, ‘afterReturning’, and ‘afterThowing’.

We’ll just use two here: “after” to observe a function invocation, and “around” to observe a property change.

Observe of function
In this example object:

var myObject = {
   myProperty:'15',		

   doSideEffect: function(x){this.myProperty = x;},		

   doSomething: function(a, b) {return a + b;}
};

We want to subscribe to the ‘doSomething’ function in this object. That is, when that function is invoked, the resulting return value should be sent to an Observable stream.

The stream:

 
var subject = new Rx.Subject();
    
var subjectAsObservable = subject.map(function(x){
       console.log("in map x:" + x);
       return x;
});
    
var subscription = subjectAsObservable.subscribe(
       function(x){ // onNext
           console.log("Next in subject: " + x.toString());
       }	
);

Now to apply an After advice to that function “joinpoint” we do:

 
var remover = 
  doAfter(myObject, 'doSomething', subject, 
    function(result) {
        console.log("after function .... ");
    }
);

myObject.doSomething(1, 2); 
remover.remove();

The output of running this is:

 
in map x:3
main.js:3
after function .... 

Note that “When adding advice to object methods, meld returns an object with a remove() method that can be used to remove the advice that was just added.”

“doAfter” function is a simple function wrapper of the ‘after‘ Meld function:

 
/**
 * @param anObject the object to advise
 * @param method the method selector
 * @param subject the subject that observes the function
 * @param afterFunction function to invoke after the advised function returns
 * @return object with remove() method
function doAfter(anObject, method, subject, afterFunction){
    return meld.after(anObject, method, function(result){
    	subject.onNext(result);
    	afterFunction(result);
    });
}

Observe of property change
Some AOP libraries support get/set advices on property changes. AspectJ in Java world supports this. Meld does not, but this can be implemented using an Around advice. This type of advice wraps a function call.

With an Around advice we can record a properties’ value before and after a function invocation. If there is a change, we generate a new event for the Observable stream.

 
remover = meld.around(myObject, getMethodNames, function(methodCall){
    var originalValue = myObject.myProperty;
    
    var result = methodCall.proceed();
    if(originalValue != myObject.myProperty){
        subject.onNext("prop changed: " + originalValue + " -> " + myObject.myProperty);
    }
    
    return result;
});

myObject.doSideEffect('25');

remover.remove();

The above uses a utillity function:

/** Utility function to get all functions in object */
function getMethodNames(obj){
   return Object.getOwnPropertyNames(obj)
        .filter(function(name){
	    return typeof obj[name] === 'function';
        });		
}

When any method in object is invoked that changes the property, we get:

in map x:prop changed: 15 -> 25
Next in subject: prop changed: 15 -> 25

Of course, there is a performance cost to this approach of monitoring property changes. This costs has not been quantified or explored.

Summary
Using AOP an object’s functions can be made into observers. An “after” advice can signal a result event, and using an “around” advice, it is possible to generate events on an Object’s properties. Useful? Unknown. Already possible in ReactiveX ecosystem, not sure.

Possible operators for the above could be:
fromFunction(object, methodSelector)
fromProperty(object, propertySelector)
fromProperty(object, methodSelector, propertySelector).

TODO
Look at the Rx.Observable.fromCallback(func, [context], [selector]) operator. Could it be used to go from function to Observable stream without using AOP, as presented here?

Software used
RxJS: version 4.0.7, rx.all.compat.js
Cujojs Meld: version 1.3.1,

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Use AOP Aspects as Mocks in JUnit tests?

This post illustrates a subtle relationship between Mock Objects and Aspect Oriented Programming. Some code examples are shown, and ends with questions for further research. This is a follow on to a prior “Using AspectJ For Testing Legacy Code“. Author: Josef Betancourt

Use AOP Aspects as Mocks in JUnit tests?

Written by:
Josef Betancourt
Date:
2013-05-29
Subject:
AOP vs Mocks for testing

Categories and Subject Descriptors
D.2.2 [Software Engineering]: Design Tools and Techniques, Object-Oriented programming

Keywords
interceptors, AOP, Mock Objects, JMockit, JUnit, Aspectj, unit test

Introduction
In Unit testing during code development and maintenance, the smallest unit of code is tested in isolation from collaborating units or subsystems. In well designed systems, testing is much simpler: outgoing interfaces are easier to manage. In legacy systems and/or badly written code, that may not be the case. Further, when testing legacy systems, changes to increase testability are not always possible.

Thus, various patterns, frameworks, and tools are used to support this isolation. Mock Object frameworks are ideal for this. Aspect Oriented Programming (AOP) is also capable of providing this isolation but is rarely mentioned. What are the differences? Is one better than another for testing?

Aspect Oriented Programming
See the wikipedia entry for more information.
In the Java world, AspectJ is the most well-known AOP implementation.

Mock Objects
Mock Objects are simulated objects that mimic the behavior of real objects in controlled ways.

The JMockit toolkit is a modern Java mocking toolkit. One distinguishing feature of JMockit is that it is powerful enough to Mock legacy unmockable code.

Example Project
A Client class uses a Service object’s query method to get a user’s name: Client.getUserName(int) –> Service.query(int).

For the test we want to substitute the name for a specific argument. When invoked with argument integer 1, the result is the string “second”, but in the test we want to return “Hello world!”. This example is just to show the technology; a hello world app, not practical in itself.

Advice.
To intercept a method call on an object in AOP an Aspect is created.

“Aspects wrap up pointcuts, advice, and inter-type declarations in a modular unit of crosscutting implementation.”
 

The Advice below is written in the annotation based AspectJ syntax introduced in AspectJ 5.

@Aspect
static class TestAspect {
		
	@Pointcut("call( * AspectJUnit.Service.query(int)) && args(i)")		
	void intercept(int i){}
		
	@Around("intercept(i)")
	public String query(ProceedingJoinPoint tjp, int i){
	    return (i == 1) ? "Hello world!"
		: (String) tjp.proceed(new Object[]{i});
	}		
	
}
 

This defines a method that will ‘replace’ the original target. The terminology used by the AspectJ programming manual:

• A Join Point is a well-defined point in the program flow.
• A Pointcut picks out certain join points and values at those points.
• An Advice is code that is executed at a join point.
 

AspectJ defines many Join Points. There is even a Handler execution join point: “When an exception handler executes. Handler execution join points are considered to have one argument, the exception being handled”.

The pointcut language is very rich and the signature patterns can be generic or very specific.

We can also use an anonymous pointcut instead:

@Aspect
static class TestAspect {
		
	@Around("call( * AspectJUnit.Service.query(int)) && args(i)")
	public String doQuery(ProceedingJoinPoint jp, int i){
	    return (i == 1) ? "Hello world!"
		: (String) jp.proceed(new Object[]{i});
	}		
	
}
 

Note that the advice, here doQuery(..), does not have to match the original target method’s name.

Mocking
In JMockit‘s State Based API an object can be mocked by using a MockUp class. Within that class any method that needs to be mocked is re-implemented and annotated with @Mock.

Since JMockit allows even private, final, or static methods to be overridden, this is not normal Java method overriding.

class MockService extends MockUp<Service> {
	@Mock
	public String query(Invocation invocation, int n) {
		return (n == 1) ? "Hello world!"
			: ((Service) invocation.
			getInvokedInstance()).query(n);
	}			
}
 

We can also define the mock using an anonymous inline class within the test method:

new MockUp<Service>() {				
        @Mock
	public String query(Invocation invocation, int n) {
	     return (n == 1) ? "Hello world!" 
              : ((Service) invocation. 
                getInvokedInstance()).query(n);
	}
};
 


Comparison
Structure and Syntax
Compare this to the prior Aspect. They are both a class definition. The new behavior is also a method. The difference is the pointcut. In JMockit the pointcut and the advice specification are combined. Note that the method signature must match the target method’s. In the AspectJ Aspect, these are separate.

@Aspect
static class TestAspect {
		
	@Around("call( * AspectJUnit.Service.query(int)) && args(i)")
	public String doQuery(ProceedingJoinPoint jp, int i){
	    return (i == 1) ? "Hello world!"
		: (String) jp.proceed(new Object[]{i});
	}		
	
}
class MockService extends MockUp<Service> {

	@Mock
	public String query(Invocation invocation, int n) {
		return (n == 1) ? "Hello world!"
			: ((Service) invocation.
			getInvokedInstance()).query(n);
	}			
}
 

The combined syntax in the Mock approach is simpler, and since a test is usually targeted at one specific method, the extra AOP pointcut expressiveness may have limited use. The Mock class can even be created in the test inline. A paper referenced below even makes the suggestion that test based pointcut definitions are an improvement to AOP since they are less fragile.

The aspect must be created as a static class and the associated advice is instrumented at compile time. The mock is integrated in the JUnit run time so it’s ‘advice’ is active only for the current test, thus, you have more opportunities for various tests of the same target. The full source code of the test using Aspects does not have as many assertions as the one written using Mocks because of this early instrumentation. See section below for listings.

There are more differences. These are of course due to the intended application of these two technologies. AOP is a more general purpose systems tool. Crosscutting concerns are an architectural artifact whereas isolation is a testing concern.

Mocks are a subset of Aspects
AOP is optimized for cross-cutting concerns. But, collaborator isolation is just an example of a single-use test-based cross-cutting concern. So Mocks are just a special use of general AOP capabilities.Mocks supply a single joinpoint, and in many implementations allow the use of an Around advice. This has been sufficient and with other features such as Expectations, the Mock Objects framework fits well into current testing approaches.

In AOP there is a richer language for selecting different kinds of joinpoints. And the pointcuts are used by different kinds of advice, such as cflow, before, after, around, and so forth. The modularization (inheritance and other features) into Aspects presents an opportunity for more high-level syntax and use, especially with the annotation based language.

Can Mock Objects use AOP techniques?
This begs the question of whether Mock Objects can or should use generic AOP techniques? If so how?

One example of the integration of AspectJ is the Spring framework, which now can use the AspectJ pointcut language in its own AOP implementation. Spring’s implementation is not specifically targeted as a test solution.

Another technology is the use of a language that can create “advice” rules and be invoked via instrumentation. This is available in Byteman. Byteman uses Event Condition Action (ECA) rules.

More recent versions of JMockit now supports a ‘wildcard’ mock method: ‘”Object $advice(Invocation)”, which if defined will match every method in the mocked class hierarchy.’ — mockit Annotation Type Mock

If AOP pointcuts do have a use-case in testing mocks, could this be an area where a pointcut DSL could be introduced in future?

 

Aspects and Mocks shown in full listings

Example Mocks in a JUnit test
package com.octodecillion.jmockit.example;

import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertThat;

import java.util.Arrays;
import java.util.List;

import mockit.Invocation;
import mockit.Mock;
import mockit.MockUp;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;

/**
 * mockit.Invocation class use example
 */
public class JMockItExample2 {

	/**  the client we are testing */
	public class Client {
		private Service service;
		
		public Client(Service service) {
			this.service = service;
		}
		
		/** invoke the 'expensive' service. */
		public String getUserName(int n){
			return service.query(n);			
		}
		
	}
	
	/** the service that does integration stuff */
	public class Service {
			/**
			 * @param key 
			 * @return 
			 */
			public String query(int n) {				
				return values.get(n);
			}
			
			private final List<String> values = 
				Arrays.asList("first","second","third");
	}
	
	/**
	 * Nested test class.
	 *
	 */
	@RunWith(JUnit4.class)
	public static class UnitTest{
		private JMockItExample2 example;
		
		@Before
		public void before(){
			example = new JMockItExample2();
		}
		
		/**   */
		@Test
		public void should_do_around_advice() {
			String expected = "second";
			String actual = example.new Service().query(1);
			assertThat(actual, is(expected));
			
			new MockUp<Service>() {				
				/**
				 * Use an Invocation to allow use of mocked object.
				 */
				@SuppressWarnings("unused")
				@Mock
				public String query(Invocation invocation, int n) {
					return (n == 1) ? "Hello world!"
						: ((Service) invocation.
						getInvokedInstance()).query(n);
				}
			};
			
			expected = "Hello world!";
			actual = example.new Service().query(1);
			assertThat(actual, is(expected));
			
			// do the around advice
			actual = example.new Client(example
					.new Service()).getUserName(1);
			
			assertThat(actual, is(expected));			

		}		

	}

}

 
Example Aspect in a JUnit test
/**  */
package com.octodecillion.jmockit.example;

import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertThat;

import java.util.Arrays;
import java.util.List;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;

/**
 * Using AspectJ in a JUnit test.
 * <p>
 * For compile issue in Eclipse: {@link http://stackoverflow.com/questions/8958267/java-lang-verifyerror-expecting-a-stackmap-frame}
 * @see http://stackoverflow.com/questions/8958267/java-lang-verifyerror-expecting-a-stackmap-frame
 * 
 */
public class AspectJUnit {

	/**  the client we are testing */
	public class Client {
		private Service service;
		
		public Client(Service service) {
			this.service = service;
		}
		
		/** invoke the 'expensive' service. */
		public String getUserName(int n){
			return service.query(n);			
		}
		
	}
	
	/** the service that does integration stuff */
	public class Service {
			/**
			 * @param key 
			 * @return 
			 */
			public String query(int n) {				
				return values.get(n);
			}
			
			private final List<String> values = 
				Arrays.asList("first","second","third");
	}
	
	@Aspect
	static class TestAspect {
		
		@Around("call( * AspectJUnit.Service.query(..)) && args(i)")
		public String doQuery(ProceedingJoinPoint tjp, int i){
			return (i == 1) ? "Hello world!"
				: (String) tjp.proceed(new Object[]{i});
		}		
		
	}
	
	/**
	 * Nested test class.
	 *
	 */
	@RunWith(JUnit4.class)
	public static class UnitTest{
		private AspectJUnit example;
		
		/**   */
		@Test
		public void should_do_around_advice() {
			String expected = "Hello world!";
			String actual = example.new Service().query(1);
			assertThat(actual, is(expected));
			
			actual = example.new Client(example
					.new Service()).getUserName(1);
			
			assertThat(actual, is(expected));			

		}
		
		@org.junit.Before
		public void setup(){
			example = new AspectJUnit();
		}

	}	

}

 

Related Reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Java testing using XStream, AspectJ, Vsdf

Problem
A unit or integration test may require the creation of complex object structures. This is even more problematic in legacy systems that were not created with testability in mind. Faced with this, it may seem better in the short term to avoid programmatic tests and continue to rely on traditional test techniques.

Solution
A possible short term approach is to use object serialization to capture the required objects as they are used in the actual system. Then a test can rebuild the objects and use them in various tests. We will not elaborate on the shortcomings of this approach.

XStream
Their are many approaches in Java that can be used for object serialization: JAXB, JavaBean serialization, and so forth. All of them have issues. For example, many of them require that the object to be serialized conform to certain requirements, like the JavaBean specification. If the objects don’t there are ways around it, but this quickly becomes complex, not only must the top level object be ‘handled’ but then its nested object graph. The XStream library does not have these requirements and so can probably handle a large percent of the use cases.

Example Test
In listing 1 below is a unit test using XStream. A class called XApp contains a scenario1 method that we wish to test. To invoke scenario1, a lot of data must be created and structured into a required object hierarchy, here we just use a simple map to represent that. Thus, the test creates an object from an XStream XML serialization and then invokes the scenario1() method of the system under test.

Listing 1, test
package com.octodecillion.testing;

import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertThat;

import java.io.File;
import java.io.IOException;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;

import com.octodecillion.testing.XApp.Inventory;
import com.thoughtworks.xstream.XStream;
import com.thoughtworks.xstream.io.xml.StaxDriver;

/**
 * @author jbetancourt
 *
 */
@RunWith(JUnit4.class)
public class InventoryTest {
	/**
	 * @throws IOException 
	 * 
	 */
	@SuppressWarnings("boxing")
	@Test
	public void scenario1_test() throws IOException {
		Inventory inventory = (Inventory) new XStream(new StaxDriver())
				.fromXML(new File(
				"inventory-scenario1.xml"));

		XApp app = new XApp();
		app.setInventory(inventory).scenario1();
		assertThat(app.getInventory().stock.size(), is(1));
	}

}

AOP
Since there are no mappings or modifications to objects, serializing with XStream takes two lines of code. We can insert this source code (or via a utility) where we need it in the system under test to capture a state for future testing. However, this requires that we litter the code with these serialization concerns. We can remove them after we capture the object, of course.

An alternative is to use Aspect Oriented Programming. With AOP we can ‘advise’ at ‘joinpoints’ to capture objects. This can be done with Load-Time Weaving. The original source is unmodified and to recapture the same objects we just rerun the system and reapply AOP LTW.

In listing 2, the AspectJ AOP Java framework is used to create an aspect to capture the data object at the setScenarioData() method in the class under test, XApp.

Listing 2, object capture
package com.octodecillion.testing;

import java.io.File;
import java.io.FileWriter;

import com.thoughtworks.xstream.XStream;
import com.thoughtworks.xstream.io.xml.StaxDriver;

/**
 * Aspect to allow XStream streaming of application data object.
 * 
 * @author jbetancourt
 *
 */
public aspect XStreamObject {
	/**
	 * Capture the App inventory object during creation of data 
	 * and stream to file.
	 * 
	 */
	@SuppressWarnings("javadoc")
	after(final XApp app) : 
		execution( protected void setScenarioData()) && 
		!within(XStreamObject) && target(app) {
		
		try {
			FileWriter writer = new FileWriter(
					new File("inventory-scenario1.xml"));
			new XStream(new StaxDriver()).toXML(app.getInventory(), writer);			
			writer.close();			
		} catch (Exception e) {
			e.printStackTrace();
		}		
	}	
}

Storing using Vsdf
One problem with using serialization for tests is where to store these. If we have a complex app, which we do or else why go through all this, there can be a lot of objects and these can be in various states depending on the test scenario requirements.

In prior posts Simple Java Data File and very simple data file I presented a concept for just this scenario. With Vsdf, multiple serializations can be stored in one file. Thus we can store all streamed object in one file, or various serialized states of an object type can be stored per Vsdf file. Some advantages of this approach is the reduction in test files, ability to change or update individual serializations.

Example Class to test
In listing 3 below is the example class to test. It is more of a “Hello World” type of app.

Listing 3, target class to test
package com.octodecillion.testing;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

/**
 * Simple XStream example.
 * 
 * @author jbetancourt
 *
 */
public class XApp {

	private Inventory inventory;
	
	/**
	 * The scenario that needs testing.
	 */
	public void scenario1() {
		// do the business case here ...
	}

	/** default constructor */
	public XApp() {
		setScenarioData();
	}

	/**
	 * The Application entry point.
	 * @param args
	 * @throws IOException 
	 */
	public static void main(final String[] args) throws IOException {
		new XApp();
	}

	/**
	 * set up the data for scenario1
	 */
	protected void setScenarioData() {
		inventory = new Inventory();
		inventory.add("322", new Vehicle() {
			@Override
			public int getNumWheels() {
				return 4;
			}

		});
	}

	/**
	 * @author jbetancourt
	 *
	 */
	public class Inventory {
		/**		 */
		Map<String, Vehicle> stock = new HashMap<String, Vehicle>();

		/**
		 * @param id
		 * @param veh
		 */
		public void add(final String id, final Vehicle veh) {
			stock.put(id, veh);
		}

	} // end class Inventory

	/**
	 * @author jbetancourt
	 *
	 */
	public interface Vehicle {

		/**
		 * @return num of wheels
		 */
		public int getNumWheels();

	} // end Vehicle

	/**
	 * @param inventory
	 * @return the app object
	 */
	public XApp setInventory(final Inventory inventory) {
		this.inventory = inventory;
		return this;
	}

	/**
	 * @return the inventory
	 */
	public Inventory getInventory() {
		return inventory;
	}

} // end class XApp

Issues
Sounds great but what happens when a class that was serialized is modified, like gets new fields? How do we handle versioning and so forth?

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Java JMockIt mocks via Spring DI

How to use Dependency Injected mock objects to allow Integration Testing of a Java application.

When working with some legacy code bases, the introduction of Dependency Injection has limitations. Thus, various forms of ‘instrumentation’ will be required to reach the “last mile”. A major difficulty with legacy code is instantiation of objects with the “new” operator.

One form of instrumentation is a modern Mocking framework, like JMockIt, another is the use of AOP with, for example, AspectJ.

As part of an evaluation of a possible approach I looked into using DI for data driven Integration Testing. Is it possible to use declarative specification of JMockIt mock objects? That is, can they be specified in a Spring XML bean configuration file and loaded via Spring Framework DI? This is in lieu of Using AspectJ to dependency inject domain objects with Spring or some other framework.

Short answer, yes. Useful technique? Probably not. But …

I was able to solve this by looking at the JMockIt source code for the mockit.internal.startup.JMockitInitialization class.

From JMockIt source repository:

final class JMockitInitialization
{
  ... snipped ...
  private void setUpStartupMocksIfAny()
   {
      for (String mockClassName : config.mockClasses) {
         Class<?> mockClass = ClassLoad.loadClass(mockClassName);

         //noinspection UnnecessaryFullyQualifiedName
         if (mockit.MockUp.class.isAssignableFrom(mockClass)) {
            ConstructorReflection.newInstanceUsingDefaultConstructor(mockClass);
         }
         else {
            new MockClassSetup(mockClass).setUpStartupMock();
         }
      }
   }
  ... snipped ...
}

See the “Using mocks and stubs over entire test classes and suites” in the tutorial for further info.

Turned out to be easy to create a Spring XML configuration and a simple mocked “hello world” proof of concept. The web server was instrumented dynamically!

Updates
One thing I could not figure out was how to destroy existing Mocks that were created in the Tomcat app server. Calling Mockit.tearDownMocks() had no effect. Admittedly, JMockIt is for use in JUnit based tests, so this may be pushing it.

Further reading

  1. Hermetic Servers This shows that the above idea is useful and not unique.
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Exception verification using fault injection via AOP

A simple example is used to show how to use Aspect Oriented Programming to provide fault injection.

Intro

A few months ago I was looking at Java code and thinking there must be a better way to see if this will really work, if there are any issues in how the program would react to exceptional conditions.  I could write some unit tests or force some problems to see if the exception handling is correct and the log file output is really useful.  But is there another way?

I got a solution after a while of thinking: Insert faults.  Now how to do that in an easy and  maintainable way? Not a new idea of course. After doing a web search I found some interesting references.

Funny, today at work I was extending an application and one of the support objects was failing. Turned out to be an index out of bounds problem. The method had adequate exception handling but, of course, an index out of bounds scenario was not tested. In this case the problem would not occur in production (hmmm), but it definitely occurred during development, and was not handled correctly.

Exception Handling

There are many references of correct exception handling strategies and best practices.  However, when creating applications, the developer must still make critical creative decisions about how to handle an exception.    That decision may not be a wise one.  And even if it is, may later prove to have been very unwise since the system requirements and implementation may have changed due to maintenance or requirement evolution.

Testing Approaches

Using various testing methodologies a subsystem can be exhaustively tested for exception handling.  Yet, most testing methods require that the subsystems be isolated in some manner.   This, though very valuable, can give fatal false confidence that the system will behave in a predictable manner in responding to exceptional conditions.

The prime example of this is within the Unit Testing methodologies.  Since Unit Tests are tests of isolated components, they do not test actual exception handling in the live system.  A perfect component can still contribute to system failure when it is used incorrectly or uses other components incorrectly.    For example, a unit test can show that class X’s methods correctly handle and when necessary throw the perfect exception.   That means nothing if components that interact with X don’t correctly use those exceptions:  they swallow, mask, or mistakenly catch them when there is nothing they can do.

Thus, one needs to use Integration Testing to test assemblages of components.  But, we are still back with the same shortcoming as with Unit Testing. Thus we would next need various forms of system testing.  And, that system testing must be a White Box test.  A Black Box test wherein an interface is exercised with various inputs (such as in web test systems or by QA personnel), will not necessarily invoke the full internal API/SPI between components that could be part of a complicated call chain.  Why?  Because, the interface, such as a browser client, will have (we hope) the validation, security, and logging layers implemented so that by the time a system test workflow enters the deep system components within the client or middleware, there is no way to influence the target component state into programmatic intentional failure mode.

If there is no way to exercise a run time trajectory, then why bother? The reason is that exceptional conditions are exceptional. Resources can be exhausted, systems not available, and so forth. The rule of Fail Fast may not be enough if collaborating components are not exceptionally responsive.

Fault Injection

To see if a system is responding to exceptional conditions, one must wait for those conditions or create them.    Analogously to Fuzz Testing we can dynamically and randomly insert faults.  In Fuzz Testing, inputs are manipulated to a system under test.  In Fault Injection we have to manipulate inputs inside the real system, within the components themselves.  And as in Fuzz Testing, we can employ various strategies to do so, such as randomized faults, etc.   Of course, this system is not the deployed production system, it is the real system in a test environment, an internal full deployment.

Some approaches are: source modification, source macros, annotations, dependency injection (IOC), and Aspect Oriented Programming (AOP). Of course, there are many other techniques as found in open/closed projects, commercial products, and the research community.

Fault Injection using AOP

Since using the real system is required, a viable approach is to use the Instrumentation already available in the Java system. We could dynamically insert code that forces exceptions. AOP as implemented by the AspectJ language can already do this.

The major advantage of using AOP is that the source is not changed in any way. AspectJ is the state of the art in the Java ecosystem for using AOP.

Other approaches

Monitoring

BTrace (a Java oriented approach comparable to DTrace) is very interesting. However, since, like DTrace, it is meant to be safe, defect injection may not be doable. But, see this post on an unsafe mode for BTrace.

Source modification

One way of injecting failure is just to edit the source code and add strategic code that will cause the instigating failure.  An advanced form of this is Mutation Testing.   Code changes could be:  arguments with wrong values, nulls, deliberately thrown exceptions, etc.  For example, one can simply create a method that throws a runtime exception:


     *** DON'T DO THIS ***

    public static void REMOVE_XXX_FROM_XXX_SOURCE(){
       if(true){
          throw new NullPointerException(
             "nt****** DELIBERATE RUNTIME EXCEPTION*****n");
       }
    }
    

Then insert invocations to this method at strategic points in the code, deploy to a test environment, and see if the expected results are obtained, such as logging output that can identify the cause, or that no side effects are recorded, such as incorrect data storage.

This is a low tech and error prone approach.  Most importantly, this is dangerous.  In the mad development rush to meet deadlines and also eat lunch, this code could make it into production!  Even with that horrific method name, it will wind up in production!  One would have to create deployment filters that stop the process if any “fault” inducing code is included.  Another reason why this is not a good approach is that it doesn’t scale very well.   Forcing exceptions is just one type of verification.  Another is setting values outside of expected ranges or states.  Thus, one would need many different kinds of source code insertions. 

Of course, Unit and Behavior based testing tools sets can supply these required verifications if included into the system as Built-In Self-Tests (BIST).

Built-In Self-Test

In the hardware realm, BIST as found in, for example, IEEE 1149.1 JTAG, has been very successful. On the software side, there is ongoing research on how to implement BIST-like capability. This would make Brad Cox’s concept of the “Software IC” even more powerful.

Macros

A somewhat viable approach to source code fault insertion is instead of inserting the faulting code,  insert “include” macros in the original source code that indicates what fault insertion should be done at a location.  A fault injection preprocessor can scan and insert the fault framework’s invocations to accomplish the requirement.  The source code build process can then simply enable or disable the use of the preprocessor.  This would however still require source code modification and an extra maintenance nightmare.  When the code changes the macros may also require change.

Annotations

Instead of macros, we could also use Annotations.  Annotations could explicitly state the runtime exceptional behavior “service level agreements” at the component level. These SLA could then be systematically manipulated to test if they really hold at the system level.

Dependency Injection

One can also inject exceptional behavior by dependency injection, using programmatic, declarative, or annotations, faulted components could be inserted (or created via Aspect Oriented Programming) into the target system.

AOP Demonstration

In listing one below, an ATM class uses a service to do some banking.

/**
 * Driver class for example. 
 * For a real system see:  https://bitbucket.org/aragost/javahg/
 * @author jbetancourt
 */
public class ATM {
	private BankService service = new BankService(1000);

	/** 
	 * Perform a banking action.
	 */
	public void transaction(){	
		{
			service.deposit(100L);
			service.withdraw(200L);
		}
		
		service.statement();		
	}
	
	/** Application entry point	 */
	public static void main(String[] args) {
		new ATM().transaction();
	}
}

Listing two is the BankService being used. Of course, a real service would be more complex.

/**
 * Example service.
 */
public class BankService {
	// Use Long to allow example's null for failure
	// injection.

	private BigDecimal balance;
	
	public BankService(long savings){
		this.balance = new BigDecimal(savings);
	}
	
	/** */
	public void deposit(Long amount){
		balance = balance.add(new BigDecimal(amount));
		System.out.println("Deposit:  " + amount);
	}
	
	/** */
	public void withdraw(Long amount){
		balance = balance.subtract(new BigDecimal(amount));
		System.out.println("Withdraw: " + amount);
	}
	
	/** */
	public void statement() {
		System.out.println("Balance:  " + balance);		
	}	
	
} // end BankService

A example Non-faulted execution result is:

Report: pointcut usage
  withdrawCut:   true ->  false
  depositCut:  false ->  false
       Deposit: 100
      Withdraw: 200
       Savings: 900

When we run the demo program we get:

Exception at the withdraw method.

Report: pointcut usage
withdrawCut:   true ->   true
depositCut:  false ->  false
Deposit: 100
Exception in thread "main" java.lang.NullPointerException
   at BankService.withdraw_aroundBody2(BankService.java:15
   at BankService.withdraw_aroundBody3$advice(BankService.java:81)
   at BankService.withdraw(BankService.java:1)
   at Main.main(Main.java:16)

Exception at the deposit method.

Report: 
pointcut usage
withdrawCut:   true ->  false
depositCut:  false ->   true
Exception in thread "main" java.lang.NullPointerException
   at BankService.deposit_aroundBody0(BankService.java:9)
   at BankService.deposit_aroundBody1$advice(BankService.java:81)
   at BankService.deposit(BankService.java:1)
   at Main.main(Main.java:15)

All pointcuts enabled.

Report: 
Report: 
pointcut usage
withdrawCut:   true ->  false
depositCut:  false ->   true
Exception in thread "main" java.lang.NullPointerException
   at BankService.deposit_aroundBody0(BankService.java:9)
   at BankService.deposit_aroundBody1$advice(BankService.java:81)
   at BankService.deposit(BankService.java:1)
   at Main.main(Main.java:15)

Implementation

Now to test the exceptional behavior we want to fault the deposit and withdraw methods.
First we have a way of specifying what we want to fault by using a JSON configuration file:

{
	"__comments":[
		"File: properties.json"
	],
	"settings":{
		"random":true
	},
	
	"cuts":{
		"depositCut":false,
		"withdrawCut":true
	}	
}

The “cuts” settings indicate which AspectJ “pointcut” to turn on. The “random” setting indicates if we want the exceptions to be randomly inserted into the code base at the enabled pointcuts.

The Abstract failure injection aspect. Subclasses (aspects) will supply the actual pointcuts to specify where to ‘do’ the injection.

/**
 * FailureInjectionAspect.aj 
 * @author jbetancourt
 * 
 */

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

/**
 * Abstract Aspect that determines if failure should occur.
 * @author jbetancourt
 * 
 */
public abstract aspect FailureInjectionAspect {
	private Map<String, Boolean> pointcutFlags = new HashMap<String, Boolean>();
	private volatile boolean initialized =false;
	
	/** 
	 * Initialize Failure injection aspect.
	 * @throws Exception 
	 * 
	 */
	protected void init() throws Exception {
		Config config = new Config();
		config.configure();
		pointcutFlags = config.getPointcutFlags();		
		initialized = true;
	}	

	/**
	 * Get boolean value for pointcut name.
	 * 
	 * @param name pointcut name.
	 * @return true if pointcut enabled.
	 * @throws IOException 
	 */
	protected boolean isPointcutEnabled(String name) {
		if(!initialized){
			try {
				init();
			} catch (Exception e) {
				throw new IllegalStateException(
						"Could not initialize object",e);
			}
		}
		
		boolean f = false;	
		Object val = pointcutFlags.get(name);
		if (null != val) {
			f = ((Boolean) val).booleanValue();
		}	

		return f;
	}
	
} // end FailureInjectionAspect.aj

Here is a aspect that uses a nulling type of injection.

/**
 * 
 * Example of an aspect for nulling an argument to a method.
 * @author jbetancourt
 *
 */
public aspect AmountFailureAspect extends FailureInjectionAspect {

	/** fault the deposit */
	private pointcut depositCut(Long amount) :
		execution(public void BankService.deposit(Long)) 
		&& if(AmountFailureAspect.aspectOf().isPointcutEnabled("depositCut"))
		&& args(amount) 
		&& !within(FailureInjectionAspect)
	;

	/** fault the withdrawal */
	private pointcut withdrawCut(Long amount) :
		execution(public void BankService.withdraw(Long)) 
		&& if(AmountFailureAspect.aspectOf().isPointcutEnabled("withdrawCut"))
		&& args(amount) 
		&& !within(FailureInjectionAspect)
	;

	/** Null the amount arg */
	void around(Long amount) : depositCut(amount)|| withdrawCut(amount) {
		amount = null;
		proceed(amount);
	}
}

Here is how to read the configuation JSON file. I use the JSON-Simple library.

import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.util.HashMap;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;

import org.json.simple.JSONObject;
import org.json.simple.JSONValue;

/**
 * 
 * @author jbetancourt
 *
 */
public class Config {
	private static final Logger logger = 
          Logger.getLogger(Config.class.getName());
	private String configFile = "srcproperties.json";
	private Map<String, Boolean> pointcutFlags = new HashMap<String, Boolean>();
	private volatile boolean initialized = false;
	private String basePath;

	/**
	 * 
	 * @param configFile2 
	 * @throws Exception
	 */
	@SuppressWarnings("unchecked")
	public void configure() throws Exception {
		basePath = new File(".").getAbsolutePath();

		String path = basePath + configFile;

		Object obj = (JSONObject) JSONValue
				.parse(new FileReader(new File(path)));
		
		Map<String, Map<String, ?>> map = (Map<String, Map<String, ?>>) obj;
		pointcutFlags = (Map<String, Boolean>) map.get("cuts");
		Map<String, Object> settings = (Map<String, Object>) map
				.get("settings");
		
		Object r = settings.get("random");
		boolean randomize = (r != null &amp;&amp; ((Boolean)r)) ;
		
		if(randomize){
			println(String.format(
			    "Pointcut usagen%16s %6s    %6s","Name        ",
			    "Prior","Current"));
			
			for(Map.Entry<String, Boolean>entry : pointcutFlags.entrySet()){
				Boolean prior = entry.getValue();
				Boolean f = Math.random() > 0.65;
				entry.setValue(f);
				println(String.format(
					"%15s: %6s -> %6s", entry.getKey(),prior,f));
			}
			println("n");		
		}		
		
		saveConfig();		
		initialized = true;		
		logger.log(Level.INFO,"Initialized: " + initialized);
	}
	
	/**
	 * Write the config settings to external JSON file.
	 * 
	 * Format is: { cutname : boolean , ... }
	 *   Example: {"withdrawCut":true,"depositCut":true}
	 * Will be used to set up behavior analysis service to 
	 * monitor response.
	 * 
	 * @throws IOException
	 */
	protected void saveConfig() throws IOException{
		String json = JSONValue.toJSONString(pointcutFlags);
		File file = new File(basePath + "runtimeProperties.json");
		FileWriter writer = new FileWriter(file);		
		writer.append(json);		
		writer.close();		
	}	

	/** getter */
	public Map<String, Boolean> getPointcutFlags() {
		return pointcutFlags;
	}

	/** setter */
	public void setPointcutFlags(Map<String, Boolean> cutFlags) {
		this.pointcutFlags = cutFlags;
	}

	/**
	 * Just a shortcut to System.out.println(String).
	 * @param s
	 */
	private void println(String s){
		System.out.println(s);
	}
}

Running at the command line

Using AspectJ is much easier in a supporting IDE like Eclipse. Below is the “mess”, unless you love the CLI, of compiling and running in a command shell. Could be made clean by creating aliases, scripts, etc.

cd src
src>javac -d ..bin -cp ..jarsjson-simple-1.1.jar  *.java
src>java -cp ..bin Main
Jun 9, 2011 3:18:24 PM Main main
INFO: Starting Main.....
   Deposit:  100
   Withdraw: 200
   Balance:  900

Now we change the flag from false to true in the runtimeInjection.json file:

src>type runtimeInjection.json | sed "s/false/true/" > temp
src>copy /Y temp runtimeInjection.json
src>del temp

Compile the aspects using the Aspectj “ajc” compiler.  Here we use 1.6 compliance level; destination of output to ..bin folder; and give the classpath.

c:\Users\jbetancourt\Documents\projects\dev\AspectsForNullTesting\src>java\aspectj1.6\bin\ajc -1.6 -d ..\bin -cp "c:\java\aspectj1.6\lib\aspectjrt.
jar;c:\javaaspectj1.6\lib\aspectjtools.jar;c:\javaaspectj1.6\lib\aspectjweaver.jar;..\jars\json-simple-1.1.jar" -sourceroots .

Now we run the same Main program.  Since the pointcut flag is true, the advise is invoked and the list argument to the service(List) is set to null.

c:UsersjbetancourtDocumentsprojectsdevAspectsForNullTesting>java -cp "c:javaaspectj1.6libaspectjrt.jar;c:javaaspectj1.6libaspectjtools
.jar;c:javaaspectj1.6libaspectjweaver.jar;jarsjson-simple-1.1.jar;bin;." Main
Jun 9, 2011 3:25:04 PM Main main
INFO: Starting Main.....
Pointcut usage
Name          Prior    Current
withdrawCut:   true ->  false
depositCut:  false ->  false
c:UsersjbetancourtDocumentsprojectsdevAspectsForNullTesting>java -cp "c:javaaspectj1.6libaspectjrt.jar;c:javaaspectj1.6libaspectjtools
.jar;c:javaaspectj1.6libaspectjweaver.jar;jarsjson-simple-1.1.jar;bin;."   Main
Jun 9, 2011 3:25:10 PM Main main
INFO: Starting Main.....
Pointcut usage
Name          Prior    Current
withdrawCut:   true ->   true
depositCut:  false ->   true
Jun 9, 2011 3:25:10 PM Config exec
INFO: Initialized: true
Exception in thread "main" java.lang.NullPointerException
at BankService.deposit_aroundBody0(BankService.java:19)
at BankService.deposit_aroundBody1$advice(BankService.java:26)
at BankService.deposit(BankService.java:1)
at Main.main(Main.java:16)

c:UsersjbetancourtDocumentsprojectsdevAspectsForNullTesting>java -cp "c:javaaspectj1.6libaspectjrt.jar;c:javaaspectj1.6libaspectjtools
.jar;c:javaaspectj1.6libaspectjweaver.jar;jarsjson-simple-1.1.jar;bin;."   Main
Jun 9, 2011 3:25:13 PM Main main
INFO: Starting Main.....
Pointcut usage
Name          Prior    Current
withdrawCut:   true ->   true
depositCut:  false ->  false
Jun 9, 2011 3:25:13 PM Config exec
INFO: Initialized: true
Deposit:  100
Exception in thread "main" java.lang.NullPointerException
at BankService.withdraw_aroundBody2(BankService.java:25)
at BankService.withdraw_aroundBody3$advice(BankService.java:26)
at BankService.withdraw(BankService.java:1)
at Main.main(Main.java:17)

Updates

  • Feb 1, 2012: Just learned about Byteman.
  • April 25, 2013: A new article on using Byteman: http://aredko.blogspot.com/2013/04/fault-injection-with-byteman-and-junit.html

Further Reading



Nicolas Lens – Sumus Vicinae (Flamma Flamma)

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Using AspectJ For Testing Legacy Code

Recently I brought up the topic of Aspect Oriented Programming (AOP) at work. I argued that it does have uses beyond mere tracing and psuedo-logging. In fact, it’s all over the place.

There is a more recent post regarding this topic: Use AOP Aspects as Mocks in JUnit tests?

Just to get familiar with it again I looked at how it could be used on a test scenario I had. A class under test directly uses a Singleton collaborating class to invoke a service call, in other words non-dependency injection. This is similar to a class using “new” to create objects it depends on.

A requirement of unit testing is the isolation of the class from its collaborators (to a pragmatic degree). Since the Singleton class has a private constructor it can’t be subclassed or proxied (even with cglib via Mock frameworks).

I created a simplified version of the scenario to show how “easy” it is to use AOP. First the class under test class in listing one:

public class ClassUnderTest {
	/**   	 */
	public void service(){
		Collaborator.getInstance().speak();
	}

	/**  	 */
	public static void main(String[] args) {
		ClassUnderTest main = new ClassUnderTest();
		main.service();
	}

} // end ClassUnderTest

And, the collaborator singleton class is in listing two:

public class Collaborator {
	private static final Collaborator instance = new Collaborator();

	/**   */
	private Collaborator(){
	}

	/**  */
	public static Collaborator getInstance(){
		return instance;
	}

	/**  */
	public void speak(){
		System.out.println(&quot;Hello!&quot;);
	}

} // end class Collaborator

The result of a compile and run is:

src>javac -d ..\bin ClassUnderTest.java Collaborator.java
src>cd ..\bin
bin>dir
bin>java ClassUnderTest
Hello!

Now, the scenario goal is to change the speak() method to instead output “Goodbye!”. The aspect in listing three accomplishes this. It defines two pointcuts, cut() that specifies the class under test, and service() that identifies the target speak() method call in the collaborator object. The around() advice allows us to substitute our own replacement ‘method’. In the ‘around’ advice, using proceed() would let the original method execute. Note the terminology used here is not Aspectively correct.

public aspect TestAspect {

	pointcut cut() :
		execution(public void ClassUnderTest.service()) &amp;&amp;
		!within(TestAspect);

	pointcut service() : cflow(cut())
	    &amp;&amp; call(public void Collaborator.speak());

	void around() : service() {
		System.out.println(&quot;Goodbye!&quot;);
		//proceed();
	}

} // end class TestAspect

Compiling:

.....src>javaaspectj1.6\bin\ajc -cp "c:\javaaspectj1.6\lib\aspectjrt.jar;c:javaaspectj1.6\lib\aspectjtools.jar;c:\javaaspectj1.6\lib\aspectjweaver.jar" -sourceroots .
Compiler took 958ms
 

Running:

....src>cd ..\bin

....bin\java -cp "c:\javaaspectj1.6\lib\aspectjrt.jar;c:\javaaspectj1.6\lib\aspectjtools.jar;c:\javaaspectj1.6\lib\aspectjweaver.jar;." ClassUnderTest
Goodbye!

Success!

Of course, our aspect in real life would not “do” the actual required test actions, but instead delegate to a test class or stub to do so. This is especially important if the collaborator object has many methods that must be advised. At first I was trying to do this directly, swapping out the use of the Singleton Collaborator class using static crosscutting, to another dynamically generated stub or mock class. Though with AspectJ you can do some powerful stuff, the private constructor puts a kink in all that.

Left to the reader is actually making this work within an actual Unit Testing framework like JUnit.

Updates

  • 30 Oct 2010:  Another approach which in many cases may be better is using the JMockit library.  It does work with real Singletons and other cases where AspectJ would be the only recourse.
  • 15 Dec 2010: If you can change the source code, a very simple test pattern is to extract the troublesome code lines into a method. Now that method can more easily be mocked.

Further Reading

  • Use AOP Aspects as Mocks in JUnit tests?
  • JMockit, http://code.google.com/p/jmockit/
  • Advanced Mocking: Capturing State with Answer and Captors“, http://canoo.com/blog/?p=1592#captcha_input
  • Aspect Oriented Software Development
  • Why use an AOP language when other languages can now do Aspects?”, Josef Betancourt, http://goo.gl/r1YB
  • Next steps with aspects“, http://www.ibm.com/developerworks/java/library/j-aopwork16/
  • Testing legacy code“, Elliotte Rusty Harold, http://www.ibm.com/developerworks/java/library/j-legacytest.html
  • AspectJ, http://eclipse.org/aspectj/
  • Nicholas Lesiecki, “Test flexibly with AspectJ and mock objects“, 01 May 2002, http://www.ibm.com/developerworks/java/library/j-aspectj2/
  • Approaches to Mocking“, Simon Stewart, http://onjava.com/pub/a/onjava/2004/02/11/mocks.html?page=1
  • Use AOP to maintain legacy Java applications“, http://www.ibm.com/developerworks/java/library/j-aopsc2.html
  • Static crosscutting“, http://eclipse.org/aspectj/doc/released/progguide/semantics-declare.html
  • Qi4j, http://www.qi4j.org/ Fascinating project and concepts, IMHO.
  • Russ Miles, AspectJ Cookbook, O’Reilly Media, Inc., 2005
  • AspectJ In Action, Ramnivas Laddad, Manning, http://manning.com/laddad2/
  • Multi-threaded Testing with AOP Is Easy, and It Finds Bugs!“, http://www.springerlink.com/content/c8m3g6v2cp49xcg6/
  • Making AspectJ development easier with AJDT“, http://www.infoq.com/articles/aspectj-with-ajdt
  • Bodden, Eric; Havelund,Klaus. “Racer: Effective Race Detection Using AspectJ“, http://www.havelund.com/Publications/issta2008.pdf
  • Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    Why use an AOP language when other languages can now do Aspects?

    Interesting research,  JOT: Journal of Object Technology – How AspectJ is Used: An Analysis of Eleven AspectJ Programs: Sven Apel, on how an AOP language, specifically AspectJ, is used in a few sample projects.  Was it useful, required?  The jury is out.

    Interestingly the author correctly mentions that some languages now have rudimentary support for handling of basic crosscutting concerns.   And, since the analyzed projects did not really need advanced AOP techniques,

    “numbers indicate that languages that provide only basic crosscutting mechanisms are appropriate to implement a large extent of the analyzed programs.”  

    Note that the study does not apply to container based AOP as used in frameworks such as JBossAOP and SpringAOP.  Several Dynamic Languages use a Metaobject Protocol.

    I used AspectJ for some utility and diagnostic help.  The language is fine and the concepts are approachable.  It’s the configuration, tools, and environment hassles that put a damper on things.   That’s why frameworks and tool support like that found in Eclipse are critical.

    Updates
    9/11/2010:
    Adding Logging around all of the Methods of a Class with Groovy“, Ted Naleid. http://naleid.com/blog/2010/09/11/adding-logging-around-all-of-the-methods-of-a-class-with-groovy/

    Further Reading

    JOT: Journal of Object Technology – How AspectJ is Used: An Analysis of Eleven AspectJ Programs, Sven Apel

    McClean, John, “Painless AOP with Groovy“, http://www.infoq.com/articles/aop-with-groovy

    Meta-object Protocol, http://en.wikipedia.org/wiki/Meta-object_protocol

    Dynamic Programming Language, http://en.wikipedia.org/wiki/Dynamic_language

    AspectJ, http://www.eclipse.org/aspectj/

    AJDT, http://www.eclipse.org/ajdt/

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.