Tag Archives: java

Prune deleted folders of repo diff report using Java

Sometimes the contents of deleted folders are not relevant. Just showing the top level deleted folder or dir is enough. For example, in a branch diff against trunk, folder moves are shown as adds and deletes. But in a diff report, the whole sub node of the deleted folder is output.

In the particular report I was creating everything under the deleted directories was just noise. How to prune that?

BTW, I searched the SVN docs and found no support for this kind of pruning. I also looked at the Git docs to see if it was different, but also found no mention of this use-case.

Example scenario:

d1 (modified)
├ d2 (deleted)
   ┝ d3 (deleted)

If we create a deleted prune report of the above, it would contain, d1 modified, and d1/d2 deleted. It would not contain d1/d2/d3.

The algorithm in psuedocode

Jan 8, 2015: I found a use-case where this algorithm does not work when used with an actual Subversion repository being accessed by SvnKit. Not due to SVNKit of course. Will update when fixed.

 

boolean deleting := false
String deletePath = “”

for each path{
    if deleting then 
            if path starts with deletePath then
                skip path
            else
                deleting = false
                deletePath = “”
                result.add(path)
            endif       
    else
        if path is directory and it is deleted then
            deleting = true
            deletePath = path
        endif
        
        result.add(path)
        
    endif
}

This can be implemented at the script level, for example, using Groovy or some other language.

SvnKit Scenario

In an Subversion Console application I wrote using ReactJS, I used SvnKit at the server side to interface to the Subversion server. Future blog post will discuss this single-page app. One of the React components was a Diff table.

Using the SvnKit library I generated an svn diff summary list. This is a list of SvnDiffStatus objects: List. Among other things, these object contain a path string of the respective resource.

To prune the excess deleted folders I first attempted to create an Object tree structure, then I gave up on that (too slow), finally a co-worker suggested I just sort the paths and do a simple contains comparison. Wow, how did I miss that? I guess I was sidetracked by “Objects” and didn’t use simple scripting techniques.

So to prune the folders we sort and prune the results of an SvnDiffSummary object, svnDiff::
List list = pruneDeletedFolders(sortDiffList(((Receiver)svnDiff.getReceiver()).getEntries()));

Where Receiver is an implementation of ISvnObjectReceiver.

(The SvnDiff entries may already be sorted, but the SvnKit API docs do not specify this).

Sort the diff list using Guava

	private List<SvnDiffStatus> sortDiffList(List<SvnDiffStatus> list) {
		return new Ordering<SvnDiffStatus>() {
			@Override
			public int compare(SvnDiffStatus left, SvnDiffStatus right) {
				return left.getPath().compareTo(right.getPath());
			}
		}.sortedCopy(list);
	}

prune the list

	private List<SvnDiffStatus> pruneDeletedFolders(List<SvnDiffStatus> list){
		String deletedPath = "";
		boolean isDeleting = false;
		
		List<SvnDiffStatus> prunedList = new ArrayList<>();

		for (Iterator<SvnDiffStatus> iterator = list.iterator(); iterator.hasNext();) {
			SvnDiffStatus diff = iterator.next();
			String path = diff.getPath();
			if (isDeleting) {
				if (path.startsWith(deletedPath)) {
					// skip this diff
				} else {
					isDeleting = false;
					deletedPath = "";
					prunedList.add(diff);
				}
			} else {
				if (isDirectory(diff) && isStatusDeleted(diff)) {
					isDeleting = true;
					deletedPath = path;
				}

				prunedList.add(diff);
			}
		}

		return prunedList;

	}

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Java dupe key Properties detect using Commons Configuration

Java Properties files can contain duplicate key entries. This can be a problem since the Java Properties loader will just keep the last value parsed.

Not a problem for small applications, but when property files grow in number and size this can become a maintenance issue. Fortunately there are many solutions for this (I hope). This post will present a very simple method that performs duplicate key detection.

When first confronted with dupe detection goal, reading the Properties file directly and determining dupes using a HashMap sounds simple. However, you would have to duplicate the Properties file format parsing. That can get complex. For example, a property entry may use “:” or “=” to specify the key, value pair. Plus, you would have to handle the line continuations feature, and other requirements.

Commons Configuration makes this simple since its Properties loader will create lists for any duplicate property key. Thus, to detect duplicates, just find which property values are lists, see listing 1 below.

Listing 1, Duplicate key detection using Commons Config Also available as a Gist

package com.octodecillion.util;

import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Properties;

import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.PropertiesConfiguration;
import org.junit.Assert;
import org.junit.Test;
import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;

/**
 * Detect duplicate properties when loading Properties from stream.
 * <p>
 * 
 * @author jbetancourt
 * @since
 */
public class DuplicatePropDetectWithCommonsConfig {

    /**
     * Detect duplicate keys in Properties resource.
     * <p>
     * <a href="">Commons Config</a> is used.
     * {@link Properties} class.
     * 
     * @see Properties#load(Reader)
     * 
     * @param reader The reader is NOT closed.
     * @param delimiterParsingDisabled 
     * @return Map of keys that are duplicated with a list value of each value of duplicate. 
     * @throws ConfigurationException 
     */
    @SuppressWarnings("unchecked")
    public Map<String, List<String>> loadWithConfig(Reader reader, boolean delimiterParsing) 
      throws ConfigurationException {
        PropertiesConfiguration config = new PropertiesConfiguration();
        final Map<String, List<String>> results = new HashMap<>();

        config.setDelimiterParsingDisabled(delimiterParsing);
        config.load(reader);

        Iterator<String> keys = config.getKeys();
        while (keys.hasNext()) {
            String theKey = keys.next();
            Object valueObject = config.getProperty(theKey);
            if (!(valueObject instanceof String)) {
                results.put(theKey, (List<String>) valueObject);
            }
        }

        return results;
    }
    
    /**
     * @author jbetancourt
     *
     */
    @RunWith(Enclosed.class)
    public static class DuplicatePropDetectWithCommonsConfigTest {
        private static final String DATA1_PROPERTIES = "/Data1.properties";

        /**
         * @throws ConfigurationException
         */
        @Test
        public void testLoadFileWithConfig() throws ConfigurationException {

            InputStream is = this.getClass().getResourceAsStream(DATA1_PROPERTIES);
            Assert.assertNotNull(is);

            DuplicatePropDetectWithCommonsConfig detector = new DuplicatePropDetectWithCommonsConfig();
            Map<String, List<String>> map = detector.loadWithConfig(new InputStreamReader(is), true);
            Assert.assertEquals(1, map.size());

            Assert.assertTrue(map.containsKey("three"));

        }
    }
}

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Java fluent inline map initialization

This is a common task, and there are many good and bad solutions out there. Java 8 even has a “Supplier” that can assist with inline map initializations.

Unfortunately, Java does not have a simple method like that found in the Groovy language. A map in Groovy is:

def map = [name: ‘Gromit’, likes: ‘cheese’, id: 1234].

 
I implemented a builder in Java 7 that is just simple syntax sugar. This is before I found Per-Åke Minborg’s great blog post Java 8, Initializing Maps in the Smartest Way on the topic, or the others found online. Perhaps I was using the wrong search. Hmm.

Using my version without all the Generics details:
Map codes = InitMap.with(new HashMap())
   .put(“xyz”,”321″).put(“abc”,”362″).toMap();

Alternatively, you can create a Map instance and use the builder, no need to invoke toMap():
Map<String,String> map = new HashMap<>();
InitMap.with(map).put(“xyz”,”321″).put(“abc”,”362″);

How does it work? InitMap is just a “partial” Decorator. When you create the InitMap object the Map parameter is stored in a field. Then when you “put” key/values, they becomes a ‘put’ to the hidden map. This allow you to chain the ‘put’s. When your done adding entries, you invoke toMap() to get the actual Map instance. The simplest thing that works.

Not too sure about the Generics stuff. Generics are like a loose thread on a coat; if you keep pulling on it, soon you’ll have no coat.*

Jump to source

Links

Footnotes

*This leads, me anyway, to wondering how do you test compile only concerns, like generics? Sure, compile errors are thrown if you try to use Generics type-safe code incorrectly. But, how do you know your generics code is meeting the requirements, or that future maintenance would not still enforce the requirements?

Hmm. Only thing I can think of is you have to write unit test that invoke a Javac compiler on strings of source code.


Source Section

  • Java 8, but not the new JDK features.
  • Eclipse Mars for development

Listing 1, InitMap and tests implementation, full source Gist

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

How to unit test Java servlets

The question of how to unit test Servlets comes up a lot. How can it be done? Should it be done? What are the options?

A unit test, in the realm of xUnit semantics, is an isolated test of the smallest testable subset of a program. Usually this translates to a test of a single method in an Object. When this object itself is part of a framework or container, such tests border on becoming Integration Tests. How could these types of objects still be ‘unit tested’?

Written by: Josef Betancourt, Date: 2015-09-17, Subject: Servlet testing

Options

Here are a few options.

POJO

When you write a servlet, ultimately the servlet object is instantiated by the server container. These objects do a lot behind the scenes that may prevent invoking methods on them when not attached to an actual container.

A servlet, or any other server based object like an EJB, provides access to problem domain services or functionality. The easiest way to test these Objects is to refactor that service into plain old Java objects, POJO.

Jakob Jenkov writes: “… push the main business logic in the servlet into a separate class which has no dependencies on the Servlet API’s, if possible”.

If your working with a framework that is likely the design approach anyway.

Servlet stub library

A library that allows creation of “server” objects can make creating stubs for testing very easy. Again, a framework should provide such a feature.

Mocking

Mocking using modern libraries like Mockito, Powermock, and JMockit, provides a very powerful approach. There is also what appears to be a more focused Mockrunner project, which has a mockrunner-servlet module.

In listing 1 below, a test is created for a target SutServlet class’s doGet. This method will set the response status to 404 if an ID request parameter is null.

Using JMockit, proxies of HttpServletRequest and HttpServletResponse are created. The request’s getParameter and the response’s setError methods are mocked. The actual unit test assertion is done in the mocked setError method.

Listing 1, JMockit use

@RunWith(JMockit.class)
public class SutServletTest_JMockit {
    
    @Test
    public void should_Set_ResourceNotFound_If_Id_Is_Null() throws Exception {
        new SutServlet().doGet(
        new MockUp<HttpServletRequest>() {
            @Mock
            public String getParameter(String id){
                return id.compareToIgnoreCase("id") == 0 ? null : "don't care";
            }
        }.getMockInstance(),
        new MockUp<HttpServletResponse>() {
            @Mock
            public void sendError(int num){
                Assert.assertThat(num, IsEqual.equalTo(HttpServletResponse.SC_NOT_FOUND));              
            }
        }.getMockInstance());
    }
     
}

JDK Dynamic Proxies

The Mock approach can also be duplicated using dynamic proxies. JDK dynamic proxy support is usable here. JDK proxies have one limitation, they can only proxy classes that extend an interface. (Still true in Java 9?). Servlets extend interfaces, so we can the proxy support in the JDK.

Listing 2, using JDK proxies

public class SutServletTest_using_jdk_proxy {
    
    private static final String DON_T_CARE = "don't care";
    private static final String SEND_ERROR = "sendError";
    private static final String GET_PARAMETER = "getParameter";

    /**  @throws Exception  */
    @Test
    public void should_Set_ResourceNotFound_If_Id_Is_Null() throws Exception {
        
        // request object that returns null for getParameter("id") method.
        HttpServletRequest request  = (HttpServletRequest)Proxy.newProxyInstance(this.getClass().getClassLoader(),
            new Class[]{HttpServletRequest.class},
                new InvocationHandler() {
                    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                        if(method.getName().compareToIgnoreCase(GET_PARAMETER) ==0){
                            return ((String)args[0]).compareToIgnoreCase("id") == 0 ? null : "oops";
                        }
                        return DON_T_CARE;
                    }
                }
        );
        
        // Response object that asserts that sendError arg is resource not found: 404.
        HttpServletResponse response  = (HttpServletResponse)Proxy.newProxyInstance(this.getClass().getClassLoader(),
            new Class[]{HttpServletResponse.class}, 
                new InvocationHandler() {
                    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                        if(method.getName().compareTo(SEND_ERROR) == 0){
                            Assert.assertThat((Integer) args[0], IsEqual.equalTo(HttpServletResponse.SC_NOT_FOUND));
                        }
                        return DON_T_CARE;              }
                }
        );
         
        new SutServlet().doGet(request,response);
    }
}

Javassist Proxies

Just for completeness, in listing 3, we use the Javassist library.

Listing 3, using Javassist proxy

public class SutServletTest_using_assist {
    
    private static final String DON_T_CARE = "don't care";
    private static final String SEND_ERROR = "sendError";
    private static final String GET_PARAMETER = "getParameter";

    /**  @throws Exception  */
    @Test
    public void should_Set_ResourceNotFound_If_Id_Is_Null() throws Exception {
        
        // request object that returns null for getParameter("id") method.
        HttpServletRequest request  = (HttpServletRequest)createObject(new Class[]{HttpServletRequest.class},
            new MethodHandler() {
                public Object invoke(Object self, Method thisMethod, Method proceed, Object[] args) throws Throwable {
                    if(thisMethod.getName().compareToIgnoreCase(GET_PARAMETER) == 0){
                        return ((String)args[0]).compareToIgnoreCase("id") == 0 ? null : "oops";
                    }
                    return DON_T_CARE;
                }
            }
        ); 
        
        // Response object that asserts that sendError arg is resource not found: 404.
        HttpServletResponse response  = (HttpServletResponse)createObject(new Class[]{HttpServletResponse.class},
            new MethodHandler() {
                public Object invoke(Object self, Method thisMethod, Method proceed, Object[] args) throws Throwable {
                    if(thisMethod.getName().compareTo(SEND_ERROR) == 0){
                        Assert.assertThat((Integer) args[0], IsEqual.equalTo(HttpServletResponse.SC_NOT_FOUND));
                    }
                    return DON_T_CARE;
                }
            }
        );
         
        new SutServlet().doGet(request,response);
    }
    
    /**
     * Create Object based on interface.
     * <p>
     * Just to remove duplicate code in should_Return_ResourceNotFound_If_Id_Is_Null test.
     * @param interfaces array of T interfaces
     * @param mh MethodHandler
     * @return Object
     * @throws Exception
     */
    private <T> Object createObject(T[] interfaces, MethodHandler mh ) throws Exception{
        ProxyFactory factory = new ProxyFactory();
        factory.setInterfaces((Class<?>[]) interfaces); // hmmm.        
        return factory.create(new Class[0], new Object[0], mh);
    }
}

Embedded server

Its also possible to start an embedded server, deploy the servlets, and then run the tests. Various Java app servers (like Tomcat and Jetty) support this and are well documented. The complexity comes when only partial integration is required. For example, we may want to have a real app server running the tests, but do we also really need a database server too? Thus, we also have to deploy stubs or mocks to this embedded server. Many resources on web for this approach, for example, “Integration Testing a Spring Boot Application“.

Another approach is the concept of the Hermetic Servers.

AOP

AOP can be used on embedded server, and this would allow “easy” mocking of integration endpoints and mocks. Such an approach was shown here “Unit test Struts applications with mock objects and AOP“.

References

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Search Java classpath for JUnit tests using Spring

A simple classpath scanner implementation that searches for classes having any method annotated with @Test or @RunWith. Spring Framework’s scanner utility is used to do this.

Many tools and dev environments support the search and invocation of JUnit tests. Some just search for Java files that end in ‘Test’, others search for files that actually contain tests. I think the latter is more accurate and useful.

Use case
One obvious one is programmatically creating test suits. The JUnit 4 approach of an annotation with a list of test classes is just plain wrong. With the code below a JUnit 3 type of test suite can be created.

Implementation
In the code below, Spring’s ClassPathScanningCandidateComponentProvider is used to scan the classpath. A custom TypeFilter is used to test each method of found classes for the annotations, and if found, the class is added to a list. This is available as a Gist.

Using Annotations to identify tests allows a path to also extend this to use more task focused annotations. For example, one can target what JUnit already offers such as Categories.

I got the idea for this from Classpath Scanning: Hidden Spring Gems.

As I mentioned, this is a simple approach. If you look at other’s such as in Eclipse, the scanners are more robust.

How would this be done without using Spring’s scanner support?

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Unit testing Java exception handling using JMockIt

How do you test that a method caught an exception? What if that catch did not have a side effect, it just logged output, or simply swallowed the exception?

Context
We have a method in a class that was written with a try catch and we need to unit test it. There are many ways this task can occur, such as the need to test legacy code or the need to write a test before a refactoring of such code.

We won’t go into the anti-pattern aspects or what constitutes proper handling of an exception here.

Error hiding is an anti-pattern in computer programming. Due to the pervasive use of checked-exceptions in Java, we must always address what to do with exceptions. Error hiding is when a catch clause does not property handle an exception.
 

In a catch block there are three common ways of handling an exception:

try{
   ... stuff ...
}catch(X){
   // 1. do something here
   // 2. maybe throw X or something else
   // 3. skip the above and do nothing
}

How are these tested?

Thrown exception
When the method reacts by throwing an exception we can test using the standard JUnit @Test(expected=SomeException.class), or for fine-grained verification, the test itself has a try catch block where we can assert the exception details.

Swallowed exception
If a method does nothing in the catch block, which is also called “swallowing” the exception, should it even be a test issue? Yes. The method is just tested normally. One of the tests must force the exception, of course. We do this since in future the method may be changed to handle the exception differently. One of the test assertions that can be made is that the forced exception is not thrown from the method.

Exception handler pointcut
Testing that an exception in the method under test was actually caught is possible, but only with Aspect Oriented Programming (AOP). One example language is AspectJ which supports the pointcut:

handler(TypePattern)
Picks out each exception handler join point whose signature matches TypePattern.
 

Behavior in catch
It gets more interesting when the catch block has ‘behavior’. This behavior has side effects. If these side effects are only local to the method, such as setting a flag false, then normal testing is adequate. If this behavior has side effects at the class or with collaborating objects, then this requires more complex testing.

It can get murky with this kind of testing. What is important is that one does not test the implementation (but sometimes that is crucial), only the interactions and requirements of the target “Unit” under test. What constitutes a “unit” is very important.

“Typically, a unit of behavior is embodied in a single class, but it’s also fine to consider a whole set of strongly-related classes as a single unit for the purposes of unit testing (as is usually the case when we have a central public class with one or more helper classes, possibly package-private); in general, individual methods should not be regarded as separate units on their own.” — Rogerio in JMockit Tutorial

.

Example
The method being tested invokes a method on a collaborating object and that object throws an exception. In the catch block, the exception is logged using the logging utility collaborator . Though not part of an explicit API, that logging may be critical to the use of a system. For example, an enterprise log monitoring system expects this logging for support or security concerns. A simple class Shop is shown in figure 1,

Figure 1, the class to test

public class Shop {
    private ShoppingSvc svc;
    
    /**
     * Get product name.
     * @param id the unique product id
     * @return the product name
     */
    public String getProduct(int id){
        String name = "";
        try {
            name = svc.getProductName(id);
        } catch (Exception e) {
            Logger.getAnonymousLogger()
			.log(Level.SEVERE, 
			"{result:\"failure\",id:\""+ id + "\"}");
        }
        
        return name;
    }
    
}

JMockit Use
JMockit supports two type of testing: behavior and state-based (or “Faking”).
Using the state based approach we create a mock for the getProductName(String) method of the collaborating (or dependent) class, ShoppingSvc. With JMockit this is easily done as an inline MockUp object with the target method mocked to throw an exception.

Listing 2, mocking

new MockUp<ShoppingSvc>() {
    @Mock
    public String getProductName(int id) throws IOException{
		throw new IOException("Forced exception for testing");
    }
};

JMockit’s behavior based support is then used to test the catch clause handling. As in other mocking frameworks, a record-replay-verify phases are used. Since the side effect of the exception handler here is the use of the logging dependency and we are not testing the logger, we ‘behaviorally’ mock the Logger class.

We can do this in the test method signature, @Mocked final Logger mockLogger. This mocks every method in the logger class. Then we set an expectation on the log method being used in the exception handler, then verify the method was actually invoked.

The full test class is shown in figure 3 below and the sample code is in a repo on GitHub:https://github.com/josefbetancourt/examples-jmockit-exceptions.

An alternative to using both state and behavior mocking is to just specify the exception throwing with the expectations. The article “Mocking exception using JMockit” shows how to do this. Of course, the JMockit Tutorial has all the details.

Listing 3, the full test class

@RunWith(JMockit.class)
public class ShopTest{
    /**
     * 
     * @param mockLogger Logger object that will be behaviorally mocked.
     */
    @Test
    public void shouldLogAtLevelSevere(@Mocked final Logger mockLogger)
    {
        /**
         * state-based mock of collaborator ShoppingSvc
         */
        new MockUp<ShoppingSvc>() {
            @Mock
            public String getProductName(int id) throws IOException{
                throw new IOException("Forced exception for testing");
            }
            
        };
        
        // the SUT  
        final Shop shop = new Shop();

        // what we expect to be invoked
        new Expectations() {{
            mockLogger.log(Level.SEVERE,anyString); 
        }};
        
        shop.getProduct(123); // actual invocation
        
        // verify that we did invoke the expected method of collaborator
        new Verifications(){{
            mockLogger.log(Level.SEVERE, anyString);  // we logged at level SEVERE
        }};
    }
}

Alternatives?
Just write better code so that you don’t need unit tests? This is mentioned in Functional Tests over Unit Tests

Test using a scripting language like Groovy? See Short on Time? Switch to Groovy for Unit Testing

Software
– JUnit: 4.12
– JMockit: 1.18
– JDK: 1.8
– Eclipse: Mars
– Maven: 3

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

maven-tomcat7-plugin not extracting the war file

The Apache Tomcat Maven Plugin has a great feature: it can create a runnable jar file with an embedded Tomcat server.

I tried this on a simple ‘Hello world!’ webapp to see if this works. It didn’t. When you run the jar: java -jar target\hello.jar, it would fail saying that the hello.war could not be found.

Of course I tried many different configurations and Maven POM file changes. Web searches did not point to an issue. Finally at the end I found something. There is a bug in the plug-in version 2.2. Errrrr.

Anyway, here is a suggestion to get around it. Made the changes, it works.

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

SvnKit E170001: Negotiate authentication failed: ‘No valid credentials provided’

In a new project, attempts to programmatically access our Subversion server using SvnKit fails with an E170001 error. But, only on one Windows 7 workstation.

After a lot of searching on web for answers finally found something that helped. I had to add system property: svnkit.http.methods=Basic,Digest,Negotiate,NTLM

So, using SvnCli, which I used to debug this, you add the property using the “-D” switch to the command line.

java -Dsvnkit.http.methods=Basic,Digest,Negotiate,NTLM -cp "SvnCli\*" org.tmatesoft.svn.cli.SVN --username *** --password *** list

I also had to add this property to the Tomcat app server.

Solution?
While this does fix the problem in this instance, since only one workstation is effected, it is probably hiding an underlying configuration setup issue.

I wonder what percentage of the nation’s GDP is spent on configuration and its issues.

Original stacktrace:

Mar 18, 2015 11:40:31 AM org.tmatesoft.svn.core.internal.util.DefaultSVNDebugLogger log
SEVERE: CLI: svn: E170001: Negotiate authentication failed: 'No valid credentials provided'
org.tmatesoft.svn.core.SVNException: svn: E170001: Negotiate authentication failed: 'No valid credentials provided'
        at org.tmatesoft.svn.cli.AbstractSVNCommandEnvironment.handleWarning(AbstractSVNCommandEnvironment.java:401)
        at org.tmatesoft.svn.cli.svn.SVNListCommand.run(SVNListCommand.java:95)
        at org.tmatesoft.svn.cli.AbstractSVNCommandEnvironment.run(AbstractSVNCommandEnvironment.java:142)
        at org.tmatesoft.svn.cli.AbstractSVNLauncher.run(AbstractSVNLauncher.java:79)
        at org.tmatesoft.svn.cli.svn.SVN.main(SVN.java:26)
        at org.tmatesoft.svn.cli.SVN.main(SVN.java:22)
svn: E170001: Negotiate authentication failed: 'No valid credentials provided'

Environment

  • Java 1.7
  • SvnKit 1.8.8
  • Tomcat 7

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Use JAXB to export Subversion log to Java

In a previous blog post I obtained the subversion XML log output. Now I need to convert that into Java objects in order to provide some special reporting requirements. Below I just present the unmarshal code.

Updates
Dec 3, 2014: cloned code into a github repository

As mentioned before, by using a high-level Java API to Subversion, like SVNKit, we can generate logs and have those already unmarshaled into objects. This is the recommended approach.

Let’s continue with the “brute force” way of accessing the output XML log output.

In listing one the Unmarshal class is a utility that hides the JAXB unmarshal code. The actual use of JAXB is a few lines, but this class provides methods that accepts various sources. For example using a file path:
Log theLog = new Unmarshal().path(“src/test/resources/log.xml”);

Unmarshal.java
package com.octodecillion.svn;

import java.io.File;
import java.io.IOException;
import java.io.Reader;
import java.net.URL;
import java.nio.charset.Charset;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;
import javax.xml.transform.stream.StreamSource;

import org.xml.sax.SAXException;

import com.google.common.base.Preconditions;
import com.google.common.io.CharSource;
import com.google.common.io.Resources;

/**
 * @author jbetancourt
 */
public class Unmarshal {
    /** */
    public Log path(String path) throws JAXBException, SAXException,
            IOException {
        Preconditions.checkNotNull(path, "'path' param is null");
        return url(new File(path).toURI().toURL());
    }

    /**  */
    public Log url(String url) throws JAXBException, SAXException, IOException {
        Preconditions.checkNotNull(url, "'url' param is null");
        CharSource charSrc = Resources.asCharSource(new URL(url),
                Charset.defaultCharset());
        return unmarshall(charSrc);
    }

    /**  */
    public Log url(URL url) throws JAXBException, SAXException, IOException {
        Preconditions.checkNotNull(url, "'url' param is null");
        CharSource charSrc = Resources.asCharSource(url,
                Charset.defaultCharset());
        return unmarshall(charSrc);
    }

    /** */
    public Log string(String xml) throws JAXBException, SAXException,
            IOException {
        Preconditions.checkNotNull(xml, "'xml' param is null");
        return unmarshall(CharSource.wrap(xml));
    }

    /** */
    public Log unmarshall(CharSource in) throws JAXBException, SAXException,
            IOException {
        Preconditions.checkNotNull(in, "'in' param is null");
        JAXBContext jaxbContext = JAXBContext.newInstance(Log.class);
        Log theLog = null;
        
        try(Reader reader = in.openStream()){
            StreamSource source = new StreamSource(reader);
            Unmarshaller unmarshaller = jaxbContext.createUnmarshaller();
            JAXBElement<Log> jxbElement = unmarshaller.unmarshal(source, Log.class);
            theLog = jxbElement.getValue();
        }
        
        return theLog;
    }
}
Listing 1, Unmarshal class

The Classes tree that captures the SVN log output XML
The log is in this format:

<log>
  <logentry revision="20950">
	<author>jbetancourt</author>
	<date>2014-11-10T20:12:11.910891Z</date>
	<paths>
		<path text-mods="false" kind="file" action="D" prop-mods="false">/2014/Acme/branches/rabbit-trap/www/images/beep.png
		</path>
	</paths>
	<msg>initial commit</msg>
   </logentry>
   <logentry revision="20948">
.....
</log>

Listing 0, example log contents

To use JAXB we create annotated classes to match the log XML.

These are simple use of JAXB. I’m sure there are better approaches. Note, the getter/setters
were not put in. Does a JAXB processor need these? Can’t it just use reflection?

The toString() in these classes return a JSON marshaling of the object. This is helpful for unit testing and debug.

Log.java
package com.octodecillion.svn;

import java.util.List;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

/**
 * @author j.betancourt
 */
@XmlRootElement
public class Log {
	@XmlElement(name = "logentry")
	List<LogEntry> entries;

        @Override
	public String toString() {
		StringBuilder bld = new StringBuilder();
		for(LogEntry entry : entries){
			bld.append(entry.toString());
		}
		
		return bld.toString();
	}	
	
}

Listing 2, Log class

LogEntry.java
package com.octodecillion.svn;

import java.util.List;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElementWrapper;
import javax.xml.bind.annotation.XmlRootElement;

import com.google.common.base.Joiner;

/**
 * @author j.betancourt
 */
@XmlAccessorType(XmlAccessType.FIELD)
@XmlRootElement
public class LogEntry {
	@XmlAttribute
	String revision;
	String author;
	String date;
	@XmlElementWrapper(name="paths")
	@XmlElement(name="path")
	List<Path>paths;
	String msg;

        // getter/setters not shown 

	@Override
	public String toString() {
        	StringBuilder bld = new StringBuilder("[");		
		bld.append(Joiner.on(",").join(this.paths)).append("]");		
		return String.format("{revision:%s,author:%s,date:%s,paths:%s,msg:%s}", this.revision,this.author,this.date,bld.toString(),this.msg);
	}
}
Listing 3, LogEntry class

Path.java
package com.octodecillion.svn;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlValue;

/**
 * @author j.betancourt
 */
@XmlAccessorType(XmlAccessType.FIELD)
@XmlRootElement
public class Path {
	@XmlValue
	String value;
	@XmlAttribute(name = "text-mods")
	String textmods;
	@XmlAttribute(name="kind")
	String kind;
	
	@XmlAttribute(name="action")
	String action;
	@XmlAttribute(name="prop-mods")
	String propmods;

	@Override
	public String toString() {
	   return String.format("{value:%s,kind:%s,action:%s,textmods:%s,propmods:%s}", 
             value.replaceAll("\\n+",""),kind,action,textmods,propmods);
	}
}
Listing 4, Path class

A JUnit 4 test

package com.octodecillion.svn;

import java.io.IOException;

import javax.xml.bind.JAXBException;

import org.junit.Assert;
import org.junit.Test;
import org.xml.sax.SAXException;

/**
 * @author j.betancourt
 */
public class UnMarshallTest {

	@Test
	public final void test() throws Exception {
		Log theLog = new Unmarshal().path("src/test/resources/log.xml");
		String actual = toSingleLine(theLog.toString());
		String expected1 = "{revision:20950,author:jbetancourt,date:2014-11-10T20:12:11.910891Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/images/beep.png,kind:file,action:D,textmods:false,propmods:false}],msg:initialcommit}{revision:20948,author:jbetancourt,date:2014-11-10T19:55:58.629641Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/images/desert.png,kind:file,action:D,textmods:false,propmods:false}],msg:changedicontint}{revision:20942,author:jbetancourt,date:2014-11-10T15:30:08.770266Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/scripts/acme/traps/rocket.js,kind:file,action:M,textmods:true,propmods:false},{value:/2014/Acme/branches/rabbit-trap/www/scripts/acme/traps/sled.js,kind:file,action:M,textmods:true,propmods:false}],msg:Added'usestrict'.}{revision:20941,author:rsmith,date:2014-11-10T15:20:41.707766Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/ads/umbrella/promo.html,kind:file,action:M,textmods:true,propmods:false},{value:/2014/Acme/branches/rabbit-trap/www/ads/images/umbrella.jpg,kind:file,action:A,textmods:true,propmods:true}],msg:promotionMerge}";
		String expected = toSingleLine(expected1);
		Assert.assertEquals("Created wrong object structure", expected, actual);
	}

	/**  */
	String toSingleLine(String s) {
		String s1 = s.replaceAll("\\n+", "");
		return s1.replaceAll("\\s+", "");
	}

}

Listing 5, JUnit test

pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.octodecillion</groupId>
  <artifactId>svnunmarshal</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>SvnUnMarshal</name>
  <description>Example of how to use JAXB to unmarshal svn log</description>
  <!-- 
  <build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.2</version>
            <configuration>
              <source>1.7</source>
              <target>1.7</target>
            </configuration>
        </plugin>       
    </plugins>
  </build>
   -->
  <dependencies>
    <dependency>
        <groupId>org.tmatesoft.svnkit</groupId>
        <artifactId>svnkit</artifactId>
        <version>1.8.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>18.0</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>xmlunit</groupId>
        <artifactId>xmlunit</artifactId>
        <version>1.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-exec</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.jmockit</groupId>
        <artifactId>jmockit</artifactId>
        <version>1.13</version>
        <scope>test</scope>
    </dependency>
  </dependencies>
</project>
Listing 6, Maven POM

Further reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Run svn command from Java using Commons exec

In a project I will need to marshal a Subversion log into Java. First step was getting Java to talk to a subversion repository. There are many options for doing this. Two of these are using a SVN API like SVNKit or invoking the command line svn executable.

Lets say you decide on the second option, invoking the svn binary, how would you do it? Easiest way is to use the Apache Commons Exec library. In listing below I use Exec and the Guava API in a Java 1.7 source level. Of course, this is not a ‘reusable’ solution, for example, the arguments to the two implemented commands are fixed. But, that is all I need right now.

Example use: new Command().log(“some/repo/somewhere”);

Expand.java
package com.octodecillion.svn;

import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Properties;

import org.apache.commons.exec.CommandLine;
import org.apache.commons.exec.DefaultExecutor;
import org.apache.commons.exec.ExecuteException;
import org.apache.commons.exec.PumpStreamHandler;

import com.google.common.base.Preconditions;
import com.google.common.io.Resources;

/**
 * Invoke SVN commands from Java.
 * Just log and list for now.
 * <p>
 * Requires the command line SVN executable.
 * These are specified in svn.properties file in class path.
 *
 * Note: Thread safety has not been tested.
 *
 * @author j.betancourt
 *
 */
public class Command {

    /**
     * Constructor.
     * @throws IOException
     */
    public Command() throws IOException {
        try(BufferedReader is = 
             (Resources.asCharSource(Resources.getResource("svn.properties"),
			Charset.defaultCharset())).openBufferedStream()
            ){

            Properties props = new Properties();
            props.load(is);
            commandLineClientLocation = props.getProperty("commandLineClientLocation",
				DEFAULT_SVN_PROGRAM_LOCATION);
            commandLineClientFileName = props.getProperty("commandLineClientFileName", 
				DEFAULT_SVN_EXE);
        }
    }

    /**
     * @see http://svnbook.red-bean.com/en/1.4/svn.ref.svn.c.log.html
     *
     * @param url
     * @return
     * @throws IOException
     * @throws ExecuteException
     * @since Nov 18, 2014
     */
    public String log(String url) throws ExecuteException, IOException {
        Preconditions.checkNotNull(url, "url param is null");
        CommandLine cmdLine = createCmdLine()
                .addArgument("log")
                .addArgument("--stop-on-copy")
                .addArgument("--verbose")
                .addArgument("--xml")
                .addArgument(url);

        return executeCommand(cmdLine);
    }

    /**
     * @see http://svnbook.red-bean.com/en/1.4/svn.ref.svn.c.list.html
     * @param baseURL
     * @return
     * @throws IOException
     * @throws ExecuteException
     * @since Nov 19, 2014
     */
    public String list(String url) throws ExecuteException,IOException {
        Preconditions.checkNotNull(url, "url param is null");
        CommandLine cmdLine = createCmdLine()
                .addArgument("list")
                .addArgument("--xml")
                .addArgument(url);
        return executeCommand(cmdLine);
    }

    private CommandLine createCmdLine() {
        return CommandLine.parse(commandLineClientFileName);
    }

    /**
     * Execute command line at working directory.
     * @param cmdLine
     * @return String that captured the error and standard output streams
     * @throws ExecuteException
     * @throws IOException
     * @since 2014-11-20
     */
    private String executeCommand(CommandLine cmdLine) 
		throws ExecuteException,IOException {
        DefaultExecutor exec = new DefaultExecutor();
        exec.setWorkingDirectory(new File(commandLineClientLocation));

        String str ="";
        try(ByteArrayOutputStream outputStream = 
              new ByteArrayOutputStream()){

            exec.setStreamHandler(new PumpStreamHandler(outputStream));
            exec.execute(cmdLine);
            str =  outputStream.toString();
        }

        return str;
    }

    private static final String DEFAULT_SVN_EXE = "svn.exe";
    private static final String DEFAULT_SVN_PROGRAM_LOCATION = 
		"\\Program Files\\CollabNet\\Subversion Client";

    private String commandLineClientLocation = DEFAULT_SVN_PROGRAM_LOCATION;
    private String commandLineClientFileName = DEFAULT_SVN_EXE;
}

 

Further reading

  • Guava
  • Apache Commons Exec
  • SVNkit
  • Calling SVN commands from a java program
  • Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.