Application logging using global inception identifier

For multithreaded programs that write to log files a best practice is to include a tracking ID. What should this ID be and how to use it? The following is presented as a ‘design pattern’.

TL;DR

For machine readability and tool use, a non-business related global inception ID (GIID) should be used for every log output. This ID is a combination of an ‘origin id’ and ‘unique id’. When each new business level or operational id is obtained or generated, it is logged to provide an ‘association’ with this GIID. This is similar to various existing practices so it is presented as a Software Design Pattern.

Author: Josef Betancourt, Jan 11, 2015

CR Categories: H.3.4 [Systems and Software]; D1.3 [Concurrent Programming]; D.2.9 [Management]; K.6 [MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS]; K.6.3 [Software Management]

  1. Context
  2. Forces
  3. Solution
  4. Consequence
  5. Implementation
  6. Appendix
  7. Further reading
  8.  

Context

App Logs

An application log file records runtime programmatic events: details, exceptions, debug, and performance related measures. This is different than specialized server log files, such as a webserver’s access logs, error.log, etc. The latter are more standardized, for example with W3C Extended Log File Format, and well supported.

App logs usually capture details at specific logging levels and named contexts. In the Java ecosystem there are plenty of libraries to support this and now the JDK supports this as part of the java.util.logging package.

Despite the advances made in logging APIs and support within operating systems and frameworks, app logs are at a primitive level of software maturity. What A. Chuvakin and G. Peterson describe as the “… horrific world of application logging” is composed of ad hoc, minimally documented, unstructured, untested, and under-specified delivered components.

Attempts to create widely used industry standards have failed and every business, software project, dev team, or industry reinvents and attempt to tackle the same problems.
 

Forces

In the context of server app logs, multiple sessions will output log information that can be intermixed. These sessions can be part of an ongoing process, such as a user interaction with a web site.

External integration points (web services, database, etc) may also be invoked. Unless each session is identified in the log and integration logs, subsequent support, debug, and auditing are very difficult.

The problem is not just tracking where and when ‘integration’ occurred or its non-functional integration criteria (i.e., timing), but the tracking of subsequent logging, if any, at that location.

App logs are used extensively during development. Their importance is illustrated by an old mime “debuggers are for wimps”. As such, logs with impeccable tracking used for design and test are a good Return On Investment (ROI).

The same is true for deployed systems. In many cases the only information available on historical behavior is in a log file.

This seems like a programming 101 lesson, but it is widely disregarded in practice. That log file output is a minor concern and sometimes not even part of a “code review” is puzzling.

Scenarios

1. A service must invoke a distributed call to another system. The service has retry logic, and logs each failure. If each log output does not identify the session or operation, the retries could get mixed with other threads. Identifying an end user’s request is a hit or miss bracketing of time stamps if the development team did not enough identifiable data in each log output.

2. A computer savvy end user or family may attempt to register into your system with multiple browsers simultaneously. This could cause problems if multiple logins are supported and an error occurs. How do you track this and debug it?

3. The app server makes a remote call to a service integration point and that service fails. How is the owner of that service informed as to the specific invocation? There are probably deployed systems where one would have to compare time stamps on log output to even coordinate where the two systems communicated and even then it is vague. Some systems may not even do any logging and the unless there is a fault of some kind.

4. You have to identify time periods based on hazy user complaints, search through multiple log files with regular expressions, then walk each output to recreate a specific error scenario. Isn’t this manual drudgery what computers were supposed to eliminate?
 

Solution

Global Inception ID

Logging with Unique identifiers is encouraged as a best practice:

“Unique identifiers such as transaction IDs and user IDs are tremendously helpful when debugging, and even more helpful when you are gathering analytics. Unique IDs can point you to the exact transaction. Without them, you might only have a time range to use. When possible, carry these IDs through multiple touch points and avoid changing the format of these IDs between modules. That way, you can track transactions through the system and follow them across machines, networks, and services.” — http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6
 

This unique identifier is generalized so that on first entry into a system or the start of a process, A Global Inception ID (GIID), is assigned to distinguish that ongoing process from others. A more descriptive term would be a Global Tracking ID, but that conjures up privacy and security concerns and is already being used in commerce for a different purpose. But ‘inception ID’ brings up visions of barcodes on people’s foreheads. Ok, how about “bludzwknxxkjysjkajskjjj”?

The term “Global” is to indicate that this ID is unique in a specific enterprise system. The uniqueness comes from its creation on a first contact basis on a specific subsystem. In essence this is a log tracking ID.

For example, a web server or an app server would be the first point of contact or request from an external User. The GIID, consisting of a combination of origin id and a unique id, would be created at this point. GIID ::= originID uniqueID

In article “Write Logs for Machines, use JSON” Paul Querna uses the term “txnId” for this type of ID:

“… this is essentially a unique identifier that follows a request across all our of services. When a User hits our API we generate a new txnId and attach it to our request object. Any requests to a backend service also include the txnId. This means you can clearly see how a web request is tied to multiple backend service requests, or what frontend request caused a specific Cassandra query.”
 

Another term for this GIID, or ‘txnId’ is Correlation ID. This terminology is used in SharePoint.

 
The correlation ID is a GUID (globally unique identifier) that is automatically generated for every request that the SharePoint web server receives.

Basically, it is used to identify your particular request, and it persists throughout further communications, if any, between the other servers in the farm. Technically, this correlation ID is visible at every level in the farm, even at a SQL profiler level and possibly on a separate farm from which your SharePoint site consumes federated services. So for example, if your request needs to fetch some information from an application server (say, if you are using the web client to edit an Excel spreadsheet), then all the other operations that occur will be linked to your original request via this unique correlation ID, so you can trace it to see where the failure or error occurred, and get something more specific than “unknown error”. — https://support.office.com/en-nz/article/SharePoint-2010-Correlation-ID-in-error-messages-what-it-is-and-how-to-use-it-5bf2dba7-43d2-484c-8ef4-e059f76e3efa

Various ‘Structured Logging’ efforts or syslog implementations already contain a ‘sending’ field specification. The GIID incorporates the sender id as the Origin ID, and this combination is more amendable to human and textual tools parsing.

 

Consequence

Size

A good candidate for a GIID must be large enough to satisfy uniqueness requirements. This could be, for example, a 36 character field. Where the log files are manually inspected with a text editor, this increases the log line which already contains many common fields like a time stamp.

Security

Unintentionally, “bad” logging practices makes it harder to track and correlate personally identifiable information (PIN). With the use the trans-system GIID, correlation between various business related identifiers is made easier.

The correlation ID is not necessarily a secret, but like other tracking objects like cookies, can be used for information discovery or questionable information storage. But, if an attack can already access your log files, there are other more serious issues?

Redundancy

What determines the first contact subsystem? A true distributed system could be configured or dynamically configured so that any system could be the first contact system. If so, then each subsystem is creating GIID and passing that GIID to other systems that are themselves creating GIIDs.

One approach to handle this is that a system will only create a GIID if none is present in the incoming request.

Feedback

For user interfaces, exposing the GIID or parts of it in exception situations can be beneficial:

“We also send the txnId to our user’s in our 500 error messages and the X-Response-Idheader, so if a user reports an issue, we can quickly see all of the related log entries.” — https://journal.paul.querna.org/articles/2011/12/26/log-for-machines-in-json/
 

Compare this to the Hunt The Wampus adventure in enterprises that only have an approximate time of an issue and must track this over multiple systems.

Accuracy

If a giid is part of a support system and as above the ID would be shared with Users in some way, would the value need some form of validity testing? Should it be tested that it is wellformed and include a checksum?

Example crc calculation for a UUID, based on textual version of id:

groovy -e "java.util.zip.Adler32 crc = new java.util.zip.Adler32(); crc.update(UUID.randomUUID().toString().getBytes());println Long.toHexString(crc.getValue())"
af9c09a3

 

Implementation

Origin ID

An OID uniquely identifies a system in an enterprise. This could be a web server or messaging system. Using a code for the actual system is recommended. Thus, instead of Acctsys, it would be a code, PE65K for example. Using a code is more ‘durable’ than a human readable name.

An extension is to also encode other information in the origin ID, such as application or subsystem identifiers.

Unique ID

This ID must not be a business related entity such as user id or account number. The simple reason is that these may occur in the logging record multiple times for different sessions or transactions. For example, user Jean Valjean with account number 24601 may log in multiple times into a web site. Tracking a specific session if a problem occurs is easier if we use a unique ID.

A business level ID may not even be relevant in another system that interacts with the origin point. In one system the ID could be one type of ID, and in the other the same party or user could be identified with a different ID.

Note that as soon as determined, accessed, or generated, a business level ID should be associated with the GIID. This could be a simple log output of that business ID which, since every log output has a GIID, will associate the business ID or value with the GIID.

Similarly when the same process communicates with another system, that system’s unique identifiers and related business IDs will also be associated with the GIID. For example, a web service will take the GIID and relate it to its own ID(s). Now a support engineer can follow the complete path of the system response to a user session.

ID creation

The easiest approach is to use the entry system’s session id created by the server. A potential problem is that this session id is not guaranteed to be unique and ends when the session expires. A UUID solves most problems.

Sample UUID generation in Groovy language:

groovy -e "println UUID.randomUUID().toString().replace('-','')"
1f788da1ac4a43bb82adb8e61cfcb205 

If the system ID is 3491 then the above UUID is used to create the GIID and use in logging:

20110227T23:34:37,901; EV={_ID:”34911f788da1ac4a43bb82adb8e61cfcb205″, USERID:”felixthecat”, ….}

Alternative to UUID use?

A UUID is a 32 character string. Could something smaller be used? Perhaps, but eventually the complexity of threaded systems would make the uniqueness constraint of any ID approach a comparable length.

Other approaches are possible. Most of these will incorporate a timestamp in some way. Note that a UUID contains a timestamp.

An example of a ‘unique’ id is used by MongoDB’s ObjectID specification. That spec calls for a 12-byte BSON type of:
• a 4-byte value representing the seconds since the Unix epoch,
• a 3-byte machine identifier,
• a 2-byte process id, and
• a 3-byte counter, starting with a random value.
An example of an ObjectID string representation is ObjectId(“507f1f77bcf86cd799439011″)

Log Framework support for GIID

The use of a GIID is a ‘cross-cutting’ concern. Requiring programmatic use of this ID would be burdensome and error-prone, even if stored in a thread-safe context.

Some logging frameworks support the concept of “nested diagnostic contexts”. This is a way of storing an ID so that interleaved logging is properly identified. See http://wiki.apache.org/logging-log4j/NDCvsMDC for more information.

Example usage

In a conventional Java server application a JSP or template system would obtain a GIID and insert it into generated pages and associated client side scripts. That GIID would also be stored in the server side session. Since the GIID is stored at the session it is accessible to the various services and components on a server.

This ID is embedded in request to other distributed servers and provides event correlation. Thus the logging systems will have access to the GIID and Client or server side errors can then display or use the GIID for tracking and reporting to assist support engineers.

Since the client also has the GIID, it can display or use this ID for customer service processing.

Of course, this would make more sense if it is a part of a wider Application Service Management (ASM) system.

Standards for IDs

Though many standards specify ID handling, modern architectures, especially web based or distributed, emphasize a stateless protocol. A GIID requirement could be one of those legerdemain stateful practices.

Development impacts

If logging is a deployed feature of an application then it too needs testing. But, since log output is an integration point, it does not fall under the “unit” testing umbrella. There is even some doubt if this should even be tested! Here is one example: Unit Testing: Logging and Dependency Injection
If log files can contain security flaws, convey data, impact support, and impair performance, then they should be tested that they conform to standards. Project management spreadsheets needs to add rows for logging concerns.

Technology

Log output can be developer tested using the appropriate XUnit framework, like JUnit.
Mocking frameworks provide a means of avoiding actually sending the output of a logger to an external ‘appender’. “Use JMockit to Unit test logging output”.
Issues
In development of a project, the log output changes rapidly as the code changes. Selecting where in the software development life cycle (SDLC) to test logging or even specify what logs should contain is difficult.
One approach is that the deployed system will not do any app logging that was not approved by the stake holders. These must be “unit” tested, and all development support logging is removed or disabled except for use in a development environment.

Deployment

There is no need to change every subsystem to use this log tracking approach. If the GIID is created somewhere in the “path” of a process or service, it adds value. Other systems can gradually start to use a tracking ID. Thus, the tools and training to use this capability can also be progressively introduced.

About this post

I was going to title this article ‘Logging in Anger’, as a response to my own experiences with application logging. Alas, there are so many issues that I had time to only focus on one as a result of a recent stint supporting an application that exhibits the typical logging anti-patterns. Example: it’s bad to get a null pointer exception, but to not know which argument to a function caused this?
 

Appendix

Structured Logging

(this article was going to add more info on incorporating a GIID into a Structured Logging framework. This section is here for refernce)
Structured Logging is a type of app log file that is data based rather than prose based. Thus, it is machine readable and amendable to high-level tools, not just a text editor.

Treating logs as data gives us greater insight into the operational activity of the systems we build. Structured logging, which is using a consistent, predetermined message format containing semantic information, builds on this technique …. We recommend adopting structured logging because the benefits outweigh the minimal effort involved and the practice is becoming the default standard. — http://www.thoughtworks.com/radar/techniques/structured-logging

 
An example, is a system where the log output uses a predetermined message format. An overview of such systems is found in chapter 5 of “Common Event Expression”, http://cee.mitre.org/docs/Common_Event_Expression_White_Paper_June_2008.pdf

Note this should not be confused with a similar sounding technology called “Log-structured file system”.
 

Further reading

  1. Log management and intelligence, http://en.wikipedia.org/wiki/Log_management_and_intelligence
  2. Logging a global ID in multiple components, http://stackoverflow.com/questions/1701493/logging-a-global-id-in-multiple-components
  3. Application Service Management (APM) system
  4. Application performance management, http://en.wikipedia.org/wiki/Application_performance_management
  5. The art of application logging, http://www.victor-gartvich.com/2012/05/art-of-application-logging.html
  6. Patterns For Logging Diagnostic Messages, http://c2.com/cgi/wiki?PatternsForLoggingDiagnosticMessages
  7. UUID, UUID
  8. How to test valid UUID/GUID?
  9. Log Data as a Valuable Tool in the DevOps Lifecycle (and Beyond), http://devops.com/features/log-data-valuable-tool-devops-lifecycle-beyond/
  10. OWASP – Logging Cheat Sheet, https://www.owasp.org/index.php/Logging_Cheat_Sheet
  11. How to Do Application Logging Right, http://arctecgroup.net/pdf/howtoapplogging.pdf
  12. Request for comment Structured Logging, http://www.mediawiki.org/wiki/Requests_for_comment/Structured_logging
  13. 6 – Logging What You Mean: Using the Semantic Logging Application Block, http://msdn.microsoft.com/en-us/library/dn440729(v=pandp.60).aspx
  14. A Review of Event Formats as Enablers of event-driven BPM, http://udoo.uni-muenster.de/downloads/publications/2526.pdf
  15. Basic Android Debugging with Logs, http://www.androiddesignpatterns.com/2012/05/intro-to-android-debug-logging.html
  16. Mapped diagnostic context vs Nested diagnostic context, http://wiki.apache.org/logging-log4j/NDCvsMDC
  17. Building Secure Applications: Consistent Logging, http://www.symantec.com/connect/articles/building-secure-applications-consistent-logging
  18. Log for machines in JSON, https://journal.paul.querna.org/articles/2011/12/26/log-for-machines-in-json/
  19. Logging Discussion, http://c2.com/cgi/wiki?LoggingDiscussion
  20. CEE, http://cee.mitre.org/docs/Common_Event_Expression_White_Paper_June_2008.pdf
  21. CEE is a Failure., https://gist.github.com/jordansissel/1983121
  22. Centralized Logging Architecture, http://jasonwilder.com/blog/2013/07/16/centralized-logging-architecture/
  23. Centralized Logging, http://jasonwilder.com/blog/2012/01/03/centralized-logging/
  24. Logging and the utility of message patterns, http://calliopesounds.blogspot.com/2014/07/the-power-of-javatextmessageformat.html?m=1
  25. Payment Application Data Security Standard, https://www.pcisecuritystandards.org/documents/pa-dss_v2.pdf
    Payment application must facilitate centralized logging.
    Note: Examples of this functionality may include, but are not limited to:
    • Logging via industry standard log file mechanisms such as Common Log File System (CLFS), yslog, delimited text, etc.
    • Providing functionality and documentation to convert the application’s proprietary log format into industry standard log formats suitable for prompt, centralized logging.
    Aligns with PCI DSS Requirement 10.5.3
  26. NIST 800-92 “Guide to Computer Security Log Management”, http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf
  27. UniqueID Generator: A Pattern for Primary Key Generation, http://java.sys-con.com/node/36169
  28. java.util.logging, http://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Money pooling based on random tiered payout

This idea was created in response to the recession. The goal was to inject cash into the economy as rapidly as possible. A reaction to the “shovel ready” bureaucratic approach.

I subsequently found that this is not an altogether new idea. Perhaps the tiers and random elements are something new.
~~~~~
Abstract
To accelerate the velocity of money flow for economic stimulus, a payout of a pool of money among contributors randomly placed in payout level tiers is advocated. The randomness of selection and increased frequency of payout creates a network effect that increases demand for goods and services. This will also have social benefits by increasing morale and unity of purpose, unlike present political inspired economic approaches.

Key words: Taxes, Economy, Unemployment

Reverse Taxes

Taxes are a necessary evil. One purpose of taxation is to fund social services that benefit all. For example, taxes pay for road maintenance. The street in front of my house gets repaired because my neighbors and I pay our taxes. However, if my own driveway needs maintenance, I have to fund that since only I own it and gain value (well, the bank owns it still, details). Yet, if I don’t maintain my driveway and it becomes a dangerous eyesore and attracts vermin, not only will my standard of living suffer, but eventually my neighbors will too: esteem, property values, and so forth. So, it behooves my neighbors to either drive me out with flaming torches and pitchforks or more civilly, encourage me to get with the program.

The problem is that families, especially in today’s economy, are not able to attend to many issues in a timely manner. They are cash strapped and, unfortunately, consumer fixated. Regardless of the reasons, there are personal as well as social-economic reasons for this. Even certain Wealth based religions and philosophies have a hand in making this situation worse.

A reverse tax is a tax that instead of going to a government agency, goes directly to private tax payers. These tax payers choose on their own what to do with it. Some will splurge, drink, or go further into debt. However, many will choose to act responsibly and improve their lives and immediate environment. Regardless of what they do, the funds will be quickly injected into the economy.

With Stochastic Reverse Taxation Subsidization (SRTS), funds can be made available that bypass the “public” route, where the individual directly chooses what needs to be funded. Since in each period those receiving payout are random (but everyone will get paid), the odds are that a portion of these people will buy goods and pay for services. This will increasingly stimulate demand with each period.

This is a direct application of changing one aspect of a system to influence the whole. In this case we are increasing the velocity of money. Instead of top-down approaches, a direct mid-level insertion is made. Thus, money is exchanged quicker, spread more rapidly without the red-tape and bureaucratic siphoning.

Implementation

Mathematics

Just how much would be available? There were 144,103,375 individual income tax returns in FY 2009. If we include the disenfranchised and those in the in the underground or shadow economy, the actual number of people who “could” file are actually much larger. Let’s just make it 200M. (the population of the USA is about 310,409,288 people). If we make the contribution $42, that is 13,037,190,096 about $13 Billion to stochastically redistribute. Choosing 42 weeks for a campaign, we will distribute $310 million dollars a week. This does not sound like a lot, but it is the quick “injection”; it is not waiting for “shovel-ready projects” that just fatten those in the loop.

If we create four payout tiers, then it is now a simple matter to determine the number of people and how much each be allocated per week. Can it be made no-loss, that is, no one would lose their $42? Probably not. We want there to be sizable distributions. Since we want some sizeable feedback back into the general system, the top tier will get a substantial return.
• Tier 1: X1 people would each get $1 million.
• Tier 2: x2 people would get thousands.
• Tier 3: x3 people would get hundreds
• Tier 4: x4 people would just get their $5 dollars back.

The randomness comes from assigning contributors to a tier. Each cycle will create a new random assignment but with algorithms to make sure every person gets to be in each tier after a few cycles. That is, the same person will not be in the same tier all the time.

Compare this to the recent tax breaks to spur the economy. What did the normal tax payer get $50? And that is a one-time year long break, I guess that would buy a few more take-out meals for lunch.

A mathematician/economist would have to create the algorithms for this process. Maybe it is not possible to do this and make it attractive for people to join in.

Note that this is different than the fixed payout algorithms or schedules like those used at eMoneyPool or Indian Chitty pools.

The problem with conventional pools is that the incentive is reduced the more people are part of the pool since the length of the ‘campaign’ is proportional to the desired payout level.

Administration

Using the existing Tax System?

The simplest approach is to just have a new checkbox on the tax form stating that one would like to participate in the Reverse Tax System. One possibility is to use a contribution value one, five, ten dollars. The payoff would be proportional. This would be the worse approach to use if we want to stimulate the economy. For that we would need a more accelerated schedule, perhaps monthly, optimally weekly. Using the Internal Revenue infrastructure is just convenient since the tax payer information is already available.

But, the tax schedule is yearly. We need a more aggressive schedule. Weekly! Not only would this be more effective in any stimulus effect, it would make the system more attractive, since there would be more ‘churn’.

Requirements and Limitations

Non-profit

This is a stimulus package for the middle class. Thus, the Wall Street crowd should not even know about it. :)

Not a lottery

This approach is similar to a multi-state lottery. Many differences however. Lotteries are for-profit and there are many losers (most!). The approach suggested here is not a payment, but a temporary allocation of funds. If the math is done correctly there is minimal loss other than normal inflationary devaluation of liquid assets within the payment schedule period and administrative costs.

Tax Free

Enough with the taxes. The payouts should be tax free. They should be totally reintroduced into the system so that demand is increased.

No competition

The effectiveness of this approach would be reduced if there are competing systems. Furthermore, it would be much easier for abuse if other systems were allowed.

No new bureaucracy

Taxes, are unavoidable, but they are also a giant spigot that feeds a growing never satiated beast. Bureaucracies grow. And grow. And become less effective, so they sprout new growths, that grow. And grow.

No debt generation

There is the possibility for a family to receive a guaranteed amount within a certain time period. So, institutions and businesses would be happy to capitalize on that. For example, offering various loans or “deals”. A great example of this are pay date loan companies. These should all be disallowed. An easy way to assure this is to not preannounce or make a big deal about it. It’s not that many large payments would be optimal anyway.

How to incorporate compound interest and investment?

Traditional “chits” or money pools are for short term financial of poor or modest groups. The modern banking and financial industry is used for more long term goals. If pools are used, money that should go into interest generating investments will suffer.

History

Something I posted on my blog on December 5, 2011, with the title “Demand Stimulation By Stochastic Reverse Taxation”. I came up with the idea for a massive “chit fund” that could pay out more often in a random schedule and people would get a payout based on a tiered system.

A few years ago I had a crazy idea. What if people pooled part of their money together and then redistributed it among the pool participants? Yes, not a new idea. A form of this even appears in “Stranger in a Strange Land” by Robert Heinlein. but what I was thinking was doing this on a massive scale, very massive. What if this massive “office chit pool” guaranteed that at each distribution at least hundreds of people would get a million bucks, others would get thousands, and the rest just a few hundred?

I abandoned the idea. It would not work, for one thing, you would need a lot of people, and the more people the longer it would be for each to be in line for the big payout. Would anyone wait years putting money into the pool just so one year they can get a substantial payout just to buy some needed item? No. It would probably be more advantages to just invest the money, at least there would be an accumulating interest payment.

Another try

Ideas, sometimes even bad ones, keep coming back. See this graph: They keep coming back

On July 19, 2010, I had another thought, what if instead of a rotation, one uses a random distribution into a payment hierarchy.

One pays into the pool, and at each distribution period one is randomly placed into a payout tier and the income distributed. In subsequent draws one is not placed into the same tier as the prior draws. When everyone has participated in each tier, the cycle could be repeated.

This is more reasonable. Using randomization, payout tiers, and shorter cycles would make this more acceptable. It would be like an income co-op.

Revisiting The Idea

Months later I revisited the idea. Could this be useful as some kind of economic stimulus? A “for the people” stimulus? Something to boost employment, increase opportunity, increased morale, and motivation? Probably not, there are no Silver Bullets. But, could it help?

Further reading
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Use JAXB to export Subversion log to Java

In a previous blog post I obtained the subversion XML log output. Now I need to convert that into Java objects in order to provide some special reporting requirements. Below I just present the unmarshal code.

Updates
Dec 3, 2014: cloned code into a github repository

As mentioned before, by using a high-level Java API to Subversion, like SVNKit, we can generate logs and have those already unmarshaled into objects. This is the recommended approach.

Let’s continue with the “brute force” way of accessing the output XML log output.

In listing one the Unmarshal class is a utility that hides the JAXB unmarshal code. The actual use of JAXB is a few lines, but this class provides methods that accepts various sources. For example using a file path:
Log theLog = new Unmarshal().path(“src/test/resources/log.xml”);

Unmarshal.java
package com.octodecillion.svn;

import java.io.File;
import java.io.IOException;
import java.io.Reader;
import java.net.URL;
import java.nio.charset.Charset;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;
import javax.xml.transform.stream.StreamSource;

import org.xml.sax.SAXException;

import com.google.common.base.Preconditions;
import com.google.common.io.CharSource;
import com.google.common.io.Resources;

/**
 * @author jbetancourt
 */
public class Unmarshal {
    /** */
    public Log path(String path) throws JAXBException, SAXException,
            IOException {
        Preconditions.checkNotNull(path, "'path' param is null");
        return url(new File(path).toURI().toURL());
    }

    /**  */
    public Log url(String url) throws JAXBException, SAXException, IOException {
        Preconditions.checkNotNull(url, "'url' param is null");
        CharSource charSrc = Resources.asCharSource(new URL(url),
                Charset.defaultCharset());
        return unmarshall(charSrc);
    }

    /**  */
    public Log url(URL url) throws JAXBException, SAXException, IOException {
        Preconditions.checkNotNull(url, "'url' param is null");
        CharSource charSrc = Resources.asCharSource(url,
                Charset.defaultCharset());
        return unmarshall(charSrc);
    }

    /** */
    public Log string(String xml) throws JAXBException, SAXException,
            IOException {
        Preconditions.checkNotNull(xml, "'xml' param is null");
        return unmarshall(CharSource.wrap(xml));
    }

    /** */
    public Log unmarshall(CharSource in) throws JAXBException, SAXException,
            IOException {
        Preconditions.checkNotNull(in, "'in' param is null");
        JAXBContext jaxbContext = JAXBContext.newInstance(Log.class);
        Log theLog = null;
        
        try(Reader reader = in.openStream()){
            StreamSource source = new StreamSource(reader);
            Unmarshaller unmarshaller = jaxbContext.createUnmarshaller();
            JAXBElement<Log> jxbElement = unmarshaller.unmarshal(source, Log.class);
            theLog = jxbElement.getValue();
        }
        
        return theLog;
    }
}


Listing 1, Unmarshal class

The Classes tree that captures the SVN log output XML
The log is in this format:

<log>
  <logentry revision="20950">
	<author>jbetancourt</author>
	<date>2014-11-10T20:12:11.910891Z</date>
	<paths>
		<path text-mods="false" kind="file" action="D" prop-mods="false">/2014/Acme/branches/rabbit-trap/www/images/beep.png
		</path>
	</paths>
	<msg>initial commit</msg>
   </logentry>
   <logentry revision="20948">
.....
</log>

Listing 0, example log contents

To use JAXB we create annotated classes to match the log XML.

These are simple use of JAXB. I’m sure there are better approaches. Note, the getter/setters
were not put in. Does a JAXB processor need these? Can’t it just use reflection?

The toString() in these classes return a JSON marshaling of the object. This is helpful for unit testing and debug.

Log.java

package com.octodecillion.svn;

import java.util.List;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

/**
 * @author j.betancourt
 */
@XmlRootElement
public class Log {
	@XmlElement(name = "logentry")
	List<LogEntry> entries;

        @Override
	public String toString() {
		StringBuilder bld = new StringBuilder();
		for(LogEntry entry : entries){
			bld.append(entry.toString());
		}
		
		return bld.toString();
	}	
	
}

Listing 2, Log class

LogEntry.java
package com.octodecillion.svn;

import java.util.List;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElementWrapper;
import javax.xml.bind.annotation.XmlRootElement;

import com.google.common.base.Joiner;

/**
 * @author j.betancourt
 */
@XmlAccessorType(XmlAccessType.FIELD)
@XmlRootElement
public class LogEntry {
	@XmlAttribute
	String revision;
	String author;
	String date;
	@XmlElementWrapper(name="paths")
	@XmlElement(name="path")
	List<Path>paths;
	String msg;

        // getter/setters not shown 

	@Override
	public String toString() {
        	StringBuilder bld = new StringBuilder("[");		
		bld.append(Joiner.on(",").join(this.paths)).append("]");		
		return String.format("{revision:%s,author:%s,date:%s,paths:%s,msg:%s}", this.revision,this.author,this.date,bld.toString(),this.msg);
	}
}


Listing 3, LogEntry class

Path.java
package com.octodecillion.svn;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlValue;

/**
 * @author j.betancourt
 */
@XmlAccessorType(XmlAccessType.FIELD)
@XmlRootElement
public class Path {
	@XmlValue
	String value;
	@XmlAttribute(name = "text-mods")
	String textmods;
	@XmlAttribute(name="kind")
	String kind;
	
	@XmlAttribute(name="action")
	String action;
	@XmlAttribute(name="prop-mods")
	String propmods;

	@Override
	public String toString() {
	   return String.format("{value:%s,kind:%s,action:%s,textmods:%s,propmods:%s}", 
             value.replaceAll("\\n+",""),kind,action,textmods,propmods);
	}
}


Listing 4, Path class

A JUnit 4 test

package com.octodecillion.svn;

import java.io.IOException;

import javax.xml.bind.JAXBException;

import org.junit.Assert;
import org.junit.Test;
import org.xml.sax.SAXException;

/**
 * @author j.betancourt
 */
public class UnMarshallTest {

	@Test
	public final void test() throws Exception {
		Log theLog = new Unmarshal().path("src/test/resources/log.xml");
		String actual = toSingleLine(theLog.toString());
		String expected1 = "{revision:20950,author:jbetancourt,date:2014-11-10T20:12:11.910891Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/images/beep.png,kind:file,action:D,textmods:false,propmods:false}],msg:initialcommit}{revision:20948,author:jbetancourt,date:2014-11-10T19:55:58.629641Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/images/desert.png,kind:file,action:D,textmods:false,propmods:false}],msg:changedicontint}{revision:20942,author:jbetancourt,date:2014-11-10T15:30:08.770266Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/scripts/acme/traps/rocket.js,kind:file,action:M,textmods:true,propmods:false},{value:/2014/Acme/branches/rabbit-trap/www/scripts/acme/traps/sled.js,kind:file,action:M,textmods:true,propmods:false}],msg:Added'usestrict'.}{revision:20941,author:rsmith,date:2014-11-10T15:20:41.707766Z,paths:[{value:/2014/Acme/branches/rabbit-trap/www/ads/umbrella/promo.html,kind:file,action:M,textmods:true,propmods:false},{value:/2014/Acme/branches/rabbit-trap/www/ads/images/umbrella.jpg,kind:file,action:A,textmods:true,propmods:true}],msg:promotionMerge}";
		String expected = toSingleLine(expected1);
		Assert.assertEquals("Created wrong object structure", expected, actual);
	}

	/**  */
	String toSingleLine(String s) {
		String s1 = s.replaceAll("\\n+", "");
		return s1.replaceAll("\\s+", "");
	}

}

Listing 5, JUnit test

pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.octodecillion</groupId>
  <artifactId>svnunmarshal</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>SvnUnMarshal</name>
  <description>Example of how to use JAXB to unmarshal svn log</description>
  <!-- 
  <build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.2</version>
            <configuration>
              <source>1.7</source>
              <target>1.7</target>
            </configuration>
        </plugin>       
    </plugins>
  </build>
   -->
  <dependencies>
    <dependency>
        <groupId>org.tmatesoft.svnkit</groupId>
        <artifactId>svnkit</artifactId>
        <version>1.8.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>18.0</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>xmlunit</groupId>
        <artifactId>xmlunit</artifactId>
        <version>1.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-exec</artifactId>
        <version>1.3</version>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.jmockit</groupId>
        <artifactId>jmockit</artifactId>
        <version>1.13</version>
        <scope>test</scope>
    </dependency>
  </dependencies>
</project>


Listing 6, Maven POM

Further reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Run svn command from Java using Commons exec

In a project I will need to marshal a Subversion log into Java. First step was getting Java to talk to a subversion repository. There are many options for doing this. Two of these are using a SVN API like SVNKit or invoking the command line svn executable.

Lets say you decide on the second option, invoking the svn binary, how would you do it? Easiest way is to use the Apache Commons Exec library. In listing below I use Exec and the Guava API in a Java 1.7 source level. Of course, this is not a ‘reusable’ solution, for example, the arguments to the two implemented commands are fixed. But, that is all I need right now.

Example use: new Command().log(“some/repo/somewhere”);

Expand.java

package com.octodecillion.svn;

import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.Properties;

import org.apache.commons.exec.CommandLine;
import org.apache.commons.exec.DefaultExecutor;
import org.apache.commons.exec.ExecuteException;
import org.apache.commons.exec.PumpStreamHandler;

import com.google.common.base.Preconditions;
import com.google.common.io.Resources;

/**
 * Invoke SVN commands from Java.
 * Just log and list for now.
 * <p>
 * Requires the command line SVN executable.
 * These are specified in svn.properties file in class path.
 *
 * Note: Thread safety has not been tested.
 *
 * @author j.betancourt
 *
 */
public class Command {

    /**
     * Constructor.
     * @throws IOException
     */
    public Command() throws IOException {
        try(BufferedReader is = 
             (Resources.asCharSource(Resources.getResource("svn.properties"),
			Charset.defaultCharset())).openBufferedStream()
            ){

            Properties props = new Properties();
            props.load(is);
            commandLineClientLocation = props.getProperty("commandLineClientLocation",
				DEFAULT_SVN_PROGRAM_LOCATION);
            commandLineClientFileName = props.getProperty("commandLineClientFileName", 
				DEFAULT_SVN_EXE);
        }
    }

    /**
     * @see http://svnbook.red-bean.com/en/1.4/svn.ref.svn.c.log.html
     *
     * @param url
     * @return
     * @throws IOException
     * @throws ExecuteException
     * @since Nov 18, 2014
     */
    public String log(String url) throws ExecuteException, IOException {
        Preconditions.checkNotNull(url, "url param is null");
        CommandLine cmdLine = createCmdLine()
                .addArgument("log")
                .addArgument("--stop-on-copy")
                .addArgument("--verbose")
                .addArgument("--xml")
                .addArgument(url);

        return executeCommand(cmdLine);
    }

    /**
     * @see http://svnbook.red-bean.com/en/1.4/svn.ref.svn.c.list.html
     * @param baseURL
     * @return
     * @throws IOException
     * @throws ExecuteException
     * @since Nov 19, 2014
     */
    public String list(String url) throws ExecuteException,IOException {
        Preconditions.checkNotNull(url, "url param is null");
        CommandLine cmdLine = createCmdLine()
                .addArgument("list")
                .addArgument("--xml")
                .addArgument(url);
        return executeCommand(cmdLine);
    }

    private CommandLine createCmdLine() {
        return CommandLine.parse(commandLineClientFileName);
    }

    /**
     * Execute command line at working directory.
     * @param cmdLine
     * @return String that captured the error and standard output streams
     * @throws ExecuteException
     * @throws IOException
     * @since 2014-11-20
     */
    private String executeCommand(CommandLine cmdLine) 
		throws ExecuteException,IOException {
        DefaultExecutor exec = new DefaultExecutor();
        exec.setWorkingDirectory(new File(commandLineClientLocation));

        String str ="";
        try(ByteArrayOutputStream outputStream = 
              new ByteArrayOutputStream()){

            exec.setStreamHandler(new PumpStreamHandler(outputStream));
            exec.execute(cmdLine);
            str =  outputStream.toString();
        }

        return str;
    }

    private static final String DEFAULT_SVN_EXE = "svn.exe";
    private static final String DEFAULT_SVN_PROGRAM_LOCATION = 
		"\\Program Files\\CollabNet\\Subversion Client";

    private String commandLineClientLocation = DEFAULT_SVN_PROGRAM_LOCATION;
    private String commandLineClientFileName = DEFAULT_SVN_EXE;
}

 

Further reading

  • Guava
  • Apache Commons Exec
  • SVNkit
  • Calling SVN commands from a java program
  • Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    FORTH language processor on comet

    The Philae spacecraft that landed on Comet 67P/Churyumov–Gerasimenko (67P) had an experiment on board that was driven by a FORTH language processor, the Intersil HS-RTX2010RH.

    The experiment/hardware was the The Surface Electric Sounding and Acoustic Monitoring Experiment (SESAME).

    What are they using for newer space platforms? ARM, INTEL, …?

    The Forth language
    Forth is a stack-based language. Interestingly PostScript is similar: http://en.wikipedia.org/wiki/PostScript#The_language

    Historical minutia
    Forth gave rise to this bumper sticker: FORTH ♡ IF HONK THEN

    Updates
    November 26, 2014: Turns out the RTX2010 is used extensively on this mission. It has ten of them and one pair controlled the flight.

    Further reading

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    Use dojo’s promise.all in non-failure mode

    The “all” function “Takes multiple promises and returns a new promise that is fulfilled when all promises have been resolved or one has been rejected.” This makes a lot of sense. If you are invoking multiple asynchronous services in parallel and want to wait for all of them before continuing with a “reactive” function, you probably want to not run this function if one of the services fails.

    In some scenarios you do want all the service invocations to complete and then deal with the results whether success or failure. Thus, it would be nice to have a version of ‘all’ that did this.

    The now deprecated DeferredList may have supported this. See this reference.

    Some() function implementation
    Below in listing one I created a function ‘some’ that transforms the input to ‘all’ to accomplish this. This simply takes each promise argument (in array or object) and chains an always(…) Thenable. This Thenable promise link takes a handler that will be invoked whether the prior promise is resolved or rejected. It acts like a ‘finally’ in a try .. catch .. finally.

    A better approach would have been to create an alternative to the ‘all’ function that did this.

    /**
     * Takes an object or array of promises and creates a new array of 
     * promises that do not fail.
     * <p>
     * The Dojo/All function is fulfilled when all promises 
     * have been resolved or when one has been rejected.
     * In certain scenarios even if some promies
     * @author J. Betancourt 
     * @since 2014/11/04
     * @param objectOrArray  Object|Array?
     * @returns {Array} of promises.
     */
    ;function some(objectOrArray) {
        "use strict";
        
        var promises = [];
        require([ "dojo/_base/array", "dojo/Deferred", 'dojo/when' ],
          function(array, Deferred, when) {
            var origPromises = [];
            if (objectOrArray instanceof Array) {
                origPromises = objectOrArray;
            } else {
                for ( var key in objectOrArray) {
                    if (Object.hasOwnProperty.call(objectOrArray, key)) {
                        origPromises.push(objectOrArray[key]);
                    }
                }
            }
            // create array with each item a promise chain.
            array.forEach(origPromises, function(item){
               promises.push(when(item).always(function(res){return res;}));
            });
        });
        
        return promises;  // array of dojo/promise/Promise
        
    } // end some(objectOrArray) function
    

    Listing 1

    Example
    Below in Listing 2, this function is used. Available on jsFiddler.

    <!DOCTYPE html>
    <html>
    <head>
    <script src="http://ajax.googleapis.com/ajax/libs/dojo/1.10.0/dojo/dojo.js"  
        data-dojo-config="async:false"></script>
    <script src="scripts/some.js"></script>
    
    <script>
        require([ "dojo/Deferred", "dojo/promise/all",
    	 "dojo/dom", "dojo/on","dojo/domReady!" ], 
            function(Deferred, all, dom, on) {
            "use strict";    
            
            function show(msg){
                dom.byId('output').innerHTML += msg + '<br/>';
            }
            
            /**
             * Use setTimeout to simulate an async process.
             * @param {string} name the name of this async invoke
             * @param {boolean} fail if true fail the promise
             * @returns {Promise} dojo/promies/Promise
             */
            function async(name, fail){
                var deferred = new Deferred();
                setTimeout(function(){
                    if(fail){
                        deferred.reject(name + "Failed!");
                    }else{
                        deferred.resolve(name);
                    }
                },1000);
                
                return deferred.promise;
            }
            
            /**
             * On click of 'StartButton' button 
    		 * wait for an array of promises. 
             */
            on(dom.byId("startButton"), "click", function() {
                var multiples = [async('V0'),
    			 async('V1',true),async('V2')];
    			 
                var p = all(some(multiples))
                .then(function(result){
                    show(result);
                    console.log(result);
                },
                function(err){
                    // this should not be reached.
                    show(err);
                    console.log(err);
                });
            });
        });
    </script>
    </head>
    <body>
        <button type="button" id="startButton">Start</button><p/>
        <div id="output" style="border: 1px black solid;width:20em;margin-bottom:20px;">Ready<br/></div>
    </body>
    </html>
    

    Listing 2

    Further reading

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    JavaScript Chaining Promises to Promises

    When you create a Promise you can chain asynchronous behavior by using the then(success,fail) method. The success and fail are callback functions invoked appropriately when the promise is resolved or rejected. Note that since ‘then(…)’ returns a promise, this technique is called promise chaining: p.then(…).then(….).

    What I discuss in this post are two subjects
    1. How to ‘stitch’ multiple promises together?
    2. How to ‘splice’ in a value into a promise chain?
    3. How to chain using a promise object as an argument?

    What if in complex application you already have promises objects and want to create promise chains linking them to each other? How would you do this with ‘then’ methods? Amazingly, in most examples of using Promises on the web this simple scenario is never presented directly. Granted it doesn’t *seem* to be relevant.

    Here is the source of a simple demo based on one of the Dojo Deferred examples. It is ‘normal’ promise chaining. I’m using this as the basis for the investigation presented here. The output displayed on page should be: completed with value: 75

    The actual chaining ‘stitching’ is setupTrans().then(function(x){return action()}).then(outputValue);

    <!DOCTYPE html>
    <html >
    <head>
    
    <script src="http://ajax.googleapis.com/ajax/libs/dojo/1.10.0/dojo/dojo.js"  data-dojo-config="async:false"></script>
    	
    <script>
    require(["dojo/when", "dojo/Deferred", "dojo/dom", "dojo/on", "dojo/domReady!"],function(when, Deferred, dom, on){
    	  var balance = 100;
    
    	  function asyncProcess(x){
    	    var deferred = new Deferred();
    	
    	    setTimeout(function(){
    	      if(x === 'plus' || x === 'minus'){
    	      	 x === 'plus' ? (balance += 25) : (balance -= 25);
    	      } 
    	      deferred.resolve(balance);
    	    }, 1000);
    	
    	    return deferred.promise;
    	  }
    
    	  function setupTrans(){
    	  	return asyncProcess('z');
    	  }
    
    	  function debitAccount(){
    		return asyncProcess('minus');
    	  }
    	
    	  function creditAccount(){
    		return asyncProcess('plus');
    	  }
    
    	  function outputValue(value){
    	    dom.byId("output").innerHTML += "<br/>completed with value: " + value;
    	  }
    	  
    	  on(dom.byId("startButton"), "click", function(){
    	  	var debit = true;
    	  	var action = debit ? debitAccount : creditAccount;
    	  	
    	  	setupTrans()
    	  	.then(function(x){return action()})
    	  	.then(outputValue);
    	  	
    	  });
    });
    </script>
    </head>
    <body class="claro">
        <h1>Output:</h1>
    	<div id="output"></div><p />
    	<button type="button" id="startButton">Start</button>
    </body>
    </html>
    

    What is ugly is the inline function. Luckily I found the “Flattening Promise Chains” article where the author shows how to get rid of the function and simplify. However that article didn’t answer my question, how can you simply and directly chain promise objects. What the article showed is that you have to use the correct structure or design. Well, of course, but …


    Answer
    I thought this would be difficult. The Dojo documentation page never mentions that instead of the first ‘then’ argument being onFulfilled handler function it could be an actual promise object. But this is mentioned in specs such as the Promises/A+ specification. Or does it? The ‘Promise resolution procedure’ seems to mix compile time view with runtime, or I’m reading it incorrectly. Q documentation is very good on this. Will have to research further.

    To directly chain separate promise chains the above code sample is modified as follows by just using an existing promise as the argument to another chains then(…) method. However, this can only be done using a promise returning function.

      on(dom.byId("startButton"), "click", function(){
      	var debit = true;
      	var action = debit ? debitAccount : creditAccount;
      	
      	setupTrans()
      	.then(action)
      	.then(outputValue);
      	
      });
    

    Or, to inline the decision:

      on(dom.byId("startButton"), "click", function(){
      	var debit = true;
      	
      	setupTrans()
      	.then(debit ? debitAccount : creditAccount)
      	.then(outputValue);
      	
      });
    


    Chaining using actual promise objects inside the then(f,r) is made easy using the approach shown next, using “scalars”.

    Chaining values into promise chains
    While looking at this subject I thought about a situation where we want to directly insert a known value into the chain so that the next ‘link’ will use this as the prior handler’s return value.

    We have a chain, and I want to put the great Answer in there. This won’t work: var p1 = promiseReturningFunction().then(…).then(’42′).then(….).

    Sure you can wrap ’42′ in an inline function then(function(){return ’42′}), but that is yucky. How about a function that does that:

         /** return a function that returns the value */
         function now(value){
        	  return function(){return value;};
          }
    

    Now you can do: var p1 = async().then(…).then(now(’42′)).then(….);

    Here is the prior source code with the changes. This is just some of the script changes. The output on screen is:

    completed with value: 75
    completed with value: Goodbye

          /* ...  */
    
          function outputValue(value){
            dom.byId("output").innerHTML += "<br/>completed with value: " + value;
          }
          
          function now(valueOrPromise){
        	  return function(){return when(value);};
          }
          
          on(dom.byId("startButton"), "click", function(){
            var debit = true;
            var action = debit ? debitAccount : creditAccount;
            
            var promise1 = setupTrans()
            .then(debit ? debitAccount : creditAccount)
            .then(outputValue)
            .then(function(){
            	console.log("now do another promise chain...");
            });
            
            promise1.then((now('Goodbye'))).then(outputValue);
            
          });
    


    Using promise object directly in then(f,r) by using now(x)
    With the Promises implementation in Dojo, you can’t use promiseObject.then(anotherPromiseObject). And you can’t use promiseObject.then(when(anotherPromiseObject)). At least, I haven’t figured how to do it.

    With the now(x) function shown above, this is easy. The code below will output:

    completed with value: 75
    completed with value: 100
    completed with value: undefined
    completed with value: finished both chains

         /** return function that returns the value 
             or apply Dojo's when(...) to return a Promise 
          */
         function now(valueOrPromise){
        	  return function(){return when(value);};
          }
          
          on(dom.byId("startButton"), "click", function(){
            var debit = true;
            var action = debit ? debitAccount : creditAccount;
            
            var promise1 = setupTrans()
            .then(action)
            .then(outputValue)
            .then(function(){
            	console.log("now do another promise chain...");
            });
            
            debit = false;
            action = debit ? debitAccount : creditAccount;
            var promise2 = setupTrans()
            .then(action)
            .then(outputValue);
            
            promise1
            .then(now(promise2)).then(outputValue)
            .then(now("finished both chains"))
            .then(outputValue);
            
          });
    });
    

    Groovy language has Promises too
    GPars brings Promises to the Groovy language.

    Further reading

    1. Flattening Promise Chains
    2. A Deeper Dive Into JavaScript Promises
    3. How-to Compose Node.js Promises with Q
    4. Promise Anti-patterns
    5. dojo/when
    6. q
    7. General Promise Resources
    8. Promises/A+
    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    Force failure of JavaScript promise in Dojo

    For development and unit testing you may need to force a promise chain to fail in various ways.

    In listing 1 below is shown a simple XHR request. Now to make sure our promise chain error handling is working correctly we want the Ajax call to fail.

    One way of achieving this to just change the url of the XHR destination. This is fine here, but is not a generic approach since not every async function or service is based on a URL or even single value.

    Another approach is shown in listing 2. Here we cancel the promise. In Dojo’s promise implementation we can cancel(reason,flag) a promise giving a reason and a flag, and even invoke a cancel handler if the promise was created with one. If this flag is true, a successful async call will result in a failure. Unfortunately, unless you type at superhuman speeds, this can’t be added at a debugger breakpoint and must be coded in temporarily.

    Listing 1

    /**
     * XHR call.
     * @returns promise
     */
    function _loadData(requestParams) {
        var promise;
        require([ "dojo/request"], function(request) {
            promise = request.post("someurl", {
                data : requestParams,
                handleAs : "json",
                preventCache : true
            })
        });
        
        return promise;
    }
    

    Listing 2

    function _loadData(requestParams) {
        var promise;
        require([ "dojo/request"], function(request) {
            promise = request.post("someurl", {
                data : requestParams,
                handleAs : "json",
                preventCache : true
            })
        });
        
        promise.cancel("force error",true);
        return promise;
    }    
    

    There are XUnit type testing frameworks for JavaScript that may be able to do this failure testing better. I have not looked into this yet.

    Further reading

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    A new kind of rowing stroke

    A few months ago I got an idea for a different kind of sport Rowing stroke. I don’t remember the exact train of thought, but it had to do with swimming. Anyway, …

    The physics of rowing and dynamics of the water are very complex. My idea is that you want to move the water past the boat faster, reduce the amount of oars in the water, and increase the glide of the boat. Thus, instead of all the rowers rowing at the same pace, the rowing is staggered.

    At the beginning of the race, the traditional rowing stroke is used. We want the maximum power to accelerate the boat quickly.

    In the middle of the race, a staggered stroke is used. Here the front rower begins the stroke and at the end of the “drive’ phase of the stroke, while the blade is still in the water, the second rower begins their stroke. This is repeated for each rower in turn.

    Toward the end of the race, the traditional synchronized power stroke is resumed.

    In essence the effect, if done smoothly, is the strokes become just one long stroke. As if only one person “drove” a very long stroke from bow to stern. Like birds in flight, the water becomes part of solution, a slipstream.

    Advantages? I’m only guessing here.
    - Rowers get more rest, and more energy available toward end.
    - The glide of the boat is increased.
    - Water is moved faster, velocity of boat.
    - Less oars in the water so less friction.

    – Josef

    Further reading

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

    JavaScript Promise example, reverse geocode location

    In prior post we wrapped the Google Map API in a Promise. Now we wrap navigator.geolocation.getCurrentPosition in a similar promise. This allows us to to neatly chain these two to get the browser’s current address.

    The example is using Dojo’s Promise implementation, but the approach is applicable to other Javascript Promise libraries or native support.

    In listing 1 below I reproduce the former getGeocode function.

    Listing 1, Get geocode using Google service

    function geoCode(request) {
        var deferred;
        require([ 'dojo/Deferred' ], function(Deferred) {
            deferred = new Deferred();        
            new google.maps.Geocoder().geocode(request, 
              function(results, status) {
                if (status == google.maps.GeocoderStatus.OK) {
                    deferred.resolve(results);
                } else {
                    deferred.reject(status);
                }
            });
            
        });
        
        return deferred.promise;
        
    }
    
    

    In listing 2 below we have a similar function

    Listing 2, get browser location.

    function getLocation(){
        var deferred;
        require([ 'dojo/Deferred' ], function(Deferred) {
            deferred = new Deferred();
            navigator.geolocation.getCurrentPosition(
                function navSuccess(position){
                    deferred.resolve(position);
                },function navFail(){
                    deferred.reject(posError);
                }
            );
        });
    
        return deferred.promise;
    }
    

    Hmmm. The code is very similar. It seems that these two functions could be done using a Promise wrapper function, like toPromise(doFunction, onSuccess, onFailure). Is there such a beast?

    Now we can use them together, as shown in listing 3. Here instead of a simple promise chaining we use the technique presented in “Break promise chain and call a function based on the step in the chain where it is broken (rejected)” to “catch” the correct error and not have it bubble up the ‘then’ chain.

    Listing 3, get browser location, then reverse geocode

    function findCurrentLocation(){
        var request;
    
        if (navigator.geolocation) {
            getNavLocation().then(
                function navSuccess(position){
                    request = {
                        'location' : 
                          new google.maps.LatLng(
                            position.coords.latitude, 
                            position.coords.longitude)
                    };
    
                    return getGeoCode(request).
                       otherwise(function geoCodeError(status){
                         throw new Error(status);
                       }
                    );
                }, function navFailed(err){
                    console.log('navLocation() failed: " + err); 
                    throw new Error(err);       
                })
                .then(function geoCodeSuccess(results){
                        console.log('navLocation() success: " + results);
                    } 
                );
        }
    }
    

    Above, .otherwise(onFailure) is just a shortcut for .then(null, onFailure)

    Of course, the code doesn’t have enough error handling and actual use of the data obtained.

    Further reading

    Creative Commons License
    This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.