Tag Archives: reactive

Version control console using ReactJS

Created a web app to generate a report from the version control repository, Apache Subversion™. Similar approach is possible targeting a different repository, like Git.

TL;DR
Someone said a process we follow could not be automated. I took that as a challenge and created a proof of concept (POC) tool.

The final GUI using ReactJS is relatively complex: five data tables that hide/show/expand/collapse. Searches on those tables, sorting, navigation links, help page, Ajax requests to access Subversion repo data, export to CSV, report generation, and client side error handling. It started as just a GUI to a report, but since it was easy, added more features: Zawinski’s law.

To top it off, the app had to automatically invoke the default workflow or no one would use it.

Result?
1. It is a complex disaster that works. And works surprisingly fast. Using ReactJS and Flux made it into a fast elegant (?) disaster that works … kind of.
2. The app served as an example of a SPA in the our dev group. But, mostly to try out the ReactiveJS approach.
3. My gut feel is that there are some fundamental problems in the client side MV* approach which leads to control flow spaghetti (a future blog post).

Since the time I wrote that app I have noticed a mild push back on React that point out the hidden complexities. There are now new ideas and frameworks, like Redux or Cycle.js. Most recently, to tackle the action-coordination, there is much digital ink written on Redux Sagas, for example: “Managing Side Effects In React + Redux Using Sagas“.

Note, though there are critiques of the ReactJS approach or implementation, this does not imply that React was not a real breakthrough in front end development.

Report generation
Creating simple reports from a version control repository can be accomplished with command line tools or querying XML output from SVN log commands. In this case generating the criteria to generate the report was the hard part. Details are not relevant here: this web app would remove a lot of manual bookkeeping tasks that our group currently must follow due to a lot of branch merging and use of reports for error tracking, verification, and traceability. Yup, long ago legacy Standard Operating Procedures (SOP) of an IT shop.

Architecture
Server
A simple Java web app was created and deployed to a Tomcat server. A Java Servlet was used at the server to receive and send JSON data to the browser based client. This server communicates with the version control repository server.

Client
The browser is the client container with ReactJS as the View layer and Flux (implemented in the McFly library) as the framework client implementation. Dojo was used as the JavaScript tools library. Dojo supplied the Promise and other cross-browser capabilities. Why Dojo? That is already in use here. If we were using JQuery, that is what I would use.

Local application service
Performance note: Since the repo query and processing occurs at the server, multiple developers accessing the service would have a performance impact. A future effort is to deploy this as an runnable Jar application (Spring Boot?) that starts an embedded app server, like Tomcat or Jetty, at the developer’s workstation. The browser would still be used as the client.

Repository Query
Some options to generate SVN reports:

1. Use a high level library to access SVN information.
2. Export SVN info to a database, SQL or NoSQL.
3. Use an OS or commercial SVN report generator.
4. Use command line XML output option to create a navigational document object model (DOM)
5. Use SVN command line to capture log output, and apply a pipeline of Linux utilities.

This was a ‘skunkworks’ project to determine if some automation of a manual process could be done and most importantly, if doable, would the resulting tool be used? The first option, seemed easiest, and was chosen. The repo was accessed with the SvnKit Java library. (For Java access to a Git repo, JGit is available).

The process approach was to generate and traverse a Log collection. A simple rule engine was executed (just a series of nested conditionals) to determine what to add from the associated Revision objects.

This seemed like a workable idea until a requirement was requested after the POC was completed: instead of listing a particular source file once per report, show multiple times per each developer who made a commit to it. An easy change if this were implemented as an SVN log query sent to a pipe of scripts. However, with the design this required going into the nuts and bolts of the “rule engine” to add support for filtering, and further changes to the model.

Yup, a POC solution can be a big ball of mud, and unfortunately can be around a long time. Incidentally, this happened with Jenkins CI; where I …

Very recently a flaw in the design will force a revisit of the algorithm again. Instead of making the ‘rule engine’ more powerful, an alternative approach is to start from a Diff collection. The diff result would be used to navigate the Log collection. A similar approach is shown here: http://www.wandisco.com/svnforum/forum/opensource-subversion-forums/general-setup-and-troubleshooting/6238-svn-log-without-mergeinfo-changes?p=36934#post36934

But, already a problem was found with diff output. There is no command line or Java library support for pruning of deleted folders. For example, if a/b/c is a hierarchy, and you delete b, c is also deleted. Now if you generate a diff, the output would contain delete entries for: a/b and a/b/c. What was needed was just a/b. Simple, you say. Sure, but this information is a OOP object graph, so can be complicated.

I solved it: a diff list of 1800 folders was reduced to just 8 folders! I’m surprised a solution or something similar was not found in a web search. I wrote about this in “Can conditional state be used in RxJava Observable streams?

Perhaps revisit the alternative approaches, like export to database? Not sure if this really would have simplified things, but instead just change where the complexity was located. Is the complexity of a software solution a constant?

Other systems take this export approach. One system I saw years ago, exports the version control history (it was CVS) into an external SQL database and then used queries to provide required views.

Client Single-Page Application
What to use as the browser client technology? From past experience, I did not want go down the path of using event handlers all over the place and complex imperative DOM updates.

Anyway, React seemed interesting and had a shorter learning curve. I looked at Angular, but it seemed to be the epitome of embedding the developer into the product (future blog post on the application development self-deception).

A few ReactJS components were created:

  • BranchSelect
  • CommentLines
  • ControlPanel
  • DiffTable
  • ErrorPanel
  • ExclusionRow
  • ExclusionTable
  • FilesRow
  • FilesTable
  • ManifestRow
  • ManifestTable
  • ProgramInfo
  • ProjectPanel
  • RevisionRow
  • RevisionTable
  • ViewController

Lessons Learned
This project progressed very quickly. React seemed very easy. But, that was only temporary. Until you understand a library or a paradigm, start with a smaller application. Really understand it. Of course, these too can fool you. For example, when this app first loads, I had to invoke the most likely use-case. There was a endless challenge of chicken/egg model flow disasters. Solved it, but can’t understand how I did it. Somehow I tamed the React flow callbacks. Or this is just a lull and will blow up in as a result of an unforeseen user interaction.

All the new cutting edge JavaScript front-end frameworks are very very complex for us average developers. Just the tooling is daunting if your coming from a vanilla JavaScript shop. Node dis! See this: ‘I’m a web developer and I’ve been stuck with the simplest app for the last 10 days

Next SPA project?
My next app will probably use Redux as the Flux framework. Or may leave Reactjs and go directly with Cycle.js which is looking very good in terms of ‘principles’ or conceptual flow and is truly Reactive, based on a ReactiveX library: RxJS.

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Reactive client application state constraint contracts

As in Design By Contract (DbC), the contents and structure of a Store mutation can be constrained by Preconditions, Postconditions, and Invariants. These constraints are part of the Store implementation.

Background
Modern JavaScript frameworks are now emphasizing a form of “global” state storage. This is not really new, but gained new impetus when React came out, which synergistically combined this with other factors, like components, virtual-DOM, and one-way data flow.

React Stores

“… contain the application state and logic. Their role is somewhat similar to a model in a traditional MVC, but they manage the state of many objects — they do not represent a single record of data like ORM models do. Nor are they the same as Backbone’s collections. More than simply managing a collection of ORM-style objects, stores manage the application state for a particular domain within the application.” — https://facebook.github.io/flux/docs/overview.html

In very large apps and multiple development teams, this Store could become a high maintenance cost and a source of defects.

Non-React Store
For example of a reusable Store implementation see Redux. Amazing video: Dan Abramov – Live React: Hot Reloading with Time Travel at react-europe 2015

Interesting Java implementation of Redux: redux-java.

Maintenance
Today’s great application and shiny new framework are tomorrow’s maintenance nightmare. Thus, it makes sense to add in features that will aid this future support.

Example
A store has a JSON map of { foo : true, fee : false, fum : {…}, … }.

A particular flow in the application will update foo to false. Is that ok?

If foo and fee are in the same ‘context’, can they both be false? Note that the ‘application’ or maintenance developer may not know that there is a relationship between foo and fee. With annotations, dependency injection, and so forth, even perusal of the source could make this determination difficult.

Sure, in a small app and simple Store, it would be easy to trace the program and find out the implicit constraints. In reality, Stores grow complex over time, the app gets complex, new requirements mean many changes, and the developer culprit has left the company. Stores will mutate to resemble the catch-all Windows’ Registry.

To compound this, the control flows are hidden within the framework being used. No matter if its today’s MVC usurper, React, or some Observable liberator, there is spaghetti flow and implicit state somewhere.

“At some point, you no longer understand what happens in your app as you have lost control over the when, why, and how of its state. When a system is opaque and non-deterministic, it’s hard to reproduce bugs or add new features.” —
http://redux.js.org/docs/introduction/Motivation.html

Constraints
One way to reduce issues is to use the techniques from data design and use schema constraints. This does not necessarily mean the use of actual schemas as in XML’s XSD or its wannabe JSON equivalent. One alternative is that the Store has an API for constraint binding to content, new life cycle functions apply the constraints. The result of a constraint violation depends on dev or production deployment, log or throw of exceptions.

Afterword
Notice how in the development world we are going back in time? Central stores, Data flow diagrams, functional programming, FSM (obfuscated), etc. Maybe Object-Oriented Programming was a great distraction like Disco. Time for some real funk. “Maceo, I want you to Blow!

Wow, this synchronicity is weird. I’ve been thinking of this post’s contents for a while. Started to write it and my twitter app on phone rings. @brianloveswords tweets about a blog post by @jlongster: “Starters and Mainainers”. Not exactly the same subject, but was about maintenance.

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Java ‘hello world’ using RxJava library

A simple RxJava example that uses an array of strings and combines them to create the traditional first program.

Jump to source code

RxJava is a NetFlix open source library that they developed as part of optimizing their architecture. The library is related to the “Reactive programming” pattern:

“RxJava is an implementation of Reactive Extensions – a library for composing asynchronous and event-based programs using observable sequences for the Java VM”.

The main data structure in ReactiveX is, as presented in “Streams: The data structure we need” by Pam Selle, is the Stream. Streams are discussed in the SICP book, 3.5 Streams.

The main resource for RX in general is http://reactivex.io/

Updates

  • Oct 18, 2015: Revised blog post and source code.
  • Dec 26, 2015: Read an article on ribot’s site, “Unit Testing RxJava Observables“, that RxJava already has experimental support for unit testing assertions.

Approach

I thought I would try my hand at it. I used the ‘Hello world’ example on the RxJava wiki’s getting started page:

public static void hello(String... names) {
      Observable.from(names).subscribe(new Action1<String>() {
          @Override
          public void call(String s) {
              System.out.println("Hello " + s + "!");
          }
      });
}

As a source of gradual complexity, I used a list of strings for the target output, and I also don’t include the end “!” in the input stream. With this I was able to use more of the library to get a feel for the syntax and how it relates to the underlying concepts.

Learning Reactive
One problem with many introductions to RxJava is that they start off with overly simple or complex examples, and these complex examples require domain knowledge of some application. BTW, the worse example of this tendency was a book on Design Patterns where the author used obscure sports racing concepts and so forth.

Another problem is that the major Operators are very abstractly documented. Just look at the and/then/when or join operators. You’d think each Rx implementation’s documents would give examples for each operator. Update: I found one tutorial that has much more RxJava examples: Intro-To-RxJava

A good resource on picking Reactive operators is A Decision Tree of Observable Operators.

Example
Below I created a JUnit test class in Java that creates an Observable from an array [“Hello”, “world”], and each test subscribes to it. That is, the test is the Observer. The subtle complexity is that the required “!” at the end of result string is not in the String array. How is that added withing the ‘Rx’ pattern? RxJava has a rich API, thus there are many ways to take that array and create the “Hello world!” string.

Concats
Test2 in the source listing 1 is my attempt to avoid ‘programmatic’ addition of the ending “!” to the String result. Instead, I concat three Observables: a concatMap that takes each streamed in String and creates a new Observable with [data, ” “], an Observable that skips the last [data,” “], and finally an Observable that just creates the “!”.

Yes, this is a lot of complex code for this simple task. The point was to use a simple task to learn the complex code. Later, the complexity would be tamed or matched to the task at hand.

How to fail JUnit test
Since I used JUnit tests to write the code, I wondered how would you detect that an RxJava operator got an error and failed the test? For example, during the onCompleted method? An error there will not invoke the onError method since the Observable is done generating any data. Thus, in Test4, I had to wrap the assertion fail in a try catch, then manually invoke the onError method. Probably semantically incorrect approach. But, that still would not make the JUnit test fail, I had to set a field so that the last statements in the test would use that to fail the test. Seems like a kludge.

Dec 26, 2015
Using RxJava’s test support is the way to go for simple assertions. Below I use TestSubscriber.

/**
 * Dan Lew's example in 'Grokking RxJava, Part 1: The Basics'.<p>
 * Converted to a test.
 * @see "http://blog.danlew.net/2014/09/15/grokking-rxjava-part-1/"
 * @throws Exception
 */
@Test
public final void test6() throws Exception{
	TestSubscriber<String> testSubscriber = new TestSubscriber<>();
	
	Observable.just("Hello, world!")
	.map(s -> s + " -Dan")
	.map(s -> s.hashCode())
	.map(i -> Integer.toString(i))
	.subscribe(testSubscriber);
	// .subscribe(s -> System.out.println(s)); // -
	testSubscriber.assertValue("-238955153");
}

I’ll leave most of the sample code as is for now. But, its best to use the library to its fullest before you create your own testing kludge.

Summary
Have not grokked RxJava yet, so the above is probably not idiomatic use or correct.

RxJava looks like a powerful system, but it is very sophisticated and may require a long grok curve. There are various YouTube videos and other presentations that give a contextual understanding of RX in general. Ben Christensen’s presentations are awesome.

Options?
There are many Reactive libraries available. Another is Pivotal’s Reactor. See this article “Pivotal’s Reactor Goes GA” A review of Reactor is “Playing with Reactor“.

The “React” term?
Many things are using some form of this term. For example, ReactJs, the M~V library from FaceBook. Sure, it relates, however the concept of event streams is not central to the library’s conceptual model, afaik.

JavaScript
Perhaps one of the most important use of RX, in terms of usage, is on the client side. One library is RxJS.

Cycle.js
There is even a framework, Cycle.js created by André Staltz, née Medeiros, that combines the “reactive” concept with RxJS to create a new paradigm of client structure called Model View Intent (MVI). Some intro videos on CycleJS library:

 


 

Dependencies

  • Java 8 (some tests are using Java 8 syntax)
  • JUnit – ‘junit:junit:4.12’
  • RxJava – ‘io.reactivex:rxjava:1.0.14’
  • Guava – ‘com.google.guava:guava:18.0’

Dev System

  • Windows 10
  • Eclipse Mars

Links

Listing 1, Full Source
Having the source as a JUnit test is helpful since the code can be used as a means to experiment and extend, while having the ability to easily test regressions.


Listening to Jake Bugg: “Broken”

On youtube:

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.