Category Archives: Version control

Using Git with BitTorrent Sync

The various cloud sync services provide a good way to backup or remote a Git repository for ‘single developer’ situations. The main advantages are that the cost is minimal and no ‘server’ is required. In this post we use something quite different, the new BitTorrent Sync (BTSync) system.

Use Cases
1. Single developer on multiple devices, PC, laptop, mobile. Also see Single Developer Git Workflow, .
2. Easy, fast, and lightweight code sharing setup. See also Sneakernet with Git
3. Ad Hoc Version Control: How to do Ad Hoc Version Control, Ad Hoc Version Control With Git

BTSync is a serverless folder syncing system. Instead of using a remote server storage system, it creates a fast private peer-to-peer file sync system using a P2P protocol. Note it is not necessarily a replacement for a server, backup system, or even other services such as DropBox, more like a welcome addition that covers some limits that others may have, such as file size limitations, speed, and privacy.

Using this type of service is very easy. I took the easy way out and “forked” a very well written blog post by Sergei Shvetsov that did the same thing, only using DropBox. Using Git with Dropbox. In this post, however, I use BTSync and I am running on Windows (most blogs show examples on *nix). Of course, experienced Git users may approach this very differently. The following uses a console shell UI.

Process
1. Create a local repo
2. Create a “bare” repo that lives in the Synced folders
3. Add the bare repo as the origin of the local repo.

Result
Now on another system, the synced folder, which contains the bare repo is available as if it was created locally. During development or other uses, since we are using the working repo and only occasionally the ‘bare’ or origin repo in the synced folder, the synced folder is not constantly transferring data over the network to any other synced locations (there can be many).

This approach is diagrammed below. (On mobile device this ASCII diagram using <PRE> tag looks horrible.) Two systems with the two Git repos:

+-----------------+                     +----------------+
|     Local       |                     |     Local      |
|     Repo        |                     |     Repo       |
+-----------------+                     +----------------+
      ^ +                                    +   ^
      | |                                    |   |
      | | push/pull                          |   |  push/pull
      | |                                    |   |
      | |                                    +   |
      + v                                    v   +
+-----------------+         BT Sync     +----------------+
|      Bare       | <-----------------+ |      Bare      |
|      repo       | +------------------>|      repo      |
+-----------------+                     +----------------+

Alternatives
Of course whenever possible direct access to remote repos from a clone is preferred, or via a server. For private use and 5 users, Bitbucket offers free code repositories.

Example
Below is a walkthrough of this process using a Windows cmd shell.

Create a local repository

C:\temp\git-on-BTSync>git init new-project
Initialized empty Git repository in C:/temp/git-on-BTSync/new-project/.git/

C:\temp\git-on-BTSync>cd new-project
C:\temp\git-on-BTSync\new-project>echo "" > README.txt

git add .
C:\temp\git-on-BTSync\new-project>git commit -m "Initial Commit"
[master (root-commit) dcb3f2b] Initial Commit
 1 file changed, 1 insertion(+)
 create mode 100644 README.txt

Create a new ‘bare’ repo inside of local BTSynced folder

new-project>mkdir C:\Users\jbetancourt\BTSync\git
C:\new-project>git init --bare C:\Users\jbetancourt\BTSync\git\new-project.git
Initialized empty Git repository in C:/Users/jbetancourt/BTSync/git/new-project.git/

Add this new bare repo as upstream remote to the local repo

new-project>git remote add dropbox C:\Users\jbetancourt\BTSync\git\new-project.git

Push local changes to the bare repo

C:\temp\git-on-BTSync\new-project>git push -u dropbox master
Counting objects: 3, done.
Writing objects: 100% (3/3), 223 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To C:\Users\jbetancourt\BTSync\git\new-project.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from dropbox.

Use another system
On another system that is running BTSync, like a laptop, the bare repository folder is already synced. Now we can clone the repo.

cd \temp
mkdir remote-workspace
cd remote-workspace
git clone -o dropbox \Users\jbetancourt\BTSync\git\new-project.git
Cloning into 'new-project' ...
done.
cd new-project
dir
11/29/2013  10:48 AM      5 README.txt

Now make some changes
On the laptop we modify a file and commit it.

echo "Hello world" > README.txt
C:\temp\remote-workspace\new-project>type README.txt
"Hello world"
C:\temp\remote-workspace\new-project>git add README.txt

C:\temp\remote-workspace\new-project>git commit -m "changed readme"
[master 452b02e] changed readme
1 file changed, 1 insertion(+), 1 deletion(-)

Now push the changes to the local bare repo

C:\temp\remote-workspace\new-project>git push
Counting objects: 5, done.
Writing objects: 100% (3/3), 260 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To \Users\jbetancourt\BTSync\git\new-project.git
   dcb3f2b..452b02e  master -> master

Back to the original repo on the PC
We pull the changes that were automatically synced via BTSync into
the bare repo.

type README.txt
""
C:\temp\git-on-BTSync\new-project&gt;git pull
remote: Counting objects: 5, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From C:\Users\jbetancourt\BTSync\git\new-project
   dcb3f2b..452b02e  master     -> dropbox/master
Updating dcb3f2b..452b02e
Fast-forward
 README.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

C:\temp\git-on-BTSync\new-project>type README.txt
"Hello world"

That was easy!

Issues with syncing git repos with BTSync?

  • BitTorrentSync has been updated and now the process of syncing folders is either easier or harder, depending on you viewpoint. There are also some limitation for the free product, like ten folder limit.
  • The machine having a source sync must be on to allow syncing, of course. Not true with a server based sync solution like Dropbox. This is only required while the local and remote folders are syncing of course. In many BTSync articles and blog posts the wrong impression is given that this is a continuous requirement. In fact, as soon as you see the sync complete, if you have the BTSync app visible you can shut down the source machine.
  • Damage is also synced: If one of the synced repos gets damaged, that damage is reproduced in all correlated syncs. This can be prevented by using BTsync’s read only share feature. This would introduce some limitations or other complexities.
  • Repository ignored files are synced
  • There was a discussion on whether the .git folder should be synced. Not sure I follow the rational.
  • I don’t know if there are any issues with BitTorrent Sync for long term work with a Git repo. People have complained of such issues with Dropbox. See the link for Mercurial use on DropBox below. In the comments of that blog post, robhamilton posts: “… found that it would break the Mercurial repo. Mercurial locks files and creates temp journal files which get sync’d by the dropbox daemon. My advice is to stop dropbox, perform your push/commit, then restart dropbox. Pulls and clones are readonly.” Is this an issue with Git? I don’t think so since we are using the bare repo approach.

Updates

    Mobile Git

  • Dec 24, 2013: I did not investigate the mobile Git use with BTSync as shown above. BTSync has a mobile app that allows the sync to mobile devices. On that device a mobile Git client can access the synced bare repo to clone into a mobile local repo. There are now a few mobile Git clients, for example, SGit.
  • June 26, 2015: Bittorrent Sync has an API: BitTorrent Gives Developers A Cloud-Free Alternative.

Some links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

“.ignore” files should not be stored in a project’s source

Many version control systems use ignore files in a folder to indicate resources that should not be tracked , the target ignored files. These ignore files could be in any folder in the project where something must be ignored, natch.

These ignore files are a separate concern, not germane to the purpose of the source code. Should we also have files that indicate how to compile? Compile and config concerns are localized, either in the build scripts or in other types of management system. Likewise, the version control metadata should also be separate from the source.

This is comparable to how in the past Subversion version control system had .svn files throughout a project’s source folders. These were used as part of their vcs “bookkeeping”. Currently the “bookkeeping” was centralized into one .svn file at the project’s root folder, like found in Git. (Not an SVN expert so this is just what I see when I use SVN).

A few that use ignore files all over the place are: Git, Mercurial, CVS, and Subversion. Yes, Subversion too, except that it hides these in this nebulous ‘properties’ objects that are not seen but can cause great pain.

To see how to use these, see for example Git’s: gitignore(5) Manual Page

Any issues?
Other than be a context leak, any issues? None directly. Yet, if you search the web, there are many pleas for making automation tools and apps work around the inclusion of these files. Extra steps must be taken to remove from views in IDE or in searches. And so forth.

What should be done?
Ignore files or properties should be handled just like everything else. They should be localized in the version control system’s metadata folder (db or workspace).

For example, with Git, the .git repository could have a folder called “ignores”:

./.git/hooks
./.git/ignores
./.git/refs
...
 

Cons
There are probably many reasons for the current approach: performance, simplicity, etc. WTHDIK

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Transform SVN log diff using Java XPath

Sometimes the output of a version control system’s tools or command options must be transformed for further use. For example, you generate a log report and must then import that into a spreadsheet or database.

Options
One of the options for doing this is with text based tools and scripting. Another approach is using the XML output of the version control system if available. With Subversion, XML can be used for the log and diff reports. Now you can use the various XML tools, such as XQuery, XPath, XSLT, and so forth.

DVCS?
Out of the box Git or Mercurial, two popular Distributed Version Control Systems (DVCS), do not output in XML as easily as SVN’s –xml option. To use XML, you have to define a ‘formatter’ or ‘pretty-print’ for them to use.

XPath
Below I use Java and XPath to transform the SVN output into Comma-separated values (CSV) files. One good reason for using an imperative approach is that this gives the opportunity to perform more complex transformations. A case in point is that in SVN the XML output format is only available for summary reports, and thus to create more comprehensive results or send to complex destinations, more processing would be required.

Source
The source is just a simple approach with minimal error handling and limited to CSV output. Originally I had two methods with internal loops that were almost identical, so I made the loop handling into a call-back that gets invoked by an Anonymous class. Kind of a simple Visitor or Strategy Pattern? Closures, as available in scripting languages like Groovy, would have been a simpler solution.

Of course a more comprehensive approach would abstract the output format or use other methods like POI to create a spreadsheet. Note that this approach requires loading the full XML into memory. For very large logs or diffs, a streaming approach would be needed.

Source also available here.

Example source SVN diff and log tranform to CSV
/**
 * 
 */
package com.octodecillion.utils;

import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.Reader;
import java.io.Writer;

import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;

import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;

/**
 * Transform SVN diff and log XML output files.
 * 
 */
@SuppressWarnings("javadoc")
public class SvnOutputTransform {

	/** Example run */
	public static void main(final String[] args) {
		try {
			SvnOutputTransform svnTransform = new SvnOutputTransform();

			svnTransform.diffSummaryToCsv("data/diff-summary.xml",
					"bin/SvnDiff.csv");

			svnTransform.logSummaryToCsv("data/log-summary.xml",
					"bin/SvnLog.csv");

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

	/** Call back interface */
	interface RowProcess {
		/** accept method of Vistor pattern */
		public void doRows(NodeList nodeList) throws Exception;
	}

	/**
	 * 
	 * @param dataFilePath
	 * @param reportFilePath
	 * @param nodeString
	 * @param processRows
	 */
	public void generateReport(final String dataFilePath,
			final String reportFilePath, final String nodeString,
			final RowProcess processRows) {

		try {
			NodeList nodeList = setup(dataFilePath, reportFilePath, nodeString);
			reportOut.println(HEADER_COLUMN);
			processRows.doRows(nodeList);
		} catch (Exception e) {
			throw new SvnTransformException("", e);
		} finally {
			finallyHandler(reportOut, fr);
		}
	}

	/**
	 * 
	 * @param dataFilePath
	 * @param reportFilePath
	 * @param xpathString
	 * @return
	 * @throws Exception
	 */
	private NodeList setup(final String dataFilePath,
			final String reportFilePath, final String xpathString) throws Exception {

		builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
		xp = XPathFactory.newInstance().newXPath();
		fr = new FileReader(dataFilePath);
		dom = builder.parse(new InputSource(fr));
		reportOut = new PrintWriter(new File(reportFilePath));

		Object nodes = xp.evaluate(xpathString, dom, XPathConstants.NODESET);
		NodeList nodeList = (NodeList) nodes;

		return nodeList;

	}

	/**
	 * Convert SVN generated diff summary xml file to CSV. 
	 * 
	 * The format of the input XML is:
	 * <diff>
	 *	<paths>
	 *		<path props="none" kind="file" item="modified">
	 *			full path of resource
	 *		</path>
	 *  </paths>
	 * </diff>
	 * 
	 * @param dataFilePath xml file generated by svn diff --xml --summary ....
	 * @param reportFilePath destination of CSV file
	 * @throws SvnTransformException 
	 */
	public void diffSummaryToCsv(final String dataFilePath,
			final String reportFilePath) {

		preconditionCheck(dataFilePath, reportFilePath);
		
		generateReport(dataFilePath, reportFilePath, "//diff/paths/path",
				new RowProcess() {
					@SuppressWarnings("synthetic-access")
					@Override
					public void doRows(final NodeList nodeList) throws Exception {
						for (int i = 0; i < nodeList.getLength(); i++) {
							Node node = nodeList.item(i);
							String kind = xp.evaluate(KIND_ATTR_NAME, node);
							String item = xp.evaluate(ITEM_ATTR_NAME, node);
							String pathEntry = xp.evaluate("text()", node);

							// row
							reportOut.println(String.format(
									"%s,%s,%s,%s,%s,%s,%s", DIFFUSER, revision,
									date, item, kind, pathEntry, message));
						}
					}
				});
	}

	/**
	 * Convert SVN generated log summary xml file to CSV.
	 * 
	 * <log>
	 * 		<logentry revision="10879">
	 * 			<author>T16205</author>
	 * 			<date>2013-03-15T18:10:07.264531Z</date>
	 * 			<paths>
	 * 				<path kind="file" action="A">
	 * 					/2013/Amica/branches/SOW114-LifeAPP/Test/Resources/Properties/Test/HomeQuotingFields.inix
	 * 				</path>
	 * 			</paths>
	 *      </logentry>
	 * </log>
	 * 
	 * @throws SvnTransformException 
	 */
	public void logSummaryToCsv(final String dataFilePath,
			final String reportFilePath) {
		preconditionCheck(dataFilePath, reportFilePath);

		generateReport(dataFilePath, reportFilePath, "//log/logentry",
				new RowProcess() {
					@SuppressWarnings("synthetic-access")
					@Override
					public void doRows(final NodeList nodeList) throws Exception {
						for (int i = 0; i < nodeList.getLength(); i++) {
							Node node = nodeList.item(i);
							String author = xp.evaluate(AUTHOR_NODE, node);
							String date = xp.evaluate(DATE_NODE, node);
							String revision = xp.evaluate("@revision", node);
							String message = "\"" + xp.evaluate(MSG_NODE, node) + "\"";

							NodeList paths = (NodeList) xp.evaluate(
									"paths/path", node, XPathConstants.NODESET);

							if (paths != null) {
								for (int k = 0; k < paths.getLength(); k++) {
									Node aPath = paths.item(k);

									String action = xp.evaluate("@action",
											aPath);
									action = actionToName(action);

									String filePath = xp.evaluate("text()",
											aPath);

									// row
									reportOut.println(String.format(
											"%s,%s,%s,%s,%s,%s", author,
											revision, date.split("T")[0],
											action, filePath, message));
								}
							}

						} // end each logentry		
					}
				});

	} // end logToCsv
	
	/**  */
	private String actionToName(final String n) {

		try {
			return ACTION.valueOf(n).getName();
		} catch (Exception e) {
			return n;
		}
	}

	private void finallyHandler(final Writer reportOut, final Reader fr) {
		if (reportOut != null) {
			try {
				reportOut.flush();
				reportOut.close();
			} catch (Exception e) {
				// 
			}
		}

		if (fr != null) {
			try {
				fr.close();
			} catch (IOException e) {
				// 				
			}
		}
	}

	/**
	 * @param dataFilePath
	 * @param reportFilePath
	 * @throws IllegalArgumentException
	 */
	private void preconditionCheck(final String dataFilePath,
			final String reportFilePath) throws IllegalArgumentException {
		if ((dataFilePath == null) || (reportFilePath == null)) {
			throw new IllegalArgumentException(String.format(
					"dataFilePath='%s',reportFilePath='%s'", dataFilePath,
					reportFilePath));
		}
	}

	/**
	 * SVN action codes.
	 * 
	 */
	enum ACTION {
		A("add"), M("modify"), D("delete"), Z("z");

		private final String name;

		private ACTION(final String name) {
			this.name = name;
		}

		public String getName() {
			return name;
		}
	}

	/** Svn transform runtime exception */
	static public class SvnTransformException extends RuntimeException {
		private static final long serialVersionUID = 1L;

		/**  */
		public SvnTransformException(final String message, final Throwable cause) {
			super(message, cause);
		}
	}

	private static final String HEADER_COLUMN = "Dev,Revision,Date,Action,Kind,Path";
	private static final String ITEM_ATTR_NAME = "@item";
	private static final String KIND_ATTR_NAME = "@kind";
	private static final String MSG_NODE = "msg";
	private static final String DATE_NODE = "date";
	private static final String AUTHOR_NODE = "author";
	private PrintWriter reportOut;
	private FileReader fr;
	private static final String DIFFUSER = "DIFF";
	private final String revision = "";
	private final String date = "";
	private final String message = "";
	private DocumentBuilder builder;
	private XPath xp;
	private Document dom;

}

 

Updates

Links

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Groovy script to bootstrap a Gradle Project

I took Ted Naleid’s “Quick Shell Function to Bootstrap a Gradle Groovy Project” example code and converted it to a Groovy script.

Updates
1. Fixed. This was not running. I was using triple quoted string, when I should have used the triple single-quoted string for the HEREDOC like usage below.
2. Gradle now includes (since 1.6) a Build Init Plugin that will create a project. This should be used and not the code in this blog post.
In Gradle 1.9 this works:

    gradle init --type java-library
    

3. Gradle IDE plugins have the ability to create Gradle based project.
4. Would be nice to be able to reuse Maven style “archetypes”.


Deprecated
But, may serve as a Groovy code example.
Listing 1.

// NewGradle.groovy
// Author: Josef Betancourt
// Based on T. Naleid's shell script 

println "Creating files for new Gradle project ..."

new File(".gitignore").withPrintWriter{ w ->
     "*.un~,*.iml,*.ipr,*.iws,build,.gradle".split(",").each{ 
         w.println(it)
     }
}

new File("build.gradle") << '''
apply plugin: 'groovy'
apply plugin: 'idea'
apply plugin: 'eclipse'
 
repositories {
    mavenCentral()
}
 
dependencies {
    groovy 'org.codehaus.groovy:groovy:1.8.6'
    compile 'org.apache.ivy:ivy:2.2.0'
}
 
task createSourceDirs(description : 
   'Create empty source directories for all defined sourceSets') 
   << {
        sourceSets*.allSource.srcDirs.flatten().each { 
            File sourceDirectory ->        
            if (!sourceDirectory.exists()) {
                println "Making $sourceDirectory"
                sourceDirectory.mkdirs()
            }
        }
}
 
idea {
    project {
        jdkName = '1.6'
    }
}

''' // end content

"cmd /c gradle createSourceDirs".execute()

"cmd /c git init".execute()

Thread.start{
	sleep 5000 // allow time for all files to be created
	new File(".").eachFile{
		println it
	}
}

Not expert Groovy, but was easy to do. The bulk of it is the creation of a “here” doc using Groovy’s triple quote string. I didn’t duplicate the last line of Naleid’s script: “ls -a1 && find src # list all created assets”.

This script is not fully cross-platform. The invocation of shell commands at the end are in the Windows format. Left as an exercise to reader is the use of inline AntBuilder to reuse Ant’s exec task. 🙂

Updates
2012-03-09: Tweaked the source. Removed use of two temp variables.
2012-03-09: Added the Eclipse plugin. Now after creating the Gradle project executing eclipse will create the eclipse project: gradle eclipse
Or instead generate the Idea project: gradle idea.

Further Reading

  1. Groovy (Programming language)
  2. Groovy
  3. Gradle
  4. Quick Shell Function to Bootstrap a Gradle Groovy Project
  5. Strings and GString
  6. Groovy JDK extensions
  7. Executing External Processes
  8. Using Gradle to Bootstrap your Legacy Ant Builds

Off Topic

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

New Subversion version will centralize working copy metadata

Subversion version 1.7 removes that gruesome use of those very numerous .svn folders in your working copy. CVS also used that approach, just named .cvs.

This is more like how many ‘newer’ version control systems (VCS), like Git or Mercurial, lay out a repository.

More info: Working Copy Metadata Storage Improvements (client)

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Single Developer Git Workflow

I’m finding Git to be very useful. Since I use it in conjunction with the “official” VCS at my job, I classify this as a SDGWF situation. Below I give an example of the simple workflow I’m currently experimenting with. The following is based on a Windows development workstation. Usually any blog posts involving Git are *nix based.

What is shown?

  1. Use of a bare repository
  2. Branching
  3. Merging

Note: This is an old post, and there are hopefully much better examples on web about single-developer workflows. The Simplified Git Process blog post is very good.

log in gui
graphic log

Two local repos
Creating the two repositories:

cd \temp\git\SdWf
C:\temp\git\SdWf>mkdir project
C:\temp\git\SdWf>mkdir depot
C:\temp\git\SdWf>cd depot

C:\temp\git\SdWfdepot>git init --bare project.git
Initialized empty Git repository in C:/temp/gitSdWf/depot/project.git/

C:\temp\git\SdWfdepot>cd ..\project
C:\temp\git\SdWf\project>git init --separate-git-dir ..\depot\project.git
Reinitialized existing Git repository in C:/temp/gitSdWf/depot/project.git/
C:\temp\git\SdWf\project>echo hello world! > hello.txt
C:\temp\git\SdWf\project>git add .
C:\temp\git\SdWf\project>git commit -m "initial commit"
[master (root-commit) e935e02] initial commit
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 hello.txt

C:\temp\git\SdWf\project>git log
commit e935e0220ab90945f6160db00eee6bfd13173b6d
Author: josef betancourt 
Date:   Mon Oct 3 19:07:29 2011 -0400
    initial commit

Note that the first repository created is “bare“, i.e., it does not have any work files. It is not created in the project work folder. At the project folder, a new Git repository is created, but with the –separate-git-dir option the first repository is used as the actual project repository.

Instead of a .git subdirectory, a .git file is created at the project folder, and this file points to the actual Git repo.

Why use this setup?
One reason is to allow tools to transparently work with the sanctioned project folder. For example, if we use the ‘normal’ repo setup, to zip the project, one would have to filter out the
.git subfolder (which could get large). Another reason is to allow backups of the working repo; one can delete a project and inadvertently delete the local repo. The last two reasons could be avoided by not creating the local repo within the project folder, of course.

Usage examples
Next day we are assigned some tasks at work. For this example, we’ll make up a naming approach:
defect: dNNNNN-name
feature: fNNNNN-name

Also, for this example, we’ll just have two defects and one feature:
d12345-validation-fails
d54321-login-box
f52123-error-report

Rather then waiting to create the branches when the branch is needed, we just create them now:

git branch d12345-validation-fails
git branch d54321-login-box
git branch f52123-error-report

Our repository now has four branches:

C:\temp\git\SdWf\project>git branch | sort
  d12345-validation-fails
  d54321-login-box
  f52123-error-report
* master

Let’s work on the first defect.

git checkout b12345-validation-fails

C:\temp\git\SdWf\project>git branch | sort
  d54321-login-box
  f52123-error-report
  master
* d12345-validation-fails

Make this branch is up to date against the master branch:

C:\temp\git\SdWf\project>git merge master
Already up-to-date.
C:\temp\git\SdWf\project>git commit -a -m "worked on defect"
[d12345-validation-fails 0aa5930] worked on defect
 1 files changed, 1 insertions(+), 1 deletions(-)

C:\temp\git\SdWf\project>git diff master

diff --git a/hello.txt b/hello.txt
index d4f3cf2..9f2b679 100644
--- a/hello.txt
+++ b/hello.txt
@@ -1 +1 @@
-hello world!
+initial hello ^M

Work on the other defect:

C:\temp\git\SdWf\project>git checkout d54321-login-box
Switched to branch 'd54321-login-box'

C:\temp\git\SdWf\project>git merge master
Already up-to-date.
C:\temp\git\SdWf\project>git checkout d54321-login-box
Switched to branch 'd54321-login-box'

C:\temp\git\SdWf\project>git merge master
Already up-to-date.

C:\temp\git\SdWf\project>echo Have to go. >> hello.txt

C:\temp\git\SdWf\project>git commit -a -m "made some fixes"
[d54321-login-box db17a9e] made some fixes
 1 files changed, 1 insertions(+), 0 deletions(-)

C:\temp\git\SdWf\project>git diff master

diff --git a/hello.txt b/hello.txt
index d4f3cf2..7fe577f 100644
--- a/hello.txt
+++ b/hello.txt
@@ -1 +1,2 @@
 hello world!
+Have to go. ^M

Now lets switch to the feature branch


C:\temp\git\SdWf\project>git checkout f52123-error-report
Switched to branch 'f52123-error-report'

C:\temp\git\SdWf\project>git merge master
Already up-to-date.

C:\temp\git\SdWf\project>echo Created new feature >> hello.txt

C:\temp\git\SdWf\project>git diff master
diff --git a/hello.txt b/hello.txt
index d4f3cf2..17244a7 100644
--- a/hello.txt
+++ b/hello.txt
@@ -1 +1,2 @@
 hello world!
+Created new feature ^M

C:\temp\git\SdWf\project>git commit -a -m "finished the feature"
[f52123-error-report aef7f49] finished the feature
 1 files changed, 1 insertions(+), 0 deletions(-)


C:\temp\git\SdWf\project>git log --graph
* commit e935e0220ab90945f6160db00eee6bfd13173b6d
  Author: josef betancourt 
  Date:   Mon Oct 3 19:07:29 2011 -0400

      initial commit

Lets log all the branches ….


C:\temp\git\SdWf\project>git log --graph --branches
* commit aef7f4990b63dbb8da4f81558ac492609afa3522
| Author: josef betancourt 
| Date:   Mon Oct 3 19:33:09 2011 -0400
|
|     finished the feature
|
| * commit db17a9eb9e0672b0e6d2b9a9bef645a5d3a6e1de
|/  Author: josef betancourt 
|   Date:   Mon Oct 3 19:29:41 2011 -0400
|
|       made some fixes
|
| * commit 0aa5930c88d619b58ce14908ba8795736988d46c
|/  Author: josef betancourt 
|   Date:   Mon Oct 3 19:26:05 2011 -0400
|
|       worked on defect
|
* commit e935e0220ab90945f6160db00eee6bfd13173b6d
  Author: josef betancourt 
  Date:   Mon Oct 3 19:07:29 2011 -0400

      initial commit

Show the detailed branch structure

C:\temp\git\SdWf\project>git show-branch
! [d12345-validation-fails] worked on defect
 ! [d54321-login-box] made some fixes
  ! [f52123-error-report] finished the feature
   * [master] initial commit
----
  +  [f52123-error-report] finished the feature
 +   [d54321-login-box] made some fixes
+    [d12345-validation-fails] worked on defect
+++* [master] initial commit

Now check out master branch and merge the first defect

git checkout master
C:\temp\git\SdWf\project>git merge --no-ff d12345-validation-fails
Merge made by recursive.
 hello.txt |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

C:\temp\git\SdWf\project>git merge --no-ff d54321-login-box
Auto-merging hello.txt
CONFLICT (content): Merge conflict in hello.txt
Automatic merge failed; fix conflicts and then commit the result.

Yuck, a merge conflict. What was the conflict?


C:\temp\git\SdWf\project>type hello.txt
<<<<<<>>>>>> d54321-login-box

C:\temp\git\SdWf\project>npp hello.txt

C:\temp\git\SdWf\project>"C:Program Files (x86)Notepad++notepad++.exe" hello.txt

Edit the file and accept the changes, then merge …


C:\temp\git\SdWf\project>type hello.txt
initial hello
hello world!
Have to go.

C:\temp\git\SdWf\project>git status
# On branch master
# Unmerged paths:
#   (use "git add/rm ..." as appropriate to mark resolution)
#
#       both modified:      hello.txt
#
no changes added to commit (use "git add" and/or "git commit -a")

C:\temp\git\SdWf\project>git commit -a -m "merged in 2nd defect"
[master cc7ebf6] merged in 2nd defect

Now merge the feature branch into master


C:\temp\git\SdWf\project>git merge --no-ff f52123-error-report
Auto-merging hello.txt
CONFLICT (content): Merge conflict in hello.txt
Automatic merge failed; fix conflicts and then commit the result.

C:\temp\git\SdWf\project>type hello.txt
initial hello
hello world!
<<<<<<>>>>>> f52123-error-report

Edit the file, then commit the merge


C:\temp\git\SdWf\project>npp hello.txt

C:\temp\git\SdWf\project>"C:Program Files (x86)Notepad++notepad++.exe" hello.txt

C:\temp\git\SdWf\project>type hello.txt
initial hello
hello world!
Have to go.
Created new feature
C:\temp\git\SdWf\project>git commit -a -m "merged in feature"
[master dc58815] merged in feature

Show the final results using a log with graphics ….


*   commit dc58815ae250ee089e9e83db3b6e301a5a4552a9
|  Merge: cc7ebf6 aef7f49
| | Author: josef betancourt 
| | Date:   Mon Oct 3 19:57:14 2011 -0400
| |
| |     merged in feature
| |
| * commit aef7f4990b63dbb8da4f81558ac492609afa3522
| | Author: josef betancourt 
| | Date:   Mon Oct 3 19:33:09 2011 -0400
| |
| |     finished the feature
| |
* |   commit cc7ebf6c79fc0abb82b2b0d1fd5e53bcbf4fb94c
|   Merge: 8e47975 db17a9e
| | | Author: josef betancourt 
| | | Date:   Mon Oct 3 19:55:25 2011 -0400
| | |
| | |     merged in 2nd defect
| | |
| * | commit db17a9eb9e0672b0e6d2b9a9bef645a5d3a6e1de
| |/  Author: josef betancourt 
| |   Date:   Mon Oct 3 19:29:41 2011 -0400
| |
| |       made some fixes
| |
* |   commit 8e479756f7db2b926259c9774d5fd8cb38d14935
|   Merge: e935e02 0aa5930
| |/  Author: josef betancourt 
|/|   Date:   Mon Oct 3 19:51:30 2011 -0400
| |
| |       Merge branch 'd12345-validation-fails'
| |
| * commit 0aa5930c88d619b58ce14908ba8795736988d46c
|/  Author: josef betancourt 
|   Date:   Mon Oct 3 19:26:05 2011 -0400
|
|       worked on defect
|
* commit e935e0220ab90945f6160db00eee6bfd13173b6d
  Author: josef betancourt 
  Date:   Mon Oct 3 19:07:29 2011 -0400

      initial commit

That’s a simple example. I’m sure I missed some steps during the text captures.

Currently I’m just using the “master” branch as the development branch. I only use branches for short term coding. I may go back to using more branches as I get better with Git.

Further reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Sneakernet with Git?

When remote connection is not possible, you can put a Git repo on a stick. But how do you merge without getting so many conflicts?

Why would you get so many conflicts? Well, since they are the same files, and you are changing them in local and remote, you will probably change the same lines. Or my present Git understanding is missing something, like a force option?

If you don’t care about branching and all that, just use file synchronization tools.

Here is an example session where the USB stick contains a Git repo:

git fetch  path-to-repo-on-stick
git merge -s recursive -Xtheirs --no-commit FETCH_HEAD

... now inspect, test, etc. ...

git commit

Explanation
1. Get fetch will get the new stuff from the “remote” repo that is on the USB stick.

2. Now we want to merge this into the local repo.
-s recursive is the strategy.
-X gives the option to the strategy
–no-commit we want to inspect and test before we actually merge.
FETCH_HEAD is where the new stuff gets placed (the commit object?)

4. We visually inspect and run tests here. Nothing broke?

6. Commit the changes in the working set, that was created at 2.

Warning
Posted by a Git newbie, so “be careful out there”.

Updates
The git-bundle command is the standard way of doing this. Find out how to use it: Git’s Little Bundle of Joy.

Does using bundling remove the excess conflicts?

Further Reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Remove CVS folders from Git version control

Open a bash shell and just do:
$ find . -type d -name CVS -exec git rm -r –cached ‘{}’ ;

Is there an easier or more direct method just using Git commands?

Note the –cached option (there are two dashes there). This will ensure we don’t remove the CVS folders in the work area, only those in the index.

Note:
If your on Windows, the msysGit or Cygwin installs include a Bash shell. Git, though usable on Windoze, still has a lot of *nixisms.

But
before doing the above, make a backup or a backup clone of the project. Maybe even do a: git fsck

Check
if everything was done correctly. Do a git status and it should print a bunch of:
# deleted: some/path/to/CVS/cvs track file

and, a reminder to add the .gitignore and .cvsignore files that you finally created, right?

Then to make sure the folders and their content are in the work area do:
$ find -type d -name CVS -print

In a Windows shell this would be: dir /S CVS
but, I didn’t test that.

Update ignores
; the .gitignore file should contain a line: CVS/
Make sure there is no trailing space at the end of the line.

I did a: git add -n . | grep CVS
to make sure nothing was going to be added.

Or instead, add this line to the .git/info/exclude file.

Don’t forget to modify the .cvsignore file to ignore the .git folder.

Now do
git commit

But wait …

WARNING:
CVS keeps file tracking version information in CVS subfolders (Subversion uses SVN). These are in each subdirectory of your working files.

It is very important that only CVS manage these. Thus, using Git (or Mercurial, btw) should have valid ignore files set up.

Just the existence of these subfolders is a good reason to abandon these systems. They make the use of external tools, even search, even more complicated. You have to use filters for everything. To be fair, good tools already have these filters in place.

Why
would someone want to do all this? Well, if you forget to create a .gitignore file before you put all your CVS officially managed project files into Git. Of course, no one forgets to create the ignore file first.

Why
would you ever use Git parallel with CVS? Well, a centralized repository discourages checking in of code. You have to have perfect code before you commit stuff. Don’t even think about branching off a branch (twigs).

An aspect of a centralized VCS is:

When you check new code in, everybody else gets it.

Since all new code that you write has bugs, you have a choice.

You can check in buggy code and drive everyone else crazy, or
You can avoid checking it in until it’s fully debugged.

Subversion always gives you this horrible dilemma. Either the repository is full of bugs because it includes new code that was just written, or new code that was just written is not in the repository.

As Subversion users, we are so used to this dilemma that it’s hard to imagine it not existing.

— from HgInit

So, if you can’t use the VCS to check in your work what do you do? Make manual backups? If you say use the IDE to manage it, you can’t. For example, an IDE (in my experience) cannot do “undo” and “redo” across separate source files, thus the ‘undo’ can only get you so far. For small changes not really an issue. But, if your working on a subsystem or module, or a small team, this can be an issue.

An alternative is
to just add a DVCS to the Integrated Development Environment (IDE) to do true local versioning (maybe embedded Git?). That way I can just indicate to the IDE that I want to mark my current work as a milestone, for example, and it will know what to do. Now I can proceed and do all the fancy little versioning stuff, like branch, log, etc. It is just local.

I don’t even need to know about the arcana of the embedded VCS, it is just the IDE doing its thing. Further, higher-order functions like tasks, bug tracking, could also relate to the IDE’s repo, not only the centralized repository.

When it is finally debugged brilliant code, as usual, I just push to the ‘real’ local version control system, which could just be the same DVCS or a centralized one like SVN or CVS, and then commit as usual.

I bet some IDE already does this. Oh well.

Updates
5Sep11: Took a look at the IntelliJ site. Their IDE does allow directory history and ability to set a version labels. Nice.

System
Git: git version 1.7.3.1.msysgit.0
OS: Windows 7 64bit Professional
PC: AMD quad-core, 8GB ram.

Further Reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Hudson/Jenkins CI Server, can’t edit a job?

I was looking at a possible use of a Continuous Integration Server to quickly set up a build process.

Downloaded the Jenkins war file, put into Tomcat and defined a simple Job that invoked an Ant file to echo “Hello World!”. Cool, that was easy. But, then I wanted to expand that job to do more. Could not find a modify or edit capability. Huh? What’s up with that?

I searched and found very little. There was even some mention of using SED to edit the Job configuration XML, yeeech! Edit using a text tool for a tree-based data structure?

Anyway, not impressed. Of course, this was a quick tryout. Or maybe Linux people are so perfect they never have to edit their work. 🙂

Is Jenkins/Hudson just a pretty face on *nix utilities?

I looked at a few other CI Servers. So far Pulse and Team City look interesting, but they are not free.

Updates
Mar 2, 2012: Used one of the latest Jenkins version. Much much better! Though I’m having issues getting Active Directory authentication going. Can log in ok, but then it uses the wrong user “name” that our PCs must use. You know how LDAP has all this distinguished this or that.

Links

  1. Jenkins
  2. Active Directory Plugin
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Generate SHA1 hash for a file

I’m starting to learn Git the DVCS or DSCM tool. For each file that Git tracks it identifies the file (the blob) by a generated SHA-1 hash ID of the file contents.

I wanted to check that hash on a file.
First this is how you list the IDs:

C:\work\workspace\SimpleServer>git ls-tree HEAD
...
100644 blob 536d1f72385998faf4e4c61f4385c40ad1138b3e    .gitignore
...

Here is how you get the hash outside of Git:

Using sha1sum.exe

serverscygwinbinsha1sum.exe workworkspaceSimpleServer.gitignore
6062906b77fc71db2960f2e0ec83704a4d3f403e *workworkspaceSimpleServer.gitignore

Using Ant via Groovy
Here is how its done using Groovy and it’s AntBuilder DSL:

new AntBuilder().checksum (
	file: "C:workworkspaceSimpleServer.gitignore",
	algorithm:"SHA-1"
)

This creates a file,.gitignore.SHA-1, with the contents: 6062906b77fc71db2960f2e0ec83704a4d3f403e

Hmmm.
The sha1sum.exe of cygwin and the Ant sha1 both agree, but Git’s value is different.

Update
Ah, the Git hash of the file is of the blob of the file as Git stores it in the Git structure.

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.