Stimulating Pixels

In the software development world, where I spend a decent amount of my time, there exists a very helpful type of tool designed for what’s called Revision or Version Control. Among other things, this type of software provides ways for multiple people to work safely on the same set of files without overwriting each other’s work and they provide a way to "roll back" to previous versions of files if you find a bug. When you are dealing with tens or hundreds of thousand (or more) of lines of code, it’s very reassuring to know that if you break something you can easily jump back to one of the earlier revisions where everything was working while you hunt down whatever went wrong.

For the past few years, the Version Control System I’ve been using is one called Subversion. While it’s much better than it’s predecessor (CVS), Subversion is still a bit clunky and some of the day-in/day-out tasks require enough thinking that it easy to loose the flow of what you are working on when you have to deal with it. The safety of having the system behind you makes it worthwhile but it’s enough of a headache that I started looking for an alternative and after some testing have decided to switch to a different system called Git.

As part of the process, I need to remove literally hundreds of directories from my projects that were created by Subversion as part of the way it manages files. While possible to go though everything and delete them individually, it’s much easier and safer to write a little program to do it automatically.

In fact, I’ve written this same basic thing at least three or four times before when I needed to do some clean up of older Subversion repositories. In the Perl programming language it only takes 5-10 minutes to create and test a basic one-off script to do this. Instead of doing that again, I’m using this as an opportunity to create my first Open Source project.

There are two reasons for this:

1. Other people who are going though the same thing might not want to have to build the script themselves. So, this might be of some use to someone else out there if they happen to find it.
2. I can use it as a chance to try a site/service called GitHub.

GitHub is one of those really excellent parts of the internet. Their primary service is to host Git repositories (which are what Version Control Systems use to do their magic). If you have something that you want to develop just for yourself they have various price points for private storage depending on what you need. What makes them really shine, though, is that if you are developing a piece of Free Open Source Software, they let you use their system at no cost.

Not only does GitHub encourage developers to produce code and software the rest of the world can use for free, but it also let’s people like me who are in the very early parts of the learning curve have a safe environment to get their feet wet with Git. Once I’ve gained enough confidence with their system to be sure I won’t completely break things, I’ll have the confidence to move my mission critical work into my own Git system.

If you are interested to see what the site looks like, the GitHub home page for the script is here. If you would like to see what the Perl code itself looks like, you can see that here.

Even though I do a ton of work with XML files, I don’t use actual XML Schema documents very often. It seems like this is the case with most people who work with XML files. After spending some time working with schemas, it’s not hard to see why. They are tedious and it is very easy to get by without having to use them. In day-to-day operation, they offer very little value. Especially considering the language is dense to start with and bulky to work with.

One plays where Schemas are very valuable is in testing software while it’s still in development or when a change is being put in place. The language has a lot of potential for catching little errors that would otherwise be missed in high level test suites.

The impression I get is that a lot of other folks are running XML without really worrying about created Schemas. One of the main reasons is that I had to search for a couple hours to find a good command line tool that validates an XML document against a schema. The one I found and have so far been pleased with is Sun’s “Multi-Schema XML Validator” known as “msv”. The most recent version can be downloaded here. Note, if you are using a Mac with it’s standard Java install, the latest version won’t work out of the gate. Instead you can use the older version which is available here as msv.zip.

The core of msv is stored in a .jar file that is included with the zip. To run msv, the command would be something like:

java -jar /path/to/msv.jar /path/to/schema.xsd /path/to/doc.xml

If the XML doc is valid, you’ll see a message like:

start parsing a grammar.

validating /path/to/doc.xml

the document is valid.

If there is an error, msv will report it and let you know what’s wrong. Sometimes these can be a little cryptic, but when they are, they have enough info to get you started in the right direction.

Controlling Subversions “svn” command with Perl can be a little tricky. The biggest issue is the fact that svn errors are sent to STDERR instead of STDOUT. If you are running the svn command via backticks, perl only captures STDOUT which means you won’t see errors. One way around this is to setup the command you fire off in Perl so that it sends STDERR messages to STDOUT so that 100% of the svn output goes back to your Perl script. This can be done by using something like:

my $response = `$cmd 2>&1`;

With that, both the STDOUT and SDTERR output is captured in “$response”. If you check out this page on perl.com, you’ll see how to determine which content is sent to STDOUT and which is sent to STDERR. This could be used to split out the responses are respond accordingly in your script. I like to approach it a little differently.

Since I already know what the svn command I send will do if it works, I don’t really care about the success messages. I only need to know if something goes awry so I can deal with it appropriately. To do this, I use svn’s “-q” argument for quite output. With this set, all the commands I use don’t return any content to STDOUT if they succeed, while the report errors to STDERR. So, if nothing comes back, everything is good, but if I get a response, something went wrong.

A simple script that tries to export would look like this:

#!/usr/bin/perl -w

use strict;

my $svnCmd = ’svn -q export http://www.example.com/svn/path’;
my $svnResp = `$svnCmd 2>&1`;

if($svnResp)
{
  print “Something went wrong. SVN Error:\n”;
  print $svnResp;
}
else
{
  print “Success!\n”;
}

Since there isn’t a svn server at the above example.com URL, running this as is would fail and respond with something like:

Something went wrong. SVN Error:
svn: PROPFIND request failed on ‘/svn/path’
svn: PROPFIND of ‘/svn/path’: 405 Method Not Allowed (http://www.example.com)

If you change the URL to an actual repository you have access to, you’ll receive the “Success!” message if you are able to output. Once you have the ability to determine the success or failure of the svn command, it is a lot easier to manage Subversion via scripts. The other benefit of this is that if you run the script in a crontab, the STDERR doesn’t get kicked over to send out emails.

Doing a little Java development on the Mac. Just making a note to myself about XOM (an XML Object Model). According to the web site, “XOM™ is a new XML object model. It is an open source (LGPL), tree-based API for processing XML with Java that strives for correctness, simplicity, and performance, in that order.”, and “XOM is designed to be easy to learn and easy to use. It works very straight-forwardly, and has a very shallow learning curve. Assuming you’re already familiar with XML, you should be able to get up and running with XOM very quickly.”

Sounds worthy of looking into.

From a higher perspective, in order to utilize XOM the xom-1.2.1.jar file (which can be downloaded from the XOM page), must be added to the Java class path. On Macs, one possible location for this is ‘/Library/Java/Extensions/’.

IBM has started releasing a new set of offerings they are calling “Skill kits” built on their Toolkit for Custom and Reusable Solution Information. The kits themselves are packages of reference information their developers have identified as being valuable to a given topic. This content is then assembled into a single package for easy consumption via a mini web server that runs locally on your machine.

When I’ve been working in a new language or dealing with part of a language I haven’t used in a long time, I go straight to the web and search for what I need. After years of doing this, I’m pretty good at getting to what I need quickly but the fact that I’m searching the open web has a few main drawbacks:

1. It’s not topic specific. Even keying in on terms to help limit the results, irrelevant information leaks in.
2. It’s not vetted. Most of the time answers you find will get solve the issue, but they may be a poor way to do it. The result could be as simply as a process running slower than it needs to, or something more severe like opening up a security hole.
3. It may not be up to date. Technology moves fast. The answer you find that worked for version 1.0 may not work or cause issues in the 1.1 release.
4. Only a partial solution may be presented. Problems in programming often involve multiple step solutions. It may take cross referencing several potential solutions and assembling various parts from each to get to the answer of a related but distinct problem.

The Skill Kits idea avoids the first two issues completely and if you have the proper version, the third issue dissolves as well. The last issue about only partial solutions being available will depend on the depth of the kit that is assembled. Once it hits a certain level, even if it doesn’t have specific answers, it will provide the framework for developing the answer.

Right now, there is only one kit listed: Project Zero WebSphere sMash skill kit, but I expect there will be at least a few more on the way. Of course, it wouldn’t have to be limited to programming. Just about any topic could be put into the framework.

Even though there is very little difference between these kits and a really good reference/tutorial site, I love the idea. Of course, this could also be the basis for building a system like Trinity used in The Matrix to learn how to fly a Huey by downloading flight skills directly into her brain. Just gotta figure out where to plug in the wires.

The online comic strip xkcd recently posted the following panel:

For me, it was my sophomore year in college instead of 11th grade. Also, I was never that good about doing my homework so I’m sure I had much fewer hours for that part of the graph. The usefulness of hacking around with Perl on my own, though, is well represented.

The other thing I would add to the graph is the semester of Typing I took in high school for an easy “A”. I never would have thought that out of all the courses I took during school Typing would be the most useful.

Two web tools that can help things out:

Fiddler – a web debugging proxy for Windows. It’s one of the Microsoft “PowerToys”. You can read more about it here.

I.E. Developer Toolbar – adds developer functions to I.E.

I haven’t had a chance to check these out yet, but both look very useful if you do web development and need to see how things go on a windows box.

I’ve written before about how WordPress doesn’t really have a way to allow you to put administration tools in a secure location unless you do the same thing with the entire blog. This concerns me since I’m often on a wireless network that is open and not mine. Say, for example, at a book store with free wireless. While surfing on an open wireless network is generally pretty benign, sending any username/password across it without them being secure/encrypted makes it very easy to steal them.

I’ve hunted around a few times before, but had never really found a good solution. While doing some work on my site, I decided to try again and this time came up with “Admin-SSL“. It’s a plug-in someone wrote for WordPress that allows you to move all the “admin” stuff to a secure location. Something that isn’t possible with the default install of WordPress (where you are either all secure or all open).

There are two examples of the power and benefit of open-source software with this plug-in. First off is the basic fact that WordPress is open which allowed for the plug-in to be created in the first place. While this isn’t limited to open source software, it’s a big help.

Second, when I installed the plug-in on my site, it didn’t work properly. Some of the software that runs my site is different where the plug-in was originally created. However, since I could look at the source code, I was able to find a fix that works and allows me to use the it. To contribute back to the overall community a little, I’ve sent a note back to the original author explaining what I ran into and how I fixed it. This gives him the opportunity to let other people know about the issue and a way to fix it. Possibly even creating a specific fix for the issue in the next version.

Stop reading…. unless you are a web geek and/or are specifically looking for a fix for Admin-SSL on version 1.3 of the Apache web server. Below are the details of the fix that works for me. YMMV.

First, the short and sweet fix to try:

When you configure Admin-SSL (at least version 1.1) on a server running Apache 1.3, under the “Other Settings” category and the “HTTPS Detection” section

change: “The name of the HTTPS $_SERVER variable”
to: “SERVER_PORT” (without the quotes)

and change: “The value of the HTTPS $_SERVER variable when HTTPS is ON”
to: “443″ (again, without the quotes)

Now some details. Admin-SSL uses the predefined $_SERVER['HTTPS'] php variable to check for secure connections while pattern matching to see if it should redirect a page to a protected URL. While this variable is available in Apache 2.x it is not in the Apache 1.3.x versions of the server.

See the list of “specials” under the “RewriteCond Directive” for reference:
Apache 2.x – http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html#rewritecond
Apache 1.3 – http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteCond

You can use an existing feature in the Admin-SSL configuration (described above) to get around this limitation assuming the port that your host uses for SSL is different from. Usually, SSL is set to run on port “443″. If your provider uses a different port, you can simply use that instead. The only exception to this is if you have a host that runs both HTTP and HTTPS over the same port. In that case, there is no way to tell the difference in the script using the above method.

All this, of course, assumes that your host provides you with a way to access your site via HTTPS with either a private or shared cert. A general practice is for them to setup URLS like:

“https://www.your-site.com/~your-username/” that would simply give you a secure version of “http://www.your-side.com/”. If you don’t see a colon followed by a number after the .com, you should be running on 443. If you see something like “https://www.your-site.com:1234/~your-username/”, that means that your host is running HTTPS on port “1234″, or whatever the number there is. That’s the number you would want to configure.

If, for some strange reason, that number is “80″, you’re going to have to fins another solution, because that’s the standard port for web traffic which means the script wouldn’t be able to tell the difference.

So, the theory behind “permalinks” is that they are supposed to stay the same. I’m breaking the rules and changing mine. The good news is that because of the way Wordpress is built, links to the old addresses won’t break.

Originally, when I set them up, the format was: /blog/[year]/[month]/[day]/[post-title]. I’m removing the [day] because it’s overkill. When looking at a web page address, it’s nice to see the year and month the page was created, but there isn’t a whole lot of reason to for the day to be there.

The developers who built WordPress included a very nice feature that handles these changes smoothly. Bacially, if it sees requests for the old style address (which would potentially be coming in from other sites), it automatically redirects them to the new location. So, if an incoming link is pointing to something like this:

http://www.stimulatingpixels.com/blog/2008/07/29/no-idea/

WordPress will see it and automatically change it to remove the now extranious “/29″ part. This is a great example of a good software design. If this wasn’t handled and the format for the permalinks was changed, any links to the old format would immediately be broken. The user of the software shouldn’t have to worry about stuff like that, and because WordPress developers thought about and designed for this, the user doesn’t have to. It just works without really thinking about it.

Another nice example of good software design is that if you remove the post-title from the address (e.g. http://www.stimulatingpixels.com/blog/2008/07/) you’ll get a list of all the articles that were posted in the corresponding month. Once again, very well thought out.

Kudos to the WordPress team.

I just discovered that the free IrfanView image software can make favicon.ico files. These are the little icons often show up that show up on web sites in the address bar or when you bookmark them. Earlier, I had pointed to an online tool that does this, but I prefer local software for this type of thing so this is a very nice discovery.