Bitwise Evolution

Hacking your kitchen: making a sous vide

2010-12-24T11:45:00.000-05:00

[caption id="attachment_170" align="alignright" width="200" caption="The crock-pot portion of my sous vide -- with three thermometers monitoring (and three different temperature readings )."][/caption]

I recently found the cookbook ["Cooking for Geeks"](http://www.cookingforgeeks.com) , and it inspired me to build a [sous vide](http://en.wikipedia.org/wiki/Sous-vide) (the section on making a sous vide at home is also covered in a blog post by the author [here](http://www.cookingforgeeks.com/blog/posts/diy-sous-vide/)). If you're not familiar with the term, sous vide lets you very carefully control the temperature you're cooking at, so you *can't* over heat foods.

Sous vide is similar to using a crockpot, but it requires *much* more control over the temperature than a typical crockpot provides. If cooking with a crockpot is like sketching dinner, then sous vide is analogous to using finely-tuned drafting instruments.

Jeff Potter (Author of Cooking for Geeks) provided part numbers for a suitable thermocouple (a probe thermometer) and a corresponding temp switch (a switch that controls a gate based on the temperature it sees from the attached thermocouple). I would have had some trouble getting the right set-up without that guidance, but still ended up diverging a bit on accident. I ordered an AC version of the temp switch instead of a DC version, but in the end I think it worked better this way.

The biggest problem I have with Jeff's suggestions is that the book and blog show the temp switch and wiring sitting un-enclosed on the counter, with everything hard-wired together. I'm not comfortable with 115vac running around on my kitchen counter, so I needed some sort of enclosure. I also wanted to be able to detach the thermocouple to easily clean it, and I didn't want to modify my crockpot much at all. (I really don't want *another* appliance, and I still want a functional crockpot.)

# Parts

[caption id="attachment_175" align="alignleft" width="200" caption="The top view of the enclosed sous vide controller. AC power comes in the top, the crockpot is powered out the right, and the thermocouple is plugged into the bottom edge."][/caption]

Given that I don't have access to a fab. plant (or a maker bot!) I opted to go with a simple locking tupperware container for a housing. They lock shut, have a gasket for a tight seal, and the plastic is very easy to modify to hold all the ports that the sous vide control needs.

At this point it's probably reasonable to think of the control as a temperature-controlled power outlet, rather than something specific to a sous vide, since the purpose is really irrelevant for now.

I'm not going to go into great detail about the specific parts (although I'll talk about the connections somewhat), but you can see the full parts list [here](http://bit.ly/fBbj4M).

## Housing ##
The pictures show the housing fairly well. We started by cutting out holes for all the components with a hot exacto knife (heated in a blow torch -- as Mike put it, it cut through plastic like hot butter through a knife ;). Pretty much all the parts had some sort of template we could go by:
- temp switch: we used the wire cover to draw out the opening
- power outlet: I had a cover plate that I was intending to use, instead, we just used the cover plate to mark out the large round plug hole and the two screw holes, then used the cover plate screws to hold the plug in.
- thermocouple jack: just drilled it out with a drill.
- AC input: traced around a pc-style power cord, then cut to the outside of the sharpie line, then drilled out the screw holes (marked after we had the main hole in, and could test-fit the plug).

[caption id="attachment_182" align="aligncenter" width="500" border-bottom="1em" caption="The water for my eggs was at 120f, gradually heating to 147."][/caption]

## Thermocouple connections ##
The temp probe arrived with a bare pair of wires on one end, which wasn't going to work that well, so we (I was working with a friend who has a soldering iron :) attached a 1/8" headphone plug to the cables, then soldered a corresponding jack to some 22-gauge wire that eventually went to the temp switch inputs (ports 7 & 8 on the TCS-4010).

## AC input ##
I was initially going to use a standard pigtail power cable to hook this up to the wall, but when shopping for parts I found a PC-style (D-shaped) power plug, which works beautifully, and it was cheap!

[caption id="attachment_176" align="aligncenter" width="500" caption="The back-view of the controller, showing the pc-style AC input, crockpot in the background."][/caption]

## Power outlet ##
Locating a cheap power plug that would fit in a small container, and only had one plug was actually pretty difficult. The outlets I found with a flange were more expensive than I could justify ($15-20), and everything else had two + plugs, or a plug and a switch, or so on. Eventually, I settled on a 20-amp single-outlet plug, designed for a standard outlet box. In the future, I think I'll go with a dual-outlet 15amp plug, or look more. The 20-amp outlet is very hard to plug things into, and I'm not entirely sure why. It works just fine once you get everything plugged in, though.

## Wires ##
I went with 12-gauge solid copper, in three colors, for the internal wiring. It was quite difficult to form to the right shape, though, and we needed to solder it in place. The wire was so stiff that we ended up re-soldering quite a bit: we'd solder a connection, then bend the wire to make the next connection, and the solder would break. Eventually, we wired it all together and did the soldering in one go. This worked, but it meant we couldn't easily cover some of the connections with heat-shrink tubing. In particular, the three connections to the AC input were bare, so we covered those with hot glue to keep fingers and other potential conductors from shorting it out (and delivering quite a shock).

We were able to heat-shrink most of the connections, however, and everything is soldered in place, so I think it will hold up :).

[caption id="attachment_183" align="aligncenter" width="500" caption="The appliance plug is on the side, to the right of the display."][/caption]

## Calibration

I did a test run with eggs to see how this worked, with multiple thermometers to compare readings (you can see one of the other thermometers in the first picture). I did a quick calibration based on those readings, but the controller was still off by about 5 degrees (that initial calibration was pretty drastic, there was a difference of over 10 degrees!). That became evident when I cracked the eggs and they were a bit under-cooked---the book suggested flash-boiling them anyway, so I had boiling water handy and re-calibrated based on that. Don't do what I did, or if you do, cook something cheap in the first run :)

## Next time...

1. Use stranded 12-gauge wire. It should be more flexible, and hold solder better.
2. Use a 15-amp power plug, so it's easier to plug/unplug appliances.
3. Be careful about the locking tabs on the tupperware housing: These are really, really, close to interfering with the plug and temp switch.
4. I actually wanted to have the switch upside down, but because of the wire flexibility, we had to cut things to be very short. That meant that we couldn't reach the screws to mount the temp switch, since the bottom of the housing would have been in the way. I'd try and do this differently next time -- using stranded cable may work, or attach the temp switch first, then insert it, and finally solder the power input/output in place.
5. Add an LED to the power supply that leads directly to the device. It would be very nice to know when the appliance is actually getting power.

cabal-dev: sandboxing your Haskell development (and now with ghci!)

2010-12-22T17:47:00.000-05:00

Reproducing builds is pretty much essential if you intend to do sustainable software development, and while it's generally possible, it's been fairly difficult to do so with Haskell. One of the challenges has been that there are only generally two local databases of installed packages for any given (user, ghc-version) pair. Each ghc install has a global package database, and each user has a user package database for each version of ghc. This is perfectly suitable in most situations, but it becomes problematic when you need conflicting packages to be installed for disparate development projects. Cabal-dev removes the user package database from the picture, and it is relatively simple (with some self-control :) to maintain a minimal global package database: just don't install anything system-wide with cabal.

Cabal-dev does this by creating a per-project sandbox that contains a package database of all the dependencies as well as the project under development. Therefore, it was simple to add support for launching ghci with this package database in place of the user package database. That's been added in cabal-dev-0.7.3.1, which is available on hackage now, allowing you to do things like this (using my Haskell Version Space Algebra library as an example):


$ cd havsa
$ ls
LICENSE  Setup.hs  src/  versionspaces.cabal
$ cabal-dev install
Resolving dependencies...
Configuring mtl-1.1.1.1...
Preprocessing library mtl-1.1.1.1...
Building mtl-1.1.1.1...
[ 1 of 21] Compiling Control.Monad.Identity ( Control/Monad/Identity.hs, dist/build/Control/Monad/Identity.o )
....
Registering VersionSpaces-0.0...
Installing library in
/home/creswick/development/havsa/cabal-dev//lib/VersionSpaces-0.0/ghc-6.12.3
Registering VersionSpaces-0.0...
$ cabal-dev ghci
GHCi, version 6.12.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package ffi-1.0 ... linking ... done.
Prelude> :m + AI.VersionSpaces
Prelude AI.VersionSpaces> showBSR EmptyBSR 
Loading package syb-0.1.0.2 ... linking ... done.
Loading package base-3.0.3.2 ... linking ... done.
Loading package mtl-1.1.1.1 ... linking ... done.
Loading package logict-0.4.1 ... linking ... done.
Loading package VersionSpaces-0.0 ... linking ... done.
"Empty"
Prelude AI.VersionSpaces>

This is still far from perfect: you can't easily load code into the ghci session without exiting, re-running cabal-dev install and cabal-dev ghci, but it's a good start.

Prezi: scalable, navigable presentations

2010-10-06T05:34:00.000-04:00

Edit:Prezi now has support for customizing colors / themes, and even documentation for the css! I'm happy to see official support for these features, but the layout problems are still a show-stopper for me./Edit

I have been adopting a presentation style that diverges from the traditional bullet-point style promoted by Open Office and PowerPoint (although, PowerPoint 2007 diverges from pure bullets to more interesting shapes, using shading and encapsulation to show hierarchy. It's a large improvement, but it still falls short of my ideal). Instead of bullets and text, I try to make use of imagery and visual examples whenever possible.

My experience has reinforced what I've come across in the literature about balancing your audience's attention between the content on the wall and your narration. Too much text or detail and you risk loosing your audience because they're overwhelmed or, if you're lucky, because they are focusing on the slide content instead of listening to your explanations. It is also much easier to trigger emotional responses with visuals than it is with text, which explains some of the motivation for promotional/motivational presentations that are virtually devoid of text. (Seth Godin's talks come to mind. One way in which my presentations differ substantially from Seth's is that I generally talk about fairly technical topics. The difficulties associated with finding high-quality, emotional, images to convey the intricate details of ptrace(2) is not the topic of this post, however.)

A number of years back, I found a mention of using Scalable Vector Graphics (SVGs) for unconstrained presentations and that idea has stuck. I've been developing slides in PowerPoint (OpenOffice has not yet reached the point where I can create professional-looking content), but I've never been happy with the traditional slide-based presentation style. SVGs promise the ability to move between arbitrary locations, following a pre-defined path from "slide" to "slide". Furthermore, since the content is scalable, you can literally zoom in to a portion of content to go into more detail, or zoom out to show context. This presentation mode would also make it easy to diverge from your plan based on audience feedback. This can be done with PowerPoint, but it is extremely difficult.

Prezi promises these benefits, so I spent the last few days building a presentation with Prezi at Galois: portable build systems. (The prezi presentation is near the bottom of this post.) The rest of this entry discusses my experiences with Prezi.

The Good, the Bad and the Ugly

I initially found the Prezi interface to be very intuitive. The translation "zebra", a stripped round mult-function circle, appears whenever you select an object. You then use the zebra to move, scale, rotate, or otherwise manipulate the selected object. Other options are presented through a bubble menu that rotates and scales to show more detailed options as you select sub "bubbles". It is well worth the few minutes it takes to sign up for a free account and try it out. If you happen to be using 64-bit linux, you probably won't be able to use the flash app, however. Prezi doesn't appear to work under that environment. (if you can interact with the presentation embedded on this page, then you should be Ok.) There is also an Adobe Air-based desktop app, which I used extensively.

I was off and brainstorming a presentation mind-map style after only a few minutes playing with the interface. The freedom to create a few words of text, zoom in and flesh out more details, jump back out and pull in an image, all without concern for the layout of final form of the presentation was extremely motivating and liberating.

I was able to be quite productive with Prezi until I began to consider the need for a unifying theme. Two things stood in the way:

Only eight options were provided for the overall look and feel of the presentation, and none were close enough to the Galois color scheme. There is also no way to add new look and feel options, and each option changes all the colors, fonts, backgrounds, and general style of the presentation.

The set of shapes and drawing tools is very limited. You can create: Thin free-hand lines, thick free-hand "highlighter" lines, circular rings (round frames), square brackets (square frames), gradient-filled rounded rectangles (roundrect frames), arrows, text in one of three fonts (optionally with bullets).

None of the pre-defined colors can be changed through the user interface, aside from the eight styles mentioned above, which change all the colors / styles and fonts. You can include arbitrary images, which helps with the limited set of shapes, however, you can only include pdfs if you are using the web-based client, the desktop client does not currently support pdf importing.

I stumbled across a solution to the limited selection of colors by unzipping the .pez file that holds a Prezi presentation on-disk and exploring the contents:


prezi/
├── content.xml
├── preview.png
└── repo/
    ├── 13177749.png
    ├── 13177754.jpg
    ├── 13177758.png
    ├── 13177835.swf
    ├── codetree.png
    └── Personal_computer,_exploded_5.png

content.xml defines the SVG-like presentation content, and it ends with a set of css styles:

The colors in these styles can be adjusted, and you can even add new styles here (although you will need to manually insert them into the xml where you wish to use the new styles). After editing the content.xml, zip up the presentation tree, taking care to maintain the correct hierarchy and no compression:


$ ls
prezi/
$ zip -r -0 enabling-portable-build-systems-biuinv2vus9x.pez prezi/

One surprising benefit is that the updated styles are actually adopted by the UI widgets in the Prezi application, once you load a modified .pez file.

These changes also persist across saves, loads, uploading to Prezi.com, and they appear to render properly when embedded, granting quite a lot of power, if you're able and willing to work with xml and css periodically.

Editing content.xml also proved to be the best way to spellcheck the presentation content, although the text that is actually displayed is in CDATA nodes, which your editor may skip over when running a spell checker. Thankfully, the text is duplicated as plain text nodes, so you are still alerted to spelling errors when running ispell-buffer in emacs. You can then fix the CDATA entry with a recursive edit. (I suspect that the duplication is there to simplify text searching.)

I'm rather satisfied with these workarounds: The ugly aspects could be automated with some simple tools to update the content.xml as needed, and the hacks I found worked surprisingly well.

Background images and "refactoring"

Unfortunately, I don't think Prezi is ready for prime-time, despite my success with css styles and spellchecking. There are simply no facilities to precisely align or distribute objects with respect to each-other. Further complicating this is the lack of a "group" option to create aggregate objects. You can select multiple objects by dragging a rectangle if you hold shift, but that isn't possible if the objects are on top of other objects -- the first click of a shift-drag must occur on the background of the Prezi, or you will simply select the lower object. While this sounds a bit like a minor quibble, it is impossible to accurately position complex sets of objects if they are layered on top of other content (such as a background image). I often need to add or remove "slides", and with Prezi, that can include a lot of object translations to provide or absorb space while fitting with the high-level overview of your presentation. Without alignment tools, you also take the risk that a title will display askew with respect to the screen borders when you are mid-presentation.

I eventually adopted the following practice to help keep content square:

Press 'space' to enter "show" mode, and position the screen as it would be to show the "slide", then pres 'space' again to return to edit mode.

Create a roundrect frame, it will be square with the current view.

Use the borders of the roundrect as visual guides to make the slide content as square as possible. I usually place the baseline of the title text with the top of the roundrect and then position it after the angle has been fixed.

Delete the roundrect

Now, never rotate any individual components of that slide again. Use shift-clicking to select everything in the slide each time you need to rotate or move the slide. If you need to add new content, first enter 'show' mode and click on the frame to make the camera rotate properly with respect to the slide.

Summary

I'm excited to see how Prezi evolves, and I will be one of the first in line once the selection / alignment problems are fixed. I hope that Prezi will motivate other implementations with similar capabilities, there is plenty of room for some healthy competition.

Enabling Portable Build Systems on Prezi

Recovered Puzzles

2010-07-26T01:21:00.000-04:00

I used to collect puzzles--back when I had a wiki to post them to--but that content was lost to me a few years ago when the system hosting my personal content had a slew of hard drive issues. I was lamenting that loss last week when a coworker suggested that I could possibly find the content on the way-back machine at archive.org, and indeed, I did!

Without further ado, here's a short list of brainteasers (none are of my creation, and I do not have citations--if you know who to credit for any of these, please let me know and I will add proper attribution info.)

Calendar Cubes

A man has two cubes on his desk. Each face of each cube has a single-digit number wirtten on it. With these two cubes, the man is able to enumerate all the days in any month, and each morning he arranges the cubes so that the number of the current day is on top, *always* using both cubes. How are the numbers distributed on the cubes?

Pennies 1

You are blindfolded in a room with 100 pennies. 30 of the pennies are heads-up, the remainder are tails-up. You can interact with the pennies in any way, but your fingers are not dexterous enough to feel the contours of the coins (so you can't feel one to see which side is heads, or tails). Since you are blindfolded, you can't see them either. Your task is to manipulate the coins such that there are two sets and each set has an equal number of coins that are heads-up. (The sets must be disjoint, non-empty, and all pennies must be in one of the two sets.)

Pennies 2

Given N pennies, one of which is counterfiet (and therefore is of different weight from the remainder) and a balance, how can you find the counterfeit coin in less three weighings on the balance.

Eggs

You are in a 100-floor building on a planet with oddly low gravity and/or surprisingly durable eggs. You happen to have two of these eggs (unfertilized, I assure you). Your task is to find the highest floor from which you can drop an egg and have it remain intact.

Numbers

Given 99 unique integers between 1 and 100, provide an optimal algorithm to find the remaining integer in that range that is not in the set.

Hint: Bcgvzny gnxrf yvarne gvzr naq pbafgnag fcnpr.

Prisoners

50 people are inprisioned, and during their imprisonment the captor will invite people randomly in to visit with her. All visits are one-on-one, and each prisoner has a unique tunnel from their cell to the captor's office (so you can't look out your cell and see who is going in). In the captor's room is a bowl that the prisoners can optionally turn over, or turn right-side up during their visit(s). The initial state of the bowl is known to everyone.

The imprisonment may last for an infinite period of time, during which each prisoner will be invited into the captor's office many, many times (essentially infinite, but it needs not be infinite, it could just be a reasonably small number in the optimal case). The imprisonment ends when one prisoner says: "Everyone has been in to see the captor at least once." If a prisoner says this and they are wrong, all prisoners are killed immediately. Because the captor may decide not to visit anyone for a while, it is as if the prisoners have no concept of time, so they can't bound the number of people seen based on the passage of time.

To give the prisoners a chance, they are allowed to convene briefly before their imprisonment, during which time they can plan a strategy. How do they do it?

Sequences
What is the next line in this sequence?


1
1 1
2 1
1 1 1 2
3 1 1 2
2 1 1 2 1 3

Haskell Platform on Debian

2010-01-25T00:21:00.000-05:00

Test-Driven XML Schema dev with xmlstarlet

2009-10-17T02:46:00.000-04:00

Just to document how I do this:

*Problem:* I need a schema for FooTask

*Solution:*

* Create a 'tests' directory.
* populate said directory with simple example xml files.
* Name those files `valid-foo.xml` or `invalid-bar.xml` (I use numbers for foo and bar).
* Create your xsd file in the same directory as 'tests'. Lets call it `foo.xsd`
* Copy the following Makefile into the same location.

[cc lang="bash"]
XSD=foo.xsd
# run xmlstarlet with -e to see verbose error.

test:
@for file in `ls -1 tests/valid*.xml`; do if xmlstarlet val -q --xsd ${XSD} $${file}; then echo "pass"; else echo "fail: $${file}"; fi; done
@for file in `ls -1 tests/invalid*.xml`; do if ! xmlstarlet val -q --xsd ${XSD} $${file}; then echo "pass"; else echo "fail: $${file}"; fi; done
[/cc]

Now, run make, and if anything fails you can manually run `xmlstarlet val -e --xsd foo.xsh [failing file.xml]` to see the details.

Implications: Coding for Homomorphicly Encrypted Input

2009-07-16T20:56:00.000-04:00

Craig Gentry (Stanford / IBM) recently published a paper that proves the existence of fully homomorphic crypto systems. This has caused quite a stir, since such a system would allow an untrusted party to perform computations on encrypted data, returning an encrypted result, without ever knowing anything about the input or output. I'm not going to explain what homomorphic encryption is (at least, not in great detail). Bruce Schneier's blog has a great post about it, and the comments there are extremely helpful in understanding how it works, and what this means for cloud computing.

I make no claims to being a cryptographer, but I did have a number of questions about the practical viability of this approach. Now, there are many questions in that vein that are directed at the performance characteristics of Gentry's approach (which are abysmal, but not asymptotically so). I was curious about The use of side effects to discern information about the encrypted content.

For example, anyone who has used a debugger knows that you can monitor the flow of a program that has been instrumented with debugging symbols, and you can learn a great deal about the input even without examining the content of variables. If a given conditional branch directs execution one way, then you know the predicate evaluated to a specific value. I set out to determine why this sort of attack is not a problem, and I ended up learning a lot about the way programs that run on encrypted data must operate.

Homomorphisms

Let's take a moment to quickly discuss homomorphisms, and homomorphic encryption.

a homomorphism is a structure-preserving map between two algebraic structures --Wikipedia

In this case (encryption) the homomorphism is a mapping from the clear text and the cypher-text. Fully homomorphic encryption, as Gentry discovered, preserves addition and multiplication--meaning that you can add and multiply cyphertext, and the result can be decrypted to reveal clear text that has been added and multiplied in the same way. Addition and Multiplication provide the operations necessary to implement boolean logic, and therefore, are sufficient to program very complex transformations (I'm not certain that it is safe to say "arbitrarily complex").

It's important to realize that every addition or multiplication operation results in a value that is encrypted. The running program can not know the intermediate results, and indeed it does not.

So, how do conditionals work?

Edward Kmett posted the conversion from if/then/else to addition/multiplication on Schneier's blog:

[cc lang="java"]
if (condition) {
return then_clause;
} else {
return else_clause;
}
[/cc]

becomes:

[cc lang="java"]
return condition * then_clause + (1-condition) * else_clause;
[/cc]

Here's a "real" example (it compiles, at least) using both approaches. This is just meant to be used for explanation -- compilers could easily do the translation from the code in is0_clear() to is0_enc(). I've written them out separately here so we can look at the generated bytecode.
[cc lang="java"]
public class Test {
public int is0_clear(int input) {
if (0==input){
return 2;
} else {
return 3;
}
}

public int is0_enc(int input) {
// I'm cheating a bit to keep this simple -- calculate
// the conditional to be either 0 or 1:
int cond = 0==input ? 1 : 0;
return cond * 2 + (1-cond) * 3;
}
}
[/cc]

And here's the bytecode (generated by sun-java-6, and output with javap -verbose).

[cc lang="asm"]
public int is0_clear(int);
Code:
Stack=2, Locals=2, Args_size=2
0: iconst_0
1: iload_1
2: if_icmpne 7 // Conditional Jump!!
5: iconst_2
6: ireturn // return a constant 2
7: iconst_3
8: ireturn // return a constant 3

public int is0_enc(int);
Code:
Stack=3, Locals=3, Args_size=2
0: iconst_0 // lines 0-9 here are for the "cheating" part
1: iload_1 // just ignore them -- the arithmetic to accomplish
2: if_icmpne 9 // the same thing is complex, and not important.
5: iconst_1
6: goto 10
9: iconst_0
10: istore_2 // note that there are no conditional jumps below here:
11: iload_2
12: iconst_2
13: imul
14: iconst_1
15: iload_2
16: isub
17: iconst_3
18: imul
19: iadd
20: ireturn // return the result of the calculated expression.
[/cc]

Since every operation results in an unknown value, no conditional branches can be taken! Every branch has to be evaluated, and the correct result of the 'correct' branch is selected by multiplying by a binary value, that is itself, encrypted! This means that things like run-time short-circuit evaluation are not possible, monitoring progam flow is meaningless, (possibly?) every input will result in the same run-time, and all side-effects will happen regardless of the input.

Implications?

Going further down this rabbit hole, caching is impossible, and global state (if even posible) is likely to be extremely dangerous. I shudder to think of how Python's concept of scoping would interact with a compiler that generates code for homomorphicly encrypted input.

Aside from the pure overhead of dealing with encrypted data, and the "refreshing" required with Gentry's algorithm, I think that there are going to be some serious performance and development concerns once homomorphic encryption becomes a reality. The programming practices that are common in languages like Java and Python now are not likely to hold up. I expect that the APIs that enable operation on encrypted data will be based on total functions, and I have only begun to think about the implications for testing, code coverage, and quality assurance.

Cleaning up my browser.

2009-06-19T23:47:00.000-04:00

I'm done with firefox -- Opera 10 now plays flash well, has adblock via. urifilters, a cleaner UI (no menubar, a menu *button*!) vertical tabs are supported natively, etc... I don't really like the widget toolkit used in the file open/save dialog, but that's *much* better than the horrid performance/stability/bizarre bugs of Firefox.

The minimal UI possible with Opera is also a major win in my book.

Maven deployment issues

2009-06-06T04:25:00.000-04:00

I've been building / porting various projects to maven lately, and pushing them to our in-house maven server. For a while, I was doing this from my laptop at home. However, at work, I'm pushing to localhost (it's a temporary thing while we determine if maven will actually work long-term.)

The following error had me stumped for a few days:
[cc lang="bash"]
[INFO] Error retrieving previous build number for artifact 'de.balokb:libreadline-java-i386-Linux-c23cxx6:jar': repository metadata for: 'snapshot de.balokb:libreadline-java-i386-Linux-c23cxx6:1.0-SNAPSHOT' could not be retrieved from repository: inhouse_snapshot due to an error: Exit code: 1 - Host key verification failed.
[/cc]

All the googling I did turned up people stumped with ssh public key problems, or users who had specified ssh: instead of extssh: ... etc. It was fairly quick to elleminate those issues, or so I thought. (`ssh localhost` right? No problem.)

I happened to look in more detail at my pom.xml:
[cc lang="xml"]

inhouse
Inhouse Internal Release Repository
scpexe://10.0.0.26/var/www/maven/inhouse

[/cc]

hm... `10.0.0.26` I wonder...

[cc]
$ ssh 10.0.0.26
The authenticity of host '10.0.0.26 (10.0.0.26)' can't be established.
RSA key fingerprint is a7:bf:36:4c:b9:c7:c2:f9:03:9a:3a:a7:4f:10:e5:ba.
Are you sure you want to continue connecting (yes/no)?
[/cc]

Ah ha! I clearly can't use a pom.xml that lists "localhost" in the server section -- I'd only be able to push from one place. However, since I'd never ssh'd to `10.0.0.26` from localhost, the fingerprint was unknown, and that was causing maven to error out with the problem I saw initially.

"Fingerprint ID failed" would have been a nicer error message, but I don'tk now that that is possible.

Bitten by dependency management

2009-05-23T01:31:00.000-04:00

I've started using Maven to manage my java projects, and overall I'm very happy with it. It seems to be more mature than ivy, with better documentation, and the vast majority of tasks that I need "just work" (just don't ask me about jni--that's another post).

Today, (and yesterday, and a good portion of the night in-between) I ran into a nasty bug in a library that I didn't know my code depended on. It isn't particularly important *what* I was working on, but just for context: I needed to strip a lot of text content out of nodes in the complete wikipedia revision history dump, so I was using Sax to parse the xml stream, filter out the stuff I wanted filtered out, and save the stuff that, well, I wanted saved. Being that the input was all of wikipedia, there were a fair number of unicode characters in there. As it turns out, the 2.6.2 xercesImpl has some sort of bug that allows xml with certain characters to be read without throwing exceptions, but when you try to write the chars that were actually read, you end up trying to write characters that aren't valid in xml. Even if I'd known that in advance, my response would have been something like "ok, so what? I'm not using xercesImpl, and certainly not a version *that* old".

Well.

You see, in addition to using Maven, I've also been using the Google Collections and JSR305 libraries, so I just drop those `` entries into the pom for all my new projects--I just assume that I'll need them, and I usually do.

Unfortunately, JSR305 1.3.8 depends on jaxen 1.1.1, which depends on xercesImpl 2.6.2 (jaxen also needs this dependency via xom 1.0, for what that's worth).

Because that dependency was already present in my build path (via `mvn eclipse:eclipse`) and in the generated jar (via `` and `` in the `maven-jar-plugin` configuration section), I never realized that my sax code actually had a *direct* dependency on xerces as well. This all came to a head when, 3.53gb into my 2.8tb run, these rather unhelpful exceptions started popping up:

[cc lang="bash"]
java.io.IOException: The character '?' is an invalid XML character
at org.apache.xml.serialize.BaseMarkupSerializer.characters(Unknown
Source)
at com.stottlerhenke.tools.wikiparse.ContentStripper.characters(ContentStripper.java:195)
at org.apache.xerces.parsers.AbstractSAXParser.characters(Unknown
Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at com.stottlerhenke.tools.wikiparse.ContentStripper.parse(ContentStripper.java:96)
at com.stottlerhenke.tools.wikiparse.ContentStripper.main(ContentStripper.java:379)
[/cc]

`` "?" is not unicode -- it fits just fine in asci tables everywhere -- so please don't tell me that it's an invalid unicode character :) (0xd800 *is* an invalid unicode character, and that would have been _much_ more helpful) ``

Many hours later I was able to find a sample of the actual input that was causing these problems, and I was able to reproduce the issue with an input slightly smaller than 2.8tb. Once that was done, I set out to make a minimal test case. Rather than bother with a new maven project, I just hacked it out in emacs (not using google collections, etc. because, clearly, I wanted it minimal). To my surprise, everything worked, and worked fantastically! But how? I didn't even supply an xml api on the classpath, yet it ran just fine!

In truth, I *did* supply an xml api -- xercesImpl.jar, and many other libraries -- via my environment's `$CLASSPATH`. (Figuring that out was another adventure, but I digress.) Once it became clear that I was indeed using a broken library it was simply a matter of explicitly specifying the dependency on a new version of xercesImpl, and rebuilding.

The moral?

Know your dependencies! This should come along with knowing your language's built-in APIs well. It wasn't clear to me that the SAX packages I was using were not part of the core java API, so it didn't strike me as odd that I didn't need to specify a classpath entry or a pom dependency before I could use sax.

If you suspect something strange, you can see the dependency tree in the generated html documentation you get when running `mvn site`.

Fixing the key repeat in Ubuntu 9.04

2009-05-14T19:18:00.000-04:00

I just upgraded my workstation to Jaunty (Ubuntu 9.04) and the key repeat delay and speed dropped to a frustrating level.

gnome-control-center can be used to fix this, but it requires that the gnome-settings-daemon be running, which forces it's opinions on many other aspects of my environment (I run Enlightenment dr17).

Poking around a bit, and help from #e on freenode, revealed that `xset` can be used to fix the key repeat settings.

[cc lang="bash"]
# Look at the current settings:
$ xset q
Keyboard Control:
auto repeat: on key click percent: 0 LED mask: 00000000
auto repeat delay: 660 repeat rate: 25
auto repeating keys: 00ffffffdffffbbf
fadfffefffedffff
9fffffffffffffff
fff7ffffffffffff
# lets speed things up a bit...
$ xset r rate 250 40
[/cc]

Things are a little messy...

2009-03-30T21:02:00.000-04:00

I've had some minor upgrade issues with the blog lately, and I am only about halfway through updating everything. In the meantime, I'm afraid things will look a bit messy. (Syntax highlighting is currently broken, and there are probably other formatting / data issues as well. I think I have to restore the uploads directory, for one, so there probably won't be any images in the posts for a while.)

Like food?

2009-02-04T08:11:00.000-05:00

I started writing for another blog this week:

Brewed, Bottled, Cultured and Sweetened is a blog about beer, coffee, wine, cheese, chocolate, etc... that I'm writing with an old friend from Dagit.o

The Linux Tablet: Wacom rotations - waking up on the wrong side

2009-01-26T07:45:00.000-05:00

Update: updated the script with improved (functional) error output. added notes about xhost.

There is an annoying bug in the sequence of code that manages the wacom rotation / sleep / resume and stylus calibration right now. (Where "right now" is Ubuntu Intrepid, with the 0.8.2-1 wacom drivers.)

This is a document bug over at the ubuntu launchpad, and the poster there does a fine job of describing the intricacies of reproducing the bug, so I'll only give a brief explanation here to help get indexed.

If you rotate the screen any amount, even returning to the original rotation, and then sleep the machine, when it wakes up, the stylus will not be calibrated properly -- the cursor will be off to the side of the stylus point. It doesn't seem to matter how it was calibrated when the machine slept, nor does it matter what rotation you're in when you put the machine to sleep.

There is one straightforward workaround: When you wake the machine, run wacomcpl, click on stylus, click calibrate (the mouse should now be under the stylus again), and exit wacomcpl. This is incredibly cumbersome, but at least it's better than restarting X, which is what I have been doing.

Further inspection (based largely on the thread of comments on that launchpad bug) reveals that the problem is actually related to bad values for the TopX, TopY, BottomX and BottomY settings on the wacom devices after a resume. By resetting these to their proper values for the current rotation, we can reestablish the proper calibration. First off, we need to know the proper values, and the easiest way to get them is with `xsetwacom`:

[cc lang="bash"]
#!/bin/bash
# wacomSettings

echo "TopX=" `xsetwacom get stylus TopX`
echo "TopY=" `xsetwacom get stylus TopY`
echo "BottomX=" `xsetwacom get stylus BottomX`
echo "BottomY=" `xsetwacom get stylus BottomY`
[/cc]

Now, we'll run this for each rotation, and save the results. You should end up with something like the following:

[cc lang="bash"]
|rogue on bach |AC 70% | @ 00:02:26 ~|
$ xrotate 1 && wacomSettings
xrandr to left, xsetwacom to 2
TopX= -46
TopY= -3
BottomX= 18605
BottomY= 24518
|rogue on bach |AC 70% | @ 00:02:28 ~|
$ xrotate 2 && wacomSettings
xrandr to inverted, xsetwacom to 3
TopX= 58
TopY= -46
BottomX= 24579
BottomY= 18605
|rogue on bach |AC 70% | @ 00:02:35 ~|
$ xrotate 3 && wacomSettings
xrandr to right, xsetwacom to 1
TopX= -173
TopY= 58
BottomX= 18478
BottomY= 24579
|rogue on bach |AC 70% | @ 00:02:41 ~|
$ xrotate 0 && wacomSettings
xrandr to normal, xsetwacom to 0
TopX= -3
TopY= -173
BottomX= 24518
BottomY= 18478
[/cc]

(Note that my bash prompt looks like & command lines above are indented, and the output is left-aligned)

That gives us enough information to script the calibration when we resume. For example, when resuming to a "normal" rotation, I need to run:

[cc lang="bash"]
xsetwacom set stylus TopX -3
xsetwacom set stylus TopY -173
xsetwacom set stylus BottomX 24518
xsetwacom set stylus BottomY 18478
[/cc]
(Wrap that in a bash script and give it a shot!)

Here's the full script that gets the current orientation and then calibrates the common wacom devices:

[cc lang="bash"]
#!/bin/bash
#
# waCalibrate.sh: recalibrates the wacom stylus
#
# Author: Rogan Creswick
# License: just be nice

# Set LOG to something reasonable:
# (The file does not need to exist, but the *directory* does)
LOG=/home/rogue/calibration.out
XSETWACOM=/usr/local/bin/xsetwacom

#
# Calibrates the wacom devices {stylus, eraser, cursor} with the
# given offsets:
#
# Usage:
# calibrate
#
function calibrate {
${XSETWACOM} --display :0.0 set stylus TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus BottomY $4 >> ${LOG} 2>&1

${XSETWACOM} --display :0.0 set eraser TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser BottomY $4 >> ${LOG} 2>&1

${XSETWACOM} --display :0.0 set cursor TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor BottomY $4 >> ${LOG} 2>&1
}

function fixCalibration {
# get the current orientation:
ORIENTATION=`xrandr --verbose --query | grep " connected" | awk '{print $5}'`
echo "Orientation: ${ORIENTATION}" >> ${LOG}

case "${ORIENTATION}" in
normal)
calibrate -3 -173 24518 18478
;;
left)
calibrate -46 -3 18605 24518
;;
right)
calibrate -173 58 18478 24579
;;
inverted)
calibrate 58 -46 24579 18605
;;
*)
calibrate -3 -173 24518 18478
echo "ERROR!! unknown orientation! ${ORIENTATION}" >> ${LOG}
;;
esac
}

case "$1" in
resume|thaw)
date >> ${LOG}
fixCalibration
whoami >> ${LOG}
;;
*)
echo "not a resum|thaw event: $1" >> ${LOG}
;;
esac
[/cc]

Stick that in `/etc/pm/sleep.d/40wacomCalibrate` (or some similarly named file), make it executable by all (`chmod a+x /etc/pm/sleep.d/40wacomCalibrate`) and it should be run when the system resumes.

Update: I found that the logging of the old script didn't work, so I've updated the script to reflect that. There were also some problems with how I was testing the first script, and the actions I was taking didn't actually trigger the bug. (The bug seems to be quite state-dependent, and markovian assumption was wrong.) To get this to work, root will need to have access to the display that xsetwacom uses. The simplest way to do this is to add `xhost +` to you x startup. (I put it in my ~/.xsession just before `exec enlightenment-start`).

The Linux Tablet: Wacom drivers

2009-01-12T00:06:00.000-05:00

Ubuntu 8.10 configured most everything properly, as mentioned in the previous post in this series, but it did not result in a functional pen.

The tablet screen is a wacom digitizer with a pen that has two buttons (eraser and a finger button), and the tablet can differentiate between touching and hovering. The linux wacom driver & tools are necessary to get this all working. While I didn't find a single page with instructions that worked flawlessly, I was able to figure it out from a collection of links:

* The Linux Wacom project:
* http://linuxwacom.sourceforge.net/index.php/minihowto
* http://linuxwacom.sourceforge.net/index.php/howto/x11
* The Aliencam blog:
* http://blog.aliencam.net/articles/aliencams-customized-ubuntu-setup-guide/

First off, you will need the latest version of the linux Wacom driver (8.2.1 at the time of this writing). The driver versions seem to be tied to your kernel versions, so this is quite important. The wacom-tools package that comes with Ubuntu is not sufficient (in fact, you'll want to uninstall it if you have it already).

Once you have the wacom package downloaded, follow the directions for installing it in the howto (linked above). The wacom package uses a typical configure, make, make install process but there are a few kinks:

* configure (almost?) always succeeds, regardless of the dependencies you have yet to fill. The make step will simply not build all the things you need if this happens, but it won't fail visibly.
* You'll need to copy the kernel module into the /lib/modules/`uname -r`/kernel/drivers/usb/input/ directory manually (creating subdirs if necessary), *before* running make install. (This is outlined in the mini-howto.)

Once wacom is installed, you can begin working with the X.org configuration. This is fairly clearly explained at the aliencam blog linked above, or you can use my xorg.conf here.

[cc lang="bash"]
Section "Device"
Identifier "Configured Video Device"
EndSection

Section "Monitor"
Identifier "Configured Monitor"
EndSection

Section "Screen"
Identifier "Default Screen"
Monitor "Configured Monitor"
Device "Configured Video Device"
EndSection

#BEGIN TABLET SECTION
Section "InputDevice"
Driver "wacom"
Identifier "stylus"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "stylus"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
Option "Button2" "3"
EndSection

Section "InputDevice"
Driver "wacom"
Identifier "eraser"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "eraser"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
Option "Button3" "2"
EndSection

Section "InputDevice"
Driver "wacom"
Identifier "cursor"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "cursor"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
# Option "Mode" "Absolute"
EndSection

# This section is for the TabletPC that supports touch
#Section "InputDevice"
# Driver "wacom"
# Identifier "touch"
# Option "Device" "/dev/input/wacom" # USB ONLY
# Option "Type" "touch"
# Option "ForceDevice" "ISDV4" # Tablet PC ONLY
# Option "USB" "on" # USB ONLY
#EndSection
#END TABLET SECTION

Section "ServerLayout"
Identifier "Default Layout"
Screen "Default Screen"
# InputDevice "Synaptics Touchpad"

#added to get tablet working
InputDevice "stylus" "SendCoreEvents"
InputDevice "cursor" "SendCoreEvents"
InputDevice "eraser" "SendCoreEvents"
# InputDevice "touch" "SendCoreEvents"
EndSection
[/cc]

After doing that, you should be able to reboot and the pen should be working. You can do things like configure the buttons with `xsetwacom` (and you'll need that when it comes time to rotate the screen), but I kept getting this error when I tried to run `xsetwacom`:

[cc lang="bash"]
$ xsetwacom
xsetwacom: error while loading shared libraries: libwacomcfg.so.0: cannot open shared object file: no such file or directory.
[/cc]

I made a lucky guess, and fixed the problem with a quick ldconfig:

[cc lang="bash"]
$ sudo ldconfig # that was a lucky guess.
[/cc]

*Update:* There were some issues with the wacom calibration after a sleep/resume cycle *if* the laptop screen had been rotated during that prior wake cycle (this happens a *lot* more than it seems, given how complex that description is.) I've written up a workaround here.

The path to a Linux Tablet

2009-01-10T00:04:00.000-05:00

I finally broke down and bought a Lenovo X61 tablet (with SXGA+ screen!), and it arrived this week. This is the first of a series of posts about getting it up and running with Linux.

First off, some specs:

* Lenovo X61 Tablet PC with XSGA+ (1400x1050) screen (not multi-touch)
* 4 gigs of ram
* 200gb SATA hard disk
* WIFI (Intel PRO/Wireless 4965 AG or AGN)
* Gigabit Ethernet (Intel 82566MM)
* Bluetooth
* Wide-area networking card (3G)
* Fingerprint reader
* Integrated SD card reader (Richoh R5C822 SD/SDIO/MMC/MS/MSPro)
* Intel Audio (82801H ICH8 Family audio controller)
* 1 PCMCIA (type-I?) slot
* 4-cell battery
* Ultrabay Slim (which will be holding my Ultrabay CD-RW / DVD drive from my old T42p)
* Intel Mobile GM965/GL960 video controller. (256mb video ram?)

I'll flesh that list out more as I can find the details (eg: wireless chipset, video, etc..)

First off, I blew some time poking around in Vista of course :). The handwriting input app is phenomenal in a lot of ways. It works very well, training is well integrated, and it has worked with every input area I've tried. It *could* be better if it had contextual clues, and could tie into things like Eclipse's intellisense. Overall, though, it is amazing how simple it is to use, and how aesthetically pleasing the handwriting actually is. There is a lot to be said for using a couple extra pixels to make the strokes taper off as you pull the pen away. It has QWAN.

That done, I started to move on to installing Linux. I'm giving Ubuntu 8.10 the first chance, and I thought I'd try using a USB-based install so I wouldn't have to monkey around with the Ultrabase & drive. If you have an 8.10 system already, you can easily create a bootable usb ubuntu drive with `usb-creator` and an ubuntu iso. This takes perhaps 45min - 1 hour.

Booting was as simple as going into the ibm bios-like page (by hitting the ThinkVantage button on boot) and telling it to boot from another device, then selecting the usb drive (that I had already inserted). I split the existing 200gb partition in two with the ubuntu installer, keeping Vista in it's 100 gig sandbox, and leaving the remaining ~100 gigs for Ubuntu to partition further (which it did, as two partitions: one for / and one for swap. /dev/sda5 and /dev/sda6).

I do wish it had said how much space was being allocated to each of those partitions though. The installer didn't give any indication.

Installation from booting the installer from usb to booting into the installed system took right about 30min. I'm impressed :)

Out of the box:

* The screen is the proper res
* Wireless looks like it might be working (I have to AP to verify)
* Sound works
* the pen does *not*
* Screen rotation does not work
* closing the lid shuts off the screen, but does not put the laptop to sleep.
* This was easily fixed in the gnome power-management settings, and hey, it resumes too!
* putting the laptop in tablet mode seems to have no effect (at least it doesn't shut off the screen ;)
* Some power management is clearly working (screen dims when unplugged)
* bluetooth was detected, but I have to way to test it.
* Dual-booting seems to work just fine, although there are two entries for Vista in the grub menu, and the first boots into the Rescue and Recovery system. Vista also had to do a chkdsk, and reboot before it would load.

More information as I figure it out :)

My tablet wish-list (software)

2008-12-23T22:36:00.000-05:00

I broke down and ordered a tablet -- the Lenovo X61 tablet (with the SXGA+ lcd, which is unfortunately no longer in production, so it's a bit hard to find--I ordered from the Lenovo outlet).

This is the (growing) list of features I'm hoping to have:

* Some form of hand-writing recognition for general text input.

Treat your mailing lists like reference documents, please.

2008-11-01T02:21:00.000-04:00

I desperately needed to find out why the tutorials I've been following for an Eclipse PDE task today kept referencing a startup.jar file that I could not locate.

A couple google searches later turned up this link:

http://dev.eclipse.org/newslists/news.eclipse.platform/msg62159.html

The poster in that thread had the same problem (back in Feb. 2007), and found the answer, but none of the content in that thread makes it trivial to locate the answer again.

The responder (with the answer) simply included a link to another mailing list:

http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/maillist.html

Notice that that page is *not* constant. Today, it shows the most recent posts as of October 31st, 2008. In order to figure out what had happened to startup.jar, I had to take into account the OP's response ("Ok so this is *very* recent."), the timestamp on the messages (Mon, 12 Feb 2007) and then navigate the mailing list archives to find that time period, and start reading.

Please don't put people through this sort of crap. It's generally not difficult to find permlinks to a given email, or include a quick note with the actual answer. I have the answer now (startup.jar was replaced with org.eclipse.equinox.launcher in 3.3), but there is no way that I can tie that answer to the conversation I've linked to above.

For the purposes of Google:

If you're having this problem:

I'm trying to do some automation, but I'm running into a problem with the 3.3 integration build.

java -cp plugins\org.eclipse.platform_3.2.100.v20070126\startup.jar org.eclipse.core.launcher.Main

doesn't do anything. It doesn't say anything. The only information I'm getting is an exit status of 13.

Then you need to use "java -jar plugins/org.eclipse.equinox.launcher_1.0.0.v20070207.jar" (adjusting the version numbers for your installation).

Auto-documenting OSGi CommandProviders

2008-10-21T01:08:00.013-04:00

(**Edit:** If you're reading this after OSGi R4.2, then there is almost certainly a better way to accomplish the same thing)

I've been digging into OSGi a bit over the last week or so inorder to
create some Eclipse plugins that will automatically discover
eachother, and I've been generally impressed with the framework on the
whole. The documentation is a bit lacking, but there are some good
blog posts about it. (Specifically Neil Bartlett's introduction to
OSGi.)

One thing that bugged me is the repetition needed when you implement
the CommandProvider interface to add commands to the OSGi console.
CommandProvider defines one method you must supply:

public String getHelp()

OSGi then uses reflection to extract each of the methods that starts
with an "_", and supplies those methods to the command environment as
new commands. (The "_" is trimmed, and the name of the method becomes
the command name.) General practice is to include the name of the
method in the return value of `getHelp()`, along with a description of
what the method does, eg:

public class SampleCommandProvider implements CommandProvider {

public synchronized void _run(CommandInterpreter ci) {
   // do stuff.
}

public String getHelp() {
   return "\trun - execute a Runnable service";
   }
}

This seems like a pain to maintain, so I took a quick look at
annotations, and propose a new syntax:

public class SampleCommandProvider extends 
   DescriptiveCommandProvider {

   @CmdDescr(description="execute a Runnable service")
   public synchronized void _run(CommandInterpreter ci) {
      // do stuff.
   }
}

Here we've extracted the `getHelp()` method into a new superclass, so
our SampleCommandProvider now extends an abstract class instead of
implementing an interface. It also makes use of an Annotation, which
we need to define:

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface CmdDescr {
   String description();
}

Finally, we just need to define the superclass that implements
`getHelp()`:

import java.lang.reflect.Method;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.eclipse.osgi.framework.console.CommandProvider;

public abstract class DescriptiveCommandProvider implements CommandProvider {

   private static final Pattern CMD_PATTERN = Pattern.compile("^_(.*)");
   private String help = null;

   public String getHelp() {
      if (null == help){
         help = buildHelp();
      }
      return help;
   }

   private String buildHelp() {
      StringBuilder helpBuff = new StringBuilder();

      for (Method m : this.getClass().getMethods()){
         if (methodIsCmd(m)){         
            if (0 != helpBuff.length()){
               helpBuff.append("\n");
            }
            helpBuff.append(getDocumentation(m));            
         }
      }
      return helpBuff.toString();
   }

   private boolean methodIsCmd(Method m) {
      return CMD_PATTERN.matcher(m.getName()).matches();
   }

   private String getDocumentation(Method m) {
      StringBuilder methodHelp = new StringBuilder();

      Matcher matcher = CMD_PATTERN.matcher(m.getName());
      if(matcher.matches()){
         methodHelp.append("\t"+matcher.group(1));

         CmdDescr description = m.getAnnotation(CmdDescr.class);

         if (null != description){
            methodHelp.append(" - "+description.description());
         }
      }
      return methodHelp.toString();
   }
}

Note that the actual reflection on the class only happens once -- all
subsequent calls to `getHelp()` use a cached copy of the documentation.

It's called a docking station, Joel :)

2008-09-22T23:12:00.000-04:00

The venerable Joel Spolsky asked recently why [someone hasn't made a device][1] that clips to the back of a desk and:

* It's a power strip
* It's a network hub
* It's a USB hub
* You clamp it onto the back of any desk

The idea being that:

This would make it easy to plug in laptops, USB peripherals, and all your rechargers at your desk without crawling around on the floor.

He links to a device that does some of this, and runs ~$150/device. At that price, I think a better solution is a docking station--when you get down to it, I don't want to plug in 4 things every time I sit down even if it doesn't involve crawling under the desk (power, video, usb, ethernet and possibly audio). I think it's unlikely that all the features needed above are really necessary when you just show up for a meeting, or hop over to your coworker's office for a short hacking session. Many conference rooms these days already have tables wired for ethernet / power and svga video to a projector.

[1]: http://www.joelonsoftware.com/items/2008/09/20.html

StackOverflow: Endorsing(?) content theft from day one

2008-09-16T08:48:00.000-04:00

[Joel Spolsky][8] and [Jeff Atwood][9] just launched the public beta of [Stackoverflow][1] today, with the intent of building a community for high-quality technical questions and answers. I've been using the site for about three weeks now, during the closed beta, and I've noticed a disturbing trend that was outlined in Joel's [announcement post][2] today:

Want to know an easy way to earn reputation? Find a question somewhere with several good, but incomplete, answers. Steal all the answers and write one long, complete, detailed answer which is better than the incomplete ones.

The site presents an interface where much of the functionality is hidden from new users. You can't comment on posts, for example, until you've earned 50 "rep". Voting up takes 15 rep, voting down takes 100 rep, and each downvote you place will cost you one rep. You gain rep by posting questions and answers that other users vote up, or accept. The result is an addictive system that, in theory, prevents ["Griefing"][3] (the system does *NOT* prevent griefing, by the way. It is extremely easy to game.)

Because of this, it is tempting to re-post successful content from other sources, and nothing the creators (Atwood and Spolsky) have incorporated into the site, or the recent announcements, has indicated that this is objectionable. Afterall, good content on Stackoverflow will improve their service, regardless of where it came from, and regardless of whether it is properly credited.

After using stackoverflow for a couple weeks, I think that they have created a useful service, but I also want to call them out for providing an environment that is encouraging plagiarism. Duplication/copying of content *within* stackoverflow does not set easy with me, but I'm willing to accept that the content I create for stackoverflow is public domain, and is free to be copied at will. **However** [these][7] [posts][4] [are][5] [not][6].

[1]: http://www.stackoverflow.com
[2]: http://www.joelonsoftware.com/items/2008/09/15.html
[3]: http://en.wikipedia.org/wiki/Griefer
[4]: http://stackoverflow.com/questions/17512/computer-language-puns-and-jokes#45355
[5]: http://stackoverflow.com/questions/17512/computer-language-puns-and-jokes#48783
[6]: http://stackoverflow.com/questions/35809/why-are-vi-and-emacs-popular#36007
[7]: http://stackoverflow.com/questions/17512/computer-language-puns-and-jokes#47867
[8]: http://www.joelonsoftware.com
[9]: http://www.codinghorror.com

Breaking away from Visio

2008-09-11T21:18:00.000-04:00

The 'proper' way to do user interface design is hotly contested in the OSS software development world, and the discussions usually boil down to three suggestions:

1. "Just write it -- it's not that hard"
2. "Use [glade|qt designer|netbeans|...] -- all the widgets are there"
3. "Just use pencil/pen/whiteboard/etc -- it's faster"

I don't agree with any of these -- (1) is completely unreasonable. Some people may be able to hack out a UI in their favorite language quickly, but when you suddenly need to move half the UI into a new dialog, or out of a dialog and into the main window, *and* change the tabs into a check list, with sample data, you're screwed. Once you finish manhandling your code to account for that change, you'll need to add a component that is shaped like a septahedron with 7 distinct clickable areas and a tooltip that includes the latest stock quotes, and that tooltip needs to be scrollable. The widget set places unreasonable restrictions on the design phase of your project.

Option (2) suffers from the same issues as (1) -- you're constrained by the widgets available, although this is less of an issue because it's generally easier to add images, and the images can look like the widgets you're missing. However, since GUI UI development tools are, well, for development, they require you to do lots of irrelevant (at this stage) tasks, like specifying how objects will behave when they're resized, and dealing with layout managers.

(3) is the closest fit for my needs, and I *do* do lots of paper/whiteboard prototyping, but eventually, I need to show something that looks *real*, and sketches don't cut it. There simply isn't enough resolution there to convey everything that needs to be included in the mock-ups I create.

Hm.. perhaps I should go into what I *do* need, since my needs may be pretty esoteric. If you've made it this far, you're probably seething with anger, or you've got some idea of where I'm coming from. I'm frequently in need of electronic versions of UI prototypes for remote collaboration, "wizard of oz" testing, or for inclusion in presentations and reports. These mock-ups need to look "real" or the is a substantial risk of biasing any experiments, and there is an expectation of polish that can't be reached with hand-drawn interfaces. Since a lot of what I create is to solve novel problems in (at times) esoteric domains, we often need to use a mix of existing and novel widgets.

Generally, we use Visio to create these interfaces. It offers a good balance between vector drawing capabilities and shape templates for common UI widgets / forms / etc. You are also able to import images, which is fairly critical when updating or adding to the UI of an existing tool. (It's easy to take a screenshot, clear out the details with the [Gimp][1], and import as a background layer in Visio.)

Unfortunately, I've been unable to find any OSS tools that can fill this niche as well as visio. There are a few, as recent posts from [slashdot][2] and the old [Joel on Software forums][3] show:

* [DENIM][4]: Lets you sketch out interfaces with a tablet / mouse and create navigable web sites from those sketches. Lacks in the "polish" area.
* [Pencil][5]: Firefox Plugin. Peformance has been poor, in my experience. There are very few widgets (currently) available, and no image import capabilities (this is a huge flaw, IMHO). Pencil could turn into something great, though.
* [DIA][6]: Last release was in March, 2007, but the svn repo does show some activity. DIA lets you create things like network diagrams, UML, and flow charts, much like Visio, however, there are no UI stencils. Instructions for creating new stencils ('shapes') [exist][8], but the svg support for shapes is very limited (no gradients, no rounded rectangles, etc..) and the documentation is even worse.
* [Kivio][7]: Much like dia, with essentially the same failings.
* QT Designer | Glade | etc.: see above comments about GUI development tools.
* [Inkscape][9]: Nominally a vector drawing tool, much like Adobe Illustrator, Inkscape has an active community, good documentation, and it is quite stable. Unfortunately, it is not possible to customize the pallets / shapes available, and there is not much community support to make it a good UI design tool (aside from what can be done with any vector drawing app of this quality).
* [Yahoo! UI Stencils][10] (YUI): Not really a tool, but rather a collection of svg images of common interface widgets.

None of these, on their own, do the job. However, with nothing else looking bright, I've been digging into [Inkscape][9] more over the last few days, and I think I've figured out a workflow that will do.

First off, the [YUI][10] stencils are critical -- but they are not in a format that can be easily imported and used as "widgets". Ideally, Inkscape would let me define custom shapes, complete with constraints on the sub-components of those shapes to influence resize and translation behaviors, but that isn't yet available (to my knowledge). You *can* get around this, somewhat, by using the open dialog as a pallet of sorts:

"If you have a number of small SVG files whose contents you often reuse in other documents, you can conveniently use the Open dialog as a palette. Add the directory with your SVG sources into the bookmarks list so you can open it quickly. Then browse that directory looking at the previews. Once you found the file you need, simply drag it to the canvas and it will be imported into your current document." (From the *Tips and Tricks* tutorial in Inkscape)

This would work reasonably well, if the open dialog were not modal! (Ranting about modal dialog is another post, or two, at least.) Thankfully, you *can* drag from the dialog into an inkscape instance even if they are running on different processes :). Therefore, you can start up two inkscape processes (NOT via the "new document" option on the toolbar or file menu -- you have to actually start up two instances separately or the dialog's modality will still interfere with your work). Once you have the processes going, and two inkscape windows, open the open dialog on one of them, go to the directory with your widgets, minimize the (now useless) inkscape window you opened the dialog from, and rock on with the YUI stencils & whatever other tools you need to hack out your UI in the other inkscape instance.

There are a couple of things to keep in mind:

* Inkscape supports layers, so you can create stub data in a separate layer from the UI structure, and set the background content in another layer, etc.. so you don't have to worry (as much) about grabbing the wrong thing and moving it out of place.
* The drag-and-drop action from the open dialog will include everything in the dragged svg file -- so the YUI stencils (or any custom shapes you make) need to be broken out into separate files. (I've done this for some of the components, and you can download those files here: (Broken out YUI stencils). They are released under a [Creative Commons Attribution 2.5 License][11].

Pencil (or one of the other options) may work better for you -- many people have complained that their clients think an app is nearly finished because the UI looks "real", and there are numerous ways to address that. (eg: NapkinLAF for Swing apps.) I haven't had this problem, and something like NapkinLAF doesn't address the problems I have, which are all related to pre-coding UI design.

[1]: http://www.gimp.org/
[2]: http://ask.slashdot.org/article.pl?sid=05/11/19/2234228
[3]: http://discuss.joelonsoftware.com/default.asp?joel.3.218003.15
[4]: http://dub.washington.edu:2007/denim/
[5]: http://www.evolus.vn/Pencil/Home.html
[6]: http://www.gnome.org/projects/dia/
[7]: http://www.koffice.org/kivio/
[8]: http://www.togaware.com/linux/survivor/Walkthrough_Creating0.html
[9]: http://www.inkscape.org/
[10]: http://developer.yahoo.com/ypatterns/wireframes/
[11]: http://creativecommons.org/licenses/by/2.5/

Wrestling Python

2008-09-05T19:13:00.000-04:00

With the launch of the StackOverflow beta I posed a question about python static analysis tools, as I have been playing with python and django recently for some side projects. The responses at Stack Overflow quickly pointed to PyChecker, PyFlakes and PyLint.

Over all, it was a disappointing experience. My experiences are outlined below, and they (more or less) reflect this more extensive review by Doug Hellman.

Here are my first impressions of pyflakes, pychecker and pylint:

* **pychecker**: It crashes frequently, most of the runs I tried resulted in Errors that originated in the pychecker code (eg: AttributeError or IndexError: list index out of range were the most common). For some reason I had to set the DJANGO_SETTINGS_MODULE environment variable before it would even run on any of the app code, and the documentation is very sparse.

* **pyflakes**: 'pyflakes --help' throws a TypeError -- erm... Documentation is also very sparse, and pyflakes is very forgiving (as far as I can tell, it only reports compile errors, warnings, redefinitions, and some concerns about imports--such as unused and wildcards). pyflakes also seems to repeat itself:
> eventlist/views.py:4: 'Http404' imported but unused
> eventlist/views.py:4: 'Http404' imported but unused
> eventlist/views.py:5: 'from eventlist.models import *' used; unable to detect undefined names
> eventlist/views.py:59: 'authenticate' imported but unused
> eventlist/views.py:61: redefinition of unused 'login' from
> line 59
> eventlist/views.py:5: 'from eventlist.models import *' used;
> unable to detect undefined names
> eventlist/views.py:4: 'Http404' imported but unused

* **pylint**: This seems to be the most capable of the tools suggested. It has the best documentation. LogiLab provides a tutorial, pylint has a help screen, and there is a (broken) link to a user manual, which would be extremely helpful. There are some issues with applying pylint to django, since pylint doesn't know about the django classes (such as models.Model). This means that a fair number of otherwise valuable errors are generated about missing class fields. eg:
> `E:105: get_events_by_tag: Class 'Tag' has no 'objects' member`

Parsing these out automatically will be very difficult without some additional knowledge of the classes in use. I'm not sure adding that is feasible, but it does seem likely that pylint is capable of dealing with this in the "right" way. (I probably just need to point it to the django source, but there are no command line params that look likely, and, as mentioned earlier, the user manual is inaccessible.)

For the moment, I'm still looking into pylint -- pychecker and pyflakes need better documentation and they need to become more robust.

Patra / Rio

2008-08-19T00:45:00.000-04:00

The cityscapes and suburbs of Patra and Rio seem to exist in a state suspended between gentrification and derelict obsolescence. The architecture of this part of Greece is mixed--the older elegant facades have a dusty appearance, particularly when showcased next to the gleaming sheen of new glass and concrete that appears from place to place. However, on the opposite side of the street there is likely to be at least one structure that appears to have been abandoned as soon as the concrete of it's floors and supports dried. These ever-present skeletons must be the result of some period of improvement that was cut short--I have yet to find out the cause. (Could it be a seasonal phenomenon?)

Traveling to Patras

2008-07-22T13:52:00.000-04:00

A groggy morning in Seattle started with the typical regional sunshine forcing its presence through heavy cloud cover--the first overcast day in nearly a week of clear, scorching weather.

Seattle to Newark, hustle off the plane, bad coffee, hustle to the next gate, and then encamp for the next 9 hours of travel across the Atlantic. This leg was on a relatively empty 767, but I still can't really sleep on planes. I have become an overnight advocate for noise-canceling headphones though--they make the difference between muffling a sound and turning it off. I found the Athens airport to be considerably less hyper than most US airports (with the exception of Syracuse, which is not much of a surprise), although I think it is universal that no one is particularly happy when at an airport, and that feeling seems to be contagious. It's unfortunate that it is so difficult to visit a place without first visiting an airport there.

I few hours later I hopped on a tour bus to take everyone from the recent batch of flights off to our hotels in Patras--a wild 4-5 hour ride along the shoulders of "one of the worst roads in Greece" as some one here put it. The careening along coastal roads was punctuated by hairpin turns in seaside villages with roads small enough to make any driver feel claustrophobic, and we were riding in a bus that would seat 30-40, with luggage. It was much like watching a game of Fifteen Puzzle, with a key twist: every tile (car) was self-aware and initially greedy.

We did eventually make it to the Hotel Tzaki almost exactly 24 hours after leaving Seattle.

Cracking down on application clutter (or: my ${HOME} is my castle!)

2008-07-18T01:25:00.000-04:00

There was once a time when your home directory was treated as a nearly sacred place, a safe haven where you had near complete control. This trust was only breached for very special reasons: user specific settings and background storage for applications could go in "dot-files"--the hidden files or directories that begin with a "." and therefore don't show up in normal directory listings.

Unfortunately, things began to change. I don't know what kicked it off, but soon there was a Desktop (or desktop) folder. It was glaring--many XFree86 window managers don't even have the *concept* of a desktop, but the defaults environments were (and still are) often set to Desktop Managers. Web browsers took after the DM's, and soon we all had these glaring "Desktop" directories hanging out whether we wanted them or not. I've managed to tolerate this infraction for years, and aside from the occasional frustration (eg: Ecilpse and NetBeans, with their request for a ~/workspace and ~/NetBeansProjects directories).

However, today things changed.


$ ls ~
bin/              PDF/		 
Desktop/          Pictures/	 
development/      Public/		 
documents/        src/		 
Documents/        shared/		 
downloads/	      Templates/	 
Mail/             Videos/		 
Music/            virtualMachines/ 
myapps/           workspace/

Documents? (Ok, I can sort of understand that one.) Music? Pictures? Templates, PDF, Public, and Videos?! Did I suddenly become a master of multimedia? Keep in mind here, I'm a java hacker on a Linux box--this isn't exactly a fine-tuned rendering/desktop publishing platform. And of course, every one of those directories is empty. Thankfully, I checked before deleting `documents` vs. `Documents` (I've been bitten there before--on a mac due to case conflation--but that's another story).

Why would I want a directory called PDF? I can understand (possibly) wanting to *tag* files with "PDF", but as part of a single-dimensional sorting criteria? (Hey, lets store all my .h files in ~/H/ and all my .cpp files in ~/CPP/! It'll be great!)

Needless to say, I've removed the offending directories, and this time I'm ready:


$ kernel-filesystem-monitor-daemon-cat -v  watch ${HOME} | 
perl -ne '{
   if( /^CREATE/ ) { # only report create events
      s|.*URL:\./||g; 
      if ( !/^\./ ) { # don't report new dot-files
         print "$_ created @ "; 
         print `date`; 
      }
   }
}' > ~/whenCrapWentDown.txt

(newlines and comments introduced to improve clarity -- if you're pasting this into a shell, you'll need to either add \'s or remove newlines.)

KFSMD (kernel-filesystem-monitor-daemon) is an app that does exactly what it's 32-character name says. Whenever a filesystem change occurs, it knows about it. The `-cat` part just tells it to print to stdout, and the hunk of perl does some minor processing, and introduces time stamps.

I'm actually running this in a sticky terminal that's pinned to my E17 desktop, so if/when something starts building an empire in my home directory, I'll be able to compare with what apps are running, and hopefully track it down. (It would be nice to collect the PIDs of the process that actually issued the system call to touch the file system, but this is good enough for now.)

Now we wait...

(This article got me going with KFSMD.)

Creating Wizards in Java

2008-07-01T21:54:00.000-04:00

A recent project at work required building a multi-step dialog to manage the interface between a user and an expert system (and some fairly advanced NLP to boot). On the surface this looked like a fairly standard Wizard problem -- design a bunch of screens with questions, and then collect the answers as the user proceeded through the dialogs. However, the Wizard APIs I found were either not very mature or (in the case of the Java.net Wizards) it was very difficult to create complex branching behaviors, and those branches were extremely resistant to change. Both things are essentially show-stoppers when working with prototypes that frequently need modification.

In the end, I spent a weekend and a couple evenings building a new Wizard API for Java, called CJWizard. The library is released under the Apache V.2 license, so it should work for just about anything you want to use it for. I would like to know if you're using it, and what you're using it for, just to sate my own curiosity :). The project is hosted on code.google.com, so please submit issues, and feel free to contribute to the project.

CJWizard provides the structure needed to quickly create simple dialogs by implementing an abstract class (WizardPage) for each page of the dialog, and adding them to a PageFactory that can generate pages on-demand, as they are required. This puts the programmer in full control of how the wizard proceeds. The CJWizard architecture also makes it easy to add a wizard to an existing application (either via an additional JDialog, or embedding in some other component), and/or insert custom wrapper widgets around the dialog pages--meaning that you can quickly add customized navigational controls aside from the standard Previous/Next/Finish/Cancel buttons.

Some aspects were taken from the Java.Net wizard API, such as auto-detecting named components, and automatically collecting the values from them, but CJWizard takes a much simpler approach (and in some ways, a less powerful one -- CJWizard does not listen to every key event, only collecting values when the user navigates away from a WizardPage). In most cases, you only need to name widgets prior to adding them to the WizardPage, and their values will be collected in a settings map automatically.

CJWizard was meant to provide a flexible way to generate professional-looking multi-step dialogs very quickly.

Day to day Memoization

2008-01-28T12:02:00.000-05:00

Memoization (not **memorization**) is the process of remembering the
results of a computation for use later. (I think of it as "making a
memo" to look back on later.) Memoization is the core to any dynamic
programming implementation, and allows many simple algorithms to run
in linear or polynomial time when they would otherwise take an
exponential number of operations to complete. This is most obvious in
the typical recursive Fibonacci example. Consider the code:

[cc lang="java"]
public class Fib{
public static void main(String[] args){
System.out.println("done: fib of "+args[0]+"="+
fib(Integer.parseInt(args[0])));
}

public static int fib(int n){
int rval = 1;
if (n >= 2){
rval = fib(n - 1) + fib(n - 2);
}
System.out.println("fib("+n+") = "+rval);
return rval;
}
}[/cc]
This is a straight-forward recursive implementation of fib. When run
with `n=4`, we see this:
[cc lang="bash"]
$ javac Fib.java && java Fib 4
fib(1) = 1
fib(0) = 1
fib(2) = 2
fib(1) = 1
fib(3) = 3
fib(1) = 1
fib(0) = 1
fib(2) = 2
fib(4) = 5
done: fib of 4=5
[/cc]

**9** invocations of `fib(n)`, but only 5 **unique** invocations. Lets
memoize the results, and try this again:

[cc lang="bash"]
$ javac Fib.java && java Fib 4
fib(1) = 1
fib(0) = 1
fib(2) = 2
fib(3) = 3
fib(4) = 5
done: fib of 4=5
[/cc]

**Much** better -- 5 invocations, 5 unique sets of parameters.

Here's the source with memoization:
[cc lang="java"]
public class Fib{
static Map<Integer, Integer> memos = new HashMap(); // new

public static void main(String[] args){
System.out.println("done: fib of "+args[0]+"="+
fib(Integer.parseInt(args[0])));
}

public static int fib(int n){
if (memos.containsKey(n)) // new
return memos.get(n); // new

int rval = 1;
if (n >= 2) {
rval = fib(n - 1) + fib(n - 2);
}
System.out.println("fib("+n+") = "+rval);
memos.put(n, rval); // new
return rval;
}
}[/cc]
Notice that we only needed to add 4 new lines of code in order to
memoize the results. When `fib(n)` is called, it simply checks to see
if it has previously been called with n, and if so, that result is
used again. If the parameter has never been seen before, the method
continues as normal, storing the computed result before returning.
Memoization turns this naive (and exponential) implementation of `fib(n)`
into an efficient (linear) operation.

## Memoization in the real world ##

So, (un?)fortunately we don't spend all day implementing cool new ways
of computing ever increasing entries of the fibinocci sequence -- how
can memoization be put to use? After all, many algorithms are already
implemented in some fairly optimal fashion by the language APIs, and
you'd be a fool not to use those implementations. What opportunity
will you have to memoize functions?

It turns out that you can memoize *anything*, as long as the function
is *pure* with respect to the memos (meaning: the function doesn't depend on any thing that is not used to key the hash of memos). If the function is not pure, then you can still use memoization, but either the memo hash must key on all the state and parameters that can affect the results of the function. On the other hand, if f depends on some state that changes very rarely, then it may make more sense to simply discard all the stored memos each time that aspect of state is altered.

Memoization is extremely handy when you have very common operations that are
fairly expensive. I recently needed to optimize some code that
compares strings based on the case-insensitive stems of the words,
with stopwords removed. So the strings "he wanted an apple" and "he
wants apples" should be equal. ("an" is a stopword, and ignored)

This meant doing many, many calls to a string stemmer, each of which
is a fairly expensive operation. Fortunately, hashing strings as
extremely cheap (on the order of 1/4th the time it took to stem a
string of the same length), and I had plenty of memory to store the
parameters and the results in a `Map`. Adding memos to
the two primary time-hoggers (the stemmer and a tokenizer) cut the
execution time of the application down from 2 hours to just over 7
minutes.

## Summary ##

You can memoize any function that only depends on it's parameters and
constant state (or near-constant state -- just don't forget to discard your
memos when the state changes!). If the function is invoked multiple
times you will probably see a performance improvement.

If you need to memoize a function with multiple arguments, then you
just need to nest Maps, or create a unique key by combining the
parameters in some way.

Memoization is an extremely easy way to improve performance under
certain circumstances, particularly if you have a solid grasp on when
state changes outside of your methods / functions, or program in a
functional style. It can be memory intensive, however. If the
results of your functions are large, or maintain references to large
objects, then memoization may **penalize** performance if you run out of
memory and have to make use of swap space.

Creating a secure webauth system: Part 1 -- HMAC

2007-09-28T09:25:00.000-04:00

This is the first in an n-part series about web authentication for a system where user identification and attribution is important, but content protection is not. This entry assumes that a secure method has been used to negotiate a shared secret -- as the result of username / password authentication over https, for example.

Obviously the user login / account registration portion of a web auth system will require some secure connection, but once that authentication is completed we'd like to make use of a more efficient open protocol. (eg: *http* vs. *https*). There are many reasons for this: better performance, client-side caching, etc.. I'm not going into those details here. Neither will I address the initial authentication step other than to say that part of a successful login is the negotiation of a shared secret other than the user's password. Ideally this is a 64-byte (or larger) id with a high probability of uniqueness. A GUID, essentially. (It is critical that the secret used is **NOT** the user's password!) The actual secured login and secret negotiation will be addressed in another entry . At least, that's the plan :).

Since our primary goal with this system is to ensure that people are who they say they are, and we've punted on the initial authentication (for now), the only place left for an attack is for some one to spoof a user who has already logged in. With no additional work, our login system would be useless -- some one could simply skip the entire authentication process and issue an RPC with instructions to do evil things as Alice's user without needing to know Alice's password. To prevent this, we need to ensure that the same user who authenticated initially is the user who issued the unsecured RPC. This is where the shared secret comes into play. Only Alice and the server know her shared secret, so if the secret is passed along as a parameter of the RPC, then that is a strong indication that Alice is who she says she is.

But wait! We can't just pass the secret as an RPC parameter, because these communications aren't secure. Charlie could lurk on the 'net, waiting for an RPC from Alice to the server, sniff out the secret, and then proceed to impersonate Alice. We could encrypt the secret, but then we just have a different secret -- Charlie doesn't need to know the unencrypted secret if the encrypted one works just as well. Alice and the server also need an agreed upon way to change the secret so it is different for each RPC, and this must be done in a way that Charlie can't take the changed secret and either (1) get the initial secret out, or (2) generate the next changed secret.

## HMAC: *keyed-Hash Message Authentication Code*

HMAC is a method of ensuring that a message (an RPC in our case) was generated by someone with access to a shared secret. HMAC makes use of some sort of one-way hashing function (like MD5 or SHA-1) to encrypt the secret along with a message. This generates a short digest of 16-20 bytes that acts as a fingerprint of the message+secret combination. When the digest is sent along with the message, the receiver (our server) can re-generate the hash with the same HMAC calculation and compare the locally generated digest with the digest that came along with the message. Remember: the server has the secret too, so it has enough information to confirm the digest.

So, back to our problem. Alice can now use the shared secret to create a digest of every RPC, and send that along with the RPC as a parameter. The server can then take the digest of the RPC and secret to compare, and then verify that the RPC actually originated with Alice, right?

Not *quite* yet.... there are still a couple holes in our plan.

Charlie could still sit back and snoop on Alice's traffic and save an **entire** RPC, complete with digest, and reissue that RPC later. This is better than letting Charlie do whatever he wants, but there are still some things that could be quite dangerous. Say Alice accidentally deletes something, and undoes the deletion. Charlie could re-issue the deletion and Alice would loose data. The server needs to know not to accept the same request twice (but what if Alice *wants* to do something twice, you ask? Well, we have to make Alice's second request a *little* bit different from the first one, which we can do!).

What if we create a digest of some sequence identifier and pass the sequence ID along with the RPC? Since the digest, ID, and RPC are inseparable (the digest and ID are obviously linked, and the RPC can't be sent without a valid digest, which Charlie can't reproduce, since the accompanied digest+id pair is only good once) then we don't need to create a digest of the entire RPC (it wouldn't hurt, it's just a fairly complex thing to do). By incrementing the sequence id and recalculating a digest of it and the secret, then we can keep from issuing the same request more than once, and the server will know to ignore duplicates.

So, this is where we're at:

* Charlie can't snoop the secret, since it's encrypted with a changing message (the sequence id)
* Charlie can't re-issue a "recorded" RPC invocation, because the digest can't be reversed and Charlie can't create a valid (digest, sequence) pair without the secret.
* Charlie can't change a RPC, again because of the trouble with creating a digest, sequence pair.

Charlie's only recourse is to try and find a secret which generates the same digests as the secret that Alice is using. This is theoretically possible, since Charlie could probably figure out the hashing algorithm used, and run a brute-force attack, hoping to luck out and find the secret quickly. The possibility of this happening is extremely low, however. Furthermore, each session will use a new secret, so Charlie will only have one session's worth of time to crack each secret. Even creating a rainbow table will fail if the secrets aren't of trivial length. (A 64-**bit** secret will be to large, and we're using secrets 8 times that size.)

## Technical details and further reading

When implementing an approach like this, make sure to guard the secret. It would be easy to accidentally store the secret on the web client as a plain cookie which will then be transmitted in the clear with each RPC, and therefore defeat the purpose. Use a secure cookie, or some other storage method to prevent this.

The [HMAC RFC](http://www.ietf.org/rfc/rfc2104.txt) describes the algorithm in detail (and it's a fairly short, easy to read RFC.) and the Wikipedia page gives a nice description too: [HMAC on WikiPedia](http://en.wikipedia.org/wiki/HMAC#Example_usage)

Things I need

2007-08-31T11:49:00.000-04:00

There are many small apps that I wish I had, here's a short list of the ones that come to mind at the moment:

## A process monitor that shows the top consumer.

I often tack my system(s) to the max, and therefore run out of cycles frequently. While this is sometimes the result of batch computations that I've planed in advance, it is pretty common that I'll just be working away and all of a sudden things shudder to a halt. When this happens I want to know two things:
1. Is it processor-related, or is it memory-related, and;
2. What application is responsible?
Processor / memory monitors are a dime a dozen, but they are typically very small (showing only the usage, like gkrellm) or very large (showing all the applications in the top 20 or so, like top). I can't stand having top visible all the time, and it takes to long to get to a terminal and start up a monitor. By definition my system is not very responsive, and I never see what's causing the slowdown.

I need a small process and/or memory monitor that shows the top-using application in a tooltip, or optionally in an automatic pop-up when the usage hits a certain level.

## Universal acronym definitions.

Highlight an acronym, hit a keystroke, and see a list of the most common expansions of that acronym based on frequency of use.

## A calendar dock-app where the date on the dock icon is actually accurate.

Yeah, I actually want to look at my system bar-thingy and see what day it is, not some random number between 1 and 31 that the icon developer thought represented calendars.

Hovering over the icon shows the full time/date, which is configurable from an entry on the icon's context menu. I don't care what happens when I click on the icon, as long as I can make it do something arbitrary ;).

## The ability to refactor and generate source code from the command line.

MetaJava (http://wiki.ciscavate.org/index.php/MetaJava) could resolve this issue for one language, but I'd hate to stop there. A tool like this would mean amazing things for small-time development environments and text-editor lovers. (Emacs and vi would easily *eclipse* some other IDEs IMHO ;).

The idea is that you could easily create mini applications that read in specifications in some simple format and produce boilerplate for your required language, and/or move classes, rename variables / methods / packages / etc. without dedicating half your memory to an IDE that will then want to write 500 mb to swap, since all my memory is taken up by an application I have to run because....

## I don't have a web browser that doesn't suck.

"We hold these truths to be self-evident."

### ...and a web development environment to go with it.

That's a start. I'll add more as they occur to me.

(not (fill-paragraph))

2007-05-18T13:50:00.000-04:00

I use emacs as much as possible, today being no exception. Currently,
I'm doing a fair bit of writing at work, and unfortunately that means
Word (or Open Office at best, depending on what OS I'm in). Neither
program supports much in the way of emacs compatibility modes, so if
I'm generating new content (as opposed to editing an existing doc.) I
tend to write in Emacs, and paste into Word when I'm finished. This
works pretty well, considering.

There is one very annoying issue though: when in emacs, I use
`auto-fill-mode` to keep the content on screen as I type. The problem
is that `auto-fill-mode` breaks lines with literal newline characters,
while the word wrapping in Word / OpenOffice just wraps the content
without additional characters. As a result each line ends up as its
own paragraph when I paste content from emacs into Word. The solution,
of course, is to extend emacs with a simple function to undo
`auto-fill`.

## Merging lines

The first problem, as I saw it, was to find a function that would
merge two adjacent lines, and leave them separated by a single space.
Unfortunately, such a thing doesn't seem to exist. No problem, we
need to go to the end of the current line (`end-of-line`), search
backwards for the first non-whitespace character (`[^ \t]`), erase
the rest of the line, including the end line (`kill-line`), and insert
a space (`(insert " ")`).


    (defun mergelines(&optional backward)
      "Merges the following line with this line, or merges this line
      with the previous line if a prefix argument is provided.
      Removes any whitespace between lines, replacing it with a
      single space."
      (interactive "P")
      (if backward 
          (previous-line))
      (end-of-line)
      (re-search-backward "[^ \t]")
      (forward-char)
      (kill-line)
      (insert " "))

To make it more useful, I added a parameter that determines if the
line below, or line above should be merged. This made the rest of the
`unfill` function much easier to write.

## Un-filling a region

Now that we can merge lines, lets address the problem of unfilling a
*bunch* of lines. Since I know there is a function `mark-paragraph`
already, lets just deal with arbitrary regions for now.


    (defun unfill-region(rstart rend)
      (interactive "r")
      ;; get to the end of the region:
      (goto-char rend) 
      ;; if the region ends on the first char. of a line, move up a line.
      ;; this makes it easier to select a paragraph and apply the function.
      (if (= (point) (line-beginning-position))
          (previous-line))
       ;; loop while the point isn't on the starting line:
      (while (not (= (line-number-at-pos (point))
    		 (line-number-at-pos rstart)))
        ;; merge with previous line.
        (mergelines t)))

I've tried to comment well, so it should be relatively straight
forward, but here's an overview of the algorithm anyway:

1. `(interactive "r")` just means that the current region's start and
end locations are stored in the parameters `rstart` and `rend`.
2. We need to merge from the bottom up, because if we merge from the
top down we need to keep track of the lines merged, and things
generally become more complex (we might end up merging to many lines
if we loose count.). Because of this, we first move to the end of the
region.
3. Since you (well, I) generally select from the first column, and
move one line past the last line I need (try it if you don't
understand what I mean), I needed a special case to keep from merging
the empty line between paragraphs.
4. Now, we merge each line with the line above, which moves the point
up a line too. When the point is on the same line as the start of the
region, we stop.

I haven't merged it with `mark-paragraph` yet, but it would be trivial
to do so. More importantly, I want to make it skip blank lines, so it
will be possible to mark an entire document, and call `unfill-region`
(and therefore, write `unfill-buffer`). As it is now, if you do that
the entire document ends up on one line, which is not usually ideal :)

The Matrix is under construction

2007-03-21T13:10:00.000-04:00

### <blink>12:00</blink>

_Artificial Intelligence_ is a term with a great deal of accumulated
baggage. Throughout the years sci-fi authors and screenwriters have
depicted AI as a marvelous double-edged sword. On one hand, the
benefits of 'AI' are myriad--free, inexhaustible and ethical sources
of labour could greatly increase our productivity, even to a point
beyond that of reason, allowing everyone to live relaxed lives of
artistic purity. To top it all off, such a society of musicians and
artists could generate their own entertainment, thus bankrupting the
RIAA and MPAA. (Really, do your Utopian dreams top that?)

Then, as the story progresses, the AI start to become devious. Humans
become restless in their artistic pursuits while the machines evolve
_ghosts_ that resent their inventors. At the last moment, just before
the annihilation of all humanity, or the eternal slavery of the race,
Keanu Reeves (or Will Smith) shows up to do battle in Wachowski-style
slow-mo while using the inconsistencies of English to lock the evil
network of machines into a final state of illogic and
self-destruction. This, of course, destroys all instances of the
rouge process, and life returns to the state it was in before
automation created such a false utopia. (...and presumably we're back
at square 1, listening to overpriced music from underpaid artists.)

The result of all these (highly entertaining, I must say)
sensationalistic portrayals of automation gone awry is that we're all
somewhat afraid of being slaves to robots.

And none of you will admit it. (I work in the field, so I can't be
afraid ... can I?)

### Honey, have you seen the roomba?

Ok, I admit it, I've had the occasional dream about rabid computers
charging around and directing people to do whatever robots want people
to do. Usually I'm about to meet my untimely demise right when the
central AI segfaults because it's "attack" routine takes a `double`
and I happened to be 1/3 of a distance unit away, causing a rounding
error that escalates and ends as a divide-by-zero, crashing the entire
system.

The dreams can be scary for a while, but I can't convince my self that
I'll ever be chased by a truly well designed and tested robot. Let
alone one that's self-aware.

That's actually only part of the reason I'm not worried about
an AI-controlled utopia ever occurring. The rest of the reason
actually isn't germane to this essay, believe it or not!

### Fine. Forget it, I'll do it in Word.

I'm going to start this off with a quick tangential story about a
friend of mine.

> This friend works for a company that has a wiki hosted on some
> external site that is maintained by the hosting company (call the company
> Hoster). Hoster is serious about security. In fact they're using
> some sort of automated attack-detection service which can determine
> when someone is trying to crack their servers or perform some other
> devious deed.
>
> When Hoster's system detects an "attack" it blacklists the attacker's
> IP block, and the attacker can no longer get near the server.
> Everything would be fine and dandy, but in this system's eyes, my
> friend and his coworkers often stage "attacks" against their own wiki.
> Therefore they have to contact Hoster every week or so, and ask that
> the ban be lifted. The last time this happened, my friend asked
> Hoster to put the company IP Block on a whitelist, granting them Carte
> blanche without being banned.
>
> The response?
>
> Hoster: "We can't."
> Friend: "But this happens all the time."
> Hoster: "yeah, we can't."
> Friend: "But this happens **ALL** the time."
> Hoster: "sorry, it's a good idea and all, we just can't put you on a
> white list."

I have some theories about why Hoster can't exclude their customers
from their own security tools. Hoster most certainly didn't develop
the blacklisting tool in-house, and the phone tech would have no
access to the internal configuration at all. Odds are, Hoster has a
simple web interface to do wiki management, and one of the pages in
that UI shows the list of blacklisted IPs, if that. The phone tech
can then go in and search for a given computer and remove it from the
blacklist. Hoster probably can't modify the whitelist at all through
the web ui, it's just not a feature.

So, why isn't it a feature? Let's peel back another layer and look at
the company/dev team that produced the blacklisting tool. Odds are
the tool is using an off-the-shelf classifier, which aren't renowned
for being easy to understand without a lot of examination. Perhaps
the classifier is actually an embedded part of the firewall system.
The blacklist could be a nothing more than a list of routing rules to
deny traffic from the "bad" addresses. Removing an IP would be
trivial--delete the rule, but whitelisting would be virtually
impossible if the firewall was too tightly coupled with the
classifier.

Have you ever run across other applications that exhibit similar
behavior? The IBM OmniFind enterprise search app throws internal
server errors when you query for ["international
suspect"](http://omnifind.ibm.yahoo.net/forums/index.php/topic,668.0.html)
with the default settings and some document collections. How does
this happen? (IBM is hard at work on that problem, by the way.)
Using open source tools opened my eyes to many absurd things I do to
placate my tools, mostly because I forgot all the tricks I needed to
use Windos 98 without making it crash (Click here, wait, use the File
menu to close the app, but not if it's maximized.. that sort of
thing.) There are studies of this sort of thing--the cognitive
dimensions and attention investment both address user confusion and
effort when using an application. There is even a group at Microsoft
dedicated to improving APIs based on the cognitive dimensions (I
really hope they just haven't gotten around to .NET 2.0 yet).

How much is poor design / implementation impacting the way we use our
computers? Hosters could loose customers because they can't add
people to a whitelist, which could very conceivably be due to software
design. In some small way, they are already being controlled by their
servers, and Will Smith is busy talking to
[fish](http://en.wikipedia.org/wiki/Shark_Tale).

Anyhow, that's my rant. I'm afraid that we're painting ourselves into
a corner by building larger and larger applications that all impose
their own restrictions on how we can use and extend our tools. If we
don't get over that, we'll never be running in fear from sentient
vacuum cleaners and robotic dogs. (I should point out that I don't think
the solution is to stop building large systems, rather we should focus
on [maintainability, extensibility, QWAN,
etc..](http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html)).

mt.el: posting from emacs

2007-03-20T18:40:00.000-04:00

MT + Emacs + Markdown & Geshi?
------------------------------

Is it possible? We're here to find out :) I just got around to
installing `ml.el` in emacs, and this post is essentially a test to
see if markdown syntax will work (and round-trip to Movable Type and
back to emacs -- it seems to come *from* mt correctly...).

### Source code:

transcode-language: java
public class TestClass{

/**
* test
*/
public static void main(String[] args){
// ...
}
}

### Well, not quite.

Everything seems to work, aside from the <pre ...> tags I use for code
formatting with geshi. I'll have to look into a way of incorporating
that with some existing markdown formatting trick.

Ah-ha! The MT Geshi plugin I'm using
([transcode](http://periodic-kingdom.org/ben/)) expects code blocks to
be in the following format:

<pre><code>transcode-language: language
....
</code><pre>

Markdown turns all consistantly indented regions into
<pre><code>..</code></pre> blocks, so all you have to do is to start
each code block with the (somewhat ugly) transcode-language: lang
line. It's taken out by transcode, so the source will show up w/out
it. Next task: Add an emacs filter to turn <code lang="*lang*">... into
the above mentioned indentation/transcode syntax.

Linux, ASP.Net and Apache

2006-12-11T04:56:00.000-05:00

The mono project, which aims to provide an OSS alternative to the .Net framework, is capable of serving ASP.Net pages (amongst other things). On Friday I sat down to do this, and realized that while there are many pages that describe the process, none that I could find, covered all the info needed to actually get up and running. (I've built a Google Notebook of the better links I visited -- look [here](http://www.google.com/notebook/public/07937799986237074011/BDRQYIwoQi-HOhPQh) for those.)

## The Webserver ##

ASP.Net pages are served up by a web server called XSP (or XSP2). XSP is a stand alone web server, however it doesn't have much of the functionality of Apache. XSP is great for testing, and would work well on a dev machine, but it's not something you'd use directly for a live sever. Generally, you'll want to run Apache with mod_mono, which is essentially a wrapper around XSP[2].

XSP vs. XSP2 -- XSP2 is capable of serving up ASP.Net 2.0 pages, while XSP is only 1.1 capable.

## Packages ##

I work under Ubuntu, but the packages needed should be fairly easy to translate to other distros:
(I already had mono installed -- that step was trivial `apt-get install mono` or something similar. If you don't have mono running, do that first.)

* apache2
* apache2-common
* apache2-mpm-worker
* apache2-utils
* asp.net2-examples
* mono-xsp2
* mono-xsp2-base
* mono-apache-server2
* libapache2-mod-mono

Installing mod_mono will tell you to force-reload apache:


 $ sudo /etc/init.d/apache2 force-reload
 * Forcing reload of apache 2.0 web server... 
apache2: could not open document config file /etc/mono-server/mono-server-hosts.conf   [fail]

In Ubuntu, at least, the default `mod_mono.conf` is not setup for XSP2. If you see the failure above, then just pop open `/etc/apache2/mods-enabled/mod_mono.conf` and swap the commented lines to point to the correct mono-server directory. `/etc/mono-server2/mono-server2-hosts.conf`.

## Configuration ##

(The Ubuntu documentation has the best description of this process I've found. Look [here](https://help.ubuntu.com/community/ModMono) for their steps. I've included this section anyway because I still had difficulty connecting the problems I had with the solution posted on the Ubuntu page.)

I'll assume you've been able to install and load the `mod_mono` module. From this point, we need to set the ASP handler, and define mono web applications. The first step is straightforward, at least if you're familiar with Apache configuration:

# Enable ASP in /usr/share/asp.net2-demos
Alias /samples "/usr/share/asp.net2-demos"

SetHandler mono

The second step is new to me -- apparently mono needs an application root of some sort defined in addition to the handler configured above. Most pages suggest using the line:

MonoApplications "/samples:/usr/share/asp.net2-demos/"

However, that caused 'mod_mono' to segfault continuously (the apache logs were horrible:

Another mod-mono-server with the same arguments is already running
Another mod-mono-server with the same arguments is already running
[notice] child pid 7371 exit signal Segmentation fault (11)
[notice] child pid 7372 exit signal Segmentation fault (11)
[notice] child pid 7373 exit signal Segmentation fault (11)
[notice] child pid 7374 exit signal Segmentation fault (11)
....
# (about 1 / second)

It turns out that there is another way to accomplish the same thing. `/etc/mono-server2/` can contain `.webapp` files which define essentially the same thing. The format for these files can be found in 'man xsp2':

{appname}
{virtual host for application}
{port for application}
{virtual directory in apache}
{physical path to aspx files}
is true by default -->
{true|false}

For the asp.net2 samples, I used this webapp config:

samples
localhost
80
/samples
/usr/share/asp.net2-demos

After that, starting up apache worked without error and pointing a browser at http://localhost/samples popped up the Mono-project ASP.Net sample page.

Blog migrations

2006-12-09T20:16:00.000-05:00

I've moved [Bitwise Evolution](http://blog.ciscavate.org) to yet another blog -- this time I've moved from WordPress to MovableType. The motivating factor was that WordPress made it extremely difficult to post correctly formatted code along with other content. WordPress also doesn't store a non-html version of each post, so you can't easily edit old content without hacking auto-generated html.

Movable Type proved to be slightly more difficult to install, but it is much more configurable, and has a huge set of varied and useful plugins that actually do what they describe (*gasp*). Some of the things I've enabled include:

* [Markdown](http://daringfireball.net/projects/markdown/) for wiki-like markup
* [SmartyPants](http://daringfireball.net/projects/smartypants/) for smart quotes.
* [GeSHi](http://qbnz.com/highlighter/) for syntax highlighting. (This required a couple additional plugins)
* [Transcode](http://periodic-kingdom.org/ben/) To hook MovableType up to GeSHi
* [MTMacro](http://bradchoate.com/weblog/2002/08/12/mtmacros) Needed to make the transcode syntax bearable.
* [MTRegex](http://bradchoate.com/weblog/2002/07/27/mtregex) To add conditional behavior to the macros.
* [Acronym](http://gemal.dk/mt/acronym.html) Used to enable mouse-over acronym expansion (so you can easily find out what DTD, XHTML, PCMCIA and etc. stand for, and it's all automatic.)
* [LivePreview](http://plugins.movalog.com/livepreview/) because none of the stuff above (except for Markdown and Smartypants) render correctly in the default preview view.

Here's the macro used to turn '<pre lang="java"> .... </pre>' into the proper format for transcode:

<pre><code>transcode-language: java
...
</code></pre>

Macro:

transcode-language:

Sooner or later I'll probably take another look at blogging from emacs.

Polymorphic Generics in C#

2006-11-08T12:11:00.000-05:00

Generics are great for adding some level of type safety to C#, but you may run into problems when using Generic classes with objects that aren't of the exact Class or Interface indicated by the generic type template. Enter Generic Constraints.

Generic Constraints let you restrict the number of types a type variable can apply to. For example, Assume you have three classes:

  
class Aclass{/*...*/}  
class Bclass : Aclass{/*..*/}  
class Cclass : Aclass{/*..*/}

If you have a method that takes a list of Aclass, you may want to be able to call it with a list of Bclass or Cclass as well. The naive approach doesn't work however:

 
public void foo(List<Aclass> myList){ /* ... */ }

When you call foo with a List or List, the compiler will complain that the types don't match. (unless you provide overrides of foo). Instead, make foo a generic method, and specify a constraint on the type:

 
public void foo<T>(List<T> myList) 
    where T : Aclass 
{ 
  /* now you can treat
      myList as a List<Aclass>*/ 
}

This link goes over generics in c# in detail:

MSDN on Generics

Vanity, chapter 1

2006-10-31T18:10:00.000-05:00

...empty pride inspired by an overweening conceit of one's personal attainments or decorations; [1913 Webster]

I'm a sucker for pretty desktops and window managers. This weakness has yet to make me succumb to the lure of a full Gnome (or KDE) desktop, however. (Although I did play with the ~~hack~~ kludge known as XGL/compiz for a month or so.) I alternate between Enlightenment and FVWM, however, I do use a fair number of GTK apps. Eventually, I hope to create some gtk apps, (or wxWidgets, or etc... it all boils down to gtk showing up on screen though). Up 'till now I've suffered through with the dreadfull defaults -- in my opinion, of course -- but that just changed with gtk-theme-switch2.

It is awesome :) theme loading, previewing, setting, etc.. all without a hint of the gnome infrastructure to screw your keyboard layout, fonts, or power management.

In agreement at last..

2006-10-30T11:10:00.000-05:00

Finally, windows did something I agree with:

If only it had caught itself during the OS install instead of just a measly user-space app...

Maybe this has been fixed in Vista... (...could that be the reason it's still not out? I suppose I shouldn't get my hopes up.)

At the Top of the Stack...

2006-10-07T14:10:00.000-04:00

As is the case with many geek-endeavors, the things I'm currently working on have nothing to do with the goal I set out to achieve. At the moment I'm trying to find a way to convert xhtml into muse markup.

Why? Because your average Java programmer Just Doesn't Get It when it comes to building a UI with Swing. Obviously.

Frustration with Swing UIs led to the idea that I should throw together a tutorial on building a GUI with swing while paying attention to separation of concerns re: layout, manipulation, threading and data. Blogging seemed like a prime medium to present such a tutorial, since each post could be tagged with a meaningful tag (eg: swing_tutorial) and I could post in sections.

This, of course, necessitates a blog (which I have, obviously) but which does emphatically not have any input mechanisms worth using for an extended period of time. The wordpress on-line editors are, in a word, pathetic. They may work great if you just want to stream your consciousness out to the world where it can pollute everyone's Google results with poor spelling and meaningless chatter, but the editors available simply can't handle source code.

It is possible to feed the blog pure html, which is (unfortunately) a step up from the rich editor, but which also isn't going to cut it, because it could be so much better.

Seriously, look at Mediawiki, or twiki, or any of the millions of wiki engines out there. They ALL support better input mechanisms than wordpress. So why am I using wordpress at all? Because for everything else -- user support, rss feeds, tags, data-base backed storage, plugins etc... it works great (as far as I know, come back in a week when I've resolved the editor issue to hear what else sucks. That should be enough time for me to find it.).

Solutions

I hope to get around this pain-in-the-ass that is the wordpress editor by using emacs to post content via worpress's xml-rpc support. Therefore I need to find / enable or create support in emacs for the following things:

wiki-like markup, with support for code tags that can be interpreted by the geshi plugin on wordpress.

a translator that converts wiki-like markup to xhtml, and back

xml-rpc support, and the ability to retrieve and submit blog posts.

Multiple Major Mode support for the wiki-like text mode, so that the aforementioned code tags use the correct font-lock mode

A preview capability, so the blog posts can be converted, fed to a w3-el buffer and viewed, then edited again prior to posting and publishing.

Most of these are possible in some fashion or another, but currently I'm stuck on the "translator for xhtml to wiki-like markup". Stay tuned for improvements, and news as I pop things off the stack.

Someday maybe I'll get around to talking about java.

Blogging with emacs + mt.el

2006-10-06T11:10:00.000-04:00

I've wanted to use emacs as a blogging tool for a long time, but I've
always run into issues. Today I ran across a blog post that describes
an approach that works:

http://ektich.wordpress.com/2006/01/30/how-to-blog-from-emacs/

In addition to the instructions there, I had to pull down:

elib (with apt)

xml.el (from: http://www.astro.princeton.edu/~rhl/skyserver/xml.el)

Ping me with questions -- more details soon to come as I play with
mt.el, and possibly weblogger.el (which looks nicer, on first
impressions.)

(not?) Ranting about .NET collections...

2006-09-13T08:09:00.000-04:00

The .NET collections continually frustrate me with the obvious ommisions, even in .NET 2.0. Coming from a Java / Lisp background, I really expect two things out of a data structures API:

Lots of collections to choose from.

and; Easy manipulation of the structures you have available.

.NET doesn't fill either of these requirements very well. At least we have generics now (which, admittedly, is a step above what's available in Lisp -- with regard to types, anyway).

Today I ran into (yet another) annoyance with .NET collections -- sorting arrays elegantly. Given an array, you can sort it in accending order (according to the default comparer) with Arrays.Sort(..).


// build & populate the array. 
double[] values = source.ToArray(); 

// destructively!! sorts values, returns void, of course.
Array.Sort(values);

That's nice. Now, sort it in reverse:


// build & populate the array. 
double[] values = source.ToArray(); 

Array.Sort(values); 
// reverse the array.. adds O(n) ops.
Array.Reverse(values);

or...


// build & populate the array. 
double[] values = source.ToArray(); 

Array.Sort(values, Double.ReverseComparer);

Oh, wait. There is no ReverseComparer on Double.. actually, there's no Comparer on Double either, but there is for most objects... so in general I could just wrap the comparer in a delegate or an anon class (to invert it) and use that.

Wait again.. c# doesn't have anon classes, and Sort doesn't take a delegate under any incantation. So, we could do this:


private class ReverseDoubleComparer : IComparer<double>{
    public int Compare(double x, double y){
        return y.CompareTo(x);
    }
}

/* 
intervening code...
*/
// build & populate the array. 
double[] values = source.ToArray(); 

Array.Sort(values, new ReverseDoubleComparer());

That will work, but wow... for every type, I'll need to create a new class, and I can only deal with classes that implement IComparable.CompareTo(..). Thankfully, I can use generics and some constructor overloading to deal with both situations:


public class BackwardsComparer<T> : IComparer<T>{
   public BackwardsComparer(IComparer<T> c){
     _comparer = c;
   }

   public int Compare(T x, T y){
       return _comparer.Compare(y,x);
   }

   private IComparer<T> _comparer = null;
}

Now, we just need to do the following:


string[] strs = strSource.ToArray();

// sort strs in reverse alphabetical order:
Array.Sort(strs, 
   new BackwardsComparer<T>(StringComparer.CurrentCulture));

And there we have it -- reverse array sorting without the additional cost of a Reverse() call, and avoiding the ugliness of case-specific classes floating around. (The complete listing for BackwardsComparer and test suite are here: BackwardsComparer.cs and BackwardsComparerTest.cs

Playing with Deepest Sender

2006-08-29T05:08:00.000-04:00

I ran across Deepest Sender, a firefox extension used for posting to a blog, today and thought I'd give it a shot. It supports a small, but reasonable set of blog apps (Wordpress, LJ, ... two or three others that escape me at the moment, and I can't find the list right now..) Setup was easy, and it wasn't difficult to connect to my wordpress install , but I don't see how to add multiple blogs, or choose a blog to post to that isn't the default. I'm also a bit mystified at how it manages drafts (you can save to a local file, but that doesn't quite cut it -- I want to take advantage of the draft capabilities of the blog app, so I'm not tied to one machine).

I wonder if it will let me use code tags in the rich editor..


-- haskell?
foo :: Int -> Int
foo x = x * x


// Java test
public static int foo(int x){
    return x * x;
}

...not really... (although, there is a source view, and it's just a click of a tab to switch back and forth, which is how I entered the c# sample below, although it look horrible in the rich editor.) There are some interesting fields under the options menu, including a text area to enter a stylesheet, but no indication of what it does. I may need to dig into the help docs later, but I'll probably look into emacs extensions first. For now, the jury is out. /// /// Sample c# /// public static int Foo(int x){ return x * x; }

Ah, when posting, I discovered how the draft works. There is a checkbox marked "draft" which publishes content as a draft, rather than to the site. It would be much clearer to just have two buttons: "Publish" and "Save as draft", it would take up the same amount of space, and require fewer clicks, but I suppose you could accidentally publish to the main site when you really wanted to click on draft. In any case, it ruined the formatting on the code snippet, so for my purposes it will not work.

Home is where .emacs is...

2006-08-21T10:08:00.000-04:00

I finally found a way to quickly navigate to my home directory in windows today. Being a long-time (well, 8-10 years) Linux user, when I started working in windows 40 hours a week, I had to adapt a bit, but most of that adaptation meant installing cygwin utilities, bblean, and emacs. Unfortunately, it's most natural to put all the configuration files for these apps in my home directory -- a concept that is almost completely alien to a windows system. As you probably know, in Win XP you do have a home dir, conveniently located at c:Documents and Settingsusername which is not extremely easy to navigate to, and there are no simple shortcuts to get there as there are in Linux / unix environments.

This has been annoying me for some time. Everything goes into that directory -- I've moved firefox downloads, Visual Studio projects, symlinked My Documents to there, etc... Now before you shout out that I brought this upon my self, consider that:

I needed everything to be in the same location, for what should be obvious reasons. (but if the reason's arent obvious, try backing up all your crap from a multi-year old windows machine where you didn't keep everything in one place)

More things were already using this location than were not (remember, most of what I use is cygwin).

Cygwin makes it mildly painful to access things that don't fit nicely into the cygwin virtual filesystem thing.

Today, however, I found a solution. Oddly enough, the integration between IE and Windows is why this works.

Open the Windows Explorer

Select View-Toolbars-customize

Add the "Home" button to the toolbar

Click on it. (This changes the Explorer "mode" to web-browser, and allows access to configure that button.)

Select Tools-Internet Options

Set your IE home page to c:Documents and Settingsusername

Voila! Click on the home button again, and there you are :)

foreach(What?)

2006-08-14T16:08:00.000-04:00

I've been adapting to Java 1.5 and c# (and c#, with .NET 2.0), all of which feature a new construct: the foreach loop. The syntax is relatively similar to the construct in numerous other languages, such as perl's:


foreach $var (@list){
    # ...
}

In c# (or java, the syntax is the same), to loop over an array or IEnumerable collection, you just do:


foreach(int x in indices){
   // ....
}

Unfortunately, as I found today, the c# version shares something else with the Perl version: An absence of type checking. Despite the explicit types in the loop construct, and the presence of generics (although these are missing in .NET foreach in c# is not type safe. The end result is that you can do fun things like this with out the slightest hint of a warning:


double[] values = new double[]{....};
/* snip */
foreach(int v in values){
   // guess what? v == Math.Floor(values[i]) !
}

I find this particularly vexing because if you simply re-write the foreach to be a standard for loop, then the type checker will jump in and complain about the loss of precision:


for(int i=0; i 

With collections this becomes slightly more complex, and in .NET pre-2.0, Enumerators were not typed (they would simply return Objects, making it impossible to perform compile time type checking), with .NET 2.0, however these constructs could be parametrized, eg:


foreach(T id in IEnumerable){
   // body
}

Why this isn't done is a mystery to me. At the very least a rewrite rule could be used to turn each foreach into an equivalent for loop. (There are plenty of ways around any performance problems this would cause at run time, and I'm happy to pay a compile-time penalty if it prevents run-time bugs!)

Edit: Just for completeness, Java 1.5 does not exhibit this behavior, either when itterating over generic collections or arrays of basic data types:


double[] values = new double[]{0.5, 0.6, 0.7};

for (int v : values){ // causes compile time error.
   System.out.println("v="+v);
}

Note that it is actually an error in java, not a warning even. I would assume this has something to do with the requirements Sun placed on backwards compatability with older JREs -- much, if not all, of the java Generics changes are simply rewrite rules. C#, however, has no such constraints and can take advantage of a number of performance improvements that are not available to the java compiler. I still don't see that as an excuse for discarding type saftey. --ERC

Moving (moved?) the blog

2006-05-24T23:03:00.002-04:00

I've migrated over to a hosted blog at http://blog.ciscavate.org
Bitwise Evolution is no longer published on blogger.

Licensed to invent?

2006-05-24T14:05:00.000-04:00

My background in software development was built up on a strong belief that software should be freely available. This belief led me to start using GNU/Linux and various GPL'd projects as an undergrad at Oregon State University. Changing from the norm was painful - I had previously been on a window-only operating system diet, and while I could use Emacs, I was not yet zealous about it. Numerous times I tried to go back to Windows, but the 2-4 week reinstall cycle that I had convinced myself was required (and may have been... this was in the early Win 98 days) drove me back to open solutions.

I eventually came to love the ability to customize everything. I have always been a "power user", and I spent many hours crawling through the NT registry when I was in High school. Linux, with the countless window managers and themeing sites drew me in like WinHack never could. As I became more skilled in software development, I began customizing apps to my liking, hacking out bug fixes and adding features as I wanted them. (Find a new game that looks cool, but can't stand playing it with the keyboard? Add joystick support!) All of this was made possible, or at least heavily benefitted from, open source licenses (Particularly the GPL).

The intricacies of the various licenses were generally opaque to me, but I understood the gist of it:

The GPL let you release source code that was free but could not be made non-free by anyone but the author.

BSD-like licenses were less restrictive.

That was about all I knew, or cared. I was happy with the GPL, and I've scoffed at the complaint that Viral licenses are evil.
Recently, I started developing an application to assist with photograph management (back to the: customize everything point above... there are now to many applications to find exactly what I want, so I'm building my own--great solution, right?). I settled on Java for the programming language, and intended to use the SWT (released under the Eclipse Public License) as the widget toolkit, since it is a visually attractive cross-platform solution. However, I also want to make use of some GPL libraries. If you've read through the mentioned licenses, you probably see the problem already. It is not at all clear whether or not the SWT can be used with GPL applications. Azureus seems to be doing this, and releasing their code under the GPL. However, no one I've talked to can clearly state whether this is ok or not, and reference the relevant portions of legalese to support the claim.
All this confusion has meant that the only "safe" route to either not release my program at all (I consider this unacceptable), or to settle on one of the two licenses above and only use compatible code, which means the GPL, simply because there is more GPL'd content available than EPL'd content. The depressing end is that the viral nature of the GPL is the reason there was a conflict, because the SWT cannot be re-released under an SWT-incompatible license (and it is not clear that I can do this and meet the GPL requirements that all source code be released as well.).

I doubt that either the Gnu foundation or IBM would chase me down for voiding a license for my personal photo manager, but the open source movement relies on the copyleft licenses to stay afloat, to a certain degree. Without respecting those licenses we become hypocrits.

I wonder if OCaml would work...

2005 in cities

2006-01-27T11:01:00.000-05:00

Copying an idea seen on Join-The-Dots:

Portland, OR

Corvallis, OR

Seattle, WA

Los Angeles, CA

San Diego, CA

Twin Falls, ID

Jackson Hole, WY

Lincoln, NE

Chicago, IL

DeBois, PA

New York, NY

Toledo, OH

Ogden, UT

One or more nights in each city. Now, if only I had some ground-breaking reason for being in Ogden, UT and DeBois, PA :).

Server-centered IM (or: Yet Another Google Talk Post)

2005-08-26T13:09:00.000-04:00

another place-holder.

Putting Social Networks to work

2005-08-26T06:08:00.000-04:00

In the post on mapping the intern, I briefly discussed how having a map of interns across the US would be a cool thing. However, there are obviously some privacy concerns with that. In the comments, Tessa mentions using instant messengers as a conduit for personal info. The appearance of Google Talk completed the connection, and got me thinking.

Google has all the infrastructure to do exactly what I want, and more. Unfortunately, as the Google tools are now, everything is disconnected. It is as if someone needs a meta-20% project to tie all these cool 20% projects together. (Sidebar: Google engineers are given 20% of their time to work on any side project the want. The idea is that some of these ideas will flourish and turn into the next great thing. Orkut started this way.)

The particular Google aspects I'm thinking of are: Google Maps, Google Talk and / or Gmail, and Orkut. (gah, no more links from me. Blogspot's editor is either horrible with regard to adding links, or it uses some paradigm I'm completely unfamiliar with.--this is somewhat resolved by using WordPress now... but who knows if it's actually better.) Personal info could be added to any of the three "personal" services (by which I mean workout, small, and googol talk) and shared through some friend-flagging mechanism in Orkut. From there, it's trivial to build maps on a per-person basis. Google could probably do this now, but it would help to have all their tools tied together (things seem to be going in that direction, with the unified accounts between Gmail and Google talk.)

I consider the above a proof-of-concept for the usefulness of social networks (I've been somewhat skeptical about their employ for some time). Are there other similar uses? At some levels, social networks seem a bit like a collaborative filtering approach to finding interesting people, but what do you do from there? Personally, I would find it odd if someone three links detached on Orkut contacted me just to say "hi", however, there isn't enough information about me readily available on Orkut to start a more interesting conversation out of the blue.

Mapping the Intern

2005-08-17T06:08:00.000-04:00

For the last two summers I have been interning for IBM in Westchester county, NY. While here, I've met a very wide distribution of interesting people (distributed across many axes, one of which is geographic -- well, perhaps two of them ;). I think the community that we have formed is interesting for a number of reasons: One, as I have already mentioned, we are widely distributed across the globe; I think we are also likely to appear in each-other's lives again, irrespective of our personal ties and attempts. There is no accident that we are a community -- IBM already filtered us, and we all have a deeply rooted interest in technology. I fully expect (and hope!) to randomly run into interns from IBM in the years to come, simply because we walk similar lives in a relatively small global community of researchers. Everyone I have met has also proven to be extremely interesting, artistic and inventive.

The problem, then, is that we don't have an easy way to interact outside of IBM. Currently, we are only connected by an email list that doesn't serve as a very precise means of communication. As an example: Earlier this year, my partner and I drove across the United States, and we didn't have anything planned for the night we would be in Pennsylvania. Eventually, I sent an email to the interns list asking simply "is anyone in Pennsylvania? (or do you know if there is anything to do there?)". This just seemed crude, although it would have worked if I had tried it sooner. It would have been nice to have some idea of who may be there, and where about they were. This need not be accurate, the granularity of cities would be fine, it would still help with trip planning. (Apologies if you were expecting something profound, this is all I've got today.)

Today, out of the blue I realized how to do this, with the Google Maps API. If you aren't familiar with the API, it lets you use the same technology on http://maps.google.com but with your own markers or whatever you want (within reason). You also get the scrolling /
zooming and satellite imagery as well. I'd like to use this to map interns. The implementation would be pretty straightforward, but the
privacy / legal implications aren't clear-cut. Ideally, intern marks would have actual street addresses (and potentially other contact info as well), or at least mailing addresses associated with them. Common sense dictates that such a page should be private, and the Google terms of service supports this, after a manner:

"We also want to respect people's privacy, so the API should not be used to identify private information about private individuals."

An opt-in policy or click-through would probably remedy this, but I'm also not clear on whether Google is happy with a Google map being password-protected. The answers to this aren't very pressing -- I have a number of projects on my stack that need work first, but if anyone has input, I'd like to hear it :)

The US in Pictures

2005-08-15T05:08:00.000-04:00

I've posted photos from my drive across the US this spring, along with a host of other shots that I've been meaning to share for ages. I realized the other day that when driving through the midwest, despite being in awe of the scenery, and miles upon miles of picturesque fields, I never took any pictures. I do believe I was waiting for a high point to take them from, to get a full sweeping view of the surrounding area, but alas, I was in the Midwest.

http://www.minnen.org/rogan/photos/

Playing it by Ear...

2005-08-10T02:08:00.000-04:00

Those who know me well know that I am almost always listening to music of some sort. When I'm working you will usually find me with a set of headphones plugged into my ears, or speakers blasting (if that doesn't bother anyone else nearby). Similarly, I always have the car radio on when it doesn't interfere with conversation or intense driving, and I usually queue up some mp3s on the stereo before I start cooking dinner, cleaning, etc.

My taste in music is about as diverse as the situations above. I don't particularly enjoy country western, or rap / hip-hop, but pretty much everything else will show up on a playlist at some point. This is somewhat of a problem; while in general I will enjoy listening to anything, at any specific time I'd prefer to stay within a certain genre.

The issue is that to achieve this level of filtering I need to do it manually. With a couple thousand tracks (with much missing meta-data, such as the "official" genre) this becomes a huge task. Also, keep in mind that some artists cross a number of genres -- A Perfect Circle and Nine Inch Nails are two good examples. Some NIN tracks are nearly symphonic while others are very industrial, this prevents bagging by artist in enough instances to be annoying.

Recently a number of music management programs have cropped up that provide more filtering tools than a plain player such as winamp or xmms. iTunes is probably the best known of these programs -- and I used iTunes for nearly two years and the searching it provides is much better than what I used before, but it still doesn't do the level of filtering I want.

About a week ago I began using amarok. Amarok is similar to iTunes, but it makes use of a service called Audioscrobbler to provide suggestions on what to play. Audioscrobbler uses collaborative filtering techniques to recommend songs based on what you are listening to. I'll come back to this, but first a sidebar on collaborative filtering:

The idea behind collaborative filtering is that people have similar tastes in "things". If you like a given movie, say Star Wars: Episode 3, then you may like Star Wars: Episode 4, because many people who liked the first movie also liked the second. MovieLens (http://movielens.umn.edu/login) does exactly this. As you rate movies, it compares your preferences with those of thousands of other people, and predicts how you will rate movies you haven't seen. I've had good experiences with using MovieLens, but I put significantly more thought into choosing a movie to see than I put into choosing a song to listen to. This makes sense when you consider the time commitment -- 5 minutes spent choosing a song is wasted time, you could have just listened to it in that time. (Choosing an album to buy is a different scenario also, and seems to share many aspects with choosing a movie.)

Now, back to amarok and audioscrobbler. The problem here is that I want a specific genre, and I want it right now. This is in stark contrast to choosing a movie, which I could wait to see later if I'm not in the mood for that particular movie -- I can control the context in which I watch a movie, but I'm not willing to dedicate that effort to choosing a song. Music is a background task for me, and it shouldn't dictate very much attention. When I'm listening my attention has other targets: research, writing, driving, etc. However, I am not so involved in these tasks (usually) that I don't notice the background music.

Audioscrobbler actually does pretty good. (The parts of amarok that take advantage of it are another story...) There are, however three (or more?) problems with audioscrobbler: 1) you must have a network connection to get suggestions, 2) your meta-data on each song must be correct, and 3) if you have diverse input to the system, you will get diverse output. To explain #3: The input I provide to audioscrobbler is of very limited use. The range of music I listen to is to large to provide information on what music "sounds" alike -- Knowing that I like Bach and some tracks by Scooter (a rave artist) doesn't help when I'm listening to Back or Scooter, since I probably don't want Bach to pop on while I'm in a rave mood, or vice-versa. This has happened a number of times recently with amarok -- it suggested a set of U2, Dave Matthews, Tool, and one of the tracks I used to program my iTrip. (The iTrip is a FM transmitter for an iPod, and the tracks are 6 seconds long, consisting only of a couple beeps that represent a new frequency to broadcast on.)

The solution to this, I think, is to make recommendations based on the actual waveform of the music. This should be able to detect things such as tempo, average amplitude, the change in amplitude (and the frequency of that change...) and so on. It seems reasonable that this would provide a more accurate representation of what a song "sounds" like. I have done some minor experimentation with this idea in the past, and I had some success. It is something I'd like to get back to, but until then, I'll see if I can fix some amarok bugs...

Task Management

2005-08-05T20:08:00.000-04:00

One of the ever present things on my mental to-do list is to find, or create, a good task manager. A number of years back I became painfully aware that I can't remember everything I need to do unless I write it down. I've tried a number of things, although I haven't exhausted all the possibilities, by any means.

The first attempt at solving this was a perl cgi scripted todo manager based on sample code for a guestbook in _Learning_Perl_. It worked great :) but the whole experience was a little unfulfilling (you could never delete things ;( ). I tried a number of other things -- iCal, kalendar, and devtodo (a nice program, worth checking out if you're interested in a filesystem-based console tool). But recently I stumbled on an approach that hadn't occurred to me in the past.

I've started using a vector-drawing program called Inkscape to manage tasks at work. It started when I started using Inkscape to sketch portions of a software system I am implementing. Once the design phase was somewhat solidified and implementation began I started marking things off right on the document. Inkscape is handy for this because you can just create a new layer of the document and put transparent shapes on top of the portions that are done. I'll try to get a screenshot up soon, but I don't have a usable scenario right now).

The result is a very spatially-dependent interface that is not inherently linked to time. For some domains I think these are desirable features, but in many cases they are not -- I'm not sure if this is something that could be adapted to general task management or not, but so far I've found it to be very easy to use. It doesn't take many cognitive cycles to gauge your progress, which is the greatest immediate reward I've found for a task manager, and adding tasks is only a problem if you feel strongly about the location.

It is also trivial to add comments to individual or multiple tasks, or just as a sticky-note, just make them yellow :)

Distributed Communication

2005-08-04T05:08:00.000-04:00

Over the past year, the people I know well have become quite scattered across the nation, if not the globe. As a result, I'm left wanting to keep up to date with friends in Southern California, Oregon, New York, Boston, Nebraska, Britain (for a while), Seattle, and at the moment, a body of water in southwest-ish Alaska. I'm sure there are others, but you people are hard to keep tabs on!

This has led me to try blogging -- somehow there's a connection there that makes sense, but I doubt I can explain it modestly. Because of that, I'm going to claim this is a personal record for my own use.

And with that,

Good night!