Just to document how I do this:
*Problem:* I need a schema for FooTask
*Solution:*
* Create a 'tests' directory.
* populate said directory with simple example xml files.
* Name those files `valid-foo.xml` or `invalid-bar.xml` (I use numbers for foo and bar).
* Create your xsd file in the same directory as 'tests'. Lets call it `foo.xsd`
* Copy the following Makefile into the same location.
[cc lang="bash"]
XSD=foo.xsd
# run xmlstarlet with -e to see verbose error.
test:
@for file in `ls -1 tests/valid*.xml`; do if xmlstarlet val -q --xsd ${XSD} $${file}; then echo "pass"; else echo "fail: $${file}"; fi; done
@for file in `ls -1 tests/invalid*.xml`; do if ! xmlstarlet val -q --xsd ${XSD} $${file}; then echo "pass"; else echo "fail: $${file}"; fi; done
[/cc]
Now, run make, and if anything fails you can manually run `xmlstarlet val -e --xsd foo.xsh [failing file.xml]` to see the details.
Saturday, October 17, 2009
Thursday, July 16, 2009
Implications: Coding for Homomorphicly Encrypted Input
Craig Gentry (Stanford / IBM) recently published a paper that proves the existence of fully homomorphic crypto systems. This has caused quite a stir, since such a system would allow an untrusted party to perform computations on encrypted data, returning an encrypted result, without ever knowing anything about the input or output. I'm not going to explain what homomorphic encryption is (at least, not in great detail). Bruce Schneier's blog has a great post about it, and the comments there are extremely helpful in understanding how it works, and what this means for cloud computing.
I make no claims to being a cryptographer, but I did have a number of questions about the practical viability of this approach. Now, there are many questions in that vein that are directed at the performance characteristics of Gentry's approach (which are abysmal, but not asymptotically so). I was curious about The use of side effects to discern information about the encrypted content.
For example, anyone who has used a debugger knows that you can monitor the flow of a program that has been instrumented with debugging symbols, and you can learn a great deal about the input even without examining the content of variables. If a given conditional branch directs execution one way, then you know the predicate evaluated to a specific value. I set out to determine why this sort of attack is not a problem, and I ended up learning a lot about the way programs that run on encrypted data must operate.
Let's take a moment to quickly discuss homomorphisms, and homomorphic encryption.
In this case (encryption) the homomorphism is a mapping from the clear text and the cypher-text. Fully homomorphic encryption, as Gentry discovered, preserves addition and multiplication--meaning that you can add and multiply cyphertext, and the result can be decrypted to reveal clear text that has been added and multiplied in the same way. Addition and Multiplication provide the operations necessary to implement boolean logic, and therefore, are sufficient to program very complex transformations (I'm not certain that it is safe to say "arbitrarily complex").
It's important to realize that every addition or multiplication operation results in a value that is encrypted. The running program can not know the intermediate results, and indeed it does not.
Edward Kmett posted the conversion from if/then/else to addition/multiplication on Schneier's blog:
[cc lang="java"]
if (condition) {
return then_clause;
} else {
return else_clause;
}
[/cc]
becomes:
[cc lang="java"]
return condition * then_clause + (1-condition) * else_clause;
[/cc]
Here's a "real" example (it compiles, at least) using both approaches. This is just meant to be used for explanation -- compilers could easily do the translation from the code in is0_clear() to is0_enc(). I've written them out separately here so we can look at the generated bytecode.
[cc lang="java"]
public class Test {
public int is0_clear(int input) {
if (0==input){
return 2;
} else {
return 3;
}
}
public int is0_enc(int input) {
// I'm cheating a bit to keep this simple -- calculate
// the conditional to be either 0 or 1:
int cond = 0==input ? 1 : 0;
return cond * 2 + (1-cond) * 3;
}
}
[/cc]
And here's the bytecode (generated by sun-java-6, and output with javap -verbose).
[cc lang="asm"]
public int is0_clear(int);
Code:
Stack=2, Locals=2, Args_size=2
0: iconst_0
1: iload_1
2: if_icmpne 7 // Conditional Jump!!
5: iconst_2
6: ireturn // return a constant 2
7: iconst_3
8: ireturn // return a constant 3
public int is0_enc(int);
Code:
Stack=3, Locals=3, Args_size=2
0: iconst_0 // lines 0-9 here are for the "cheating" part
1: iload_1 // just ignore them -- the arithmetic to accomplish
2: if_icmpne 9 // the same thing is complex, and not important.
5: iconst_1
6: goto 10
9: iconst_0
10: istore_2 // note that there are no conditional jumps below here:
11: iload_2
12: iconst_2
13: imul
14: iconst_1
15: iload_2
16: isub
17: iconst_3
18: imul
19: iadd
20: ireturn // return the result of the calculated expression.
[/cc]
Since every operation results in an unknown value, no conditional branches can be taken! Every branch has to be evaluated, and the correct result of the 'correct' branch is selected by multiplying by a binary value, that is itself, encrypted! This means that things like run-time short-circuit evaluation are not possible, monitoring progam flow is meaningless, (possibly?) every input will result in the same run-time, and all side-effects will happen regardless of the input.
Going further down this rabbit hole, caching is impossible, and global state (if even posible) is likely to be extremely dangerous. I shudder to think of how Python's concept of scoping would interact with a compiler that generates code for homomorphicly encrypted input.
Aside from the pure overhead of dealing with encrypted data, and the "refreshing" required with Gentry's algorithm, I think that there are going to be some serious performance and development concerns once homomorphic encryption becomes a reality. The programming practices that are common in languages like Java and Python now are not likely to hold up. I expect that the APIs that enable operation on encrypted data will be based on total functions, and I have only begun to think about the implications for testing, code coverage, and quality assurance.
I make no claims to being a cryptographer, but I did have a number of questions about the practical viability of this approach. Now, there are many questions in that vein that are directed at the performance characteristics of Gentry's approach (which are abysmal, but not asymptotically so). I was curious about The use of side effects to discern information about the encrypted content.
For example, anyone who has used a debugger knows that you can monitor the flow of a program that has been instrumented with debugging symbols, and you can learn a great deal about the input even without examining the content of variables. If a given conditional branch directs execution one way, then you know the predicate evaluated to a specific value. I set out to determine why this sort of attack is not a problem, and I ended up learning a lot about the way programs that run on encrypted data must operate.
Homomorphisms
Let's take a moment to quickly discuss homomorphisms, and homomorphic encryption.
a homomorphism is a structure-preserving map between two algebraic structures --Wikipedia
In this case (encryption) the homomorphism is a mapping from the clear text and the cypher-text. Fully homomorphic encryption, as Gentry discovered, preserves addition and multiplication--meaning that you can add and multiply cyphertext, and the result can be decrypted to reveal clear text that has been added and multiplied in the same way. Addition and Multiplication provide the operations necessary to implement boolean logic, and therefore, are sufficient to program very complex transformations (I'm not certain that it is safe to say "arbitrarily complex").
It's important to realize that every addition or multiplication operation results in a value that is encrypted. The running program can not know the intermediate results, and indeed it does not.
So, how do conditionals work?
Edward Kmett posted the conversion from if/then/else to addition/multiplication on Schneier's blog:
[cc lang="java"]
if (condition) {
return then_clause;
} else {
return else_clause;
}
[/cc]
becomes:
[cc lang="java"]
return condition * then_clause + (1-condition) * else_clause;
[/cc]
Here's a "real" example (it compiles, at least) using both approaches. This is just meant to be used for explanation -- compilers could easily do the translation from the code in is0_clear() to is0_enc(). I've written them out separately here so we can look at the generated bytecode.
[cc lang="java"]
public class Test {
public int is0_clear(int input) {
if (0==input){
return 2;
} else {
return 3;
}
}
public int is0_enc(int input) {
// I'm cheating a bit to keep this simple -- calculate
// the conditional to be either 0 or 1:
int cond = 0==input ? 1 : 0;
return cond * 2 + (1-cond) * 3;
}
}
[/cc]
And here's the bytecode (generated by sun-java-6, and output with javap -verbose).
[cc lang="asm"]
public int is0_clear(int);
Code:
Stack=2, Locals=2, Args_size=2
0: iconst_0
1: iload_1
2: if_icmpne 7 // Conditional Jump!!
5: iconst_2
6: ireturn // return a constant 2
7: iconst_3
8: ireturn // return a constant 3
public int is0_enc(int);
Code:
Stack=3, Locals=3, Args_size=2
0: iconst_0 // lines 0-9 here are for the "cheating" part
1: iload_1 // just ignore them -- the arithmetic to accomplish
2: if_icmpne 9 // the same thing is complex, and not important.
5: iconst_1
6: goto 10
9: iconst_0
10: istore_2 // note that there are no conditional jumps below here:
11: iload_2
12: iconst_2
13: imul
14: iconst_1
15: iload_2
16: isub
17: iconst_3
18: imul
19: iadd
20: ireturn // return the result of the calculated expression.
[/cc]
Since every operation results in an unknown value, no conditional branches can be taken! Every branch has to be evaluated, and the correct result of the 'correct' branch is selected by multiplying by a binary value, that is itself, encrypted! This means that things like run-time short-circuit evaluation are not possible, monitoring progam flow is meaningless, (possibly?) every input will result in the same run-time, and all side-effects will happen regardless of the input.
Implications?
Going further down this rabbit hole, caching is impossible, and global state (if even posible) is likely to be extremely dangerous. I shudder to think of how Python's concept of scoping would interact with a compiler that generates code for homomorphicly encrypted input.
Aside from the pure overhead of dealing with encrypted data, and the "refreshing" required with Gentry's algorithm, I think that there are going to be some serious performance and development concerns once homomorphic encryption becomes a reality. The programming practices that are common in languages like Java and Python now are not likely to hold up. I expect that the APIs that enable operation on encrypted data will be based on total functions, and I have only begun to think about the implications for testing, code coverage, and quality assurance.
Friday, June 19, 2009
Cleaning up my browser.
I'm done with firefox -- Opera 10 now plays flash well, has adblock via. urifilters, a cleaner UI (no menubar, a menu *button*!) vertical tabs are supported natively, etc... I don't really like the widget toolkit used in the file open/save dialog, but that's *much* better than the horrid performance/stability/bizarre bugs of Firefox.
The minimal UI possible with Opera is also a major win in my book.
Saturday, June 06, 2009
Maven deployment issues
I've been building / porting various projects to maven lately, and pushing them to our in-house maven server. For a while, I was doing this from my laptop at home. However, at work, I'm pushing to localhost (it's a temporary thing while we determine if maven will actually work long-term.)
The following error had me stumped for a few days:
[cc lang="bash"]
[INFO] Error retrieving previous build number for artifact 'de.balokb:libreadline-java-i386-Linux-c23cxx6:jar': repository metadata for: 'snapshot de.balokb:libreadline-java-i386-Linux-c23cxx6:1.0-SNAPSHOT' could not be retrieved from repository: inhouse_snapshot due to an error: Exit code: 1 - Host key verification failed.
[/cc]
All the googling I did turned up people stumped with ssh public key problems, or users who had specified ssh: instead of extssh: ... etc. It was fairly quick to elleminate those issues, or so I thought. (`ssh localhost` right? No problem.)
I happened to look in more detail at my pom.xml:
[cc lang="xml"]
inhouse
Inhouse Internal Release Repository
scpexe://10.0.0.26/var/www/maven/inhouse
[/cc]
hm... `10.0.0.26` I wonder...
[cc]
$ ssh 10.0.0.26
The authenticity of host '10.0.0.26 (10.0.0.26)' can't be established.
RSA key fingerprint is a7:bf:36:4c:b9:c7:c2:f9:03:9a:3a:a7:4f:10:e5:ba.
Are you sure you want to continue connecting (yes/no)?
[/cc]
Ah ha! I clearly can't use a pom.xml that lists "localhost" in the server section -- I'd only be able to push from one place. However, since I'd never ssh'd to `10.0.0.26` from localhost, the fingerprint was unknown, and that was causing maven to error out with the problem I saw initially.
"Fingerprint ID failed" would have been a nicer error message, but I don'tk now that that is possible.
The following error had me stumped for a few days:
[cc lang="bash"]
[INFO] Error retrieving previous build number for artifact 'de.balokb:libreadline-java-i386-Linux-c23cxx6:jar': repository metadata for: 'snapshot de.balokb:libreadline-java-i386-Linux-c23cxx6:1.0-SNAPSHOT' could not be retrieved from repository: inhouse_snapshot due to an error: Exit code: 1 - Host key verification failed.
[/cc]
All the googling I did turned up people stumped with ssh public key problems, or users who had specified ssh: instead of extssh: ... etc. It was fairly quick to elleminate those issues, or so I thought. (`ssh localhost` right? No problem.)
I happened to look in more detail at my pom.xml:
[cc lang="xml"]
[/cc]
hm... `10.0.0.26` I wonder...
[cc]
$ ssh 10.0.0.26
The authenticity of host '10.0.0.26 (10.0.0.26)' can't be established.
RSA key fingerprint is a7:bf:36:4c:b9:c7:c2:f9:03:9a:3a:a7:4f:10:e5:ba.
Are you sure you want to continue connecting (yes/no)?
[/cc]
Ah ha! I clearly can't use a pom.xml that lists "localhost" in the server section -- I'd only be able to push from one place. However, since I'd never ssh'd to `10.0.0.26` from localhost, the fingerprint was unknown, and that was causing maven to error out with the problem I saw initially.
"Fingerprint ID failed" would have been a nicer error message, but I don'tk now that that is possible.
Saturday, May 23, 2009
Bitten by dependency management
I've started using Maven to manage my java projects, and overall I'm very happy with it. It seems to be more mature than ivy, with better documentation, and the vast majority of tasks that I need "just work" (just don't ask me about jni--that's another post).Today, (and yesterday, and a good portion of the night in-between) I ran into a nasty bug in a library that I didn't know my code depended on. It isn't particularly important *what* I was working on, but just for context: I needed to strip a lot of text content out of nodes in the complete wikipedia revision history dump, so I was using Sax to parse the xml stream, filter out the stuff I wanted filtered out, and save the stuff that, well, I wanted saved. Being that the input was all of wikipedia, there were a fair number of unicode characters in there. As it turns out, the 2.6.2 xercesImpl has some sort of bug that allows xml with certain characters to be read without throwing exceptions, but when you try to write the chars that were actually read, you end up trying to write characters that aren't valid in xml. Even if I'd known that in advance, my response would have been something like "ok, so what? I'm not using xercesImpl, and certainly not a version *that* old".
Well.
You see, in addition to using Maven, I've also been using the Google Collections and JSR305 libraries, so I just drop those `
Unfortunately, JSR305 1.3.8 depends on jaxen 1.1.1, which depends on xercesImpl 2.6.2 (jaxen also needs this dependency via xom 1.0, for what that's worth).
Because that dependency was already present in my build path (via `mvn eclipse:eclipse`) and in the generated jar (via `
[cc lang="bash"]
java.io.IOException: The character '?' is an invalid XML character
at org.apache.xml.serialize.BaseMarkupSerializer.characters(Unknown
Source)
at com.stottlerhenke.tools.wikiparse.ContentStripper.characters(ContentStripper.java:195)
at org.apache.xerces.parsers.AbstractSAXParser.characters(Unknown
Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at com.stottlerhenke.tools.wikiparse.ContentStripper.parse(ContentStripper.java:96)
at com.stottlerhenke.tools.wikiparse.ContentStripper.main(ContentStripper.java:379)
[/cc]
`
Many hours later I was able to find a sample of the actual input that was causing these problems, and I was able to reproduce the issue with an input slightly smaller than 2.8tb. Once that was done, I set out to make a minimal test case. Rather than bother with a new maven project, I just hacked it out in emacs (not using google collections, etc. because, clearly, I wanted it minimal). To my surprise, everything worked, and worked fantastically! But how? I didn't even supply an xml api on the classpath, yet it ran just fine!
In truth, I *did* supply an xml api -- xercesImpl.jar, and many other libraries -- via my environment's `$CLASSPATH`. (Figuring that out was another adventure, but I digress.) Once it became clear that I was indeed using a broken library it was simply a matter of explicitly specifying the dependency on a new version of xercesImpl, and rebuilding.
The moral?
Know your dependencies! This should come along with knowing your language's built-in APIs well. It wasn't clear to me that the SAX packages I was using were not part of the core java API, so it didn't strike me as odd that I didn't need to specify a classpath entry or a pom dependency before I could use sax.
If you suspect something strange, you can see the dependency tree in the generated html documentation you get when running `mvn site`.
Thursday, May 14, 2009
Fixing the key repeat in Ubuntu 9.04
I just upgraded my workstation to Jaunty (Ubuntu 9.04) and the key repeat delay and speed dropped to a frustrating level.
gnome-control-center can be used to fix this, but it requires that the gnome-settings-daemon be running, which forces it's opinions on many other aspects of my environment (I run Enlightenment dr17).
Poking around a bit, and help from #e on freenode, revealed that `xset` can be used to fix the key repeat settings.
[cc lang="bash"]
# Look at the current settings:
$ xset q
Keyboard Control:
auto repeat: on key click percent: 0 LED mask: 00000000
auto repeat delay: 660 repeat rate: 25
auto repeating keys: 00ffffffdffffbbf
fadfffefffedffff
9fffffffffffffff
fff7ffffffffffff
# lets speed things up a bit...
$ xset r rate 250 40
[/cc]
gnome-control-center can be used to fix this, but it requires that the gnome-settings-daemon be running, which forces it's opinions on many other aspects of my environment (I run Enlightenment dr17).
Poking around a bit, and help from #e on freenode, revealed that `xset` can be used to fix the key repeat settings.
[cc lang="bash"]
# Look at the current settings:
$ xset q
Keyboard Control:
auto repeat: on key click percent: 0 LED mask: 00000000
auto repeat delay: 660 repeat rate: 25
auto repeating keys: 00ffffffdffffbbf
fadfffefffedffff
9fffffffffffffff
fff7ffffffffffff
# lets speed things up a bit...
$ xset r rate 250 40
[/cc]
Monday, March 30, 2009
Things are a little messy...
I've had some minor upgrade issues with the blog lately, and I am only about halfway through updating everything. In the meantime, I'm afraid things will look a bit messy. (Syntax highlighting is currently broken, and there are probably other formatting / data issues as well. I think I have to restore the uploads directory, for one, so there probably won't be any images in the posts for a while.)
Wednesday, February 04, 2009
Like food?
I started writing for another blog this week:
Brewed, Bottled, Cultured and Sweetened is a blog about beer, coffee, wine, cheese, chocolate, etc... that I'm writing with an old friend from Dagit.o
Monday, January 26, 2009
The Linux Tablet: Wacom rotations - waking up on the wrong side
Update: updated the script with improved (functional) error output. added notes about xhost.
There is an annoying bug in the sequence of code that manages the wacom rotation / sleep / resume and stylus calibration right now. (Where "right now" is Ubuntu Intrepid, with the 0.8.2-1 wacom drivers.)
This is a document bug over at the ubuntu launchpad, and the poster there does a fine job of describing the intricacies of reproducing the bug, so I'll only give a brief explanation here to help get indexed.
If you rotate the screen any amount, even returning to the original rotation, and then sleep the machine, when it wakes up, the stylus will not be calibrated properly -- the cursor will be off to the side of the stylus point. It doesn't seem to matter how it was calibrated when the machine slept, nor does it matter what rotation you're in when you put the machine to sleep.
There is one straightforward workaround: When you wake the machine, run wacomcpl, click on stylus, click calibrate (the mouse should now be under the stylus again), and exit wacomcpl. This is incredibly cumbersome, but at least it's better than restarting X, which is what I have been doing.
Further inspection (based largely on the thread of comments on that launchpad bug) reveals that the problem is actually related to bad values for the TopX, TopY, BottomX and BottomY settings on the wacom devices after a resume. By resetting these to their proper values for the current rotation, we can reestablish the proper calibration. First off, we need to know the proper values, and the easiest way to get them is with `xsetwacom`:
[cc lang="bash"]
#!/bin/bash
# wacomSettings
echo "TopX=" `xsetwacom get stylus TopX`
echo "TopY=" `xsetwacom get stylus TopY`
echo "BottomX=" `xsetwacom get stylus BottomX`
echo "BottomY=" `xsetwacom get stylus BottomY`
[/cc]
Now, we'll run this for each rotation, and save the results. You should end up with something like the following:
[cc lang="bash"]
|rogue on bach |AC 70% | @ 00:02:26 ~|
$ xrotate 1 && wacomSettings
xrandr to left, xsetwacom to 2
TopX= -46
TopY= -3
BottomX= 18605
BottomY= 24518
|rogue on bach |AC 70% | @ 00:02:28 ~|
$ xrotate 2 && wacomSettings
xrandr to inverted, xsetwacom to 3
TopX= 58
TopY= -46
BottomX= 24579
BottomY= 18605
|rogue on bach |AC 70% | @ 00:02:35 ~|
$ xrotate 3 && wacomSettings
xrandr to right, xsetwacom to 1
TopX= -173
TopY= 58
BottomX= 18478
BottomY= 24579
|rogue on bach |AC 70% | @ 00:02:41 ~|
$ xrotate 0 && wacomSettings
xrandr to normal, xsetwacom to 0
TopX= -3
TopY= -173
BottomX= 24518
BottomY= 18478
[/cc]
(Note that my bash prompt looks like & command lines above are indented, and the output is left-aligned)
That gives us enough information to script the calibration when we resume. For example, when resuming to a "normal" rotation, I need to run:
[cc lang="bash"]
xsetwacom set stylus TopX -3
xsetwacom set stylus TopY -173
xsetwacom set stylus BottomX 24518
xsetwacom set stylus BottomY 18478
[/cc]
(Wrap that in a bash script and give it a shot!)
Here's the full script that gets the current orientation and then calibrates the common wacom devices:
[cc lang="bash"]
#!/bin/bash
#
# waCalibrate.sh: recalibrates the wacom stylus
#
# Author: Rogan Creswick
# License: just be nice
# Set LOG to something reasonable:
# (The file does not need to exist, but the *directory* does)
LOG=/home/rogue/calibration.out
XSETWACOM=/usr/local/bin/xsetwacom
#
# Calibrates the wacom devices {stylus, eraser, cursor} with the
# given offsets:
#
# Usage:
# calibrate
#
function calibrate {
${XSETWACOM} --display :0.0 set stylus TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus BottomY $4 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser BottomY $4 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor BottomY $4 >> ${LOG} 2>&1
}
function fixCalibration {
# get the current orientation:
ORIENTATION=`xrandr --verbose --query | grep " connected" | awk '{print $5}'`
echo "Orientation: ${ORIENTATION}" >> ${LOG}
case "${ORIENTATION}" in
normal)
calibrate -3 -173 24518 18478
;;
left)
calibrate -46 -3 18605 24518
;;
right)
calibrate -173 58 18478 24579
;;
inverted)
calibrate 58 -46 24579 18605
;;
*)
calibrate -3 -173 24518 18478
echo "ERROR!! unknown orientation! ${ORIENTATION}" >> ${LOG}
;;
esac
}
case "$1" in
resume|thaw)
date >> ${LOG}
fixCalibration
whoami >> ${LOG}
;;
*)
echo "not a resum|thaw event: $1" >> ${LOG}
;;
esac
[/cc]
Stick that in `/etc/pm/sleep.d/40wacomCalibrate` (or some similarly named file), make it executable by all (`chmod a+x /etc/pm/sleep.d/40wacomCalibrate`) and it should be run when the system resumes.
Update: I found that the logging of the old script didn't work, so I've updated the script to reflect that. There were also some problems with how I was testing the first script, and the actions I was taking didn't actually trigger the bug. (The bug seems to be quite state-dependent, and markovian assumption was wrong.) To get this to work, root will need to have access to the display that xsetwacom uses. The simplest way to do this is to add `xhost +` to you x startup. (I put it in my ~/.xsession just before `exec enlightenment-start`).
There is an annoying bug in the sequence of code that manages the wacom rotation / sleep / resume and stylus calibration right now. (Where "right now" is Ubuntu Intrepid, with the 0.8.2-1 wacom drivers.)
This is a document bug over at the ubuntu launchpad, and the poster there does a fine job of describing the intricacies of reproducing the bug, so I'll only give a brief explanation here to help get indexed.
If you rotate the screen any amount, even returning to the original rotation, and then sleep the machine, when it wakes up, the stylus will not be calibrated properly -- the cursor will be off to the side of the stylus point. It doesn't seem to matter how it was calibrated when the machine slept, nor does it matter what rotation you're in when you put the machine to sleep.
There is one straightforward workaround: When you wake the machine, run wacomcpl, click on stylus, click calibrate (the mouse should now be under the stylus again), and exit wacomcpl. This is incredibly cumbersome, but at least it's better than restarting X, which is what I have been doing.
Further inspection (based largely on the thread of comments on that launchpad bug) reveals that the problem is actually related to bad values for the TopX, TopY, BottomX and BottomY settings on the wacom devices after a resume. By resetting these to their proper values for the current rotation, we can reestablish the proper calibration. First off, we need to know the proper values, and the easiest way to get them is with `xsetwacom`:
[cc lang="bash"]
#!/bin/bash
# wacomSettings
echo "TopX=" `xsetwacom get stylus TopX`
echo "TopY=" `xsetwacom get stylus TopY`
echo "BottomX=" `xsetwacom get stylus BottomX`
echo "BottomY=" `xsetwacom get stylus BottomY`
[/cc]
Now, we'll run this for each rotation, and save the results. You should end up with something like the following:
[cc lang="bash"]
|rogue on bach |AC 70% | @ 00:02:26 ~|
$ xrotate 1 && wacomSettings
xrandr to left, xsetwacom to 2
TopX= -46
TopY= -3
BottomX= 18605
BottomY= 24518
|rogue on bach |AC 70% | @ 00:02:28 ~|
$ xrotate 2 && wacomSettings
xrandr to inverted, xsetwacom to 3
TopX= 58
TopY= -46
BottomX= 24579
BottomY= 18605
|rogue on bach |AC 70% | @ 00:02:35 ~|
$ xrotate 3 && wacomSettings
xrandr to right, xsetwacom to 1
TopX= -173
TopY= 58
BottomX= 18478
BottomY= 24579
|rogue on bach |AC 70% | @ 00:02:41 ~|
$ xrotate 0 && wacomSettings
xrandr to normal, xsetwacom to 0
TopX= -3
TopY= -173
BottomX= 24518
BottomY= 18478
[/cc]
(Note that my bash prompt looks like & command lines above are indented, and the output is left-aligned)
That gives us enough information to script the calibration when we resume. For example, when resuming to a "normal" rotation, I need to run:
[cc lang="bash"]
xsetwacom set stylus TopX -3
xsetwacom set stylus TopY -173
xsetwacom set stylus BottomX 24518
xsetwacom set stylus BottomY 18478
[/cc]
(Wrap that in a bash script and give it a shot!)
Here's the full script that gets the current orientation and then calibrates the common wacom devices:
[cc lang="bash"]
#!/bin/bash
#
# waCalibrate.sh: recalibrates the wacom stylus
#
# Author: Rogan Creswick
# License: just be nice
# Set LOG to something reasonable:
# (The file does not need to exist, but the *directory* does)
LOG=/home/rogue/calibration.out
XSETWACOM=/usr/local/bin/xsetwacom
#
# Calibrates the wacom devices {stylus, eraser, cursor} with the
# given offsets:
#
# Usage:
# calibrate
#
function calibrate {
${XSETWACOM} --display :0.0 set stylus TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set stylus BottomY $4 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set eraser BottomY $4 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor TopX $1 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor TopY $2 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor BottomX $3 >> ${LOG} 2>&1
${XSETWACOM} --display :0.0 set cursor BottomY $4 >> ${LOG} 2>&1
}
function fixCalibration {
# get the current orientation:
ORIENTATION=`xrandr --verbose --query | grep " connected" | awk '{print $5}'`
echo "Orientation: ${ORIENTATION}" >> ${LOG}
case "${ORIENTATION}" in
normal)
calibrate -3 -173 24518 18478
;;
left)
calibrate -46 -3 18605 24518
;;
right)
calibrate -173 58 18478 24579
;;
inverted)
calibrate 58 -46 24579 18605
;;
*)
calibrate -3 -173 24518 18478
echo "ERROR!! unknown orientation! ${ORIENTATION}" >> ${LOG}
;;
esac
}
case "$1" in
resume|thaw)
date >> ${LOG}
fixCalibration
whoami >> ${LOG}
;;
*)
echo "not a resum|thaw event: $1" >> ${LOG}
;;
esac
[/cc]
Stick that in `/etc/pm/sleep.d/40wacomCalibrate` (or some similarly named file), make it executable by all (`chmod a+x /etc/pm/sleep.d/40wacomCalibrate`) and it should be run when the system resumes.
Update: I found that the logging of the old script didn't work, so I've updated the script to reflect that. There were also some problems with how I was testing the first script, and the actions I was taking didn't actually trigger the bug. (The bug seems to be quite state-dependent, and markovian assumption was wrong.) To get this to work, root will need to have access to the display that xsetwacom uses. The simplest way to do this is to add `xhost +` to you x startup. (I put it in my ~/.xsession just before `exec enlightenment-start`).
Monday, January 12, 2009
The Linux Tablet: Wacom drivers
Ubuntu 8.10 configured most everything properly, as mentioned in the previous post in this series, but it did not result in a functional pen.
The tablet screen is a wacom digitizer with a pen that has two buttons (eraser and a finger button), and the tablet can differentiate between touching and hovering. The linux wacom driver & tools are necessary to get this all working. While I didn't find a single page with instructions that worked flawlessly, I was able to figure it out from a collection of links:
* The Linux Wacom project:
* http://linuxwacom.sourceforge.net/index.php/minihowto
* http://linuxwacom.sourceforge.net/index.php/howto/x11
* The Aliencam blog:
* http://blog.aliencam.net/articles/aliencams-customized-ubuntu-setup-guide/
First off, you will need the latest version of the linux Wacom driver (8.2.1 at the time of this writing). The driver versions seem to be tied to your kernel versions, so this is quite important. The wacom-tools package that comes with Ubuntu is not sufficient (in fact, you'll want to uninstall it if you have it already).
Once you have the wacom package downloaded, follow the directions for installing it in the howto (linked above). The wacom package uses a typical configure, make, make install process but there are a few kinks:
* configure (almost?) always succeeds, regardless of the dependencies you have yet to fill. The make step will simply not build all the things you need if this happens, but it won't fail visibly.
* You'll need to copy the kernel module into the /lib/modules/`uname -r`/kernel/drivers/usb/input/ directory manually (creating subdirs if necessary), *before* running make install. (This is outlined in the mini-howto.)
Once wacom is installed, you can begin working with the X.org configuration. This is fairly clearly explained at the aliencam blog linked above, or you can use my xorg.conf here.
[cc lang="bash"]
Section "Device"
Identifier "Configured Video Device"
EndSection
Section "Monitor"
Identifier "Configured Monitor"
EndSection
Section "Screen"
Identifier "Default Screen"
Monitor "Configured Monitor"
Device "Configured Video Device"
EndSection
#BEGIN TABLET SECTION
Section "InputDevice"
Driver "wacom"
Identifier "stylus"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "stylus"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
Option "Button2" "3"
EndSection
Section "InputDevice"
Driver "wacom"
Identifier "eraser"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "eraser"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
Option "Button3" "2"
EndSection
Section "InputDevice"
Driver "wacom"
Identifier "cursor"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "cursor"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
# Option "Mode" "Absolute"
EndSection
# This section is for the TabletPC that supports touch
#Section "InputDevice"
# Driver "wacom"
# Identifier "touch"
# Option "Device" "/dev/input/wacom" # USB ONLY
# Option "Type" "touch"
# Option "ForceDevice" "ISDV4" # Tablet PC ONLY
# Option "USB" "on" # USB ONLY
#EndSection
#END TABLET SECTION
Section "ServerLayout"
Identifier "Default Layout"
Screen "Default Screen"
# InputDevice "Synaptics Touchpad"
#added to get tablet working
InputDevice "stylus" "SendCoreEvents"
InputDevice "cursor" "SendCoreEvents"
InputDevice "eraser" "SendCoreEvents"
# InputDevice "touch" "SendCoreEvents"
EndSection
[/cc]
After doing that, you should be able to reboot and the pen should be working. You can do things like configure the buttons with `xsetwacom` (and you'll need that when it comes time to rotate the screen), but I kept getting this error when I tried to run `xsetwacom`:
[cc lang="bash"]
$ xsetwacom
xsetwacom: error while loading shared libraries: libwacomcfg.so.0: cannot open shared object file: no such file or directory.
[/cc]
I made a lucky guess, and fixed the problem with a quick ldconfig:
[cc lang="bash"]
$ sudo ldconfig # that was a lucky guess.
[/cc]
*Update:* There were some issues with the wacom calibration after a sleep/resume cycle *if* the laptop screen had been rotated during that prior wake cycle (this happens a *lot* more than it seems, given how complex that description is.) I've written up a workaround here.
The tablet screen is a wacom digitizer with a pen that has two buttons (eraser and a finger button), and the tablet can differentiate between touching and hovering. The linux wacom driver & tools are necessary to get this all working. While I didn't find a single page with instructions that worked flawlessly, I was able to figure it out from a collection of links:
* The Linux Wacom project:
* http://linuxwacom.sourceforge.net/index.php/minihowto
* http://linuxwacom.sourceforge.net/index.php/howto/x11
* The Aliencam blog:
* http://blog.aliencam.net/articles/aliencams-customized-ubuntu-setup-guide/
First off, you will need the latest version of the linux Wacom driver (8.2.1 at the time of this writing). The driver versions seem to be tied to your kernel versions, so this is quite important. The wacom-tools package that comes with Ubuntu is not sufficient (in fact, you'll want to uninstall it if you have it already).
Once you have the wacom package downloaded, follow the directions for installing it in the howto (linked above). The wacom package uses a typical configure, make, make install process but there are a few kinks:
* configure (almost?) always succeeds, regardless of the dependencies you have yet to fill. The make step will simply not build all the things you need if this happens, but it won't fail visibly.
* You'll need to copy the kernel module into the /lib/modules/`uname -r`/kernel/drivers/usb/input/ directory manually (creating subdirs if necessary), *before* running make install. (This is outlined in the mini-howto.)
Once wacom is installed, you can begin working with the X.org configuration. This is fairly clearly explained at the aliencam blog linked above, or you can use my xorg.conf here.
[cc lang="bash"]
Section "Device"
Identifier "Configured Video Device"
EndSection
Section "Monitor"
Identifier "Configured Monitor"
EndSection
Section "Screen"
Identifier "Default Screen"
Monitor "Configured Monitor"
Device "Configured Video Device"
EndSection
#BEGIN TABLET SECTION
Section "InputDevice"
Driver "wacom"
Identifier "stylus"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "stylus"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
Option "Button2" "3"
EndSection
Section "InputDevice"
Driver "wacom"
Identifier "eraser"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "eraser"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
Option "Button3" "2"
EndSection
Section "InputDevice"
Driver "wacom"
Identifier "cursor"
Option "Device" "/dev/ttyS0" # serial ONLY
Option "Type" "cursor"
Option "ForceDevice" "ISDV4" # Tablet PC ONLY
# Option "Mode" "Absolute"
EndSection
# This section is for the TabletPC that supports touch
#Section "InputDevice"
# Driver "wacom"
# Identifier "touch"
# Option "Device" "/dev/input/wacom" # USB ONLY
# Option "Type" "touch"
# Option "ForceDevice" "ISDV4" # Tablet PC ONLY
# Option "USB" "on" # USB ONLY
#EndSection
#END TABLET SECTION
Section "ServerLayout"
Identifier "Default Layout"
Screen "Default Screen"
# InputDevice "Synaptics Touchpad"
#added to get tablet working
InputDevice "stylus" "SendCoreEvents"
InputDevice "cursor" "SendCoreEvents"
InputDevice "eraser" "SendCoreEvents"
# InputDevice "touch" "SendCoreEvents"
EndSection
[/cc]
After doing that, you should be able to reboot and the pen should be working. You can do things like configure the buttons with `xsetwacom` (and you'll need that when it comes time to rotate the screen), but I kept getting this error when I tried to run `xsetwacom`:
[cc lang="bash"]
$ xsetwacom
xsetwacom: error while loading shared libraries: libwacomcfg.so.0: cannot open shared object file: no such file or directory.
[/cc]
I made a lucky guess, and fixed the problem with a quick ldconfig:
[cc lang="bash"]
$ sudo ldconfig # that was a lucky guess.
[/cc]
*Update:* There were some issues with the wacom calibration after a sleep/resume cycle *if* the laptop screen had been rotated during that prior wake cycle (this happens a *lot* more than it seems, given how complex that description is.) I've written up a workaround here.
Saturday, January 10, 2009
The path to a Linux Tablet
I finally broke down and bought a Lenovo X61 tablet (with SXGA+ screen!), and it arrived this week. This is the first of a series of posts about getting it up and running with Linux.
First off, some specs:
* Lenovo X61 Tablet PC with XSGA+ (1400x1050) screen (not multi-touch)
* 4 gigs of ram
* 200gb SATA hard disk
* WIFI (Intel PRO/Wireless 4965 AG or AGN)
* Gigabit Ethernet (Intel 82566MM)
* Bluetooth
* Wide-area networking card (3G)
* Fingerprint reader
* Integrated SD card reader (Richoh R5C822 SD/SDIO/MMC/MS/MSPro)
* Intel Audio (82801H ICH8 Family audio controller)
* 1 PCMCIA (type-I?) slot
* 4-cell battery
* Ultrabay Slim (which will be holding my Ultrabay CD-RW / DVD drive from my old T42p)
* Intel Mobile GM965/GL960 video controller. (256mb video ram?)
I'll flesh that list out more as I can find the details (eg: wireless chipset, video, etc..)
First off, I blew some time poking around in Vista of course :). The handwriting input app is phenomenal in a lot of ways. It works very well, training is well integrated, and it has worked with every input area I've tried. It *could* be better if it had contextual clues, and could tie into things like Eclipse's intellisense. Overall, though, it is amazing how simple it is to use, and how aesthetically pleasing the handwriting actually is. There is a lot to be said for using a couple extra pixels to make the strokes taper off as you pull the pen away. It has QWAN.
That done, I started to move on to installing Linux. I'm giving Ubuntu 8.10 the first chance, and I thought I'd try using a USB-based install so I wouldn't have to monkey around with the Ultrabase & drive. If you have an 8.10 system already, you can easily create a bootable usb ubuntu drive with `usb-creator` and an ubuntu iso. This takes perhaps 45min - 1 hour.
Booting was as simple as going into the ibm bios-like page (by hitting the ThinkVantage button on boot) and telling it to boot from another device, then selecting the usb drive (that I had already inserted). I split the existing 200gb partition in two with the ubuntu installer, keeping Vista in it's 100 gig sandbox, and leaving the remaining ~100 gigs for Ubuntu to partition further (which it did, as two partitions: one for / and one for swap. /dev/sda5 and /dev/sda6).
I do wish it had said how much space was being allocated to each of those partitions though. The installer didn't give any indication.
Installation from booting the installer from usb to booting into the installed system took right about 30min. I'm impressed :)
Out of the box:
* The screen is the proper res
* Wireless looks like it might be working (I have to AP to verify)
* Sound works
* the pen does *not*
* Screen rotation does not work
* closing the lid shuts off the screen, but does not put the laptop to sleep.
* This was easily fixed in the gnome power-management settings, and hey, it resumes too!
* putting the laptop in tablet mode seems to have no effect (at least it doesn't shut off the screen ;)
* Some power management is clearly working (screen dims when unplugged)
* bluetooth was detected, but I have to way to test it.
* Dual-booting seems to work just fine, although there are two entries for Vista in the grub menu, and the first boots into the Rescue and Recovery system. Vista also had to do a chkdsk, and reboot before it would load.
More information as I figure it out :)
First off, some specs:
* Lenovo X61 Tablet PC with XSGA+ (1400x1050) screen (not multi-touch)
* 4 gigs of ram
* 200gb SATA hard disk
* WIFI (Intel PRO/Wireless 4965 AG or AGN)
* Gigabit Ethernet (Intel 82566MM)
* Bluetooth
* Wide-area networking card (3G)
* Fingerprint reader
* Integrated SD card reader (Richoh R5C822 SD/SDIO/MMC/MS/MSPro)
* Intel Audio (82801H ICH8 Family audio controller)
* 1 PCMCIA (type-I?) slot
* 4-cell battery
* Ultrabay Slim (which will be holding my Ultrabay CD-RW / DVD drive from my old T42p)
* Intel Mobile GM965/GL960 video controller. (256mb video ram?)
I'll flesh that list out more as I can find the details (eg: wireless chipset, video, etc..)
First off, I blew some time poking around in Vista of course :). The handwriting input app is phenomenal in a lot of ways. It works very well, training is well integrated, and it has worked with every input area I've tried. It *could* be better if it had contextual clues, and could tie into things like Eclipse's intellisense. Overall, though, it is amazing how simple it is to use, and how aesthetically pleasing the handwriting actually is. There is a lot to be said for using a couple extra pixels to make the strokes taper off as you pull the pen away. It has QWAN.
That done, I started to move on to installing Linux. I'm giving Ubuntu 8.10 the first chance, and I thought I'd try using a USB-based install so I wouldn't have to monkey around with the Ultrabase & drive. If you have an 8.10 system already, you can easily create a bootable usb ubuntu drive with `usb-creator` and an ubuntu iso. This takes perhaps 45min - 1 hour.
Booting was as simple as going into the ibm bios-like page (by hitting the ThinkVantage button on boot) and telling it to boot from another device, then selecting the usb drive (that I had already inserted). I split the existing 200gb partition in two with the ubuntu installer, keeping Vista in it's 100 gig sandbox, and leaving the remaining ~100 gigs for Ubuntu to partition further (which it did, as two partitions: one for / and one for swap. /dev/sda5 and /dev/sda6).
I do wish it had said how much space was being allocated to each of those partitions though. The installer didn't give any indication.
Installation from booting the installer from usb to booting into the installed system took right about 30min. I'm impressed :)
Out of the box:
* The screen is the proper res
* Wireless looks like it might be working (I have to AP to verify)
* Sound works
* the pen does *not*
* Screen rotation does not work
* closing the lid shuts off the screen, but does not put the laptop to sleep.
* This was easily fixed in the gnome power-management settings, and hey, it resumes too!
* putting the laptop in tablet mode seems to have no effect (at least it doesn't shut off the screen ;)
* Some power management is clearly working (screen dims when unplugged)
* bluetooth was detected, but I have to way to test it.
* Dual-booting seems to work just fine, although there are two entries for Vista in the grub menu, and the first boots into the Rescue and Recovery system. Vista also had to do a chkdsk, and reboot before it would load.
More information as I figure it out :)
Subscribe to:
Posts (Atom)