September 29, 2002
Debugging for the blind

Every hacker watching The Matrix would know this: While the greenish glyphs streaming down the screen in the hacker submarine look really cool they do not represent in any significant way the use of visual information when hacking.
The reason: Our perception of visual information is geared for an enormous ability to orchestrate information spatially and this is done at the cost of a very poor visual resolution for temporal information.
We all know from the cinema what the approximate maximal resolution of visual information is : Approx 24 Hz, the rate of display for standard film. If it were better, movies would not look to us like fluent motion.

Our shape recognition ability on the other hand is almost unlimited and the brain even has some amazing computing related tricks where we have very high spatial resolution in the focus area of vision, which comes at the expense of general sensitivity (amateurs guess : Sincy you need a certain number of photons for a difference over space to be present you need a higher level of lighting to realize good spatial resolution). Our peripheral vision on the other hand is extremely sensitive, but has less resolution.

So a better way to construct a new age visual hacking device would be to keep the complicated glyphs - which we can easily learn to recognize - for focal vision and add peripheral information that is important but only as background information that may require us to shift our attention.

An idea for debugging could by glyphs representing various levels of function from the highest to the lowest - all visible at the same time - and then use the peripheral information for auxiliary windows. In the case of a debugger you could have variable watches etc. in the peripheral view and they would only flicker if some unexpected value was met.

I think complex glyphs would be a workable model for representing aspect oriented programming. In linguistic terms we would be moving from the standard indo-european model of language form to some of the standard cases of completely different grammers (insert technical term here) where meanings that are entire sentences in indo-european languages are represented as complex words through a complicated system of prefixing, postfixing and inflection. Matrix-like complex glyphs would be good carriers for this model of language.

Aspect oriented programming is reminiscent of this way of thinking of meaning, in that you add other aspects of meaning and interpretation of programming as modifiers to the standard imperative flow of programming languages. Design By Contract is another case in point. Every direct complex statement has a prefix and a postfix of contract material.

What would still be missing from the debugging process would be some sense of purpose of the code. And that's where the temporal aspects of hacking that the glyph flows in The Matrix represent come into play. A group of scientists have experimented with turning code into music. The ear, in contrast to the eye, has excellent temporal resolution in particular for temporal patterns, i.e. music. That's a nice concept. You want your code to have a certain flow. You want nested parentheses for instance and that could easily be represented as notes on a scale. While you need to adopt coding conventions to absorb this visually, failure to return to the base of the scale would be very clear to a human listener.
In fact, while our visual senses can consume a lot more information than our aural senses, the aural senses are much more emotional and through that emotion - known to us everyday in e.g. musical tension, the aural senses can be much more goal oriented than the visual. This would be a beautiful vision for sound as a programming resource.

They should make some changes in The Matrix Reloaded. The perfect futurist hackers workbench would consist of a largish number of screens. The center screens would present relatively static, slowly changing, beautiful complex images representing the state of the computing system at present. The periphery would have images more resembling static noise, with specific color flares representing notable changes in state away from the immediate focus. I.e. changes that require us to shift our attention.
While working, this code-immersed hacker would listen to delicate code-induced electronica and the development and tension in the code/music would of course be the tension in the film as well, and this then would tie the emotions of the hacker as observer of The Matrix - i.e. the software world within the world of the film - neatly to the emotions of the moviegoer.

Posted by Claus at September 29, 2002 12:47 PM