Designing for Low-Latency Direct-Touch Input

Designing for Low-Latency Direct-Touch Input


Software designed for direct-touch interfaces often utilize a metaphor of direct physical manipulation of pseudo “real- world” objects. However, current touch systems typically take 50-200ms to update the display in response to a physical touch action. Utilizing a high performance touch demonstrator, subjects were able to experience touch latencies ranging from current levels down to about 1ms. Our tests show that users greatly prefer lower latencies, and noticeable improvement continued well below 10ms. This level of performance is difficult to achieve in commercial computing systems using current technologies. As an alternative, we propose a hybrid system that provides low-fidelity visual feedback immediately, followed by high-fidelity visuals at standard levels of latency.


In this paper, we have described sources of latency, and demonstrated how several of these can be eliminated in building a demonstrator system capable of 1ms touch latency. Further, we have described the results of tests which showed that users were able to perceive order-of magnitude improvements in latency over current-generation hardware. Our results suggest that performance beyond 1ms may still yield improvement that is perceptible to users.

We have constructed a prototype Accelerated Touch system, wherein a traditional direct-touch layer is paired with a low-latency layer that displays nearly immediate visual feedback on user interaction, independent of application logic, but visually tied to the underlying UI widget. We have further described the design of this visual language to satisfy the various constraints of a dedicated low-latency touch processor, and we have described a potential architecture for a direct-touch system that pairs Accelerated Touch with more traditional touch interaction.

A common complaint that we heard from people who used our system extensively was that it “broke” them – that they now find the latency of current generation devices completely unacceptable. The implication is that improving latency might be an effective competitive strategy for de- vice vendors. It is our hope that this paper will spark innovation in the design of hardware and software capable of lower latency of response to user input.

We see a wealth of future work in further investigating the limits of human perception of touch-screen computer systems, and better understanding the effect of performance parameters such as latency on the usage of touch-screens. For example, are there performance benefits for input under reduced latency? Further, we have conflated latency and frame rate, future devices may decouple these two parameters, and could optimize one or the other; further investigation is needed into the effects of such a change.

My thoughts

To gain increase in latency requires that all three elements in a system (the sensor, the software and the display) are addressed. Obviously only the software can be looked at in the case of user design as attempted to redesign capacitive screens and operating systems is way beyond the scope of my research. Interestingly this research does touch upon Card, Mackinlay and Robertsons earlier work that suggests 10Mhz and 100ms are the limits of acceptability. With advances in technology and with my focus of designing interfaces for use with audio applications then I would suggest that the 100ms limit be at least halved to be seen as acceptable. Definitely an area which requires more research and is of interest to me.

Patterns for the design of musical interaction with everyday mobile devices

Patterns for the design of musical interaction with everyday mobile devices


The growing popularity of mobile devices gave birth to a still emergent research field, called Mobile Music (music with mobile devices). Our particular research investigates such re-purposing of ordinary mobile devices for use in musical activities. In this paper we propose the use of patterns in the design of musical interaction with these devices. We introduce the musical interaction patterns that came out of our investigation so far, and describe the exploratory prototypes which served as inspiration and, at the same time, as test-bed for these proposed interaction patterns.


In our work we identified four musical interaction patterns that can be implemented in common mobile devices. This small, initial set of patterns obviously does not mean to be a thorough taxonomy of musical interaction in general. We are also still on the process of compiling other related pattern sets: for interactions made possible by musical ubiquitous computing environments (i.e., involving cooperation, emergence, location awareness, awareness of contextual sound/music resources, etc.) and for musical interfaces (which instantiate musical interaction patterns, possibly using existing UIDPs). Nevertheless, the four patterns listed here already account for musical interaction in ubiquitous environments when a single mobile device is the user interface, plus they suit designs that need to ensure that music can still be made with a mobile device even with no access to pervasive musical resources (in case those are not available or are unreachable, e.g., due to connectivity limitations).

We have also been conducting preliminary tests on patterns comprehensibility, to observe if the proposed patterns can be learned quickly by designers from outside the CM area. Some other tests are being made to confirm the independence of patterns in relation to different types of musical activities, e.g. by comparing user performance and quality of use when carrying out the same musical activity following two different interaction patterns. These tests and their results will be the subject of forthcoming papers.

A pattern-oriented approach for interaction design in mobile music is an effort towards a necessary switch from the current technology-oriented perspective to a more user-centred perspective of CM as a whole, and this paper is just a step towards this goal. However, much work is still needed in order to extend the scope of current CM research to cope with many well-known HCI concerns. We are convinced that a better understanding of HCI issues in CM research and development is a good starting point, not only to identify the capabilities and limitations of future work, but mainly to establish a common ground for discussing several interesting questions that are still open.

My thoughts

Main Argument

Interaction patterns should be developed to allow conceptual designs to be implemented on mobile devices in the field of mobile music. These patterns allow for metaphor construction better suited for musical interaction in the context of ubiquitous musical activities. An interaction pattern language can demonstrate how design problems may be solved according to sound user-centred design principles.

Secondary arguments

Interfaces for musical performance or interaction should be designed with human input centred design rather than traditional computer centred design. Interfaces for musical performance should fit to our physical, embodied natures, rather than only operating in the realm of symbolic processing.

Interim conclusion

The need exists to apply HCI methods for interaction design of these solutions or as a way for improving user experience.

Main conclusion

Four main patterns of musical interaction for musical interaction in ubiquitous environments when a single mobile device is the user interface have been established as taxonomy of musical interaction. Further work to observe patterns outside of these areas have been established as well as comparing user performance and quality of use when carrying out the same musical activity following two different interaction patterns.

My research aims to further Fores by establishing which situations specific interaction patterns can be used in when used as a human input centred design element (ergo the traditional mixer element of a midi setup).



Gaze-supported multi-modal interactions

Gaze-supported multi-modal interactions


While eye tracking is becoming more and more relevant as a promising input channel, diverse applications using gaze control in a more natural way are still rather limited. Though several researchers have indicated the particular high potential of gaze-based interaction for pointing tasks, often gaze- only approaches are investigated.


This paper presented a detailed description of a user-centred design process for gaze-supported interaction techniques for the exploration of large image collections. For this purpose, gaze input was combined with additional in- put modalities: (1) a keyboard and (2) a mobile tilt-enabled multi-touch screen. The integration of user feedback at such an early stage of the design process allowed for the development of novel and more natural gaze-supported interaction techniques. While gaze acted as a pointing modality, the touch and tilt actions complemented the interaction for a multifaceted interaction. Based on user-elicited interaction techniques we developed an extended multimedia retrieval system, Gaze Galaxy, that can be controlled via gaze and touch-and-tilt input to explore large image collections. First user impressions on the implemented interaction techniques

were gathered and discussed. Results indicate that gaze in- put may serve as a natural input channel as long as certain design considerations are taken into account. First, gaze data is inherently inaccurate and thus interaction should not rely on precise positions. Using the gaze positions for setting a fish-eye lens and zooming in at the point-of-regard were described as intuitive. Secondly, users should be able to confirm actions with additional explicit commands to pre- vent unintentional actions.

My thoughts

This research could be easily adapted for use in other media applications and situations. Browsing and selecting sound files in a DAW; editing parameters in a DAW; selecting from a list of users or contacts and many other possibilities. Infancy technologies like Gaze will require lots of research and development to bring to market (so to speak) but high risk brings high reward in such cases. This is definitely an area of research that interests me and I will be actively looking at some demo’s or applications that can be brought to a small control group.

New Website

Welcome to the new Audioedge website. This is the place I will post stuff relating to the things I find interesting in life, namely decent dance music, interesting mountain bike trails and interface design. It’s also where I will upload any images too, as I would prefer to keep control over that sort of thing. I’m still on facebook, as you can see on my contact page, but it’s more for direct messaging people which is handy. Anyway enjoy the website!


Add spacers to your Mac OS X Dock

Found this handy hint somewhere on the net

To add a Blank Icon (Space) to the dock in Mac OS X, open up and enter this. Enter it as many times as you want spaces. Error on too many.

defaults write persistent-apps -array-add '{"tile-type"="spacer-tile";}'
killall Dock

To remove them, just drag them up and out of the Dock like any other icon. To move them around, click and drag.

Marin Trail – Llanwrst

Marin Trail – 25km – Llanwrst

Heres my review of it in 3 words. Climbs. Rocks. Views.

Getting there: from Chester, take the A55 head to Abergele, take the A580 then to Llanwrst. Takes about 55 mins, lovely drive

The Marin Trail

Reading reviews about this trail it seemed to be a perfect one for me. Long enough to test you, but short enough not to push you over the edge. The climbs would be rewarded with stunning views and fast downhill sections, it said. Wrong. Two out of three aint bad someone once said but missing out on fast downhill really irks me. The start of the trail is right from the car park (which is free to park and eerily quiet) and takes you deep into the forest and straight into some epic climbing with really technical rock sections (remember the word rocks).

Start of the trail

Each ‘section’ of the trail has its own climb, each climb varies in steepness and length. I would say there was 4 or 5 ‘sections’. Each starts with a ‘You are here’ style board, which is good because  you can see how much progress your making (but equally is bad because you see the upcoming climbs). I would say the first climb is one of the toughest and its pretty dull as you on a doubletrack, sort of road, in the forest. I was passed by one other rider on this initial climb but he said “Hello mate” and carried on past. The riders on this trail seemed alot more friendly than the Llandegla elites.

Just before the first big climb

Reaching the top of the first climb is rewarded with a lovely view over the mountain range, and a much needed blast of cool air as you emerge out of the forest. The white signpost ‘Marin’ is easy to spot (take note signposters of Delamere!) and upon following you are immediatley heading downards and dodging rocks and roots. This first downhill section is steep with multiple rock pools, meaning you have to sit right back on your bike, slow your descent and pick your path. This was my first encounter with one of the trails most annoying features. Each downhill section seems to have these speed sapping rocks. This first one was fun, because it was at a really steep angle, with some drop offs and I was just getting used to how to handle them. This section didnt last long, and before I knew it I had popped out of the forest and was back on some double track/road feature following round the mountain.

Road after first descent

This path ambles along for the next few miles before delivering a fast but boring descent into the next stage of climbing. This is where you hit the next ‘You are here’ map, which has a car park attached to it (the section not the map!). The climb here is fairly steep, possible the steepest one (I keep saying that!), and is very sapping as it winds its way up the mountain. The reward then is to come out literally on top of the mountain, with spectacular views and a descent in front of you. Again though we are thwarted from gaining any speed, or jumping opportunities, as the jagged rockpools are really challenging on this bit. You have to pick your route carefully as its easy to hit a rock and fly over your bars. The amount of energy this section takes out of you is immense, and I had to stop just to get some breath back, and this is on a descent!

Coming out of the bottom bit of this second descent you are rewarded with some berms and a faster downhill section but this only lasts seconds before you see a sign saying “Beware of cars” and your back on the road moving towards your next climbing section.

Using the timer on my phone to bad effect

Heading towards the third ‘You are here sign’ takes you onto an actual road and you pass by several farm houses as it takes you out of the forest. This section is quite a fast downhill allowing you to get some nice speed up, albeit on a public road. This speed takes you up back into the forest, and as you see the next white Marin trail marker, you only have to peddle a small amount to get back on the trail and into the familiar rocks. This section is quite level, with a few climbing sections, so the rockpools are even more difficult to traverse as you have to pedal over them. Picking your path is essential here as its easy to bottom out your pedals. Really you start getting a bit bored of picking your way through rocks at this point, and this section leads you back out onto a dirt road – heading up another steep climb. Eventually it points you back into the forest, more rocks, this time going uphill. Very demanding. Then you hit a plateau with a steep descent and no rocks! Unreal, obviously at this point I was very wary of hitting top speed on a downhill bit due to the rocks, so I cautiously went through this section. The bottom of this section you go over a bridge and hit a rock wall.

Marin rockwall

Really not impressed with this section. What are you meant to do here? Seriously I think pro mountain bikers would struggle here. This is after some of the toughest climbing, and your more than half way round by now. Anyway it was carry-the-bike time as this rock section stretched out, up and along for a good 100m or so. Eventually you can get back on your bike, and through a singletrack in the forest of rooty, rocky stuff. This then leads to another downhill bit, which also has a warning that people may be crossing, and you know your about to pop out onto a road again. After popping out here I was greeted with a crew of about 6 cockney riders all with maps out looking rather lost. “Excuse me mate is this the right way”, erm, “Its my first time round lads, you starting it or finishing it” I asked. They replied “Starting it mate”. “Ah well its a circle just follow the signs” I said as I rode off. They also muttered how stupid that rockwall section was, and I also mentioned there were no fast downhill parts and I could sense enthusiasm dropping.

This section then leads you back onto some doublepath/road which by now had alot of walkers on it and I wondered if I had missed a sign, but I hadn’t. That was pretty much that, and as the trail ends it gives you some more rocky rooty parts but it kind of fizzles out. The only other thing is because its a loop I missed the part where I should have come off and ended up doing the first tricky rock section again (which was good because I did it much better second time round). I then bumped into a load of miners (random!), and asked them the way to the car park as I didn’t fancy going round twice. They said turn round and just go down the hill. This was a super fast descent but on a gravel road, so no chance for jumps etc. And that was that, back to the car park and the car. I’d classify this trail as one for the fitness freaks, with plenty of climbing and even the downhill parts are sapping. Theres loads of singletrack and rockgardens, and to be honest I was sick of rocks by the end. Not a trail I will be rushing back too, but one I will revisit again just for those views.

Interface Design, Technology and Mountain Bike, Tunes: News, Reviews and Reports