Leap Motion – A Gestural Controller For Music, Audio-Visual & Multimedia Performance?

Leap Motion
Share This:

How does Leap Motion work?

Leap Motion is a peripheral device for gestural control via a USB 2.0 connection. Leap Motion is small, measuring 13mm x 31mm x 77mm, and weighing approximately 45 grams.

The Leap Motion device uses a combination of two CCD cameras and three infrared LEDs. The cameras are utilised to track hand positioning, with the infrared LEDs used to calculate depth information. The device is powered by USB.

The Leap Motion device is designed to track the positioning of fingers, hands and other objects (such as pens, pointing devices etc.) within an 8 cubic foot space, to a distance of around 1m from the device.

Leap Motion is built to work with both OS X (10.7+) and Windows (7/8) operating systems. Ubuntu is also supported (12.04).

https://forums.leapmotion.com/showthread.php?420-The-unofficial-Leap-FAQ

Leap Motion is designed to work with applications that are downloaded through the Airspace Store. Apps are designed which offer a range of functionality, and offer both a replacement to using current technology for navigating a desktop or operating system, as well as apps which offer innovative ways to control and manipulate data in space.

Touchless For Mac is an application that allows control of the OS X desktop environment through the Leap Motion Controller. Using a ‘virtual trackpad in the air’, the app allows the user to navigate using pointing and clicking, scrolling and zooming techniques.

Roshambo is an game based on the game ‘Rock, Paper, Scissors’. This offers an innovative gameplay approach, allowing the user to use similar physical movements against the computer opponent as they would against a ‘real-world’ opponent.

How can Leap Motion work for music?

Available with the commercial launch of Leap Motion (July 22nd 2013), were three apps developed to allow MIDI control of software and external devices through the Leap Motion interface. Here is a summary of the functionality of these devices.

AirMIDI – AirMIDI is an application which allows the user to trigger chords and notes from within a scale through estimating the spacing of notes in the air. The scale can be tuned to major/minor and 7th chords. There is an option within the user interface to select from octaves, to trigger a sustain mode and to select keys.

Of most interest to an audiovisual performer is the ability to assign X, Y and Z axis parameters to DAW and synthesiser controls. This is achieved through assigning MIDI cc commands. AirMIDI is an application that runs in the background, which allows control of a limited number of parameters without interaction with the AirMIDI GUI.

AirMIDI is produced by Tenacious Frog, a Singapore based start-up company specialising in music apps. As of August 2013, AirMIDI is available to purchase for $3.59.

AeroMIDI – AeroMIDI is an application that allows the user / performer to manipulate 3D cubes in a virtual space. Up to 10 cubes can be manipulated, with 3 parameters assigned to each cube. This allows therefore up to 30 parameters to manipulated using the AeroMIDI interface.

Unlike AirMIDI, AeroMIDI is an application that relies on visual feedback from the application, in the shape of the 3D cubes. The cubes provide the actual GUI for performance use, which has implications for audio-visual performance. With a performers focus on the 3D cubes, there is limited opportunity to also be in control of either music or audio-visual applications within a performance environment.

AeroMIDI is produced by Acoustica, a US based software developer. As of August 2013, AirMIDI is available to purchase for $36.02

Geco MIDI – GecoMIDI is designed to monitor both hand movements in space. Through measuring open and closed hand movements, as well as detecting movements in space, up to 40 different parameters can be manipulated. These parameters can be assigned to any assignable MIDI control message on 16 channels. The software includes a virtual MIDI port within the OSX environment to allow MIDI messages to be sent across other applications that receive MIDI.

As of August 2013, Geco MIDI is available to purchase for $12.00

How effective is Leap Motion as a music controller?

The Leap Motion has been tested by researchers in Switzerland and has been found to be highly accurate. Leap Motion themselves claim an accuracy of 0.01, although this has not been proved.

It can be summarized that it was not possible to achieve the theoretical accuracy of 0.01 mm under real conditions but a high precision (an overall average accuracy of 0.7mm) with regard to gesture-based user interfaces. Comparable controllers in the same price range, e.g., the Microsoft Kinect, were not able to achieve this accuracy. It should be mentioned that the presented evaluation results were executed on the preliminary version of the Leap Motion controller and can differ from the accuracy and repeatability of the consumer product.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3690061/

As discussed in the research, the Leap Motion has a level of accuracy that is not matched by comparable technology such as the Microsoft Kinect. One could argue that as a full body recognition tool, the Kinect does not aim for such pin-point accuracy.

One needs to consider when using Leap Motion that pin-point accuracy does not necessarily mean an accurate performance. As a camera based device, there is the potential for individual fingers to get lost in the data tracking capabilities of the Leap Motion. Unlike a glove device which can track individual fingers with ease (dependent on hardware of course), there is the opportunity for data confusion.

Leap Motion apps for control of music in performance

AirMIDI is created and marketed as an alternative to using a keyboard for data entry, and can be configured for input of musical data based on scale patterns and chord shapes. Dependent on the type of music being produced, there may be some scope for exploring musical creation, although this device is not designed to be used primarily as a versatile musical controller. AirMIDI does have limited scope for control of devices through the ability to assign movements on the X, Y & Z axis to control change parameters, although the application is not designed with flexibility in mind.

AeroMIDI – As discussed previously, AeroMIDI is an application that relies on visual feedback from the application, in the shape of the 3D cubes. This process negates the opportunity to utilise the software for audio-visual controller use, unless a monitor is provided for user feedback, alongside whatever software interface is utilised for audio-visual software.

Geco MIDI – Geco MIDI, or GECO, is designed solely as a application for converting hand gestures to MIDI. Its core function is identifying multiple simultaneous gestures and assigning them to MIDI channel and control change data. This then allows GECO data to be used by any OS X MIDI capable application GECO creates a MIDI port that is recognised as an input within MIDI applications allowing the Leap Motion to be utilised as an input device within MIDI applications.

And here’s some more Leap Motion links…

http://howto.cnet.com/8301-11310_39-57595735-285/five-ways-to-set-up-and-optimize-the-leap-motion-controller/

http://www.pcworld.com/article/2044909/leap-motion-first-impressions-theres-work-to-do.html

Share This: