Why Music Apps Will Sound Better Than Ever In Android Jelly Bean

Android devices may at last get the kind of sound performance that makes music and audio apps satisfying to use. We've suffered through generations of the OS and hardware that were quite the opposite. But material, measurable changes to software, combined with more rigorous standards for hardware makers, could change all of that soon. And using the free, cross-platform libpd library, you can be ready now to take advantage of what's coming.

The just-released developer preview of Android Jelly Bean is packed with refinements that make it faster and slicker for the everyman. Turns out musicians have a lot to gain too. Our friends at Create Digital Music compiled this comprehensive rundown of new features targeted at the audio-inclined.

If you're using an app that involves sound or music, the performance of the underlying OS and hardware will make a big difference. That's doubly true if you're using an app that simulates a musical instrument, because you're more likely to notice how responsive it is in real time. If the hardware, OS and apps add too much overhead, you can experience symptoms like clicks and pops — or, in order to prevent those glitches, app developers might have to create big sound buffers that make the apps less responsive. The challenge for the Android platform then is to eliminate that overhead and get sound from the app to your ears as directly as possible.

In brief, here's what's changing in Android 4.1 Jelly Bean and supported devices:

Low latency audio playback capability, via a new software mixer and other API improvements. Latency targets below 10ms. For now, low-latency playback (which would in turn impact round-trip latency) refers only to the Samsung Galaxy Nexus; other devices should get similar performance enhancements, but there's no official word yet on which or when. Strict maximum latency requirements for third-party vendors at some time in the future. Enhanced OS features too: USB audio device support, multichannel audio (including via HDMI), recording features and more.

But let's talk a bit about why this is important — in terms that are hopefully understandable whether you're a developer or not.

It's tough to argue with the importance of this if you know anything about what makes good sound experience in software. Human beings are on average gifted with the same common powers of perception and hearing. You don't have to be a "pro musician" to notice when sound and music apps aren't responsive. Untrained ears will respond immediately — and unforgivingly — to crackles, pops and delays in sound. (That last issue, described as "latency", is subtler but no less important — you tap on a screen and you expect to have an immediate response. Users often respond negatively to even minute increases of delay, whether consciously or not.)

So when we talk about "high performance audio", we really mean something for anyone using sound in apps. It's easy to understand why it's important. It's just hard to do from an engineering standpoint.

Raising the Bar, Lowering the Latency

"Latency" isn't one metric. Think of speed on a race car. Every little adjustment to the engine, gearing, weight and aerodynamics has an impact. These results are cumulative. You can't just get one factor right; you have to get all of them right. And that's what made Android's dismal audio performance complex to describe in the past. It has been a combination of factors — incomplete functionality in the developer API, a system mixer that added to latency, and device-specific issues being three major culprits.

Apple has done an excellent job with this on iOS, which contributes to their near-complete dominance of mobile for music apps, and justifiably so. But that should not be taken to mean that it's impossible to achieve low-latency audio performance when working with a variety of hardware vendors. The Windows (or Linux) PC is a great example — both of what works (extreme low latency across devices using an API like ASIO) and what doesn't (general-purpose mixers that drive latencies past a tenth of a second).

Based on what Android developers are saying, the platform is at last moving in the right direction. In fact, it's in stark contrast to what we currently know about Windows 8 on mobile. The very same issues I raised last week in my criticism of what's documented in Windows RT are the ones Android developers are at last addressing. Windows RT and the WinRT/Metro library for desktop and mobile, based on current information, not only lacks "nice-to-have" features like USB and multichannel audio on a new generation of Windows tablets, but also would seem to set unacceptable latency targets — 100ms+, or where Android was a year ago. We're hoping to find out that there's more we don't know, but we're awaiting more information from Microsoft. In review: Music Developer on Windows 8: A Leap Forward for Desktops; A Leap Backward for Metro, WinRT?

In that article on Windows 8, I talked about the importance of having the ability to route sound through a system mixer without other sounds pre-empting what you're doing. That's what WinRT and Windows RT each appear to be lacking (despite their name), and what Android may finally get. (It's what iOS, Windows, Linux and Mac OS X all have in some form.)

Back to the car metaphor, imagine you're in a drag race. Would you want to hit the accelerator to the floor on a nice, deserted street? Or would you like to do it in the middle of the on-ramp on the I-5 in downtown LA during rush hour?

Generic system mixers often look too much like the latter scenario. And this is what Android's development team explained in a Q&A at Google IO last week. Calling the project "a work in progress", they claimed a sub-10 ms playback target is their goal. ("Warm playback" I believe means once an audio engine is started and there's something to play in the sound buffer; someone else correct me if they have a different interpretation.)

As one Android developer puts it, regarding system mixers: "Once you get to that level, anything at all that's in the system that's pre-empting you becomes problematic."

Android developers are clearly making progress on the software. The tougher challenge is likely to be coordinating with hardware vendors. On the Samsung Galaxy Nexus handset — a device over which Google has more control — they've already improved latency from 100ms in "Ice Cream Sandwich" (4.0) to "about 12ms" in "Jelly Bean" (4.1), and want to go oven better. 12ms is usable; sub-10ms could really attract sound developers to the platform. Note: I'd like to hear more about the Nexus 7 tablet, but for now there has been no mention of sound performance on that hardware, which is made by Asus, not Samsung.

The "Fast Mixer" will work with SoundPool, ToneGenerator and OpenSL APIs. We're most excited about the OpenSL API, as we've already heard developers getting performance gains on previous OSes using that tool. (More on how to use it with the free libpd library below.)

The key variable here is when you'll actually see devices reaching these targets in the field. Unfortunately, that may be more of a waiting game. Google says they want to "get to a point" at which they're mandating maximum latency, but it's unclear when that will be.

Given the wildly variable experience on current devices, my guess is that developers targeting Android will be pretty tough on minimum requirements. If at least sub-15ms latencies become the norm on Jelly Bean devices, I could see making the 4.1 version of the OS a prerequisite, thus avoiding complaints from users when your app doesn't behave as expected.

Jelly Bean in general promises some encouraging improvements: at last, we see a focus on hardware accessories, high-quality audio, and high-quality animation and high framerates. These are some of the things that make iOS so much fun to use and so satisfying to users and developers alike. Pundits have rightly echoed Apple in saying that software isn't just about specs. But these aren't just empty specs: they're the concrete measurement of the qualities that people experience on a sensory level when using software.

More Audio Goodness

I'm actually equally enthusiastic about visual changes, but on the sound side, 4.1 delivers a lot of much-needed functionality.

Some of this is more consumer-oriented, but here's the full list:

USB Audio support, exposed through the Android Open Accessory Development Kit Multichannel audio over HDMI (Android-based surround sound installation, anyone?) Media codec access at the platform hardware and software level (with lots of cool low-level features) Audio record triggering Audio preprocessing Audio chaining, which means (among other things) support for seamless playback Media routing (a bit closer to what Apple does with AirPlay)

Out of the list, of course, what's interesting to us musicians is really HDMI multichannel and USB audio. I'll be looking into how USB audio is implemented, whether they do USB audio class support as on iOS and — just for kicks — whether USB MIDI support is possible.

All of this is described in the Jelly Bean overview, although developers will want to go ahead and grab the SDK for further detail.

OpenSL and libpd

For you developers out there, OpenSL is a big deal. (And users, just… trust us on this. You'll soon be experience better Android apps as a result.)

Peter Brinkmann is the principle developer of libpd. For those of you just joining us, libpd is the free and open-source embeddable version of Pure Data that now runs across desktop and mobile OSes. It actually is core, vanilla Pure Data, not a fork, but provides additional support around that core for making it easier to use Pd patches inside other apps. Peter has been working hard on OpenSL support — and has found reason to be excited about it coming.

Peter notes that OpenSL is working better than standard Java audio output across all test devices. You can use it with the libpd library, but other Java, C and Processing for Android developers should benefit too.

Oh yeah, and the design of the OpenSL branch should be useful on platforms other than Android too: Peter notes that he already has "a prototype using PortAudio that took about 15 minutes to implement and runs nicely on my Mac. Porting this to other platforms is just a matter of doing the grunt work of adjusting makefiles and such. In a nutshell, this promises to turn libpd into a viable stand-alone audio solution for Java, without necessarily needing to use something like JavaSound." (Sorry for quoting your email, Peter, but in the hopes other people might help the two of us Peters on this, I'm going to put this out there for everyone.)

Well worth reading; Peter has a series on libpd and OpenSL ES. It shows a bit of the future of Android audio development and also the future of cross-platform free software development for sound for Pd (and other platforms too) well beyond Android.

libpd and OpenSL ES, Part I: Squaring the circle libpd and OpenSL ES, Part II: Yet another JACK-like API for Android libpd and OpenSL ES, Part III: Receiving messages libpd and OpenSL ES, Part IV: Extending the API

And to take advantage of this on Android in libpd on Android:

I just pushed a new branch of pd-for-android that supports either AudioTrack/AudioRecord for FroYo or earlier, or OpenSL ES for Gingerbread or later. I managed to square the circle and make the entire transition more or less transparent to developers. If you limit yourself to the part of the API that I discuss in my book (i.e., everything but the low-level audio processing methods), then your apps won't need to be adjusted at all.

As it happens, the ability to switch between audio engines could be relevant in future when using JACK, Core Audio, ASIO and the like on other operating systems.

For everything you need for libpd, including Peter's superb book that will get you started on Android and iOS development, head to our minisite:

http://libpd.cc

And stay tuned. We'll keep bringing you audio and music news for users and developers alike, whatever the platform.

AMIGA, perhaps?

Republished with permission from Create Digital Music, a daily news and feature site for people making music with technology. Editor Peter Kirn is also an electronic musician and music developer, and has contributed to the libpd library.


Comments

    This is great news! Now if only someone finds a way for me to connect an interface with midi and balanced outputs to my transformer... oh and an app does something similiar to Native Instruments Kontakt and it could replace a laptop for use on stage as a sampler.

    By the time Jelly Bean goes mainstream Windows 8 will have been released and my interest in Android will have well and truly waned. W8 will no doubt burn a big hole in Android and IOS sooner or later. Hopefully I'll be able to update the ICS ROM in my tablet to JB eventually though, W8 won't run on that.

      I get your sentiment - I can see my next tablet upgrade being Win8 (and simultaneously being my next laptop upgrade) but I can't see WinPhone8 beating Android on a phone just yet.

    This article is about audio latency in Jelly Bean. Far out. Windows fanboy's on this site are worst that Apple fanboy and I though they where bad. Thanks for the informative article was wondering when Android would address audio latency as the only option for music creation on the go was iOS. WP7 has the same audio latency as ICS so will be interesting to se if WP8 has address audio latency.

    Good to hear!
    I used to listen to music all the time through my Shure SE530s on my iPhone 3GS. Sound was decent enough but not great (I miss the sound from my iRiver H340!).
    Sold the 3GS, bought a Samsung Galaxy S II. It sounded so underwhelming & weak & flat that I stopped listening to music. Batteyr drain was another issue. On top of that, I couldn't find a player app that I really liked, and I missed the system-wide integration that iOS had.
    Flagship phones shouldn't sacrifice decent DACs & amps for the sake of thinness!
    (am now using a dedicated MP3 player for music - SansaClip Zip w/ RockBox)

      The HTC desire had the same problem but HTC seem to have fixed that with the one x

        The original Galaxy and Nexus S both used a wolfson DAC. I think the Galaxy S3 does as well. All you need on those handsets for great audio is a Voodoo-enabled kernel and the free Voodoo Control app.

        I love the audio playback on my Nexus S.

    lol symbian already had multi-channel audio through hdmi. the n8 has dolby 5.1

    Why would any current Android phone user get excited about this? It's not like the phone you have now will ever be updated to Jelly Bean anyways. You want better music on your Android, then better start saving to upgrade to the latest and greatest handset, because that's the only way you'll get it.....

      why troll like that?

Join the discussion!

Trending Stories Right Now