iOS4 Background Audio Revisited

The other day I went ahead and revisited streaming audio apps for iOS4. I blogged previously about how it is the biggest pain in the ass to get working….properly. And it still is.  Background buffering was a pain to do, but I finally worked out how to apply it to AudioStreamer (a github project by mattgallagher). The solution in this StackOverflow post is incredibly useful.

So, I decided I’d checkout AudioStreamer from git, take a look at other people’s forks, try to merge all the fixes and features then add my own. That’s what I did the other day.

My current github fork of AudioStreamer is here: http://github.com/DigitalDJ/AudioStreamer (and it’s pretty damn stable, just sayin’).

Basically, my fork cherry-picks some commits from other users, notably:

- Shoutcast metadata [jfricker]
- MIME type detection [andybee]
- HE-AACv2 [idevsoftware]
- Level Metering [idevsoftware]
- NSThread memory leak [mattgallagher]

Then I set out to try and fix the bugs I had encountered…and fix up the sample app so that it worked fully with iOS4 multitasking:

- Fixed interruption crashes (crashes when you use iPod app then re-use the sample app, or receive a phone call)
- Background buffering (!!! AudioStreamer now buffers properly in the background)
- Play/pause from iPod controls (The sample app now works properly with the iPod controls)
- Stop all UI updating and timers while backgrounded (The sample app now correctly follows guidelines to stop all UI updating and unnecessary timers to save battery life)

- Retina display example (The sample app now has a retina display example…just to be iPhone 4 compatible)

- Support for Pause button in UI (Pausing now works properly, even for streams that don’t support it)
- Local Notifications (outside app) on error (Notification popup if there is a lack of connectivity when playing music outside the application)

There’s just one more bug for me to squash, and frankly I can’t figure it out. Notifications are sent from AudioStreamer to the UI  in case of an error, the UI then handles the notification with a UIAlertView. Sometimes the alerts refuse to show up, or instead, multiple show up, or…..sometimes it just works as intended (i.e. one alert popup). I’m not sure if it’s an iOS bug or something in my code. It’s rather odd as it isn’t easily reproducible and often a reboot of the device will fix the issue. Very very odd.

, , ,

Flash for iPhone 4? Yes! Frash 0.02!

DISCLAIMER: The compiled binaries are based upon ALPHA code by COMEX. Code from the github repository can change minutely creating a significantly more stable version of code. DO NOT expect this compiled code to work on every Flash object or even be the slightest bit stable.

So, jailbreak for the iPhone 4 is out and comex seems to have gotten around to updating the Frash source on github for both iOS and iPhone 4. Hooray! I set out on a mission to compile it from scratch. Here I’ll detail the steps you need to produce to compile it and provide a pre-compiled binary for the less so technical (or lazy). This is essentially Frash version 0.02, but as noted before, the github repository can be updated at any time fixing any number of bugs.

As far as I know this will only work with armv7 devices, so: iPhone 3GS, iPod Touch 3G, iPad, iPhone 4. iOS4 / iPad 3.2.x only.

HOW TO INSTALL

1. Download the precompiled binary (Frash-0.02.deb | Mirror) and SFTP it to /tmp on your iPhone/iPad

2. SSH into your phone as root and type:

dpkg -i /tmp/Frash-0.02.deb

NOTE: If you had Frash 0.01 installed on your iPad, remove it first (look for it in Cydia packages). Also, I haven’t yet tested this on iPad, but it certainly works on iPhone 4 with iOS 4.0.1.

3. Reboot your device if you want, but you shouldn’t need to.

HOW TO COMPILE (ADVANCED USERS)

What you’ll need:

  • Mac OS X 10.6 with iPhone SDK 4.0 / 4.0.1 installed
  • git-osx-installer
  • libflashplayer.so from an Android 2.2 (Froyo) image (no link for this due to copyright issues, Google it!)
  • DebMaker-osx
  • ldid

Instructions:

1. Install git (just run the downloaded DMG) and check out the Frash git repository

git clone http://github.com/comex/frash.git
libflashplayer.so

2. Patch Xcode to compile Dynamic Libraries for iOS

a) Edit /Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Specifications/iPhoneOSProductTypes.xcspec

Make sure to place the code snippet in between the first “(” and the first “{“

    // Dynamic library
    {   Type = ProductType;
        Identifier = com.apple.product-type.library.dynamic;
        Class = PBXDynamicLibraryProductType;
        Name = "Dynamic Library";
        Description = "Dynamic library";
        IconNamePrefix = "TargetPlugin";
        DefaultTargetName = "Dynamic Library";
        DefaultBuildProperties = {
            FULL_PRODUCT_NAME = "$(EXECUTABLE_NAME)";
            MACH_O_TYPE = "mh_dylib";
            REZ_EXECUTABLE = YES;
            EXECUTABLE_SUFFIX = ".$(EXECUTABLE_EXTENSION)";
            EXECUTABLE_EXTENSION = "dylib";
            PUBLIC_HEADERS_FOLDER_PATH = "/usr/local/include";
            PRIVATE_HEADERS_FOLDER_PATH = "/usr/local/include";
            INSTALL_PATH = "/usr/local/lib";
            DYLIB_INSTALL_NAME_BASE = "$(INSTALL_PATH)";
            LD_DYLIB_INSTALL_NAME = "$(DYLIB_INSTALL_NAME_BASE:standardizepath)/$(EXECUTABLE_PATH)";
            DYLIB_COMPATIBILITY_VERSION = "1";
            DYLIB_CURRENT_VERSION = "1";
            FRAMEWORK_FLAG_PREFIX = "-framework";
            LIBRARY_FLAG_PREFIX = "-l";
            LIBRARY_FLAG_NOSPACE = YES;
            STRIP_STYLE = "debugging";
            GCC_INLINES_ARE_PRIVATE_EXTERN = YES;
            CODE_SIGNING_ALLOWED = YES;
        };
        PackageTypes = (
            com.apple.package-type.mach-o-dylib   // default
        );
    },

b) Edit /Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Specifications/iPhoneOSPackageTypes.xcspec

Make sure to place the code snippet in between the first “(” and the first “{“

    // Mach-O dynamic library
    {   Type = PackageType;
        Identifier = com.apple.package-type.mach-o-dylib;
        Name = "Mach-O Dynamic Library";
        Description = "Mach-O dynamic library";
        DefaultBuildSettings = {
            EXECUTABLE_PREFIX = "";
            EXECUTABLE_SUFFIX = "";
            EXECUTABLE_NAME = "$(EXECUTABLE_PREFIX)$(PRODUCT_NAME)$(EXECUTABLE_VARIANT_SUFFIX)$(EXECUTABLE_SUFFIX)";
            EXECUTABLE_PATH = "$(EXECUTABLE_NAME)";
        };
        ProductReference = {
            FileType = compiled.mach-o.dylib;
            Name = "$(EXECUTABLE_NAME)";
            IsLaunchable = NO;
        };
    },

3. Patch Xcode to allow compilation without a Provisioning Profile (i.e. not paying the $99 Apple tax)

a) Open /Developer/Platforms/iPhoneOS.platform/Info.plist and replace all instances of “XCiPhoneOSCodeSignContext” with “XCCodeSignContext”

4. Create a self-signed certificate to sign the Frash libraries

a) Open Applications > Utilities > Keychain Access

b) From the Keychain Access menu choose Certificate Assistant > Create a Certificate

c) Choose any name, set the type to “Self Signed Root”, check “Let me override defaults” and make it a “Code Signing” certificate

d) Specify any serial number, set the expiry to whatever you want. Longer than a year if you want to compile other iPhone libraries in the future.

e) Fill in any details for Certificate Information

f) Skip to the end of the wizard, using the default settings

5. Install ldid

a) Copy the ldid file to /usr/bin and make it executable (as root)

sudo cp ldid /usr/bin/
sudo chmod +x /usr/bin/ldid

6. Install DebMaker

a) Extract the DebMaker zip and extract dpkg-dep for later use

sudo cp DebMaker.app/Contents/Resources/dpkg-deb /usr/bin/
sudo chmod +x /usr/bin/dpkg-deb

7. Edit the default Xcode configuration for Frash

a) Open Player2/Player2.xcodeproj in Xcode

b) Make sure to switch the current configuration to Device 3.2 | Release

c) Open up “Get Info” for the Device 3.2 | Release configuration and  set the “Code Signing Identity” to the self-signed certificate made earlier and ensure the project is compiling for iPhone Device 3.2. Ignore warnings about setting to compile for armv6, keep it compiling for armv7.

d) Save the project and exit Xcode

8. Time to compile! Comex provides the build steps in the README for iPhone OS 3.2 but it needs to be changed slightly:

sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk /var/sdk
sudo export PATH=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:$PATH
sudo cp -a /System/Library/Frameworks/IOSurface.framework/Versions/A/Headers Player2/IOSurface
sudo cp -a /System/Library/Frameworks/IOSurface.framework/Versions/A/Headers food/IOSurface
sudo cp -a /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreText.framework/Versions/A/Headers food/CoreText
sudo cp -a /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk/System/Library/Frameworks/IOKit.framework/Versions/A/Headers/ food/IOKit
sudo cp -a /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk/System/Library/Frameworks/IOKit.framework/Versions/A/Headers/ Player2/IOKit
sudo cp /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk/Entitlements.plist .
sudo make -C rpc
sudo make -C utils
sudo make -C libgcc
sudo make -C food
sudo xcodebuild -alltargets -project Player2/Player2.xcodeproj

9. Now package it up into a deb

a) Edit DEBIAN/control and add a conflicts line to the end of the file. This will conflict with the Frash 0.01 deb released previously. You should remove that before installing this.

Conflicts: net.ozzapoo.frash

b) Place your copy of libflashplayer.so in the frash directory.

c) Modify fakeinstall to include libflashplayer.so by adding a line before “chown -R 501 iroot/var/mobile”

cp -a libflashplayer.so iroot/var/mobile/frash/

d) Make fakeinstall executable and run it as root. You should end up with a frash.deb file.

chmod +x fakeinstall
sudo ./fakeinstal
mv frash.deb Frash-0.02.deb

10. Hooray! Now follow the novice install section using the deb you just compiled (instead of the precompiled version, duh) to install Frash!

, , ,

iOS4 Multitasking: Developer’s Hell

I finally got around to installing iOS4, so I decided I’d go ahead and investigate what it takes to update the Fresh app (streaming radio) to support multitasking. Currently, my conclusion is that implementing ANY of the multitasking APIs is 110% hell unless you’re starting from scratch.

First of all, your application has to play nice while it’s sitting in the background. This means that you’re basically required to implement a whole lot of “if backgrounded” checks, unless you want to soak up the battery. This involves stopping unnecessary timers, updating the UI when the application comes back to the foreground (rather than per a specific event), and saving non-critical alerts for when the user pops back into the application. This basically requires a crapton of refactoring…and we all love how that introduces bugs. If you have any sort of complicated app, you’ll be having to manage state saving so that fast app switching works nicely. Oh, and not to mention that iOS never guarantees that your application stays backgrounded. It can be killed depending on resource usage, available memory or even just idle time. Every task (other than a few specific exceptions) is time limited, so you are constantly having to check and request more time for your task.

Now, this particular application is meant to take advantage of the background audio multitasking API. Now, as an aside, Apple does not provide any “simple” API to stream audio. What they provide is a primitive and generic audio buffer/queue that suits both local files and network streams. Basically, this means that if you want to stream audio over the network, you have to implement the protocol, manually send the requests to get data (via HTTP for example), then feed that data into a buffer (where you have to handle the size of incoming data vs. the size of the buffer), THEN…play the audio from the buffer or handle any errors that occurred along the way. Now, in any half decent modern API, surely, you’d have a perfected version of the algorithm where you could simply plug in a URL and instruct a magical black box to play it. But alas no, each developer is to implement the over complicated, primitive audio queue… just for a streaming radio application. Now combine that with the following multitasking API, and you’re in developer hell.

One of the official pages on backgrounding is nearly impossible to find on the Apple site, so here it is: http://developer.apple.com/iphone/library/documentation/iPhone/Conceptual/iPhoneOSProgrammingGuide/BackgroundExecution/BackgroundExecution.html Now, if you look closely, you’ll see that most sections are detailed nicely….except background audio, fantastic. So, taking a quick look at that section reveals that if you simply “flag” your application as one that streams audio, you’re good to go. Hooray right? Well, sort of, not really, no.

Hurdle 1: Hooking into iPod controls

Surprisingly, this is the EASIEST part of implementing background audio. It seems to “just work” as long as you have fairly simple start/stop/pause methods for your radio streamer. One gotcha is that state changes can now take place from both your play/stop button and the iPod controls… so, basically you should be using some sort of event system for state changes and to reupdate the UI whenever your app hits the foregound again.

Hurdle 2: Interruptions

Interruptions have existed since OS2. They cause your music to stop, usually when someone rings. However, now your application has to act more like the iPod application…except you can’t! Unlike the iPod application, which automatically restarts after you finish your call, the only way your application can automatically restart it’s stream is if the user brings it back to the foreground. Not only that, I’ve found that interruption events are very inconsistent. Sometimes they fire simultaneously with the interruption, sometimes they won’t fire until you bring the application to the foreground. On the times that they don’t fire with the interruption, I find that some sort of crazy auto deallocation of the streamer occurs which I can only fix with a terrible workaround. I am yet to find the actual cause of this.

Hurdle 3: iPhone Simulator

It sucks. Testing background audio with it is impossible! Even in the GM build, there is a bug that mutes all sound from your application when it is backgrounded. This makes it rather difficult to tell whether or not your application IS ACTUALLY SITTING IN THE BACKGROUND.

Hurdle 4: You’re free to do whatever you want…except when you need do something

So, as explained before, if you flag your application for streaming audio, your application can go ahead and play audio and do what it needs to do without any particular time limit or suspended state. Cool! Sort of like a free pass to do whatever the hell I want, right? Wrong. There’s a HUGE exception here: “if your application stops playing audio while in the background, your application is suspended.” Wow, that’s fantastic. So let me get this right…if my STREAMING AUDIO APPLICATION happens to need to BUFFER (i.e. STOP PLAYING AUDIO) while on a dodgy cellular connection, my application is essentially…fucked. Yes, that’s right. If you have a little bit of a network hiccup, and need to buffer, the user will need to jump back into the application to restart the audio stream! Not even the iPod controls (which are meant for external use) allow you restart the stream. I haven’t yet found a fix for this yet, but I’m assuming I’m going to have to go back to that mindfuck of a primitive audio queue to fill it with some sort of dummy audio while it buffers, so that iOS doesn’t fucking suspend my application in the middle of playing music. Doesn’t this found like fun?

Hurdle 5:Backwards Compatibility

Finally, as always, you need to aim for backwards compatibility. Apple claims to have completely culled OS2.x, but let’s face it, there are still 1st Generation iPod Touch users that DIDN’T pay the iOS3/4 upgrade tax, that want to use apps. You have to, at a minimum, ensure your application works on OS3 devices, which means making sure that some method calls, that are iOS4 specific, actually exist. Then using the simulator to test………. /facepalm.

FML.

, , ,

Creating a custom iOS4 IPSW with Game Center

iOS4 has GM’d, but Apple is yet to finalize their Game Center app. Currently, there are two different versions of iOS 4.0. Firstly, the publicly available version and an developer’s version. The developers version is identical to the official update, but includes the Game Center app for developers. Unfortunately, tools like redsn0w, sn0wbreeze and PwnageTool don’t support the developers IPSW. Fortunately, PwnageTool is incredibly customizable for patching firmware, so a few customizations makes it accept the developers IPSW.

1) Download and extract PwnageTool 4.01

2) View the package contents of PwnageTool.app and browse to PwnageTool.app/Contents/Resources/FirmwareBundles/<Your iPhone/iPod firmware version>.bundle

3) Within this bundle is an Info.plist, open it and make the following modifications to the following keys:

3a) RootFilesystem: needs to match the root FS dmg filename in the developer IPSW. Find this by opening the IPSW with a ZIP application and find the filename of the DMG that is approximately 300-400MB in size (it should be something like xxx-xxxx-xxx.dmg)

3b) RootFilesystemKey: each root FS is encrypted. Finding the key is difficult. Instead, search for the DMG filename in Google and you should find a few places that host the key for that particular DMG.

3c) RootFilesystemMountVolume: usually this is something like ApexXXXXX.iPhoneOS. Change this to ApexXXXXX.iPhoneDeveloperOS.

3d) SHA1: this needs to match the SHA1 of the developer IPSW. You can find this by using the command “openssl sha1 <IPSW filename>”

4) Now you can save the Info.plist, open up PwnageTool and select the developer IPSW. Hooray!

, ,