Contribute  :  Advanced Search  :  Directory  :  Forum  :  FAQ's  :  My Downloads  :  Links  :  Polls  
AFP548 Changing the world one server at a time.
Welcome to AFP548
Thursday, July 07 2011 @ 03:27 am MDT

launchd in Depth

ArticlesWhen Apple announced the UNIX based Mac OS X lots of sysadmins took notice. For years the Linux crowd had been trying, and failing, to get UNIX to the general desktop audience. I ran Linux for a while on Beige G3 and it required a trip to Open Firmware and a bootconf loader on a floppy to get it going at the time. When I installed my first Rhapsody build, all I had to do was put the Caps Lock key on to get all the UNIXy goodness. The times, they were a changing.

Fast forward several years and Apple is now the highest volume UNIX vendor in the world. Furthermore the base of Mac OS X is open source and Apple uses many OSS projects in Darwin. Apple also has created projects and turned them out into the OSS world. (I'm well aware of the issues that some in the community have with Apple's OSS participation level, it's just not the focus of this article.) Recently Apple has loosed a new beast onto the world and it's knocked many people for a loop.

Welcome, launchd. Seriously.

Read on for more...

Apple's position as the leading UNIX vendor, and as a unified solution provider, gives them some unique opportunities. One of these has been to take a look at some of the traditional mishmashes of UNIX and try to sort them out a bit. Lookupd is one of these efforts and it provides general resolver services for everything from DNS to GID lookups on Mac OS X. Applications don't need to know anything about how the machine is configured, they just ask lookupd for the info. If you look at Mac OS X, and OpenStep before it, you will see that the OS has been striving for this sort of consolidation of services for a while now.

With Tiger, Apple has started to tackle the system of files and processes that start a UNIX system. Before it was often hard to tell how everything got cranked up. Was a daemon started by SystemStarter? Was it started by xinetd? Was it started by RC? Was it started by cron? Was it started by init or mach_init? Was it started by watchdog? Even worse, what if you needed to add a service? It was fairly clear that SystemStarter was what Apple wanted for startup items, but what about launch on demand? Xientd seems obvious here, but not everything can be launched by xinetd. What about service control? How do you easily start and stop services so that the rest of the system is aware of them?

In all honesty, this is a problem that Microsoft had figured out a while ago. There is nothing in Panther and earlier that compares to the Services control panel, or MMC snap-in, on Windows. Mac OS X still doesn't have that easy level of control, but the foundations are now there with launchd and its buddy launchctl. (Incidentally, this is a great opportunity for an industrious AppleScripter to whip out a Studio app.)

Meet the team
When Apple decided to overhaul the 30 year old system for starting and maintaining processes on UNIX it had to think big. launchd can do the jobs of init, mach_init, xinetd, RC, SystemStarter, watchdog, and cron. It is Apple's stated intention that they want to eliminate all of those services in favor of launchd at some point in the future. There are two main programs in the launchd system, launchd and launchctl. Briefly:

launchd manages the daemons at both a system and user level. Similar to xinetd, launchd can start daemons on demand. Similar to watchdogd, launchd can monitor daemons to make sure that they keep running. launchd also has replaced init as PID 1 on Mac OS X and as a result it is responsible for starting the system at boot time.

launchctl is our window into the world of launchd. Using launchctl we can load and unload daemons, start and stop launchd controlled jobs, get system utilization statistics for launchd and its child processes, and set environment settings.

A third, and the most confusing, part of the system are the plist files that define the services to be loaded into launchd with launchctl. Stored in the LaunchAgents or LaunchDaemons of any Library, the XML files have around thirty different keys that can be set, and the syntax is not very pretty. All of the options are laid out in the launchd.plist man page, which hides them nicely by using an obscure name. Even after reading the substantial man page you still have no real clear idea of how to use the keys to construct a valid plist, and that's where thieving from the default Apple plists comes in handy.

Time to dig in...

When it comes down to it, launchd has two main tasks. The first is to boot the system and the second is to load and maintain services. Let's take a look at the boot part of the job first.

When Mac OS X boots it goes through quite a few steps. Here is a very simplified view of the 10.4 system startup.

1. The BootROM activates, does its thing to the hardware, and then loads BootX.
2. BootX loads the kernel, spins the gear, loads any needed kexts, and then the kernel loads launchd.
3. launchd then runs /etc/rc, scans through /System/Library/LaunchDaemons and /Library/LaunchDaemons and acts on the plists as needed, and finally starts the loginwindow.

Easy huh?

The first one we should look at here is step 2. This is a huge change for Mac OS X, and *NIX in general, in that some flavor of init is not loaded here. Indeed, launchd is now PID 1 on Mac OS X.

In step three, launchd scans through a few different directories for jobs to run. There are two different folders types that are scanned. The LaunchDaemons folders should contain items that will run as root, generally background processes. The LaunchAgents folders should contain jobs, called agent applications, that will run as a user or in the context of userland. Often these can be scripts, other foreground items, and they can even include a user interface. When we get to our example we will be creating a user-specific LaunchAgent job. These directories are all kept in the typical Libraries of Mac OS X.

launchd is very different from SystemStarter in that it may not actually launch all the daemons at boot time. Key to launchd, and similar to xinted, is the idea of launch on demand daemons. When launchd scans through the job plists at boot time it reserves and listens on all of the ports requested by those jobs. If so indicated in the plist by the "OnDemand" key, the daemon is not actually loaded at the time. Rather launchd will listen on the port, start the daemon when needed, and shut it down when it is not. After a daemon is loaded, launchd will keep track of it and make sure it is running if needed. In this way it is like watchdogd, and shares watchdogd's requirement that processes do not attempt to fork or daemonize on their own. If a process goes into the background launchd will lose track of it and attempt to relaunch it.

This is why Tiger boots so fast. The system only has to register the daemons that are to run, not actually launch them. In fact the progress bar that appears is just a placebo that doesn't really show anything other than the passage of time. This explains why the bar never reaches the end before the login window appears sometimes. (For fun you can run it any time you like by executing /usr/libexec/WaitingForLoginWindow.) In addition to incoming requests, there are other ways to define launch on demand and we will take a look at them in just a few moments.

The hardest part to manage during a launchd boot is dependancies. SystemStarter had a very simple system of dependancies that used the "Uses", "Requires", and "Provides" keys in the plist of a startup item. There are two main strategies when creating launch dependancies on 10.4. You can use IPC which will allow the daemons to talk amongst themselves to work it out or you can watch files or paths for changes. Using IPC is much more subtle than the SystemStarter's keys and requires more work from the developer, but it should lead to cleaner and quicker startups. It does however, put dependancies out of the reach of many admins, myself included, as we aren't actually creating daemons. If you have a timed script that needs to be run you can still use a SystemStarter item at this time. Don't get too comfortable with that though as it has been deprecated in 10.4

plist Time
When launchd scans a folder, or a job is submitted with launchctl, it depends on a properly formatted plist file that describes the job to be run. This is, for most sysadmins, is the most difficult area of launchd to grasp. SystemStarter used simple shell scripts to launch daemons. launchd requires the admin to be able to properly format an XML file and while not exceedingly difficult, it can be confusing.

The first stop you should make is the man page for launchd.plist. Give it a good read and don't worry if it makes your eyes bug out. It does that to everyone at first. At the very end it gives an example of a plist for your perusal. This example is very simple but it breaks down some of the keys you will become familiar with. At a minimum there are two required launchd.plist keys. "Label" is the string that defines how launchd will refer to the job, and "ProgramArguments" breaks down how to execute your program. That's it, but it's not enough to generate a useful job. Really we want to toss a few more keys in there to add to the flexibility of the job and to make it more efficient.

Some other useful keys:
  • "UserName" allows you to run the job as a user other than the one who submitted it to launchd.
  • "inetdCompatibility" indicates that the daemon expects to be run as if it was launched by inetd.
  • "Program" allows you to name the path to your executable. By using this key you can save the ProgramArguments key for, well, flags and arguments.
  • "OnDemand" is a boolean that will allow you to define if a job runs continuously or not.
  • "RootDirectory" will let you chroot the job into another directory.
  • "ServiceIPC" is where you specify if the daemon can speak IPC to launchd.
  • "WatchPaths" allows you to have launchd start a job based on a path modification.
  • "QueueDirectories" almost the same as a WatchPath, a queue will only watch an empty directory for new files.
  • "StartInterval" Used to schedule a job that runs on repeating schedule.
  • "StartCalendarInterval" You can use this to schedule jobs. The syntax is similar to cron.
  • "HardResourceLimits" Will allow you to restrict the resources consumed by any job.
  • "LowPriorityIO" Tells the kernel that this task is of a low priority when doing file system I/O.
  • "Sockets" This array can be used to specify what socket the daemon will listen on for launch on demand. Check out the cool Bonjour key that can be used to register the socket with the mDNSResponder.

  • Phew! That's not all of them, but I think that these represent a useful selection. Read the man page for all of them and the options to be used. Now that we know about some keys, lets do something neat with them.

    In our example were are going to create a very simple user agent application that moves files and folders from one directory to another. It's a very basic sort of idea, but it's also a good building block for more complex file processing.

    First we need to come up with the script that our launchd job will execute. For our purposes something simple like:
    mv /Users/josh/in/* /Users/josh/out/
    exit 0
    will work. It simply takes anything in the "in" directory and moves it to the "out" one. Stick it wherever you like. Since this is a user specific agent I just dropped it in my home folder and named it "mover".

    Now we need to create our launchd job plist. First create ~/Library/LaunchAgents, then go find a launchd plist file in /System/Library/LaunchDeamons that seems pretty similar to what you want to do. For this example I stole the cron plist and copied it to my new LaunchAgents folder. Fire up your editor of choice and start making changes.

    In the end we should have something that looks a bit like this:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "">
    <plist version="1.0">
    All we really needed to do was to give it a new label, modify the executable that is called, change the cron file's RunAtLoad key into an OnDemand one, and make the WatchPaths point to the correct place. Because we started with an existing file we don't need to worry about the header or figuring the structure out on our own. Why a WatchPaths key instead of a QueueDirectories key? The QueueDirectories depends on the directory being empty to look for added files. If you open the directory in the Finder it will make a .DS_Store file and throw your job into a horrible loop. The WatchPaths is a bit different in that it watches a path for modifications and doesn't require the directory to be empty. WatchPaths can also keep an eye on a file. Take a look at the default cron plist for a nice example of this.

    Note that in this example I also added a ProgramArguments key. I did this as it makes the job more flexible for future changes. Something to keep in mind with this key is that each flag or argument to be passed to the program must be in its own string of the array. The org.postfix.master.plist file is a good example of this. You can't simply say:
    	<string>/usr/libexec/postfix/master -e 60</string>
    but instead must use:
    which can be a bit confusing at first, but is nothing too hard once you do it the first time. Really you should take a look at the default plists that Apple has in /System/Library/LaunchDaemons. They are full of good examples of just about everything that you might want to do. For example, the job controls running the daily periodic job. This used to be something that was controlled by cron, so it's a good way to see how to schedule something with launchd. Take a look:
    Be aware that launchd is nowhere near as flexible as Vixie cron. Luckily cron is still on the system for us to use if we wish.

    If all of this seems a bit scary still, there is a nice shareware GUI out there that can handle the file generation for you. Take a look at the Launchd Editor from codepoetry to see what it can do.

    Now that we have our executable and plist setup how do we load our new job into launchd? We could logout and back in, but that's a bit intrusive to just fire up a new service. This is where launchd's buddy, launchctl comes into play.

    Enter launchctl

    One of the big problems before launchd was that there was no easy or consistent way to control system services. SystemStarter was almost cool, but it never seemed to work properly in most cases for controlling services after startup. The problem with the other facilities for service control was that they are strewn across the OS with no central way to manage them. launchctl is Apple's way of fixing this.

    On its own, launchctl can take commands from the command line, from standard in, or operate in interactive mode. If you come up with a set of commands that you want to make permanent you can store them in ~/.launchd.conf or /etc/launchd.conf. By using these files you can setup the environment that launchd operates in at either a user or global level. You can use sudo launchctl to make easy changes on a global scale, something that you couldn't do on 10.3 and earlier.

    Let's start out with something simple though, loading our job. The syntax is basic, in our case:

    launchctl load ~/Library/LaunchAgents/com.afp548.mover.plist

    will load our job. We can confirm it's loaded with:

    launchctl list

    which will show us the launchd jobs that our user has submitted. If you want to see the system job list simply run the same command with sudo.

    Now that our job is loaded, try dropping something in your "in" folder. If all is well it will vanish and then appear in your "out" folder in a manner similar to this spiffy movie (H264 codec required). For what it is worth, the folder in the movie has a bunch of loose files in it and it's running on a 400 MHz G4. As you see the response time of launchd is great. Again this is a very simple example, but it's useful and is a great launching pad for bigger tasks. For example launchd could watch a drop folder for graphic files that it processes with sips and then ftp transfers to a web server. The options are practically endless.

    To remove a job we use the inverse of the load command...

    launchctl unload -w ~/Library/LaunchAgents/com.afp548.mover.plist

    Notice that I added the optional "-w" flag this time. This flag adds or removes the "disabled" key from the plist at the appropriate time. If you don't use this when unloading a job it will automatically load the next time you login or reboot, depending on where your job file is located. You use the same "-w" flag when loading the job to remove the disabled key automatically.

    Just like lookupd, you can also use launchctl in interactive mode. Simply run it without any arguments and you will be delivered to the launchd prompt. Type "help" to see the list of commands that you can issue. They are the same as if you were calling them directly, but now you don't need to keep typing launchctl over and over again. A standard ctrl-d will escape the launchd shell.

    If you want to view the resource usage of any particular job, or launchd as a whole, you can use the "getrusage" command. When using this command you need to specify if you are interested in launchd itself or its children. Again, you use sudo to see the global resource usage. So if I wanted to see the resource usage of all of launchd's kids:

    sudo launchctl getrusage children

    is all I need. If you change our demo job plist to use a QueueDirectories key rather than the WatchPaths one there is a very high probability that a race condition will appear. This can be really bad as it can run wild and suck up all the CPU time on your system. Which leads us right into our next launchctl subcommand.

    Using the "limit" command we can view and set limits on the launchd environment. With no resource specified this displays three columns representing the resources, their soft limits, and finally, their hard limits. Again you can use sudo to define or view these for the system as a whole. The limits you set with launchctl are the same limits you can set in a job plist, but they apply to the entire launchd environment rather than a single job. There are details in the launchd.plist or setrlimit man pages, but I'll list them briefly here:

  • "cpu" The maximum CPU time, in seconds, to be used by a process.
  • "filesize" The file size limit, in bytes, that a process may create.
  • "data" The data segment size limit, in bytes, for the process.
  • "stack" The stack segment size limit, in bytes, for the process.
  • "core" The core file size limit, in bytes, for the process.
  • "rss" This sets the resident set size that controls how much physical memory the process may have. The OS will prefer to reclaim memory from applications that are over their soft limit when things get tight.
  • "memlock" The maximum size an app may lock into memory.
  • "maxproc" The maximum number of processes for a particular user ID.
  • "maxfiles" Obviously, the maximum number of open files for a process.

  • To set these limits simply use the limit command but add a resource type and name a limit. If you only enter one number it will be used for both the soft and hard limits. If you want them to be different you need to enter both the soft and hard limits in order.

    launchctl limit maxfiles 256 512

    Will set the maxfiles limit to 512, while retaining the default soft limit of 256 files.

    There are other similar launchctl commands that we can use to set logging levels, change the stdout and stderr of launchd, set environment variables, or change the umask of launchd. For example if I wanted to redirect the stdout of just my launchd jobs to a file I could say:

    launchctl stdout /Users/josh/Library/Logs/launchd.out

    It is important to remember that these commands are job persistant, but not launchd persistent. Meaning that the general launchd settings will not survive a reboot. This is easy to fix by dropping a conf file in the appropriate place. For global settings you should use /etc/launchd.conf and for user settings you should use ~/.launchd.conf. Just enter your launchctl commands, without the launchctl part, in the file one line at a time. If I wanted to combine my two examples above into a permanent setting I would put:

    # This is my launchd.conf file.
    limit maxfiles 256 512<br>
    stdout /Users/josh/Library/Logs/launchd.out
    in the appropriate launchd.conf file.


    At this point you are probably thinking that launchd is pretty cool and you would be right, but I do have a few concerns.

    First of all launchd replaced init and xinetd with one process. This is a bit scary as we now basically have init listening in a bunch of different ways for something to tell it to start a job. The security implications of this aren't really known yet with launchd being as young as it is.

    Secondly, and in the same vein, launchd is process 1 and it has the potential to take down the whole system. I've already seen unconfirmed reports of a ssh scan on a network causing launchd to freak out and make systems inaccessible. Having at least some sort of resource limit set on jobs might help here.

    These risks and unknowns could have been minimized by gradually phasing launchd in, but Apple has chosen to drop it on us like a bomb. I know of people who are migrating services back to xinetd when they do installs. It's not because launchd is flawed, but because it is a bit of an unknown for the time being.

    That said I'm really excited by the potential of launchd. We are now one GUI away from a Windows-style "Services" utility.

    Wrapping up

    Wow, that was a lot to take in at once eh? launchd is here, and it is the future of service management on Mac OS X. Apple has opened the project in the hopes that other *NIX vendors will take a look at it and incorporate it into their own OSes. With launchd being the work of Jordan Hubbard it probably carries a bit more clout than other Apple OSS projects in the community. All in all it is the most aggressive attempt by anyone to reign in the herd of cats that booting and maintaining a UNIX system has become, and for that we should be standing and cheering.

    As always, have fun and read the man pages.

    man launchd
    man launchd.conf
    man launchd.plist
    man launchctl
    man launchd_debugd

    That last one is a nice little nugget, but it is a bit of a trick as the plist it references isn't installed by default. First you need to get the xml file from the darwin source here (Apple ID login required). Change its name from to, copy it to /System/Library/LaunchDaemons, load it, and then point your web browser at port 12345 on the Mac in question to get a cool output of all loaded jobs and their settings.

    (If you don't have an Apple ID, or don't want to agree to the Apple Open Source license, I've put a copy of the finished launchd_helperd.plist in our file database.)

    (Ed. note: For the ease of sharing, this document is released with the GFDL license as well as our regular Creative Commons license.)

    Story Options


    launchd in Depth | 43 comments | Create New Account
    The following comments are owned by whomever posted them. This site is not responsible for what they say.
    launchd vs StartupItems
    Authored by: Anonymous on Saturday, July 09 2005 @ 07:53 am MDT
    thanks for this article.
    just a question... if StartupItems is depracated why do Apple still use it to launch
    Apache and other services ? Is this for the security reasons you're talking about
    in your article ? (talk about the client version)
    launchd in Depth
    Authored by: Anonymous on Sunday, July 10 2005 @ 03:50 pm MDT
    "Loose" is not a synonym for "Lose".

    The loose command of the language causes me to lose all interest in
    finishing the article.
    • launchd in Depth - Authored by: Anonymous on Monday, July 11 2005 @ 08:35 am MDT
    • launchd in Depth - Authored by: Anonymous on Sunday, July 24 2005 @ 10:23 am MDT
    • launchd in Depth - Authored by: Anonymous on Tuesday, August 02 2005 @ 02:45 pm MDT
    • launchd in Depth - Authored by: Anonymous on Tuesday, October 11 2005 @ 08:39 pm MDT
    • launchd in Depth - Authored by: Anonymous on Friday, May 05 2006 @ 08:33 pm MDT
    re: launchd in Depth
    Authored by: Anonymous on Monday, July 11 2005 @ 04:19 am MDT

    First, thanks for taking the time to write such an informative article. Some
    may be nit-picking it for grammar or typos, but in so doing I believe they're
    really missing the point, mainly that these sorts of community-generated
    articles are a GREAT way of providing "missing manual" types of
    documentation that probably wouldn't be available any other way, or certainly
    not as quickly. This should be encouraged, not picked for small nits.

    That said, there are a few assertions made in the article which deserve
    clarification, purely for technical accuracy:

    First, you note in your "concerns" section that init and [x]inetd are now
    running as one job and that the security implications of this are unknown.
    While this may be true in some sense, it does bear noting that both daemons
    historically ran as root and that two daemons are more difficult to audit than
    one, so consolidating multiple root daemons, each capable of vending "root
    children", actually LESSENS security concerns since you have that many less
    possible attack vectors. Yes, launchd is young, but a lot of security review
    went into it and I'm quite happy that we have less code to audit in this
    particular realm.

    Second, the story about ssh scans causing massive malfunction does seem
    somewhat apocryphal. I haven't seen any bugs filed on this topic and, of
    course, would welcome any hard data on this, if it's subsequently confirmed to
    be a problem (which I'm inclined to doubt).

    Third, it's difficult to "phase in" something like launchd while also making it
    useful for something. We did "phase it in" in the sense that we made sure
    ALL of the existing mechanisms continued to work, from cron to xinetd, and if
    launchd had just sat there as an inert lump, doing nothing more than sit side-
    by-side with those other mechanisms, do you really think anyone would have
    even adopted it? I suspect not. Launchd wasn't just dropped 'like a bomb', it
    was designed with significant attention to backwards compatibility while still
    making it something that actually performed a useful consolidating role.

    Finally, as much as I would like to take credit for launchd, I did not write it. I
    merely manage the BSD Technology group, which did write it, and had some
    input into its design. I am, of course, very proud of the evolution represented
    by launchd and think it has a bright future. While it works well now, it's
    naturally still evolving and will hopefully become even more powerful, easier to
    use, and boast better documentation with the passage of time.

    - Jordan Hubbard
    Another launchd Presentation
    Authored by: Anonymous on Thursday, July 14 2005 @ 04:57 pm MDT

    There is a online video presentation on launchd & GUI plist editor Launchd
    Editor, done by the University of Utah, Student Computing Labs that might be
    of interest...
    launchd in Depth
    Authored by: Wire on Monday, August 08 2005 @ 03:13 pm MDT
    Why are not launchd-agents in "user- domain" unloaded when the user log
    out? Is there a way to unload them when he user log out?
    • launchd in Depth - Authored by: Anonymous on Monday, November 14 2005 @ 04:09 pm MST
    • launchd in Depth - Authored by: Anonymous on Tuesday, June 09 2009 @ 12:26 pm MDT
    combining interval options?
    Authored by: blake on Thursday, October 06 2005 @ 06:52 pm MDT
    I have made a few launchd jobs now and starting to like this new tool.

    But say I want a script to run every 5 minutes only during business hours
    9am-5pm Monday-Friday.

    Can you combine StartInterval with StartCalendarInterval to get the desired
    StartCalendarInterval usage???
    Authored by: Anonymous on Tuesday, November 22 2005 @ 07:46 am MST
    Has anyone been able to get the StartCalendarInterval option to be able to
    work with Launchd plist files?
    I can get the example in this article working which uses the WatchDirectories
    plist tag, but no matter what I do I cannot get the StartCalendarInterval tag
    to work.
    I have simply copied the com.periodic-daily.plist file to my own in ~/Library/
    LaunchAgents/. It loads successfully and shows in the list when you run
    launchctl list, but nothing happens at the specified times. I have it running a
    simply script that creates a text file, but it never executes? (nothing is written
    to the logs either.)

    Any ideas?
    launchd in Depth
    Authored by: Anonymous on Thursday, January 19 2006 @ 07:40 am MST
    Another fine launchd plist editor:

    - Steve Jacobs
    • launchd in Depth - Authored by: Anonymous on Wednesday, June 07 2006 @ 03:03 pm MDT
    • launchd in Depth - Authored by: Anonymous on Saturday, September 13 2008 @ 10:20 pm MDT
    WatchPaths doesn't actually watch the explicit path you tell it.
    Authored by: jrg on Thursday, March 04 2010 @ 08:44 am MST
    Seems, in all the testing, someone forgot that files can be renamed.

    Under 10.5.8, if you have WatchPaths set for a log file and that log file gets rotated (standard operating procedure) launchd keeps watching the old log file and doesn't monitor the new one.

    The only way around this, that I can find, is to unload and reload the plist (sadly no one thought to offer way to reload/HUP jobs, or unload/reload them by their job_label. Next time I'll see if 'launchctl stop' will do the trick.)
    launchd in Depth - Easy launchd plist
    Authored by: Anonymous on Thursday, September 16 2010 @ 06:16 am MDT
    This site can create plist files for Mac OS X
    in a few minutes.
    I hope it is helpful!

    launchd in Depth
    Authored by: Anonymous on Friday, June 03 2011 @ 07:43 am MDT
    I have an application which is written using Qt and I want to run it as a service.
    I could create a .plist and it is running fine. but the application icon is visible on the dock.
    Could you please let me know how to remove that icon from the dock.
    I am new to Mac environment.