Download Install Tutorial Docs FAQ Tools WikiLicense Team IRC Planet Involvement Shop Book

Custom Engine Plugins

The cherrypy.engine object is a WSPBus object, which provides a complete generic publish/subscribe system. The default channels govern the start/stop state of the process, but you are free to add any other channels you need, register listeners for any channel, builtin or custom, and publish to any channel.

Plugins are a way to collect multiple listeners into a single class. They are especially useful to encapsulate the startup and shutdown of HTTP sockets, database pools, memcache connections, or any other resource your application needs that requires initialization and cleanup in sync with other such objects. For example, you might want to start up a MySQL database pool, and also a thread to "ping" those connections occasionally (MySQL has had a history of closing idle conns). It doesn't make any sense to start the ping thread before the pool; it could lead to errors in the ping thread if the pool hasn't started up in time.

Many conventional web applications include a "startup script" that imports these sorts of components and initializes them in the correct order. But those scripts are notoriously ugly, with lots of code inlined, and even more notoriously difficult to debug when things get started out of order. Lots of them also assume the process will be killed outright, and therefore make no attempt to cleanly close open sockets and files.

By making a custom Plugin, you can avoid this sort of ugliness: all of the startup and shutdown logic is modularized into a single class, leaving your startup script to merely list them and turn them on. Starting in 3.1.1, if you attach your custom plugin to cherrypy.engine, it also becomes automatically configurable. Wrapping an external component in a Plugin also encourages you to make it able to idle (during maintenance). You also get execution protection (if other components fail), and simple log integration with other components.

Tutorial

In cherrypy.process.plugins, there is a SimplePlugin base class (which several of the BuiltinPlugin's use). If your component uses the default channels ('start', 'stop', 'exit', 'graceful', and 'log'), you should probably start by subclassing SimplePlugin. For each channel, if your subclass has a method of the same name, then the subscribe() method will register it as a listener on that channel.

Let's pretend we are making a Plugin to manage a SQLAlchemy pool of Postgres connections. (Though SQLAlchemy integration with CherryPy usually means something else entirely: each request enclosed in a transaction.)

We'll start by creating 'start' and 'stop' methods using SimplePlugin:

from cherrypy.process import plugins

class SQLAlchemyDB(plugins.SimplePlugin):
    """A WSPBus plugin that controls a SQLAlchemy engine/connection pool."""
    
    def __init__(self, bus, url=None, pool_size=5):
        plugins.SimplePlugin.__init__(self, bus)
        self.url = url
        self.pool_size = pool_size

    def start(self):
        self.db = create_engine(self.url, pool_size=self.pool_size, strategy='threadlocal')
    
    def stop(self):
        self.db.dispose()
        self.db = None

Safe Methods

The above works pretty well just by itself. In a startup script, you can call:

db = SQLAlchemyDB(cherrypy.engine, 'postgres://scott:tiger@localhost:5432/mydatabase')
db.subscribe()

...and the SQLA connection pool will start when you call cherrypy.engine.start and stop when you call cherrypy.engine.stop (or when some other code calls it for you in response to, say, a KeyboardInterrupt?). But if you tried to call engine.start again in the same process, you'd probably get a nasty error. There's also a glitch if, for some odd reason, SQLAlchemyDB.stop was called twice. In general, you should write your listener functions to be called safely any number of times in any order. Let's make our example safer:

from cherrypy.process import plugins

class SQLAlchemyDB(plugins.SimplePlugin):
    """A WSPBus plugin that controls a SQLAlchemy engine/connection pool."""
    
    def __init__(self, bus, url=None, pool_size=5):
        plugins.SimplePlugin.__init__(self, bus)
        self.url = url
        self.pool_size = pool_size
        self.db = None

    def start(self):
        if self.db is None:
            self.db = create_engine(self.url, pool_size=self.pool_size, strategy='threadlocal')
    
    def stop(self):
        if self.db is not None:
            self.db.dispose()
            self.db = None
    
    def graceful(self):
        if self.db is not None:
            self.db.dispose()

We've also added behavior for the 'graceful' event here. The 'dispose' method actually closes the existing pool and creates a new one, which is just what we want in this case.

There's a difference between the 'stop' event and the 'exit' event, by the way. The former can be called repeatedly, and is intended to bring the engine and its listeners/components to an idle state, perhaps in order to fix a database or restart a memcache box. The app may then be start'ed again without terminating the CherryPy process. The 'exit' channel is published only once, when the process is being terminated. You usually want to use 'stop'; see the implementation of plugins.PIDFile for an example of when you need 'exit' instead (i.e. os.remove(self.pidfile)).

Logging

Our plugin is pretty quiet so far. It's recommended that all listener methods log what they're doing--by default, these messages go to CherryPy's error log, which is usually either stderr or a file. So we sprinkle some of those around:

from cherrypy.process import plugins

class SQLAlchemyDB(plugins.SimplePlugin):
    """A WSPBus plugin that controls a SQLAlchemy engine/connection pool."""
    
    def __init__(self, bus, url=None, pool_size=5):
        plugins.SimplePlugin.__init__(self, bus)
        self.url = url
        self.pool_size = pool_size
        self.db = None

    def start(self):
        if self.db is None:
            self.bus.log("Creating the SQLAlchemy engine.")
            self.db = create_engine(self.url, pool_size=self.pool_size, strategy='threadlocal')
        else:
            self.bus.log("Skipping creation of the SQLAlchemy engine since one already exists.")
    
    def stop(self):
        if self.db is not None:
            self.bus.log("Disposing of the SQLAlchemy engine.")
            self.db.dispose()
            self.db = None
    
    def graceful(self):
        if self.db is not None:
            self.bus.log("Restarting the SQLAlchemy engine.")
            self.db.dispose()

Priority

Any subscribed listener can have a priority attribute (from 0 to 100) to help run them in the correct order. By default, all listeners run at priority 50. But if, for example, we needed our SQLAlchemyDB.start method to run after the HTTP server (priority=75) has started, we simply set the priority attribute on the 'start' method (note it's the method that gets the priority, not the class):

    def start(self):
        if self.db is None:
            self.bus.log("Creating the SQLAlchemy engine.")
            self.db = create_engine(self.url, pool_size=self.pool_size, strategy='threadlocal')
        else:
            self.bus.log("Skipping creation of the SQLAlchemy engine since one already exists.")
    start.priority = 80

Configuration

Starting in CherryPy 3.1.1, if we stick the plugin onto cherrypy.engine, we can turn it on and off, and also configure the url and pool_size, from a config file:

# startup script
cherrypy.engine.sqla_db = SQLAlchemyDB(cherrypy.engine)

# mysite.conf
[global]
engine.sqla_db.on: True
engine.sqla_db.url: 'postgres://scott:tiger@localhost:5432/mydatabase'
engine.sqla_db.pool_size: 20

Hosted by WebFaction

Log in as guest/cpguest to create tickets