jamesh is currently certified at Master level.

Name: James Henstridge
Member since: N/A
Last Login: 2012-08-22 08:36:51

FOAF RDF Share This

Homepage: http://www.jamesh.id.au/


A GNOME hacker. Author of gnorpm, gnome-python,
libglade, pygtk and others. Former maintainer of dia.
Contributor to various gnome packages.

New diary entry | Planet Gnome | Planet Ubuntu


Articles Posted by jamesh

Recent blog entries by jamesh

Syndication: RSS 2.0

Extracting BIOS images and tools from ThinkPad update ISOs

With my old ThinkPad, Lenovo provided BIOS updates in the form of Windows executables or ISO images for a bootable CD.  Since I had wiped Windows partition, the first option wasn’t an option.  The second option didn’t work either, since it expected me to be using the drive in the base I hadn’t bought.  Luckily I was able to just copy the needed files out of the ISO image to a USB stick that had been set up to boot DOS.

When I got my new ThinkPad, I had hoped to do the same thing but found that the update ISO images appeared to be empty when mounted.  It seems that the update is handled entirely from an El Torito emulated hard disk image (as opposed to using the image only to bootstrap the drivers needed to access the CD).

So I needed some way to extract that boot image from the ISO.  After a little reading of the spec, I put together the following Python script that does the trick:

import struct
import sys


def find_image(fp):
    # el-torito boot record descriptor
    fp.seek(0x11 * SECTOR_SIZE)
    data = fp.read(SECTOR_SIZE)
    assert data[:0x47] == b'\x00CD001\x01EL TORITO SPECIFICATION' + b'' * 41
    boot_catalog_sector = struct.unpack('<L', data[0x47:0x4B])[0]

    # check the validation entry in the catalog
    fp.seek(boot_catalog_sector * SECTOR_SIZE)
    data = fp.read(0x20)
    assert data[0:1] == b'\x01'
    assert data[0x1e:0x20] == b'\x55\xAA'
    assert sum(struct.unpack('<16H', data)) % 0x10000 == 0

    # Read the initial/default entry
    data = fp.read(0x20)
    (bootable, image_type, load_segment, system_type, sector_count,
     image_sector) = struct.unpack('<BBHBxHL', data[:12])
    image_offset = image_sector * SECTOR_SIZE
    if image_type == 1:
        # 1.2MB floppy
        image_size = 1200 * 1024
    elif image_type == 2:
        # 1.44MB floppy
        image_size = 1440 * 1024
    elif image_type == 3:
        # 2.88MB floppy
        image_size = 2880 * 1024
    elif image_type == 4:
        # Hard disk image.  Read the MBR partition table to locate file system
        data = fp.read(512)
        # Read the first partition entry
        (bootable, part_type, part_start, part_size) = struct.unpack_from(
            '<BxxxBxxxLL', data, 0x1BE)
        assert bootable == 0x80 # is partition bootable?
        image_offset += part_start * 512
        image_size = part_size * 512
        raise AssertionError('unhandled image format: %d' % image_type)

    return fp.read(image_size)

if __name__ == '__main__':
    with open(sys.argv[1], 'rb') as iso, open(sys.argv[2], 'wb') as img:

It isn’t particularly pretty, but does the job and spits out a 32MB FAT disk image when run on the ThinkPad X230 update ISOs. It is then a pretty easy task of copying those files onto the USB stick to run the update as before. Hopefully owners of similar laptops find this useful.

There appears to be an EFI executable in there too, so it is possible that the firmware update could be run from the EFI system partition too.  I haven’t had the courage to try that though.

Syndicated 2012-11-13 08:58:00 from James Henstridge

u1ftp: a demonstration of the Ubuntu One API

One of the projects I’ve been working on has been to improve aspects of the Ubuntu One Developer Documentation web site.  While there are still some layout problems we are working on, it is now in a state where it is a lot easier for us to update.

I have been working on updating our authentication/authorisation documentation and revising some of the file storage documentation (the API used by the mobile Ubuntu One clients).  To help verify that the documentation was useful, I wrote a small program to exercise those APIs.  The result is u1ftp: a program that exposes a user’s files via an FTP daemon running on localhost.  In conjunction with the OS file manager or a dedicated FTP client, this can be used to conveniently access your files on a system without the full Ubuntu One client installed.

You can download the program from:


To make it easy to run on as many systems as possible, I packaged it up as a runnable zip file so can be run directly by the Python interpreter.  As well as a Python interpreter, you will need the following installed to run it:

  • On Linux systems, either the gnomekeyring extension (if you are using a GNOME derived desktop), or PyKDE4 (if you have a KDE derived desktop).
  • On Windows, you will need pywin32.
  • On MacOS X, you shouldn’t need any additional modules.

These could not be included in the zip file because they are extension modules rather than pure Python.

Once you’ve downloaded the program, you can run it with the following command:

python u1ftp-0.1.zip

This will start the FTP server listening at ftp://localhost:2121/.  Pointing a file manager at that URL should prompt you to log in, where you can use your standard Ubuntu One credentials and start browsing your files.  It will verify the credentials against the Ubuntu SSO service and issue an OAuth token that it stores in the keyring.  The OAuth token is then used to authenticate requests to the file storage REST API.

While I expect this program to be useful on its own, it was also intended to act as an example of how the Ubuntu One API can be used.  One way to browse the source is to simply unzip the package and poke around.  Alternatively, you can check out the source directly from Launchpad:

bzr branch lp:u1ftp

If you come up with an interesting extension to u1ftp, feel free to upload your changes as a branch on Launchpad.

Syndicated 2012-07-05 08:19:00 from James Henstridge

Packaging Python programs as runnable ZIP files

One feature in recent versions of Python I hadn’t played around with until recently is the ability to package up a multi-module program into a ZIP file that can be run directly by the Python interpreter.  I didn’t find much information about it, so I thought I’d describe what’s necessary here.

Python has had the ability to add ZIP files to the module search path since PEP 273 was implemented in Python 2.3.  That can let you package up most of your program into a single file, but doesn’t help with the main entry point.

Things improved a bit when PEP 338 was implemented in Python 2.4, which allows any module that can be located on the Python search path can be executed as a script.  So if you have a ZIP file foo.zip containing a module foo.py, you could run it as:

PYTHONPATH=foo.zip python -m foo

This is a bit cumbersome to type though, so Python 2.6 lets you run directories and zip files directly.  So if you run

python foo.zip

It is roughly equivalent to:

PYTHONPATH=foo.zip python -m __main__

So if you place a file called __main__.py inside your ZIP file (or directory), it will be treated as the entry point to your program.  This gives us something that is as convenient to distribute and run as a single file script, but with the better maintainability of a multi-module program.

If your program has dependencies that you don’t expect to find present on the target systems, you can easily include them up in the zip file along side your program.  If you need to provide some data files along side your program, you could use the pkg_resources module from setuptools or distribute.

There are still a few warts with this set up though:

  • If your program fails, the trace back will not include lines of source code.  This is a general problem for modules loaded from zip files.
  • You can’t package extension modules into a zip file.  Of course, if you’re in a position where the target platforms are locked down tight enough that you could reliably provide compiled code that would run on them, you’d probably be better off using the platform’s package manager.
  • There is no way to tell whether a ZIP file can be executed directly with Python without inspecting its contents.  Perhaps this could be addressed by defining a new file extension to identify such files.

Syndicated 2012-05-21 07:31:05 from James Henstridge

NBN talk at PLUG

Earlier in the week, I attended a PLUG discussion panel about the National Broadband Network.  While I had been following some of the high level information about the project, it was interesting to hear some of the more technical information.

The evening started with a presentation by Chris Roberts from NBN Co, and was followed by a panel discussion with Gavin Tweedie from iiNet and Warrick Mitchel from AARNet.

James Bromberger introducing the panel: Chris Roberts (NBN Co), Gavin Tweedie (iiNet) and Warrick Mitchel (AARNet)

One question I had was when they’ll get round to building out the network where I live.  There is a rollout map on the NBN Co site, but it currently only shows plans for works that will commence within a year.  Apparently they plan to release details on the three year plan by the end of this month, so hopefully my suburb will appear in that list.

The NBN is being built on top of three methods of connection: GPON fibre for built up areas, fixed LTE wireless (non roaming) for the smaller towns where it is not economical to provide fibre, and satellite broadband for the really remote areas.  All three connection methods provide a common interface to service providers, so companies that provide services over the network are not required to treat the three methods differently.  The wireless and satellite connections will initially run at 12Mb/s down and 1Mb/s up, while fibre connections can range from 25/5 to 100/40 (with the higher connection speeds incurring higher wholesale prices).  It should be quite an improvement over the upload speed  I’m currently getting on ADSL2.

Chris brought in some sample “User Network Interface” (UNI) boxes that would be used on premises with a fibre connection.  It provided 4 gigabit Ethernet ports, and 2 telephony ports.

The inside of a current generation NBN interface box

Rather than the 4 Ethernet ports being part of a single network as you’d expect for similar looking routers, each port represents a separate service.  So the single box can support connections to 4 retail ISPs, or for any other services delivered over the network (e.g. a cable TV service).  You would still need a router to provide firewall, NAT and wifi services, but since it only requires Ethernet for the WAN port there should be a bit more choice in routers than if you limit yourself to ones with ADSL modems built in.  In particular, it should be easier to find a router capable of running an open firmware like OpenWRT or CeroWRT.

The box also acts as a SIP ATA, where each of the two telephony ports can be configured to talk to the servers of different service providers.

It is also possible for NBN Co to remotely monitor the UNI boxes in people’s houses, so they can tell when they drop off the network.  This means that they have the ability to detect and respond to faults without relying on customer complaint calls like we do for the current Telstra copper network.

Since the NBN is supposed to provide a service equivalent to the current copper telephone network, the UNI box is paired with a battery pack to keep the telephony ports active during black outs, similar to how a wired telephone draws power from the exchange.  This battery pack is somewhat larger than the UNI box, holding a 7.2 Ah lead acid battery.  At 10W, this can keep the box running for around 8 hours.  The battery pack will automatically cut power before it is completely drained, but has an emergency switch to deliver the remaining energy at the expense of ruining the battery.

Next PLUG Event

If you’re in Perth, why not come down to the next PLUG event on March 26th?  It is an open source themed pub quiz at the Moon & Sixpence.  Last year’s quiz was a lot of fun, and I expect this one will be the same.

Syndicated 2012-03-17 14:47:37 from James Henstridge

pygpgme 0.3

This week I put out a new release of pygpgme: a Python extension that lets you perform various tasks with OpenPGP keys via the GPGME library.  The new release is available from both Launchpad and PyPI.

There aren’t any major new extensions to the API, but this is the first release to support Python 3 (Python 2.x is still supported though).  The main hurdle was ensuring that the module correctly handled text vs. binary data.  The split I ended up on was to treat most things as text (including textual representations of binary data such as key IDs and fingerprints), and treat the data being passed into or returned from the encryption, decryption, signing and verification commands as binary data.  I haven’t done a huge amount with the Python 3 version of the module yet, so I’d appreciate bug reports if you find issues.

So now you’ve got one less reason not to try Python 3 if you were previously using pygpgme in your project.

Syndicated 2012-03-11 15:04:20 from James Henstridge

293 older entries...


jamesh certified others as follows:

  • jamesh certified miguel as Master
  • jamesh certified DV as Journeyer
  • jamesh certified federico as Master
  • jamesh certified hp as Master
  • jamesh certified timg as Journeyer
  • jamesh certified vicious as Master
  • jamesh certified terral as Journeyer
  • jamesh certified raph as Master
  • jamesh certified itp as Journeyer
  • jamesh certified yakk as Master
  • jamesh certified mathieu as Journeyer
  • jamesh certified dcm as Master
  • jamesh certified mjs as Master
  • jamesh certified bernhard as Journeyer
  • jamesh certified MJ as Journeyer
  • jamesh certified listen as Journeyer
  • jamesh certified campd as Journeyer
  • jamesh certified advogato as Master
  • jamesh certified sopwith as Master
  • jamesh certified lupus as Journeyer
  • jamesh certified rakholh as Journeyer
  • jamesh certified asmodai as Journeyer
  • jamesh certified alex as Master
  • jamesh certified mbp as Journeyer
  • jamesh certified harold as Journeyer
  • jamesh certified Raphael as Journeyer
  • jamesh certified Radagast as Journeyer
  • jamesh certified LotR as Journeyer
  • jamesh certified goran as Journeyer
  • jamesh certified BeeWarlock as Apprentice
  • jamesh certified krylan as Apprentice
  • jamesh certified msw as Master
  • jamesh certified amk as Journeyer
  • jamesh certified lauris as Master
  • jamesh certified andersca as Master
  • jamesh certified thom as Apprentice
  • jamesh certified fdrake as Journeyer
  • jamesh certified Sarah as Apprentice
  • jamesh certified bratsche as Journeyer
  • jamesh certified jrb as Master
  • jamesh certified Telsa as Journeyer
  • jamesh certified gleblanc as Journeyer
  • jamesh certified timj as Master
  • jamesh certified martin as Master
  • jamesh certified zilch as Journeyer
  • jamesh certified jdub as Master
  • jamesh certified effbot as Master
  • jamesh certified MCArkan as Journeyer
  • jamesh certified gman as Journeyer
  • jamesh certified blizzard as Master
  • jamesh certified malcolm as Journeyer
  • jamesh certified trs80 as Apprentice
  • jamesh certified fxn as Journeyer
  • jamesh certified yosh as Master
  • jamesh certified tromey as Master
  • jamesh certified sri as Apprentice
  • jamesh certified macricht as Journeyer
  • jamesh certified RossBurton as Journeyer
  • jamesh certified fcrozat as Journeyer
  • jamesh certified kristian as Journeyer
  • jamesh certified Darin as Master
  • jamesh certified Ankh as Master
  • jamesh certified Hallski as Master
  • jamesh certified AndrewDSmart as Apprentice
  • jamesh certified hadess as Journeyer
  • jamesh certified jdahlin as Journeyer
  • jamesh certified arvind as Journeyer
  • jamesh certified kiko as Journeyer
  • jamesh certified mwh as Master
  • jamesh certified Jody as Master
  • jamesh certified louie as Master
  • jamesh certified larsrc as Journeyer
  • jamesh certified nlevitt as Journeyer
  • jamesh certified snorp as Journeyer
  • jamesh certified elanthis as Apprentice
  • jamesh certified ndw as Master
  • jamesh certified funrecords as Journeyer
  • jamesh certified mjg59 as Journeyer
  • jamesh certified mpesenti as Master
  • jamesh certified mrd as Journeyer
  • jamesh certified jmason as Master
  • jamesh certified wingo as Journeyer
  • jamesh certified msevior as Master
  • jamesh certified seb128 as Master
  • jamesh certified keybuk as Master
  • jamesh certified gicmo as Journeyer
  • jamesh certified robertc as Master
  • jamesh certified jblack as Journeyer
  • jamesh certified sivang as Apprentice
  • jamesh certified desrt as Master

Others have certified jamesh as follows:

  • ole certified jamesh as Master
  • raph certified jamesh as Journeyer
  • goran certified jamesh as Master
  • yosh certified jamesh as Journeyer
  • campd certified jamesh as Journeyer
  • LotR certified jamesh as Journeyer
  • bernhard certified jamesh as Journeyer
  • egad certified jamesh as Journeyer
  • andrei certified jamesh as Journeyer
  • Radagast certified jamesh as Journeyer
  • Raphael certified jamesh as Master
  • faassen certified jamesh as Master
  • bombadil certified jamesh as Journeyer
  • harold certified jamesh as Journeyer
  • mathieu certified jamesh as Master
  • booch certified jamesh as Master
  • mbp certified jamesh as Master
  • feldspar certified jamesh as Journeyer
  • timj certified jamesh as Master
  • dcm certified jamesh as Journeyer
  • listen certified jamesh as Master
  • alex certified jamesh as Master
  • mjs certified jamesh as Master
  • jochen certified jamesh as Master
  • duncan certified jamesh as Master
  • MJ certified jamesh as Master
  • zhp certified jamesh as Master
  • lupus certified jamesh as Journeyer
  • rakholh certified jamesh as Master
  • asmodai certified jamesh as Journeyer
  • lordsutch certified jamesh as Master
  • mtearle certified jamesh as Master
  • psj certified jamesh as Master
  • camber certified jamesh as Master
  • yakk certified jamesh as Master
  • tpot certified jamesh as Master
  • taral certified jamesh as Journeyer
  • djs certified jamesh as Master
  • nils certified jamesh as Master
  • ztf certified jamesh as Master
  • mlsm certified jamesh as Journeyer
  • bagfors certified jamesh as Master
  • nelsonrn certified jamesh as Master
  • lauris certified jamesh as Master
  • rodrigo certified jamesh as Master
  • jpick certified jamesh as Master
  • jae certified jamesh as Master
  • jsheets certified jamesh as Master
  • andersca certified jamesh as Master
  • ryuch certified jamesh as Master
  • jules certified jamesh as Master
  • djcb certified jamesh as Master
  • mwh certified jamesh as Master
  • tca certified jamesh as Journeyer
  • nixnut certified jamesh as Master
  • glenn certified jamesh as Master
  • timg certified jamesh as Master
  • lerdsuwa certified jamesh as Master
  • fdrake certified jamesh as Master
  • grem certified jamesh as Master
  • sh certified jamesh as Master
  • gbowland certified jamesh as Master
  • jdub certified jamesh as Master
  • dneighbors certified jamesh as Master
  • amk certified jamesh as Journeyer
  • menthos certified jamesh as Master
  • cuenca certified jamesh as Master
  • exa certified jamesh as Master
  • hub certified jamesh as Master
  • murrayc certified jamesh as Master
  • nzkoz certified jamesh as Master
  • gleblanc certified jamesh as Master
  • jonkare certified jamesh as Master
  • johnsonm certified jamesh as Master
  • baruch certified jamesh as Master
  • cwinters certified jamesh as Master
  • bratsche certified jamesh as Master
  • zilch certified jamesh as Master
  • johnnyb certified jamesh as Master
  • MCArkan certified jamesh as Master
  • ricardo certified jamesh as Master
  • Senra certified jamesh as Master
  • jao certified jamesh as Master
  • fxn certified jamesh as Master
  • trs80 certified jamesh as Master
  • gman certified jamesh as Master
  • braden certified jamesh as Master
  • walters certified jamesh as Master
  • hadess certified jamesh as Master
  • rkrishnan certified jamesh as Master
  • async certified jamesh as Master
  • sand certified jamesh as Master
  • DarthEvangelusII certified jamesh as Master
  • arvind certified jamesh as Master
  • jdahlin certified jamesh as Master
  • cactus certified jamesh as Master
  • kiko certified jamesh as Master
  • Jody certified jamesh as Master
  • edd certified jamesh as Master
  • larsrc certified jamesh as Master
  • Hallski certified jamesh as Master
  • kristian certified jamesh as Master
  • fcrozat certified jamesh as Master
  • pasky certified jamesh as Master
  • nlevitt certified jamesh as Master
  • elanthis certified jamesh as Master
  • dolphy certified jamesh as Master
  • blm certified jamesh as Master
  • rhestilow certified jamesh as Master
  • zotz certified jamesh as Master
  • Mmarquee certified jamesh as Master
  • strider certified jamesh as Master
  • mrd certified jamesh as Master
  • mdupont certified jamesh as Master
  • mpesenti certified jamesh as Master
  • polak certified jamesh as Master
  • ebf certified jamesh as Master
  • monkeyiq certified jamesh as Master
  • gobry certified jamesh as Master
  • lathiat certified jamesh as Master
  • pycage certified jamesh as Master
  • jmason certified jamesh as Master
  • gheet certified jamesh as Master
  • memeyou certified jamesh as Master
  • wingo certified jamesh as Master
  • mathrick certified jamesh as Master
  • badger certified jamesh as Master
  • gicmo certified jamesh as Master
  • e8johan certified jamesh as Master
  • lucasr certified jamesh as Master
  • jarashi certified jamesh as Master
  • nikole certified jamesh as Master
  • freax certified jamesh as Journeyer
  • pvanhoof certified jamesh as Journeyer
  • jsgotangco certified jamesh as Master
  • jblack certified jamesh as Master
  • behdad certified jamesh as Master
  • mako certified jamesh as Master
  • Burgundavia certified jamesh as Master
  • cinamod certified jamesh as Master
  • gpoo certified jamesh as Master
  • ianclatworthy certified jamesh as Master
  • yosch certified jamesh as Master
  • desrt certified jamesh as Master
  • ctrlsoft certified jamesh as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page