Quickutil: A New Software Distribution Paradigm

Updated on 04 July 2013 to reflect major changes in Quickutil.

Utilities in Common Lisp

What is a utility? I would say it is a small, context-free, useful bit of code that is self-contained, and doesn’t warrant an entire new library. Let me define each of the parts of this definition in a little more detail.

  • Small: The utility should be no more than a single function/macro/etc. or two, maybe depending on a few helper functions.
  • Self-contained: Aside from depending on other associated utilities, it should not depend on external libraries or require external resources.
  • Context-free: In this discussion, a utility should not depend on particular application or library-specific code. For example, if you are writing an image library, a function that helps format data for the header of the image is not a utility; these functions are called helper functions.
  • Useful: This is more subjective, but the utility should have a broad range of application. flatten is a function that has a wide range of uses, but reverse-half-of-array isn’t quite as broad in application.

Several utilities may have a common theme but don’t achieve a common goal.

Most Common Lispers love to write utilities. They are usually a rite of passage when learning macros. One can’t resist writing their own while or until, or even one’s own version of a for loop. Lisp supports so many programming paradigms, and extending those paradigms or even adding new ones often leads to new (or, as is often the case, re-invented) utilities.

I love writing utilities. A lot of my libraries are utilitarian in nature: recur, defrec, letrec, policy-cond, and of course my personal utility library qtility.

The problem is, there are a lot of utility libraries—just look at the CLiki page. Zach Beane, author and maintainer of Quicklisp, is pretty open to which libraries he allows in Quicklisp, but utility libraries must have a very good reason to be included.

The multitude of utility libraries isn’t just a problem, it is also the fact a lot of utilities are re-invented on-the-fly. Many projects have a utilities.lisp file, which contains a bucket of definitions that don’t quite fit anywhere else. They have these definitions because pulling in five separate utility libraries—five new, possibly future changing dependencies—seems qualitatively overkill for a relatively small amount of functionality. Sometimes they contain functions and macros—sometimes buggy even—that have been written countlessly before.

These problems combined are what I like to call the utility problem.

Alexandria

Alexandria is a project and is one of the first coordinated attempts to fix this. It is an open source library that generally contains a range of high quality utility functions, vetted and maintained by a small group of project members. This library has largely been successful in adoption; it is one of Quicklisp’s most downloaded libraries. (This is also a testament to the ubiquity of utilities.)

Unfortunately, however, the project operates at a glacial pace. As of writing, there are approximately five patches on the mailing list, each of which are about a half a year old. The library seeks to be peer reviewed, but of the 150+ functions, there are only three “blessed” macros, and seven “under review” (source).

Aside from being slow-moving, the kinds of utilities that get admitted to the library are up to the strong discretion of the group of maintainers.

There have been talks to mediate the latter problem, to split off Alexandia into another library called Byzantine or alexandia-experimental, which is where utilities go before they get to join the official club.

Likely because of all of the aforementioned problems, people still make their own utility libraries.

At the International Lisp Conference of 2012, Eitarow Fukamachi (of Clack and Quickdocs fame) and I, Robert Smith, set out with another approach to the utility problem, inspired by Quicklisp. We did not want to create Yet Another Utility Library (which, ostensibly, is what Alexandria was), but something entirely new and different.

Introducing Quickutil

Quickutil is a new way to publish and distribute utilities. It has been described as “Quicklisp but for functions” by a few early testers. For example, after loading Quickutil, you can immediately load utilities you need, including old utilities such as those from Alexandria and split-sequence:

> (ql:quickload :quickutil)
> (qtlc:utilize :utilities '(:split-sequence))
> (qtl:split-sequence #\Space "Hello from Quickutil!")
("Hello" "from" "Quickutil!")
21

Quickutil is not just a utility library, it is a client for acquiring utilities. The function qtlc:utilize locally extracts and compiles the requested utilities into the current image.

Quickutil only compiles the utilities needed, without extra overhead. For example, when you utilize binomial-coefficient, Quickutil will load, in the correct order, the functions it needs: range-product, factorial, and binomial-coefficient. Quickutil does not load any other functions. Moreover, if binomial-coefficient is utilize again, it is essentially a no-op; only utilities that haven’t yet been acquired are compiled and loaded.

What does this mean? It means that there’s no need to fret over bloat. When you only need one utility, you get only one utility, not hundreds of them. It also means it’s okay to include experimental or unpopular utilities in Quickutil; they’ll only be loaded and used by the people who want them. As a result of this, Quickutil encourages the submission of almost any utility, subject to only a few simple rules, mostly outlined in the previous definition of a utility.

What’s the cost of adoption? For most consumers of the Quickutil, it is just:

  1. Make your ASDF project depend on quickutil.
  2. Add a qtlc:utilize form, and it’ll load all of the needed utilities in the quickutil or qtl package.

If you have a project with tight bootstrapping requirements, or for some reason can’t use ASDF, there’s another option: You can save the utilities locally by calling qtlc:save-utils-as:

> (qtlc:save-utils-as "utils.lisp"
     :package "MY-UTILS"
     :package-nickname "UTL"
     :utilities '(:split-sequence :weave :riffle))
#P"utils.lisp"

and then in a new Lisp instance:

> (load "utils.lisp")
> (apply 'concatenate 'string
         (utl:riffle (utl:split-sequence #\Space "a   b  c      d")
                     ","))
"a,b,c,d"

You don’t even need to include quickutil as a dependency to your project; just load the resulting file at some point.

This method of using Quickutil, using save-utils-as, is not recommended unless it’s absolutely necessary. By saving utilities, one doesn’t get the benefit of automatically updated utilities and one has to make sure there aren’t package collisions.

The Future

Quickutil has just begun. There are still many things to improve, and many ideas to test. Already, we are working on new ideas: utility versioning, namespaces, micro-libraries, un-vetted submission, and curation.

The function-level distribution of software is, as far as we know, a novel idea, and could maybe extend to other projects. We hope that you will give Quickutil a try, submit to us your own utilities to be included, and have an open mind about the idea of a library.

Explore Quickutil

Right now, Quickutil is in public beta, and has not been released to Quicklisp yet. To get the download link, to find utilities, and to submit new ones, please go to

http://quickutil.org

Quickutil is and will always be open source. Please submit bug reports to

Quickutil Issues

3 comments to Quickutil: A New Software Distribution Paradigm

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Before you post, please prove you are sentient.

what is 5 plus 3?