Yehuda Katz is a member of the Ruby on Rails core team, and lead developer of the Merb project. He is a member of the jQuery Core Team, and a core contributor to DataMapper. He contributes to many open source projects, like Rubinius and Johnson, and works on some he created himself, like Thor.
@nk @mattknox @dhh there is also risk in whether most people will want to turn it off, further eroding convention over configuration
Getting Comfortable With Rubinius’ Pure-Ruby Internals
February 18th, 2011
You probably know that Rubinius is a Ruby whose implementation is mostly written in Ruby. While that sounds nice in theory, you may not know what that means in practice. Over the past several years, I’ve contributed on and off to Rubinius, and feel that as Rubinius has matured since the 1.0 release, a lot of its promise has gelled.
One of the great things about Rubinius is that it exposes, as first-class concepts, a lot of things that are merely implicit in other Ruby implementations. For instance, Ruby methods have a variable scope which is implicitly accessible by blocks created in the scope. Rubinius exposes that scope as Rubinus::VariableScope. It also exposes Ruby bindings as richer objects that you can create yourself and use. In this post, I will talk about Rubinius::VariableScope, how it works, and how it fits into the Rubinius codebase.
Prologue
For those of you who don’t know much about Rubinius, it’s a Ruby implementation whose core classes are mostly written in Ruby. Some functionality, such as the core of the object model, as well as low-level methods, are written in C++ and exposed into Ruby via its primitive system. However, the functionality is typically fairly low-level, and the vast majority of the functionality is written on top of those primitives. For example, a Ruby String in Rubinius is a mostly pure-Ruby object with a backing ByteArray, and ByteArray’s implementation is mostly written as a primitive.
For instance, String’s allocate method looks like this:
def self.allocate str = super() str.__data__ = Rubinius::ByteArray.new(1) str.num_bytes = 0 str.characters = 0 str end
If you just want to get to the meat of the matter, feel free to skip this section and go directly to the Internals section below.
In general, Rubinius’ code is broken up into three stages. The first stage, called alpha, creates just enough raw Ruby constructs to get to the next stage, the bootstrap stage. It defines methods like Class#new, Kernel#__send__, Kernel#raise, Kernel#clone, basic methods on Module, Object, String, and Symbol, and some classes on the Rubinius module, used for internal use. It’s a pretty short file, weighing it at under 1,000 lines, and I’d encourage you to read it through. It is located in kernel/alpha.rb.
Next, the bootstrap stage creates the basic methods needed on many of Ruby’s core classes. In some cases, it defines a simpler version of a common structure, like Rubinius::CompactLookupTable and Rubinius::LookupTable instead of Hash. It also contains Rubinius constructs like the Rubinius::CompiledMethod, Rubinius::MethodTable, Rubinius::StaticScope and Rubinius::VariableScope. Many of these methods are defined as primitives. Primitive methods are hooked up in Ruby; check out the bootstrap definition of Rubinius::VariableScope:
module Rubinius class VariableScope def self.of_sender Ruby.primitive :variable_scope_of_sender raise PrimitiveFailure, "Unable to get VariableScope of sender" end def self.current Ruby.primitive :variable_scope_current raise PrimitiveFailure, "Unable to get current VariableScope" end def locals Ruby.primitive :variable_scope_locals raise PrimitiveFailure, "Unable to get VariableScope locals" end # To handle Module#private, protected attr_accessor :method_visibility end end
The bootstrap definitions are in kernel/bootstrap. The file kernel/bootstrap/load_order.txt defines the order that bootstrap methods are loaded.
Next, Rubinius fleshes out the core classes, mostly using Ruby methods. There is some primitive use in this stage, but it’s fairly few and far between. For instance, the Hash class is written entirely in Ruby, using a Rubinius::Tuple for its entries. This code can be found in kernel/common.
Some additional code lives in kernel/platform, which contains platform-specific functionality, like File and POSIX functionality, and kernel/delta, which runs after kernel/common. For the full description of the bootstrapping process, check out Rubinius: Bootstrapping.
Internals
One of the really nice things about Rubinius is that many concepts that are internal in traditional Ruby are exposed directly into the Ruby runtime. In many cases, this is so that more of Rubinius can be implemented in pure Ruby. In other words, if a core method in MRI needs some behavior, it is often implemented directly in C and unavailable to custom Ruby code.
One good example is the behavior of $1 in a Ruby method. If you call the =~ method, MRI’s internal C code will walk back up to the caller and set an internal match object on it. If you write your own method that uses the =~ method, there is no way for you to set the match object on the caller. However, since Rubinius itself implements these methods in Ruby, it needs a way to perform this operation directly from Ruby code. Check out the Rubinius implementation of =~:
class Regex def =~(str) # unless str.nil? because it's nil and only nil, not false. str = StringValue(str) unless str.nil? match = match_from(str, 0) if match Regexp.last_match = match return match.begin(0) else Regexp.last_match = nil return nil end end end
There are a few interesting things going on here. First, Rubinius calls StringValue(str), which is a Ruby method that implements Ruby’s String coercion protocol. If you follow the method, you will find that it calls Type.coerce_to(obj, String, :to_str), which then, in pure-Ruby, coerces the object into a String. You can use this method anywhere in your code if you want to make use of the String coercion protocol with the standard error messages.
Next, Rubinius actually invokes the match call, which ends up terminating in a primitive called search_region, which binds directly into the Oniguruma regular expression engine. The interesting part is next: Regexp.last_match = match. This code sets the $1 property on the caller, from Ruby. This means that if you write a custom method involving regular expressions and want it to obey the normal $1 protocol, you can, because if Rubinius needs functionality like this, it is by definition exposed to Ruby.
VariableScope
One result of this is that you can get access to internal structures like the current “variable scope”. A variable scope is an object that wraps the notion of which locals are available (both real locals and locals defined using eval), as well as the current contents of each of those variables. If the current context is a block, a VariableScope also has a pointer to the parent VariableScope. You can get access to the current VariableScope by using VariableScope.current.
def foo x = 1 p [:root, Rubinius::VariableScope.current.locals] ["a", "b", "c"].each do |item| p [:parent, Rubinius::VariableScope.current.parent.locals] p [:item, Rubinius::VariableScope.current.locals] end end foo
The output will be:
[:root, #<Rubinius::Tuple: 1>] [:parent, #<Rubinius::Tuple: 1>] [:item, #<Rubinius::Tuple: "a">] [:parent, #<Rubinius::Tuple: 1>] [:item, #<Rubinius::Tuple: "b">] [:parent, #<Rubinius::Tuple: 1>] [:item, #<Rubinius::Tuple: "c">]
A Rubinius::Tuple is a Ruby object that is similar to an Array, but is fixed-size, simpler, and usually more performant. The VariableScope has a number of additional useful methods, which you can learn about by reading kernel/bootstrap/variable_scope.rb and kernel/common/variable_scope.rb. One example: locals_layout gives you the names of the local variables for each slot in the tuple. You can also find out the current visibility of methods that would be declared in the scope (for instance, if you call private the VariableScope after that call will have method_visibility of :private).
Perhaps the most exciting thing about VariableScope is that you can look up the VariableScope of your calling method. Check this out:
def root x = 1 ["a", "b", "c"].each do |item| call_me end end def call_me sender = Rubinius::VariableScope.of_sender p [:parent, sender.parent.locals] p [:item, sender.locals] end root
In this case, you can get the VariableScope of the caller, and get any live information from it. In this case, you can tell that the method was called from inside a block, what the locals inside the block contain, and what block contexts exist up to the method root. This is extremely powerful, and gives you a level of introspection into a running Ruby program heretofore not possible (or very difficult). Better yet, the existence of this feature doesn’t slow down the rest of the program, because Rubinius uses it internally for common Ruby functionality like the implementation of private and protected.
Implementing caller_locals
Now that we understand the VariableScope object, what if we want to print out a list of all of the local variables in the caller of a method.
def main a = "Hello" [1,2,3].each do |b| say_hello do puts "#{a}: #{b}" end end end def say_hello # this isn't working as we expect, and we want to look at # the local variables of the calling method to see what's # going on yield ensure puts "Goodbye" end
Let’s implement a method called caller_locals which will give us all of the local variables in the calling method. We will not be able to use VariableScope.of_sender, since that only goes one level back, and since we will implement caller_locals in Ruby, we will need to go two levels back. Thankfully, Rubinius::VM.backtrace has the information we need.
Rubinius::VM.backtrace takes two parameters. The first is the number of levels back to start from. In this case we want to start two levels back (the caller will be the say_hello method, and we want its caller). The second parameter indicates whether the Location objects should contain the VariableScope, which is what we want.
NOTE: Yes, you read that right. In Rubinius, you can obtain a properly object-oriented backtrace object, where each entry is a Location object. Check out kernel/common/location.rb to see the definition of the class.
In this case, if we pass true, to Rubinius::VM.backtrace, each Location will have a method called variables, which is a Rubinus::VariableScope object.
def caller_locals variables = Rubinius::VM.backtrace(2, true).first.variables end
The next thing you need to know is that each block in a method has its own VariableScope and a parent pointing at the VariableScope for the parent block.
Each VariableScope also has a pointer to its method, an object that, among other things, keeps a list of the local variable names. These lists are both ordered the same.
To collect up the locals from the caller, we will walk this chain of VariableScope objects until we get nil.
def caller_locals variables = Rubinius::VM.backtrace(2, true).first.variables locals = [] while variables names = variables.method.local_names values = variables.locals locals << Hash[*names.zip(values).flatten] variables = variables.parent end locals end
This will return an Array of local variable Hashes, starting from the caller’s direct scope and working up to the method body itself. Let’s change the original method to use caller_locals:
def say_hello # this isn't working as we expect, and we want to look at # the local variables of the calling method to see what's # going on. Let's use caller_locals. p caller_locals yield ensure puts "Goodbye" end
If we run the program now, the output will be:
[{:b=>1}, {:a=>"Hello"}]
Goodbye
[{:b=>2}, {:a=>"Hello"}]
Goodbye
[{:b=>3}, {:a=>"Hello"}]
GoodbyeThis is just the tip of the iceberg of what is possible because of Rubinius’ architecture. To be clear, these features are not accidents; because so much of Rubinius (including things like backtraces), it is forced to expose details of the internals to normal Ruby program. And because the Rubinius kernel is real Ruby (and not some kind of special Ruby subset), anything that works in the kernel will work in your Ruby code as well.
Clarifying the Roles of the .gemspec and Gemfile
December 16th, 2010
TL;DR
Although apps and gems look like they share the concept of “dependency”, there are some important differences between them. Gems depend on a name and version range, and intentionally don’t care about where exactly the dependencies come from. Apps have more controlled deployments, and need a guarantee that the exact same code is used on all machines (dev, ci and production).
When developing a gem, use the gemspec method in your Gemfile to avoid duplication. In general, a gem’s Gemfile should contain the Rubygems source and a single gemspec line. Do not check your Gemfile.lock into version control, since it enforces precision that does not exist in the gem command, which is used to install gems in practice. Even if the precision could be enforced, you wouldn’t want it, since it would prevent people from using your library with versions of its dependencies that are different from the ones you used to develop the gem.
When developing an app, check in your Gemfile.lock, since you will use the bundler tool across all machines, and the precision enforced by bundler is extremely desirable for applications.
Since my last post on this topic, a lot of people have asked follow-up questions about the specific roles of the Gemfile, used by bundler, and the .gemspec, used by Rubygems. In short, people seem to feel that there is duplication between the two files during gem development, and end up using yet another tool to reduce the duplication.
Ruby Libraries
Ruby libraries are typically packaged up using the gem format, and distributed using rubygems.org. A gem contains a number of useful pieces of information (this list is not exhaustive):
- A bunch of metadata, like name, version, description, summary, email address, etc.
- A list of all the files contained in the package.
- A list of executables that come with the gem and their location in the gem (usually bin)
- A list of directories that should be put on the Ruby load path (usually lib)
- A list of other Ruby libraries that this library needs in order to function (dependencies)
Only the last item, the list of dependencies, overlaps with the purpose of the Gemfile. When a gem lists a dependency, it lists a particular name and a range of version numbers. Importantly, it does not care where the dependency comes from. This makes it easy to internally mirror a Rubygems repository or use a hacked version of a dependency that you install to your system. In short, a Rubygems dependency is a symbolic name. It is not a link to a particular location on the Internet where the code can be found.
Among other things, this characteristic is responsible for the resilience of the overall system.
Also, gem authors choose a range of acceptable versions of their dependencies, and have to operate under the assumption that the versions of the dependencies that the author tested against may change before the gem is used in deployment. This is especially true about dependencies of dependencies.
Ruby Applications
A Ruby application (for instance, a Rails application), has a complex set of requirements that are generally tested against a precise set of third-party code. While the application author may not have cared much (up front), which version of nokogiri the application used, he does care a lot that the version used in development will remain the version used in production. Since the application author controls the deployment environment (unlike the gem author), he does not need to worry about the way that a third-party will use the code. He can insist on precision.
As a result, the application author will want to specify the precise gem servers that the deployment environment should use. This is in contrast to the gem author, who should prefer long-term resilience and mirroring to the brittleness that comes with a short-term guarantee.
The application author often also wants a way to tweak shared community gems for the particular project at hand (or just use the “edge” version of a gem which hasn’t yet been released). Bundler handles this problem by allowing application authors to override a particular gem and version with a git repository or a directory. Because the gem author has specified the gem’s dependencies as a name and version range, the application developer can choose to satisfy that dependency from an alternate source (in this case, git).
Now, the application developer must be able to demand that the gem comes from that particular source. As you can see, there are critical differences between the desires of the gem developer (which are focused around longevity and flexibility) and the app developer (which are focused around absolute guarantees that all of the code, including third-party code, remains precisely the same between environments).
Rubygems and Bundler
The Rubygems libraries, CLI and gemspec API are built around the needs of the gem author and the Rubygems ecosystem. In particular, the gemspec is a standard format for describing all of the information that gets packed with gems then deployed to rubygems.org. For the reasons described above, gems do not store any transient information about where to find dependencies.
On the other hand, bundler is built around the needs of the app developer. The Gemfile does not contain metadata, a list of files in the package, executables exposed by the package, or directories that should be put on the load path. Those things are outside the scope of bundler, which focuses on making it easy to guarantee that all of the same code (including third-party code) is used across machines. It does, however, contain a list of dependencies, including information about where to find them.
Git “Gems”
For the reasons described above, it can sometimes be useful for application developers to declare a dependency on a gem which has not yet been released. When bundler fetches that gem, it uses the .gemspec file found in the git repository’s root to extract its metadata as well as discover additional dependencies declared by the in-development gem. This means that git gems can seamlessly be used by bundler, because their .gemspec allows them to look exactly like normal gems (including, importantly, having more dependencies of its own). The only difference is that the application developer has told bundler exactly where to look for the gem in question.
Gem Development
Active development on gems falls in between these two views of the world, for two reasons:
- Active development on a gem often involves dependencies that have not yet been released to rubygems.org, so it becomes important to specify the precise location to find those dependencies, during development
- During gem development, you cannot rely on rubygems to set up an environment with everything on the load path. Instead, you need a hybrid environment, with one gem coming from the file system (the current gem under development), some gems possibly coming from git (where you depend on a gem that hasn’t yet been released), and yet additional gems coming from traditional sources. This is the situation that Rails development is in.
Because gems under active development will eventually become regular gems, they will need to declare all of the regular metadata needed by gem build. This information should be declared in a .gemspec, since the Gemfile is an API for declaring dependencies only.
In order to make it easy to use the bundler tool for gem development without duplicating the list of dependencies in both the .gemspec and the Gemfile, bundler has a single directive that you can use in your Gemfile (documented on the bundler site):
source "http://www.rubygems.org"
gemspecThe gemspec line tells bundler that it can fine a .gemspec file alongside the Gemfile. When you run bundle install, bundler will find the .gemspec and treat the local directory as a local, unpacked gem. It will find and resolve the dependencies listed in the .gemspec. Bundler’s runtime will add the load paths listed in the .gemspec to the load path, as well as the load paths of any of the gems listed as dependencies (and so on). So you get to use bundler while developing a gem with no duplication.
If you find that you need to develop against a gem that hasn’t yet been released (for instance, Rails often develops against unreleased Rack, Arel, or Mail gems), you can add individual lines in the Gemfile that tell bundler where to find the gems. It will still use the dependencies listed in the .gemspec for dependency resolution, but it now knows exactly where to find the dependency during gem development.
source "http://www.rubygems.org" gemspec # if the .gemspec in this git repo doesn't match the version required by this # gem's .gemspec, bundler will print an error gem "rack", :git => "git://github.com/rack/rack.git"
You wouldn’t want to include this information in the .gemspec, which will eventually be released to Rubygems after the in-development gem has also been released. Again, this makes the overall system more resilient, as it doesn’t depend on transient external URLs. This information is purely something used to set up a complete environment during development, which requires some precision.
This is also why we recommend that people do not check in their Gemfile.lock files in gem development. That file is emitted by bundler, and guarantees that all gems, including dependencies of dependencies, remain the same. However, when doing gem development, you want to know immediately when some change in the overall ecosystem breaks your setup. While you may want to insist on using a particular gem from its git location, you do not want to hardcode your development to a very specific set of gems, only to find out later that a gem released after you ran bundle install, but compatible with your version range, doesn’t work with your code.
It’s not a way to guarantee that your code works on all versions that you specified in your .gemspec, but the incentive to lock down the exact versions of each gem simply doesn’t exist when the users of your code will get it via gem install.
Update
Just to clarify, you should check in your Gemfile.lock in applications, and not in gems. The Gemfile.lock remembers the exact versions and sources of every piece of third-party code that you use. For applications, you want this. When developing a gem, this can obscure issues that will occur because gems are deployed (using the gem command) without the benefit of bundler.
Ruby 2.0 Refinements in Practice
November 30th, 2010
First Shugo announced them at RubyKaigi. Then Matz showed some improved syntax at RubyConf. But what are refinements all about, and what would they be used for?
The first thing you need to understand is that the purpose of refinements in Ruby 2.0 is to make monkey-patching safer. Specifically, the goal is to make it possible to extend core classes, but to limit the effect of those extensions to a particular area of code. Since the purpose of this feature is make monkey-patching safer, let’s take a look at a dangerous case of monkey-patching and see how this new feature would improve the situation.
A few months ago, I encountered a problem where some accidental monkey-patches in Right::AWS conflicted with Rails’ own monkey-patches. In particular, here is their code:
unless defined? ActiveSupport::CoreExtensions class String #:nodoc: def camelize() self.dup.split(/_/).map{ |word| word.capitalize }.join('') end end end
Essentially, Right::AWS is trying to make a few extensions available to itself, but only if they were not defined by Rails. In that case, they assume that the Rails version of the extension will suffice. They did this quite some time ago, so these extensions represent a pretty old version of Rails. They assume (without any real basis), that every future version of ActiveSupport will return an expected vaue from camelize.
Unfortunately, Rails 3 changed the internal organization of ActiveSupport, and removed the constant name ActiveSupport::CoreExtensions. As a result, these monkey-patches got activated. Let’s take a look at what the Rails 3 version of the camelize helper looks like:
class String def camelize(first_letter = :upper) case first_letter when :upper then ActiveSupport::Inflector.camelize(self, true) when :lower then ActiveSupport::Inflector.camelize(self, false) end end end module ActiveSupport module Inflector extend self def camelize(lower_case_and_underscored_word, first_letter_in_uppercase = true) if first_letter_in_uppercase lower_case_and_underscored_word.to_s.gsub(/\/(.?)/) { "::#{$1.upcase}" }.gsub(/(?:^|_)(.)/) { $1.upcase } else lower_case_and_underscored_word.to_s[0].chr.downcase + camelize(lower_case_and_underscored_word)[1..-1] end end end end
There are a few differences here, but the most important one is that in Rails, “foo/bar” becomes “Foo::Bar”. The Right::AWS version converts that same input into “Foo/bar”.
Now here’s the wrinkle. The Rails router uses camelize to convert controller paths like “admin/posts” to “Admin::Posts”. Because Right::AWS overrides camelize with this (slightly) incompatible implementation, the Rails router ends up trying to find an “Admin/posts” constant, which Ruby correctly complains isn’t a valid constant name. While situations like this are rare, it’s mostly because of an extremely diligent library community, and a general eschewing of applying these kinds of monkey-patches in library code. In general, Right::AWS should have done something like Right::Utils.camelize in their code to avoid this problem.
Refinements allow us to make these kinds of aesthetically pleasing extensions for our own code with the guarantee that they will not affect any other Ruby code.
First, instead of directly reopening the String class, we would create a refinement in the ActiveSupport module:
module ActiveSupport refine String do def camelize(first_letter = :upper) case first_letter when :upper then ActiveSupport::Inflector.camelize(self, true) when :lower then ActiveSupport::Inflector.camelize(self, false) end end end end
What we have done here is define a String refinement that we can activate elsewhere with the using method. Let’s use the refinement in the router:
module ActionDispatch module Routing class RouteSet using ActiveSupport def controller_reference(controller_param) unless controller = @controllers[controller_param] controller_name = "#{controller_param.camelize}Controller" controller = @controllers[controller_param] = ActiveSupport::Dependencies.ref(controller_name) end controller.get end end end end
It’s important to note that the refinement only applies to methods physically inside the same block. It will not apply to other methods in ActionDispatch::Routing::RouteSet defined in a different block. This means that we can use different refinements for different groups of methods in the same class, by defining the methods in different class blocks, each with their own refinements. So if I reopened the RouteSet class somewhere else:
module ActionDispatch module Routing class RouteSet using RouterExtensions # I can define a special version of camelize that will be used # only in methods defined in this physical block def route_name(name) name.camelize end end end end
Getting back to the real-life example, even though Right::AWS created a global version of camelize, the ActiveSupport version (applied via using ActiveSupport) will be used. This means that we are guaranteed that our code (and only our code) uses the special version of camelize.
It’s also important to note that only explicit calls to camelize in the physical block will use the special version. For example, let’s imagine that some library defines a global method called constantize, and uses a camelize refinement:
module Protection refine String do def camelize() self.dup.split(/_/).map{ |word| word.capitalize }.join('') end end end class String #:nodoc: using Protection def constantize Object.module_eval("::#{camelize}", __FILE__, __LINE__) end end
Calling String#constantize anywhere will internally call the String#camelize from the Protection refinement to do some of its work. Now let’s say we create a String refinement with an unusual camelize method:
module Wycats refine String do def camelize result = dup.split(/_/).map(&:capitalize).join "_#{result}_" end end end module Factory using Wycats def self.create(class_name, string) klass = class_name.constantize klass.new(string.camelize) end end class Person def initialize(string) @string = string end end Factory.create("Person", "wycats")
Here, the Wycats refinement should not leak into the call to constantize. If it did, it would mean that any call into any method could leak a refinement into that method, which is the opposite of the purpose of the feature. Once you realize that refinements apply lexically, they create a very orderly, easy to understand way to apply targeted monkey patches to an area of code.
In my opinion, the most important feature of refinements is that you can see the refinements that apply to a chunk of code (delineated by a physical class body). This allows you to be sure that the changes you are making only apply where you want them to apply, and makes refinements a real solution to the general problem of wanting aesthetically pleasing extensions with the guarantee that you can’t break other code. In addition, refinements protect diligent library authors even when other library authors (or app developers) make global changes, which makes it possible to use the feature without system-wide adoption. I, for one, am looking forward to it.
Postscript
There is one exception to the lexical rule, which is that refinements are inherited from the calling scope when using instance_eval. This actually gives rise to some really nice possibilities, which I will explore in my next post.
What’s Wrong with “HTML5″
October 7th, 2010
In the past year or so, the term “HTML5″ has increasingly been picked up by the tech press as the successor to “DHTML”, “Web 2.0″ or “Ajax”. When used by the tech press, it is becoming a generic term for “the next generation of web technology”, except that the term “HTML5″ is less precise than even that.
Consider the first paragraph of an article about HTML published this week:
HTML5 is the hot topic nowadays. Everyone from Apple to Google and everyone else in between have shown their support for the standard. Word has it that HTML5 is the Adobe Flash-killer. It seems that the World Wide Web Consortium [W3C] — which is the main international standards organization for the World Wide Web — doesn’t agree. If anything, the W3C doesn’t think that HTML5 is “ready for production yet.”
…
The problem with HTML5 appears to be that it currently lacks a video codec. In addition, digital rights management [DRM] is also not supported in HTML5, which obviously makes a problem for various companies.
My engineer friends have all but given up on pushing back against the “HTML5″ moniker. After all, it’s just a term for the tech press. Everyone who knows anything understands that it means just as little as requests for “Ajaxy animations” a few years back, right? I had started to agree with this line of reasoning. It’s true: there’s no point in being pedantic on this front for the sake of being pedantic.
Unfortunately, the term, and therefore the way the technology is understood by tech writers, is causing some fairly serious problems.
First, keep in mind that unlike “Ajax” or “Web 2.0″, which were pretty benign, vague terms, HTML5 sounds like a technology. It can have “beta” versions, and one day it will be “complete” and “ready for production”. Take a look at this snippet from an InfoWorld article:
His advice on HTML5 was endorsed by industry analyst Al Hilwa of IDC.
“HTML 5 is at various stages of implementation right now through the Web browsers. If you look at the various browsers, most of the aggressive implementations are in the beta versions,” Hilwa said. “IE9 (Internet Explorer 9), for example, is not expected to go production until close to mid-next year. That is the point when most enterprises will begin to consider adopting this new generation of browsers.”
And because HTML5 is portrayed as a “Flash killer”, whether it is “complete” or in “beta” sounds really relevant. In comparison, “Web 2.0″ was never portrayed as a technology, but rather the ushering in of a new era of web technologies which would allow people to build new kinds of applications.
The truth is, the “completion” of HTML5 is absolutely irrelevant. At some point, the W3C will approve the HTML5 spec, which will mean nothing about the overall availability of individual features. At some point, the other related specs (like Web Storage and Web Sockets) will be approved, and also mean nothing about the overall availability of individual features.
In response to this conundrum, most of my friends immediately throw out the idea of getting more specific with the tech press. And they’re right. The tech press is not going to understand Web Sockets or local storage. They’re having trouble even grasping HTML5 video.
The thing is, the core problem isn’t that the name is too fuzzy. It’s what the name implies. HTML5 sounds like a technology which goes through a beta period and is finally complete. Instead, what the tech press calls “HTML5″ is really a continual process of improving the web browsers that people use. And honestly, that’s how I’d like to see the tech press cover us. Not as group of people working towards a singular milestone that will change the web as we know it, but as a group that has gotten our groove back.
Tech reporters: please stop talking about the current state of web technologies as an event (the approval of the HTML5 spec). There are interesting stories happening all the time, like Scribd, YouTube, and Gmail leveraging newer web technologies to improve their products. Little guys are doing the same every day. Not everything is about whether “Flash is dead yet”. It’s worth talking about the tradeoffs that guys like Hulu make, which prevent them from moving to web technologies. But make no mistake: large swaths of the web will be using next-generation browser features well before the last guy does. The process of getting there is an interesting story, and one you should be covering.
Bundler: As Simple as What You Did Before
September 30th, 2010
One thing we hear a lot by people who start to use bundler is that the workflow is more complicated than it used to be when they first start. Here’s one (anonymized) example: “Trying out Bundler to package my gems. Still of the opinion its over-complicating a relatively simple concept, I just want to install gems.”
Bundler has a lot of advanced features, and it’s definitely possible to model fairly complex workflows. However, we designed the simple case to be extremely simple, and to usually be even less work than what you did before. The problem often comes when trying to handle a slightly off-the-path problem, and using a much more complex solution than you need to. This can make *everything* much more complicated than it needs to be.
In this post, I’ll walk through the bundler happy path, and show some design decisions we made to keep things moving as smoothly as before, but with far fewer snags and problems than the approach you were using before. I should be clear that there are probably bugs in some cases when using a number of advanced features together, and we should fix those bugs. However, they don’t reflect core design decisions of bundler.
In the Beginning
When you create a Rails application for the first time, the Rails installer creates a Gemfile for you. You’ll note that you don’t actually have to run bundle install before starting the Rails server.
$ gem install rails Successfully installed activesupport-3.0.0 Successfully installed builder-2.1.2 Successfully installed i18n-0.4.1 Successfully installed activemodel-3.0.0 Successfully installed rack-1.2.1 Successfully installed rack-test-0.5.6 Successfully installed rack-mount-0.6.13 Successfully installed tzinfo-0.3.23 Successfully installed abstract-1.0.0 Successfully installed erubis-2.6.6 Successfully installed actionpack-3.0.0 Successfully installed arel-1.0.1 Successfully installed activerecord-3.0.0 Successfully installed activeresource-3.0.0 Successfully installed mime-types-1.16 Successfully installed polyglot-0.3.1 Successfully installed treetop-1.4.8 Successfully installed mail-2.2.6.1 Successfully installed actionmailer-3.0.0 Successfully installed thor-0.14.2 Successfully installed railties-3.0.0 Successfully installed rails-3.0.0 22 gems installed $ rails s => Booting WEBrick => Rails 3.0.0 application starting in development on http://0.0.0.0:3000 => Call with -d to detach => Ctrl-C to shutdown server [2010-09-30 12:25:51] INFO WEBrick 1.3.1 [2010-09-30 12:25:51] INFO ruby 1.9.2 (2010-08-18) [x86_64-darwin10.4.0] [2010-09-30 12:25:51] INFO WEBrick::HTTPServer#start: pid=26380 port=3000
If you take a look at your directory, you’ll see that Bundler noticed that you didn’t yet have a Gemfile.lock, saw that you had all the gems that you needed already in your system, and created a Gemfile.lock for you. If you didn’t have a necessary gem, the Rails server would error out. For instance, if you were missing Erubis, you’d get Could not find RubyGem erubis (~> 2.6.6). In this case, you’d run bundle install to install the gems.
As you can see, while bundler is providing some added features for you. If you take your application to a new development or production machine with the Gemfile.lock, you will always use exactly the same gems. Additionally, bundler will resolve all your dependencies at once, completely eliminating the problem exemplified by this thread on the Rails core mailing list. But you didn’t have to run bundle install or any bundle command unless you were missing some needed gems. This makes the core experience of bundler the same as “I just want to install gems”, while adding some new benefits.
Adding Gems
When you add a gem that is already installed on your system, you just add it to the Gemfile and start the server. Bundler will automatically pick it up and add it to the Gemfile.lock. Here again, you don’t need to run bundle install. To install the gem you added, you can run bundle install or you can even just use gem install to install the missing gems.
Once again, while bundler is handling a lot for you behind the scenes (ensuring compatibility of gems, tracking dependencies across machines), if you have all the gems that you need (and specified in your Gemfile) already installed on your system, no additional commands are required. If you don’t, a simple bundle install will get you up to date.
Deployment
After developing your application, you will probably need to deploy it. With bundler, if you are deploying your application with capistrano, you just add require "bundler/capistrano" to your deploy.rb. This will automatically install the gems in your Gemfile, using deployment-friendly settings. That’s it!
Before bundler, you had two options:
- Most commonly, “make sure that all the gems I need are on the remote system”. This could involve including the
gem installin the capistrano task, or evensshing into the remote system to install the needed gems. This solution would work, but would often leave no trail about what exactly happened. This was especially problematic for applications with a lot of dependencies, or applications with sporadic maintenance - Using the GemInstaller gem, which allowed you to list the gems you were using, and then ran
gem installon each of the gems. Heroku used a similar approach, with a.gemsmanifest. While this solved the problem of top-level dependencies, it did not lock in dependencies of dependencies. This caused extremely common problems withrackand activesupport, and a long tail of other issues. One particularly egregious example is DreamHost, which directs users to use a brittle patch to Rails
In both of these cases, there was no guarantee that the gems you were using in development (including dependencies of dependencies like Rack and Active Support) remained the same after a deploy. One recent example that I saw was that carrierwave added an updated dependency on the newest activesupport, which depends on Ruby 1.8.7. Without making any change to his .gems manifest, Heroku failed to deploy his application on their Ruby 1.8.6 stack.
With bundler, this sort of problem can never happen, because you’re always running exactly the same versions of third-party code in production as you are in development and staging. This should increase your confidence in deployments, especially when you come back to an application months after it was originally developed.
Tweaked Gems
Some times, you will need to make small changes to one of your dependencies, and deploy your application with those changes. Of course, in some cases you can make the tweaks via monkey-patching, but in other cases, the only real solution is to change the gem itself.
With bundler, you can simply fork the gem on Github, make your changes, and change the gem line in your Gemfile to include a reference to the git repository, like this:
# change this gem "devise" # to this gem "devise", :git => "git://github.com/plataformatec/devise.git"
This behaves exactly the same as a gem version, but uses your tweaks from Github. It participates in dependency resolution just like a regular gem, and can be installed with bundle install just like a regular gem. Once you’ve added a git repository, you run bundle install to install it (since the normal gem command can’t install git repositories).
You can use :branch, :tag, or :ref flags to specify a particular branch, tag or revision. By default, :git gems use the master branch. If you use master or a :branch, you can update the gem by running bundle update gem_name. This will check out the latest revision of the branch in question.
Conclusion
Using bundler for the sorts of things that you would have handled manually before should be easier than before. Bundler will handle almost all of the automatable record-keeping for you. While doing so, it offers the incredible guarantee that you will always run the same versions of third-party code.
There are a lot of advanced features of Bundler, and you can learn a lot more about them at the bundler web site.
Here’s to the Next 3 Years
September 14th, 2010
I officially joined Engine Yard on January 1, 2008, about a week before we announced our Series A funding, becoming its twenty-second employee. I was Engine Yard’s very first “Engineering” hire, and I would spend the next year working on Ezra’s Merb project, finally releasing Merb 1.0 at MerbCamp that October.
When I joined Engine Yard, I had already been working on the jQuery project for a couple years, having started visualjquery.com at a time in jQuery’s history before jQuery had version numbers. I learned a lot from John Resig about building and motivating open source communities, and I did my first professional work writing jQuery in Action.
The Merb project was my first full-time open source work, and I met some of my earliest open source friends while working on it. When I first met Carl, he was working on a rewrite of the Merb router, the one part of Merb I was still scared to work on. I met Andy Delcambre, who now works on Engine Yard’s AppCloud, after he volunteered to help with Merb.
I met Loren Segal, well-known for the YARD documentation tool, while working to give Merb documentation I could be proud of. And I became the first Github user outside of Chris, PJ and Tom in order to get the Merb project on git. If you’re paying attention, that made me Github user nil.id, and was the source of some hilarious bugs.
In Hampton Catlin’s first Ruby survey in 2008, about one in eight respondents said that they preferred Merb, and by the end of the year, the Rails and Merb teams were coming to blows. During this same time, Engine Yard had taken on a Series B round of funding (this time for $15 million), and had grown from a small company of a couple dozen to a company pushing 100. We had Sales and Marketing departments. We also had an Engineering department, which had spent a large part of the year on our first fully automated product, Engine Yard Solo.
On a personal level, I started getting heavily involved in Open Source other than Merb, especially DataMapper and Rubinius. I started speaking at conferences, starting with a talk on DataMapper at MountainWest RubyConf in 2008. Engine Yard also threw its first Hackfest at that event, an event we would repeat in 2009 and again in 2010. This event, and the work our great community marketing, Open Source and support teams would do over the next several years elevated Engine Yard’s reputation as a team of experts that would keep your app running no matter what.
As Engine Yard grew, a lot of people asked us why we were supporting Merb when so much of our business depended on the success of the Rails community. By late 2008, it was probably the most common question I got when speaking at conferences. In truth, the best I could tell was that Engine Yard was worried about the future of Rails, and wanted to make sure that Ruby remained successful. What we didn’t realize at the time was that the one full-time employee (me) that Engine Yard had on Merb was more than the total number of worldwide full-time developers on Rails. In short, Rails was a fully volunteer project, and that had something to do with some of the issues that drove people to Merb in the first place.
At the end of 2008, Tom Mornini, the CTO of Engine Yard, recognized that if we could join forces (and this was definitely a big if), we could help shore up Rails into the future. I remember the first time Tom asked me about it in the stairwell of our South Park office. I thought he was off the wall to even suggest it. After all, it didn’t seem like the teams could get along at all. But the Merb team had adopted so much of the Rails philosophy that at its core, the only question that remained was how aggressive the Rails project would get to clean things up.
In the years since Ezra first announced Merb, Rails had gotten thread-safe, Rack support, and some improvements to the internals. After hashing out the details (in some very contentious conversations), we decided to give it a try, announcing we would merge the projects on December 23, 2008. That announcement quite literally changed the trajectory of my life. In the almost two years since that announcement, I have worked harder than I have ever worked before, and am more proud of the work I have done than I have ever been before.
Perhaps more importantly, I have worked with some of the best open source developers in the business without whom Rails 3 would have been a pale shadow of itself. Including me, Rails 3 brought on eight new committers. The Rails 3 project did more than improve the quality of the codebase: it reinvigorated the community of developers actively contributing. And by moving toward a model that allowed Rails to have more dependencies, Rails became the center of a thriving Ruby community, where anyone can reuse the code that makes Rails tick for their non-Rails projects.
In the summer of 2009, José Valim became a Google Summer of Code student dedicated to rewriting the Rails generators so that users of DataMapper, RSpec, and Haml could use the standard Rails generators. A lot of people thought it was too optimistic to do in a summer, but José finished the job well before the end of the summer, giving him time to continue his contributions.
Around the same time, Mikel Lindsaar decided to write a ground-up mail implementation, releasing the imaginatively named mail gem in October 2009. This project, which involved writing a parser that could parse the gigabytes of mail in the Trec and Enron email databases, strained credulity, and would enable us to give ActionMailer the facelift it needed for Rails 3.0.
Santiago Pastorino showed up out of nowhere, with a level of pluck that is rarely seen in the Ruby community. Despite a noticeable language barrier, Santiago went from a warning fixer to a Rails committer. This month, he became an official member of the Rails core team. While working on Rails, I learned a lot from a lot of people, but Santiago’s raw determination and skill should stand out as an example to the Ruby community.
Everyone knows Aaron Patterson, and I was just as surprised as everyone else when he started the long, dedicated task of improving the performance of the Arel library. He already has quite the reputation in the Ruby community, and even has commit access to Ruby itself, so I’m personally gratified to work with him on Rails. If you haven’t yet watched it, watch his Worst Ideas Ever presentation from RubyConf last year. Be prepared to laugh.
Xavier Noria has been a one-man documentation machine, helping to grow and maintain the Rails guides, and constantly reminds everyone that documentation is just as essential to the codebase as tests. Without him, Rails 3 might have been great internally, but nobody would know it. Documentation folks often get little love in open source projects, but to me, the work Xavier does is key to our success.
More recently, I’ve been extremely impressed by Piotr Sarnacki’s work (for Ruby Summer of Code) on finally delivering on the promise of truly mountable engines. His work is already merged into Rails master, and will ship with Rails 3.1. As with the rest of the guys I’ve talked about, he took on a pretty complicated and entangled task, and managed to get more done in less time than people expected. Keep an eye on him!
In short, the Rails 3 project was about more than code; it was about reshaping the Rails community. I’m proud to have been a part of growing the Rails ecosystem, and humbled by the number of people who have worked so hard for so little.
Now that we released Rails 3, I have had an opportunity to think about what to do next. I could, of course, continue full-time work on Rails for Engine Yard, which has been a fantastic place to do Open Source for the past several years. I am still extremely interested in open web technologies, and want to do my part to help the open web succeed. Having robust, mature server side frameworks that are easy for anyone to use is certainly one large piece of the puzzle, and I’ve really enjoyed helping to bring the core of Rails up to date with modern web technologies.
But as Avi Bryant pointed out last year at DjangoCon, the technology of the open web is shifting toward the client side. That doesn’t change the need for robust, mature server-side frameworks. Even the richest web applications get and sync their data with servers, mostly over the HTTP protocol. And although a lot of people have opined that rich clients can simply wire directly into something like CouchDB or MongoDB, applications with rich clients tend to have the same kinds of complex needs (like authorization) as more tranditional document-based applications have. I see Rails remaining an excellent server-side solution even as clients become richer over the next few years.
There’s still a lot to do for Rails in the future, and I want to be a part of that. On the other hand, the field of rich client development is just getting starting, with new mobile devices like the iPhone, Android and iPad giving us an almost blank canvas to work on. Unfortunately, a lot of the really big bets in this space are on selling licenses to either entire libraries or the development tools needed to efficiently work with the tools.
What the open web needs is a robust, easy to use framework with development tools that are trivial to get started with and awesome to work with. And both the library and the developer tools need to be offered for free, like Rails or jQuery. Since 2008, I have poured my soul into helping to build an amazing open source Ruby web framework and the ecosystem that powers it. Now, I want to bring that same energy and enthusiasm to building great tools that leverage the power of the modern open web and a free and open source community around them.
That’s why I was so excited when I saw that Charles Jolley, who worked on Apple’s Mobile Me product, announced that he was leaving Apple to form a new company dedicated to those same principles. Charles had helped build Mobile Me on SproutCore, a free and open source framework, and had managed to make sure that Apple continued to contribute back to the open source project under the MIT license.
In his blog post announcing that he was leaving Apple, Charles said “SproutCore is now and will always be totally free and open source. I think this business of charging for a commercial license is not an effective way to grow a project. Sure you make a little cash, but at what expense to the community? My goal is to make SproutCore and all of the developer tools that surround it totally free to everyone. All I ask is that you participate in the community somehow to make things a little better for those who come after you.”
This philosophy closely mirrored my thinking on the future of client-side development, and seeing a mature client-side framework open a permissive open source philosophy really excited me. Since the beginning of 2010, I was starting to think that if I wanted a permissively licensed client-side toolkit, I was going to have to build it and its community myself. I already knew about SproutCore–early versions of its build tools were built on Merb, and had met Charles a few times. After reading his blog post, I emailed him to find out what he was thinking about doing and he told me about his plans for SproutCore.
The SproutCore library itself, today, is definitely recovering from having been developed almost exclusively inside Apple. It comes with a number of amazing low-level primitives, and you can build a large, complex application on it. That said, it never had a big enough new user community to get a very friendly public API. Thankfully, it’s much, much easier to build a great public API than it is to build the core of a web application toolkit. Starting this week, I will be joining Charles’ company, Strobe, and will get right to work on the next generation of the SproutCore tools.
I have to say, I’m excited about what comes next.
Postscript
Because you’re probably wondering, I will still be active in the Rails community, and will be doing work for Engine Yard on Rails 3.1. I plan to use Rails 3 and Engine Yard for the server-side elements of the projects I will be doing, and really want the new Rails 3.1 features I talked about at Windy City Rails. I will also be helping Engine Yard start a new Technical Advisory Board, which will help provide community feedback for its cloud products, and guidance about what open source efforts could use the Engine Yard bump. I forsee being part of both the Rails and Engine Yard family for years to come.
Announcing Handlebars.js
September 9th, 2010
For a number of years, I’ve been about as active in client-side development (through jQuery) as I have been active in server-side development (through Merb and then Rails). Recently, I’ve released a couple of libraries (jQuery Offline and Rack::Offline), designed to make it easier to build applications that can withstand both flaky connections and a complete lack of connectivity. Of course, those libraries leveraged features in modern browsers, and have gotten a bunch of use.
For jQuery Offline, the basic strategy is that instead of requesting a new JSON object before you can show anything on the page, you first try to retrieve that data from cache, show some stale content. In parallel, you’d download the new content (if possible), and update the display with the content in question. You’ve probably seen this pattern at work in applications like the official Twitter application for the iPhone, which shows stale content when you first boot the app, and downloads new content if you have Internet connectivity in the background.
To take the JSON and convert it into some HTML for the page, I recommended jquery-tmpl or mustache.js. Both of these libraries allow you to take a JSON object and convert it on the fly into HTML, which you can insert into the page. Both libraries are still very good, and I could easily imagine problem-free uses for both of them.
Of the two libraries, I really liked Mustache’s approach. Mustache envisions a template as something that is completely code-free, and populated via a JavaScript Object. This Object can contain normal values, or functions which are used as helpers.
<script type="text/x-mustache-template" name="posts"> {{#posts}} <h1><a href="{{url}}">{{title}}</a></h1> {{#div}} {{body}} {{/div}} {{/post}} </script>
You’d then populate this template with a JavaScript object:
var view = {posts: [ { id: 1, url: "/posts/1", title: "Hello world!", div: function(text, render) { return "<div class='post entry'>" + render(text) + "</div>" }, post: "Hello world. This is my first post" }, { id: 2, url: "/posts/2", title: "Goodbye world!", div: function(text, render) { return "<div class='post entry'>" + render(text) + "</div>" }, post: "Goodbye world. Sadly, this is going to be my last post" } ]}; var template = $("script[name=posts]").html(); var html = Mustache.to_html(template, view)
The basic idea here is that the template itself is completely free of code, and just uses keys from the JavaScript Object to convert it into HTML. The JavaScript Object can have some functionality, and even code, to help the template along. I really like the idea of limiting the template to referencing an element of the JavaScript object, and completely firewalling any kind of code to the view object. Among other things, this allows a single template to run in both Ruby on the server-side and JavaScript on the client side.
When using Mustache, I ran into a few issues. One minor issue was that the base implementation of mustache.js was essentially interpreted, rather than compiled. This means that the implementation is slower than you might expect of a template language, especially one as limited as Mustache.
More concerningly, the base Mustache syntax was too limited for a lot of things I wanted to use it for. I wasn’t immediately sure if it was just me, so I asked a bunch of people about their opinions of Mustache. One of the people who replied was Alan Johnson, who had been using Mustache a lot and volunteered his thoughts. Without much prompting, Alan’s concerns about the limitations of Mustache (in both our cases, born from real attempts to use it) closely mirrored mine. My thinking validated for the moment, I got to work trying to extend Mustache to get around these limitations.
Before I continue, here are some of the more important problems I had:
- Function values (like
divabove) needed to be located in the same place in the hierarchy as object being used as a context. This made providing a set of global helpers (like a framework would want to do) quite difficult. - Even if the syntax was modified to make function helpers global, the functions would not have access to the current context, so they would be of limited value (imagine trying to change the url key above into a helper)
- I commonly wanted access to a key in a parent scope when working in a child scope. For instance, if a post had many comments, I would sometimes want access to the post’s title (or other details) while rendering the child
There were other small irritations, but these were definitely the major ones. A lot of the people I spoke to solved these problems by rewriting the JavaScript Object before passing it into the Mustache system. Some of my initial thinking, in fact, revolved around making that sort of rewriting easier. In the end, though, I wanted to extend the Mustache syntax with a few new features that would make it behave more like what I expected.
From the beginning, I wanted my extensions to be a strict superset of the official Mustache manual, which is pretty clear in some areas and somewhat more vague in other areas. I really like the Mustache syntax and philosophy, and wanted to make sure whatever I came up with would not diverge philosophically from what I enjoyed about Mustache itself.
Compiled, Not Interpreted
The first step was to make the Mustache syntax compiled, not interpreted. There is another project (called Mu) which is a compiled variant of Mustache, but it was explicitly built for Node.js, and is built around server-side JavaScript, rather than client-side JavaScript. I also wanted to build it with some of the extensions I wanted to add in mind from the beginning.
The parser/compiler is a lot like the ERB compiler, and has a lot of what you’d expect from a parser/compiler (getChar, peek, addText, addExpression, etc.). I designed it from the beginning to be extended, taking some of the architecture from Erubis, which separates out how different kinds of tags are added to the source it is building up. We’ll probably have to make a few tweaks before extensions can be easily dropped in, but I already have a couple in mind that will help flesh out the API.
Extended Paths
The first extension to Mustache is “extended paths”. In Mustache, paths are always relative to the current context (which can change with iteration), and are always a single key. For instance, the url path from the above example references the url key in the current post that Mustache is iterating over.
Extended paths allow you to go deeper into the Object (byline/name) or up the hierarchy (../title). This doesn’t diverge from the core Mustache philosophy of avoiding code in templates, but it does allow templates to more flexibly reference other parts of the Object, apart from the context they are in the process of rendering. A lot of people implement this feature by rewriting the Object before passing it into the template–Handlebars allows that rewriting to happen on the fly.
Global Lambdas with Access to the Current Context
Mustache allows the Object to contain lambdas, but they must appear in the same context as the rest of the keys being rendered. If iterating over a list of projects, for instance, each project must have a key referencing the lambda, even if it’s always the same. Handlebars allows you to provide a global Object of lambdas, which it will look in when it does not find a key in the current context. This allows you to create helpers to use throughout your Template. Again, people implemented this feature by rewriting the Object, and Handlebars is just making it possible to achieve the same goal without that rewriting.
After adding global lambdas, those helpers might want to do different things based on the context. Handlebars invokes its helper functions with a this that reflects the current context. We can rewrite the above example as:
var view = {posts: [ { id: 1, title: "Hello world!", div: function(text, render) { return "<div class='post entry'>" + render(text) + "</div>" }, post: "Hello world. This is my first post" }, { id: 2, title: "Goodbye world!", div: function(text, render) { return "<div class='post entry'>" + render(text) + "</div>" }, post: "Goodbye world. Sadly, this is going to be my last post" } ]}; var helpers = { url: function() { return "/posts/" + this.id; } }
Of course, this example is pretty simple, and you could have just done {{title}}, but this illustrates that helpers can use the current context to control the output.
Helpers Take a Context
Helpers could be made even more generic if they could take a context (referenced as a path) as an argument. As I’ve shown above, helpers are always executed with their current rendering context as this, but it could be convenient to pass a different context, referenced by its path, to the function.
{{#projects}}
<p>Back: {{url ..}}</p>
<div>
{{body}}
</div>
{{/project}}In this example, we want each project to include a link back to the parent project. By passing in a context (in this case, the object referenced by the extended path ..) we can make our helpers even more generic.
var helpers = { url: function(context) { return "/" + context.type + "/" + context.id; } }
Block Helpers
The Mustache manual is a bit light on how lambda functions should work, referencing a render method that it doesn’t pass into the function. I wanted block helpers to take a function that was essentially identical to the compiled function for the entire template, and would be populated in the same way a regular template gets populated. A good example use for this type of helper is a custom list.
<script type="text/x-handlebars-template" name="items"> {{#list items}} <h1>{{title}}</h1> <div>{{body}}</div> {{/list}} </script>
var view = { items: [ { title: "Hello world", body: "This is my first post" }, { title: "Goodbye cruel world", body: "This will be my last post ;(" } ] } var helpers = { list: function(context, fn) { var buffer = "<ul class='list'>\n"; for(var i=0, j=context.length; i < j; i++) { buffer.push( "<li>" + fn(context[i]) + "</li>\n" ) } buffer.push("</ul>"); } } var source = $("script[name=items]").html(); var template = Handlebars.compile(source); template(view, helpers)
This will output:
<ul class='list'> <li><h1>Hello world</h1> <div>This is my first post</div></li> <li><h1>Goodbye world</h1> <div>This will be my last post ;(</div></li> </ul>
As you can see, block helpers can now invoke the sub-template many times with different contexts. And block helpers, like regular helpers, can receive a context, expressed by an extended path, as a parameter.
Iteration and Booleans Are Just Block Helpers
One thing that bugged me was that iteration and booleans were special-cased in the normal Mustache specification. I didn’t want to change the behavior of iteration or booleans, but I did want to make them behave the same as regular block helpers. By making block helpers more powerful, I was able to achieve this with one small additional change.
In particular, rather than hardcode the behavior of iteration and booleans, I created a new feature allowing a particular function to be used when a block helper could not be found. This way, every use of {{#foo}} was a block helper, but iteration was just invoking the special helperMissing block helper. In both this case and the case of conditionals, the missing helper was treated as a context to be looked up as a path.
The helperMissing function itself is quite simple:
Handlebars.helperMissing = function(object, fn) { var ret = ""; if(object === true) { return fn(this); } else if(object === false) { return ""; } else if(toString.call(object) === "[object Array]") { for(var i=0, j=object.length; i < j; i++) { ret = ret + fn(object[i]); } return ret; } else { return fn(object); } };
First, if the object referenced is true, we invoke the template block, using the current context. If the object referenced is false, we return an empty String.
If the object referenced is an Array, we iterate over the elements of the Array, invoking the template block once in the context of each element of the Array. This looks a lot like the items block helper we created above.
Finally, if the object referenced is anything else, we invoke the template block in the context of that object. Because we made block helpers more powerful, it is easy to implement the hardcoded boolean, iteration and context-switching behavior of the Mustache spec in terms of the block helpers. Note that if you wanted to change this behavior, you would just need to override Handlebars.helperMissing with a new function of your own design.
Better Inverted Sections
Finally, Mustache allows the specification of inverted sections, which behave like the reverse of a block. For instance, where {{#boolean}} will run the block if boolean is true (or a non-falsy value), and skip it if boolean is false, {{^boolean}} will run the block if boolean is false, and skip it if boolean is true (or non-falsy value).
A very common usage of inverted sections is to behave like if/else blocks:
{{#project}}
<h1>{{name}}</h1>
<div>{{body}}</div>
{{/project}}
{{^project}}
<h1>No projects</h1>
{{/project}}Because of how common this is, Handlebars has a shortened syntax for this case:
{{#project}}
<h1>{{name}}</h1>
<div>{{body}}</div>
{{^}}
<h1>No Projects</h1>
{{/project}}Additionally, you can specify the inverse form of a custom block helper by assigning a not property on the helper function.
{{#list projects}}
<h1>{{name}}</h1>
<div>{{body}}</div>
{{^}}
<h1>No Projects</h1>
{{/project}}var helpers = { list: function(items, fn) { if(!items.length) return ""; var buffer = "<ul>"; for(var i=0, j=items.length; i < j; i++) { buffer += fn(items[i]); } buffer += "</ul>" return buffer; } } helpers.list.not = function(items, fn) { if(!items.length) { return fn(); } }
Wrapping Up
Long story short: the goal of handlebars.js is to create a syntax-compatible superset of Mustache that solves a lot of problems I’ve had when working with Mustache (especially as part of a larger library).
I want to personally thank Alan Johnson for contacting me early and fleshing out my initial prototype into a workable, releasable library. His experience with Mustache, as well as his willingness to spend time on the nitty gritty details really made the difference. Thanks!
As always, please submit bug reports to the GitHub Issues tracker associated with the project. I really love the new GitHub pull request system, so please submit patches using it.
Automatic Flushing: The Rails 3.1 Plan
September 7th, 2010
preamble: this post explains, in some detail, how we will implement a nice performance boost for Rails developers. Understanding the details might help gain the full benefits of the optimization, but you will gain some benefits even if you have no idea how it works.
As you’ve probably seen, DHH announced that we’d be looking at flushing in Rails 3.1 to improve the client-side performance of typical Rails applications.
The most obvious solution, and one that already exists in plugin form, is to allow a layout to have a new flush method, which would immediately flush the contents of the layout to the browser. By putting the flush method below the JavaScript and CSS includes, the browser could begin downloading and evaluating those static assets while the server continues building the page.
Unfortunately, this solution has a major problem: it requires a fairly significant change in the current model of how people build applications. In general, for performance optimizations (including client-side optimizations), we like to make the default as fast as possible, without asking people to understand a brand new paradigm, centered around the optimization.
The problem lies in the fact that a Rails layout is essentially a template with a bunch of holes to fill in.
<html>
<head>
<title><%= yield :title %></title>
<%= javascript_include_tag :defaults %>
<%= yield :extra_javascripts %>
<%= stylesheet_link_tag :defaults %>
<%= yield :extra_stylesheets %>
</head>
<body>
<%= yield :sidebar %>
<%= yield %>
</body>
</html>I this simple example, each yield is a slot that is filled in by the template (usually via content_for). In order to achieve this, Rails evaluates the template first, which populates a Hash with each piece of content. Next, it renders the layout, and each yield checks the Hash for that content. In short, because of the way layouts work, Rails renders the template first, and then the layout.
To get around this, one option would be to say that everything before the flush must not use yield, and must be able to run before the template. Unfortunately, it’s somewhat common for people to set up a content_for(:javascripts) in a template, to keep the JavaScript needed for a particular snippet of HTML close to the HTML. This means that not only does the user have to be careful about what can go above and below the flush, he can no longer use content_for for things high up in the template, which is a fairly significant change to the overall design of Rails applications.
For Rails 3.1, we wanted a mostly-compatible solution with the same programmer benefits as the existing model, but with all the benefits of automatic flushing. After a number of very long discussions on the topic, José Valim came up with the idea of using Ruby 1.9 fibers to jump back and forth between the template and layout.
Let’s start by taking a look at a very simplified version of the current Rails rendering pipeline. First, we set up a Buffer object purely for logging purposes, so we can see what’s happening as we push things onto the buffer.
module Basic class Buffer < String def initialize(name, context) @name = name end def <<(value) super puts "#{@name} is pushing #{value.inspect}" end end end
Next, we create a simple version of ActionView::Base. We implement the content_for method simply, to print out a bit of logging information and stash the value into the @content_for Hash. Note that the real version is pretty similar, with some added logic for capturing the value of the block from ERB.
module Basic class ViewContext def initialize @buffer = Buffer.new(:main, self) @content_for = {} end def content_for(name, value = nil) value = yield if block_given? puts "Setting #{name} to #{value.inspect}" @content_for[name] = value end def read_content(name) @content_for[name] end end end
Next, we create a number of methods on the ViewContext that look like compiled ERB templates. In real life, the ERB (or Haml) compiler would define these methods.
module Basic class ViewContext def layout @buffer << "<html><head>" @buffer << yield(:javascripts).to_s @buffer << yield(:stylesheets).to_s @buffer << "</head><body>" @buffer << yield.to_s @buffer << yield(:not_existant).to_s @buffer << "</body></html>" @buffer end def template buffer = Buffer.new(:template, self) content_for(:javascripts) do "<script src='application.js'></script>" end content_for(:stylesheets) do "<link href='application.css' rel='stylesheet' />" end puts "Making a SQL call" sleep 1 # Emulate a slow SQL call buffer << "Hello world!" content_for(:body, buffer) end end end
Finally, we define the basic rendering logic:
module Basic class ViewContext def render template layout { |value| read_content(value || :body) } end end end
As you can see, we first render the template, which will fill up the @content_for Hash, and then call the layout method, with a block which pulls the value from that Hash. This is how yield :javascripts in a layout works.
Unfortunately, this means that the entire template must be rendered first, including the (fake) slow SQL query. We’d prefer to flush the buffer after the JavaScripts and CSS are determined, but before the SQL query is made. Unfortunately, that requires running half of the template method, then continuing with the layout method, retaining the ability to resume the template method later.
You can think of the way that templates are currently rendered (in Rails 2.x and 3.0) like this:
Unfortunately, this makes it very hard to get any more performance juice out without asking the end-developer to make some hard choices. The solution we came up with is to use Ruby 1.9 fibers to allow the rendering to jump back and forth between the template and layout.
Instead of starting with the template and only rendering the layout when ready, we’ll start with the layout, and jump over to the template when a yield is called. Once the content_for that piece is provided by the template, we can jump back to the layout, flush, and continue rendering. As we need more pieces, we can jump back and forth between the template and layout, flushing as we fill in the holes specified by the yield statements.
The implementation is mostly straight-forward:
require "fiber" module Fibered class ViewContext < Basic::ViewContext def initialize super @waiting_for = nil @fiber = nil end def content_for(name, value = nil) super @fiber.resume if @waiting_for == name end def read_content(name) content = super return content if content begin @waiting_for = name Fiber.yield ensure @waiting_for = nil end super end def layout @fiber = Fiber.new do super end @fiber.resume @buffer end def render layout { |value| read_content(value || :body) } template @fiber.resume while @fiber.alive? @buffer end end end
For our fibered implementation, we’ll inherit from Basic::ViewContext, because we want to be able to use the same templates as we used in the original implementation. We update the content_for, read_content, layout and render methods to be fiber-aware. Let’s take them one at a time.
def layout @fiber = Fiber.new do super end @fiber.resume @buffer end
First, we wrap the original implementation of layout in a Fiber, and start it right away. Next, we modify the read_content method to become Fiber-aware:
def read_content(name) content = super return content if content begin @waiting_for = name Fiber.yield ensure @waiting_for = nil end super end
If the @content_for Hash already has the content, return it right away. Otherwise, say that we’re waiting for the key in question, and yield out of the Fiber. We modify the render method so that the layout is rendered first, followed by the template. As a result, yielding out of the layout will start the template’s rendering.
def render layout { |value| read_content(value || :body) } template @fiber.resume while @fiber.alive? @buffer end
Next, modify the content_for method so that when the content we’re waiting for is provided, we jump back into the layout.
def content_for(name, value = nil) super @fiber.resume if @waiting_for == name end
With this setup, the layout and template will ping-pong back and forth, with the layout requesting data, and the template rendering only as far as it needs to go to provide the data requested.
Finally, let’s update the Buffer to take our fibered implementation into consideration.
module Basic class Buffer < String def initialize(name, context) @name = name @fibered = context.fibered? end def <<(value) super if @fibered puts "Flushing #{value.inspect}" if @fibered else puts "#{@name} is pushing #{value.inspect}" end end end class ViewContext def fibered? false end end end module Fibered class ViewContext def fibered? true end end end
Now that we’re rendering the layout in order, we can flush as we go, instead of being forced to wait for the entire template to render before we can start flushing.
It’s worth mentioning that optimal flushing performance will be based on the order of the content_for in your template. If you run your queries first, then put the expensive template rendering, and only finally do the content_for(:javascript) at the end, the flushing behavior will look like this:
Instead of flushing quickly, before the SQL call, things are barely better than they are in Rails 2.3, when the entire template must be rendered before the first flush. Because things are no worse, even in the worst-case scenario, we can make this the default behavior. Most people will see some benefit from it, and people interested in the best performance can order their content_for blocks so they cause the most beneficial flushing.
Even for people willing to put in the effort, this API is better than forcing a manual flush, because you can still put your content_for blocks alongside the templates that they are related to.
Look for this feature in Rails 3.1!
Small Caveat
For the purposes of this simplified example, I assumed that content_for can only be run once, immediately setting the value in the @content_for Hash. However, in some cases, people want to accumulate a String for a particular value. Obviously, we won’t be able to flush until the full String for that value is accumulated.
As a result, we’ll be adding a new API (likely called provide), which will behave exactly the same as content_for, but without the ability to accumulate. In the vast majority of cases, people will want to use provide (for instance, provide :sidebar), and get all the benefits of autoflushing. In a few cases, people will want to be able to continue accumulating a String, and will still be able to use content_for, but the template will not return control to the layout when that happens.
Also note that this (fairly detailed) explanation is not something that you will need to understand as a Rails developer. Instead, you will continue to go about your development as before, using provide if you have just a single piece of content to add, and content_for if you have multiple pieces of content to add, and Rails will automatically optimize flushing for you as well as we can.
A Tale of Abort Traps (or Always Question Your Assumptions)
August 24th, 2010
For a few months now, the bundler team has been getting intermittent reports of segfaults in C extensions that happen when using bundler with rvm. A cursory investigation revealed that the issue was that the C extensions were compiled for the wrong version of Ruby.
For instance, we would get reports of segfaults in nokogiri when using Ruby 1.9 that resulted from the $GEM_HOME in Ruby 1.9 containing a .bundle file compiled for Ruby 1.8. We got a lot of really angry bug reports, and a lot of speculation that we were doing something wrong that could be obviously fixed.
I finally ran into the issue myself, on my own machine a couple days ago, and tracked it down, deep into the fires of Mount Doom. A word of warning: this story may shock you.
It Begins
I usually use rvm for my day-to-day work, but I tend to not use gemsets unless I want a guaranteed clean environment for debugging something. Bundler takes care of isolation, even in the face of a lot of different gems mixed together in a global gemset. Last week, I was deep in the process of debugging another issue, so I was making pretty heavy use of gemset (and rvm gemset empty).
I had switched to Ruby 1.9.2 (final, which had just come out), created a new gemset, and run bundle install on the project I was working on, which included thin in the Gemfile. I had run bundle install to install the gems in the project, and among other things, bundler installed thin “with native extensions”. I use bundler a lot, and I had run this exact command probably thousands of times.
This time, however, I got a segfault in Ruby 1.9, that pointed at the require call to the rubyeventmachine.bundle file.
Debugging
Now that I had the bug on a physical machine, I started debugging. Our working hypothesis was that bundler or Rubygems was somehow compiling gems against Ruby 1.8, even when on Ruby 1.9, but we couldn’t figure out exactly how it could be happening.
The first thing I did was run otool -L on the binary (rubyeventmachine.bundle):
$ otool -L rubyeventmachine.bundle rubyeventmachine.bundle: /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/libruby.1.dylib (compatibility version 1.8.0, current version 1.8.7) /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8) /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
So that’s weird. I had run bundle install from inside an rvm gemset, yet it was being compiled against the libruby that ships with OSX. I had always suspecting that the problem was a leak from another rvm-installed Ruby, so this was definitely a surprise.
Just out of curiosity, I ran which bundle:
$ which bundle /usr/bin/bundle
Ok, now I knew something was rotten. I printed out the $PATH:
$ echo $PATH /Users/wycats/.rvm/gems/ruby-1.9.2-p0/bin:... $ ls /Users/wycats/.rvm/gems/ruby-1.9.2-p0/bin asdf* erubis* rake2thor* sc-build-number* autospec* prettify_json.rb* redcloth* sc-docs* bundle* rackup* ruby-prof* sc-gen* edit_json.rb* rake* sc-build* sc-init*
In other words, a big fat WTF.
I asked around, and some people had the vague idea that there was a $PATH cache in Unix shells. Someone pointed me at this post about clearing the cache.
Sure enough, running hash -r fixed the output of which. I alerted Wayne of rvm to this problem, and he threw in a fix to rvm that cleared the cache when switching rvms. Momentarily, everything seemed fine.
Digging Further
I still didn’t exactly understand how this condition could happen in the first place. When I went digging, I discovered that shells almost uniformly clear the path cache when modifying the $PATH. Since rvm pushes its bin directory onto the $PATH when you switch to a different rvm, I couldn’t understand how exactly this problem was happening.
I read through Chapter 3 of the zsh guide, and it finally clicked:
The way commands are stored has other consequences. In particular, zsh won’t look for a new command if it already knows where to find one. If I put a new ls command in /usr/local/bin in the above example, zsh would continue to use /bin/ls (assuming it had already been found). To fix this, there is the command rehash, which actually empties the command hash table, so that finding commands starts again from scratch. Users of csh may remember having to type rehash quite a lot with new commands: it’s not so bad in zsh, because if no command was already hashed, or the existing one disappeared, zsh will automatically scan the path again; furthermore, zsh performs a rehash of its own accord if $path is altered. So adding a new duplicate command somewhere towards the head of $path is the main reason for needing rehash.
By using the hash command (which prints out all the entries in this cache), I was able to confirm that the same behavior exists in bash, but it seems that the which command (which I was using for testing the problem) implicitly rehashes in bash.
During this time, I also took a look at /usr/bin/bundle, which looks like this:
$ cat /usr/bin/ruby #!/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby # # This file was generated by RubyGems. # # The application 'bundler' is installed as part of a gem, and # this file is here to facilitate running it. # require 'rubygems' version = ">= 0" if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then version = $1 ARGV.shift end gem 'bundler', version load Gem.bin_path('bundler', 'bundle', version)
As you can see, executable wrappers created by Rubygems hardcode the version of Ruby that was used to install them. This is one of the reasons that rvm keeps its own directory for installed executables.
A Working Hypothesis
It took me some time to figure all this out, during which time I was speaking to a bunch of friends and the guys in #zsh. I formed a working hypothesis. I figured that people were doing something like this:
- Install bundler onto their system, and use it there
- Need to work on a new project, so switch to rvm, and create a new gemset
- Run
bundle install, resulting in an error - Run
gem install bundler, to install bundler - Run
bundle install, which works, but has a subtle bug
Let’s walk through each of these steps, and unpack exactly what happens.
Install bundler on their system
This results in an executable at /usr/bin/bundle that looks like this:
#!/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby # # This file was generated by RubyGems. # # The application 'bundler' is installed as part of a gem, and # this file is here to facilitate running it. # require 'rubygems' version = ">= 0" if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then version = $1 ARGV.shift end gem 'bundler', version load Gem.bin_path('bundler', 'bundle', version)
There are two important things here. First, it hardcodes the shebang to the system Ruby location. Second, it uses Rubygems to look up the location of the executable shipped with bundler, which will respect GEM_HOME.
Switch to rvm, and create a new gemset
Switching to an rvm gemset does a few relevant things. First, it prepends ~/.rvm/gems/ruby-1.9.2-p0@gemset/bin onto the $PATH. This effectively resets the shell’s built-in command cache. Second, it sets $GEM_HOME to ~/.rvm/gems/ruby-1.9.2-p0. The $GEM_HOME is where Ruby both looks for gems as well as where it installs gems.
Run bundle install, resulting in an error
Specifically,
$ bundle install /Library/Ruby/Site/1.8/rubygems.rb:777:in `report_activate_error': Could not find RubyGem bundler (>= 0) (Gem::LoadError) from /Library/Ruby/Site/1.8/rubygems.rb:211:in `activate' from /Library/Ruby/Site/1.8/rubygems.rb:1056:in `gem' from /usr/bin/bundle:18
Most people don’t take such a close look at this error, interpreting it as the equivalent of command not found: bundle. What’s actually happening is a bit different. Since bundle is installed at /usr/bin/bundle, the shell finds it, and runs it. It uses the system Ruby (hardcoded in its shebang), and the $GEM_HOME set by rvm. Since I just created a brand new gemset, the bundler gem is not found in $GEM_HOME. As a result, the line gem 'bundler', version in the executable fails with the error I showed above.
However, because the shell found the executable, it ends up in the shell’s command cache. You can see the command cache by typing hash. In both bash and zsh, the command cache will include an entry for bundle pointing at /usr/bin/bundle. zsh is a more aggressive about populating the cache, so you’ll see the list of all commands in the system in the command cache (which, you’ll recall, was reset when you first switched into the gemset, because $PATH was altered).
Run gem install bundler, to install bundler
This will install the bundle executable to ~/.rvm/gems/ruby-1.9.2-p0@gemset/bin, which is the first entry on the $PATH. It will also install bundler to the $GEM_HOME.
Run bundle install, which works, but has a subtle bug
Here’s where the fun happens. Since we didn’t modify the cache, or call hash -r, the next call to bundle still picks up the bundle in /usr/bin/bundle, which is hardcoded to use system Ruby. However, it will use the version of Bundler we just installed, since it was installed to $GEM_HOME, and Rubygems uses that to look for gems. In other words, even though we’re using system Ruby, the /usr/bin/bundle executable will use the rvm gemset’s Bundler gem we just installed.
Additionally, because $GEM_HOME points to the rvm gemset, bundler will install all its gems to the rvm gemset. Taken together, these factors make it almost completely irrelevant that the system Ruby was used to install gems. After all, who cares which Ruby installed the gems, as long as they end up in the right place for the rvm’s Ruby.
There are actually two problems. First, as we’ve seen, the version of Ruby that installs a gem also hardcodes itself into the shebang of the executables for the gem.
Try this experiment:
$ rvm use 1.9.2
$ rvm gemset create experiment
$ rvm gemset use experiment
$ /usr/bin/gem install rack
$ cat ~/.rvm/gems/ruby-1.9.2-p0@experiment/bin/rackup
#!/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
#
# This file was generated by RubyGems.
#
# The application 'rack' is installed as part of a gem, and
# this file is here to facilitate running it.
#
require 'rubygems'
version = ">= 0"
if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
version = $1
ARGV.shift
end
gem 'rack', version
load Gem.bin_path('rack', 'rackup', version)As a result, any gems that bundler installed will put their executables in the gemset’s location, but still be hardcoded to use the system’s Ruby. This is not actually the problem we’re encountering here, but it could cause some weird results.
More concerningly, since our system Ruby installed the gems, it will also link any C extensions it compiles against its own copy of libruby. This shouldn’t be a major issue if you’re using a version of Ruby 1.8 in rvm, but it is a major issue if you’re using Ruby 1.9.
Now, we have a gem stored in your rvm’s Ruby that has a .bundle in it that is linked against the wrong Ruby. When running your application, Ruby 1.9 will try to load it, and kaboom: segfault.
Postscript
The crazy thing about this story is that it’s a lot of factors conspiring to cause problems. The combination of the shell command cache, $GEM_HOME making it look like SYSTEM Ruby was doing the right thing, and hardcoding the version of Ruby in the shebang line of installed gems made this segfault possible.
Thankfully, now that we figured the problem out, the latest version of rvm fixes it. The solution: wrap the gem command in a shell function that calls hash -r afterward.
Amazing.
Using >= Considered Harmful (or, What’s Wrong With >=)
August 21st, 2010
TL;DR Use ~> instead.
Having spent far, far too much time with Rubygems dependencies, and the problems that arise with unusual combinations, I am ready to come right out and say it: you basically never, ever want to use a >= dependency in your gems.
When you specify a dependency for your gem, it should mean that you are fairly sure that the unmodified code in the released gem will continue to work with any future version of the dependency that matches the version you specified. So for instance, let’s take a look at the dependencies listed in the actionpack gem:
activemodel (= 3.0.0.rc, runtime) activesupport (= 3.0.0.rc, runtime) builder (~> 2.1.2, runtime) erubis (~> 2.6.6, runtime) i18n (~> 0.4.1, runtime) rack (~> 1.2.1, runtime) rack-mount (~> 0.6.9, runtime) rack-test (~> 0.5.4, runtime) tzinfo (~> 0.3.22, runtime)
Since we release the Rails gems as a unit, we declare hard dependencies on activemodel and activesupport. We declare soft dependencies on builder, erubis, i18n, rack, rack-mount, rack-test, and tzinfo.
You might not know what exactly the ~> version specifier means. Essentially, it decomposes into two specifiers. So ~> 2.1.2 means >= 2.1.2, < 2.2.0. In other words, it means “2.1.x, but not less than 2.1.2″. Specifying ~> 1.0, like many people do for Rack, means “any 1.x”.
You should make your dependencies as soft as the versioning scheme and release practices of your dependencies will allow. If you’re monkey-patching a library outside of its public API (not a very good practice for libraries), you should probably stick with an = dependency.
One thing for certain though: you cannot be sure that your gem works with every future version of your dependencies. Sanely versioned gems take the opportunity of a major release to break things, and until you have actually tested against the new versions, it’s madness to claim compatibility. One example: a number of gems have dependencies on activesupport >= 2.3. In a large number of cases, these gems do not work correctly with ActiveSuport 3.0, since we changed how components of ActiveSupport get loaded to make it easier to cherry-pick.
Now, instead of receiving a version conflict, users of these gems will get cryptic runtime error messages. Even worse, everything might appear to work, until some weird edge-case is exercised in production, and which your tests would have caught.
But What Happens When a New Version is Released?
One reason that people use the activesupport >= 2.3 is that, assuming Rails maintains backward-compatibility, their gem will continue to work in newer Rails environments without any difficulty. If everything happens to work, it saves you the time of running their unit tests against newer versions of dependencies and cutting a new release.
As I said before, this is a deadly practice. By specifying appropriate dependencies (based on your confidence in the underlying library’s versioning scheme), you will have a natural opportunity to run your test suite against the new versions, and release a new gem that you know actually works.
This does mean that you will likely want to release patch releases of old versions of your gem. For instance, if I have AuthMagic 1.0, which worked against Rails 2.3, and I release AuthMagic 2.0 once Rails 3.0 comes out, it makes sense to continue patching AuthMagic 1.0 for a little while, so your Rails 2.3 users aren’t left out in the cold.
Applications and Gemfiles
I should be clear that this versioning advice doesn’t necessarily apply to an application using Bundler. That’s because the Gemfile.lock, which you should check into version control, essentially converts all >= dependencies into hard dependencies. However, because you may want to run bundle update at some point in the future, which will update everything to the latest possible versions, you might want to use version specifiers in your Gemfile that seem likely to work into the future.