The Biggest Mistake Everyone Makes With Closures

Whether they’re called “closures” or “lambdas” or “Procs” or “anonymous functions”, many popular programming languages have a hidden gotcha when it comes to combining them with loops, and nearly everyone will fall victim to it at one time or another.

The Problem

The pitfall is this: because most languages reuse the same variable scope between individual loop iterations, closures created in different iterations will capture the same loop variable.

Usually that is not what you want or expect. Most often, all the closures you created end up seeing the same (final) value for the loop variable.

Some Examples

Each of the following examples loops from 1 to 3, creating an anonymous function to return each value. In each case, we might expect the first of these to return 1, but in practice we see something else entirely…

Ruby
k = []
for x in 1..3
  k.push(lambda { x })
end

Contrary to naïve expectation, k[0].call() returns 3 instead of 1.

(This can happen just as easily with blocks passed to methods like Thread.new as it can lambda.)

Python
k = []
for x in xrange(1, 4):
  k.append(lambda: x)

Here, too, k[0]() returns 3.

(Python ranges don’t include the final value.)

Perl
my @k = ();
for (my $x = 1; $x < 4; $x++) {
  push(@k, sub { return $x; })
}

In this case, $k[0]->() returns 4, the value of $x after the last iteration.

JavaScript
k = [];
for (var x = 1; x < 4; x++) {
  k.push(function () { return x; });
}

Once again, with a C-style for loop, k[0]() returns 4.

More “functional” approaches won’t necessarily save you either:

Ruby
k = (1..3).map { |x| lambda { x } }

k[0].call() is still 3.

Edit: This depends. It works fine in Ruby 1.9. In Ruby 1.8, you will get either 1 or 3 depending on whether x has already been introduced lexically earlier in the method’s scope (e.g. due to a preceding assignment or for x ... in). I would not recommend writing code which is fragile in this way.

Python
k = [lambda: x for x in xrange(1, 4)]

k[0]() is still 3.

Perl
my @k = map { sub { return $_; } } (1..4);

Now $k[0]->() is the undefined value (undef()).

Some Exceptions

While the problem with capturing re-used scopes is widespread, it is worth noting that not ALL languages with explicit looping suffer from it:

R5 Scheme
(define k '())
(do ((x 1 (+ x 1)))
    ((= x 4) '())
  (set! k (cons (lambda () x) k)))
(set! k (reverse k))

((car k)) evaluates to 1, as we expected.

(This is for demonstration purposes only. Rest assured that I never normally write Scheme this way.)

Also, in some languages, only some loop constructs reuse scopes:

Perl
my @k = ();
for my $x (1..4) {
  push(@k, sub { return $x; });
}

Believe it or not, this time $k[0]->() is 1!

So all that being said, what’s the best way to avoid the problem in general?

The Solution

The best and most general solution is to use a named helper function to construct your closure in a fresh lexical environment.

Ruby
def make_value_func(value)
  lambda { value }
end
k = (1..3).map { |x| make_value_func(x) }

Now k[0].call() returns 1.

Python
def make_value_func(value):
  return lambda: value
k = [make_value_func(x) for x in range(1, 4)]

k[0]() finally returns 1.

Perl
sub make_value_func {
  my ($value) = @_;
  return sub { return $value; };
}
@k = map { make_value_func($_); } (1..4);

$k[0]->() returns 1 here as well.

JavaScript
function make_value_func(value) {
  return function () { return value; };
}
var k = [];
for (var x = 1; x < 4; x++) {
  k.push(make_value_func(x));
}

And finally, here k[0]() returns 1 in JavaScript too.

The approach has at least two extra advantages beyond dealing with the problem of creating closures over changing loop variables:

  • Better readability (if you name your helper function descriptively)
  • Memory savings, in some languages (since the closure won’t be holding on to every single local variable visible inside the loop)