State of the Lambda

THIS DOCUMENT HAS BEEN SUPERSEDED BY VERSION 4 AND IS PROVIDED FOR

HISTORICAL CONTEXT ONLY

Brian Goetz, 10 October 2010

This is an updated proposal to add lambda expressions (informally,"closures") to the Java programming language.  This sketch is built onthestraw-man proposalmade by Mark Reinhold in December2009 and theprevious iterationposted in July 2010.

1.  Background; SAM types

The Java programming language already has a form of closures:anonymous inner classes.  There are a number of reasons these areconsideredimperfect closures, primarily:

Bulky syntax

Inability to capture non-final local variables

Transparency issues surrounding the meaning of return, break,

continue, and 'this'

No nonlocal control flow operators

It isnota goal of Project Lambda to addressallof these issues.The current draft addresses (1) quite substantially, ameliorate (2) byallowing the compiler to infer finality (allowing capture ofeffectively finallocal variables), and ameliorate (3) by making'this' within a lambda expression be lexically scoped.  It is unlikelythat we will go further at this time (e.g., we do not intend toaddress nonlocal flow control at all, nor allow arbitrary capture ofmutable variables.)

The standard way for Java APIs to define callbacks is to use an

interface representing the callback method, such as:

public interface CallbackHandler {

public void callback(Context c);

}

The CallbackHandler interface has a useful property: it has asingle

abstract method.  Many common interfaces and abstract classes havethis property, such as Runnable, Callable, EventHandler, orComparator.  We call these classesSAM types.  This property,SAM-ness, is a structural property identified by the compiler, as isnot represented in the type system.

The biggest pain point for anonymous inner classes is bulkiness.  To

call a method taking a CallbackHandler, one typically creates an

anonymous inner class:

foo.doSomething(new CallbackHandler() {

public void callback(Context c) {

System.out.println("pippo");

}

});

The anonymous inner class here is what some might call a "vertical

problem": five lines of source code to encapsulate a single statement.

2.  Lambda expressions

Lambda expressions are anonymous functions, aimed at addressing the

"vertical problem" by replacing the machinery of anonymous inner

classes with a simpler mechanism.  One way to do that would be to add

function types to the language, but this has several disadvantages:

Mixing of structural and nominal types;

Divergence of library styles (some libraries would continue to use

callback objects, while others would use function types).

Generic types are erased, which would expose additional places where

developers are exposed to erasure.  For example, it would not be

possible to overload methods m(T->U) and m(X->Y), which would be

confusing.

So, we have instead chosen to take the path of "use what you know" --

since existing libraries use SAM types extensively, we leverage the

notion of SAM types and use lambda expressions to make it easier

create instances of callback objects.

Here are some examples of lambda expressions:

#{ -> 42 }

#{ int x -> x + 1 }

The first expression takes no arguments, and returns the

integer 42; the second takes a single integer argument, named x, and

returns x+1.

Lambda expressions are delimited by #{ } and have argument lists, the

arrow token ->, and a lambda body.  For nilary lambdas the arrow token

can be omitted, meaning the first example can be shortened to

#{ 42 }

The method body can either be a single expression or an ordinary list

of statements (like a method body).  In the single-expression form, no

return or semicolon is needed.  These simplification rules are based

on the expectation that many lambda expressions will be quite small,

like the examples above, and the "horizontal noise" such as the return

keyword become a substantial overhead in those cases.

3.  SAM conversion

One can describe a SAM type by its return type, parameter types, and

checked exception types.  Similarly, one can describe the type of a

lambda expression by its return type, parameter types, and exception

types.

Informally, a lambda expression e isconvertible-toa SAM type S ifan anonymous inner class that is a subtype of S and that declares amethod with the same name as S's abstract method and a signature andreturn type corresponding to the lambda expressions signature andreturn type would be considered assignment-compatible with S.

The return type and exception types of a lambda expression areinferred by the compiler; the parameter types may be explicitlyspecified or they may be inferred from the assignment context (seeTarget Typing, below.)

When a lambda expression is converted to a SAM type, invoking the

single abstract method of the SAM instance causes the body of the

lambda expression to be invoked.

For example, SAM conversion will happen in the context of assignment:

CallbackHandler cb = #{ Context c -> System.out.println("pippo") };

In this case, the lambda expression has a single Context parameter,

has void return type, and throws no checked exceptions, and is

therefore compatible with the SAM type CallbackHandler.

4.  Target Typing

Lambda expressions canonlyappear in context where it will beconverted to a variable of SAM type.  These include assignmentcontext, cast target context, and method invocation context.  So thefollowing examples are valid examples of statements using lambdaexpressions:

Runnable r = #{ System.out.println("Blah") };

Runnable r = (Runnable) #{ System.out.println("Blah") };

executor.submit( #{ System.out.println("Blah") } );

The following use of lambda expressions is forbidden because it does

not appear in a SAM-convertible context:

Object o = #{ 42 };

In a method invocation context, the target type for a lambda

expression used as a method parameter is inferred by examining the set

of possible compatible method signatures for the method being invoked.

This entails some additional complexity in method selection;

ordinarily the types of all parameters are computed, and then the set

of compatible methods is computed, and a most specific method is

selected if possible.  Inference of the target type for lambda-valued

actual parameters happens after the types of the other parameters is

computed but before method selection; method selection then happens

using the inferred target types for the lambda-valued parameters.

The types of the formal parameters to the lambda expression can also

be inferred from the target type of the lambda expression.  So we can

abbreviate our callback handler as:

CallbackHandler cb = #{ c -> System.out.println("pippo") };

as the type of the parameter c can be inferred from the target type

of the lambda expression.

Allowing the formal parameter types to be inferred in this way

furthers a desirable design goal: "Don't turn a vertical problem into

a horizontal problem."  We wish that the reader of the code have to

wade through as little code as possible before arriving at the "meat"

of the lambda expression.

The user can explicitly choose a target type by using a cast.  This

might be for clarity, or might be because there are multiple

overloaded methods and the compiler cannot correctly chose the target

type.  For example:

executor.submit(((Callable) #{ "foo" }));

If the target type is an abstract class, there is no place to put

constructor arguments, so we cannot invoke other than the no-arg

constructor.  However, such cases always have the option to use inner

classes.

5.  Lambda bodies

In addition to the simplified expression form of a lambda body, a

lambda body can also contain a list of statements, similar to a method

body, with several differences: the break and continue statements are

not permitted at the top level (break and continue are of course

permitted within loops, but break and continue labels must be inside

the lambda body).  The "return" statment can be used in a

multi-statement lambda expression to indicate the value of the lambda

expression; this is a "local" return.  The type of a multi-statement

lambda expression is inferred by unifying the types of the values

returned by the set of return statements.  As with method bodies,

every control path through a multi-statement lambda expression must

either return a value (or void) or throw an exception.

6.  Local variable capture

The current rules for capturing local variables of enclosing contextsin inner classes are quite restrictive; only final variables may becaptured.  For lambda expressions (and for consistency, probably innerclass instances as well), we relax these rules to also allow forcapture ofeffectively finallocal variables.  (Informally, a localvariable is effectively final if making it final would not cause acompilation failure; this can be considered a form of type inference.)

It is our intent tonotpermit capture of mutable local variables.The reason is that idioms like this:

int sum = 0;

list.forEach(#{ e -> sum += e.size(); });

are fundamentally serial; it is quite difficult to write lambda bodies

like this that do not have race conditions.  Unless we are willing to

enforce (preferably statically) that such lambdas not escape their

capturing thread, such a feature may well cause more trouble than it

solves.

7.  Lexical scoping

Unlike inner classes, lambda expressions are lexically scoped, meaning

that the body of a lambda expression are scoped just like a code block

in the enclosing environment, with local variables for each formal

parameter.  The 'this' variable (and any associated

OuterClassName.this variables) has the same meaning as it does

immediately outside the lambda expression.

Lambda bodies that reference either the 'this' variables or members of

enclosing instances (those that are implicitly qualified with a 'this'

variable, in which case the references are treated as if they used the

appropriate 'this' variable) are treated as capturing the appropriate

'this' variable as per capture of effectively final local variables.

This has a potentially beneficial implication for memory management;where inner class instances always hold a strong reference to theirenclosing instance, lambdas that do not capture members from theenclosing instance do not have this behavior.  This characteristic ofinner class instances can often be a source of memory leaks (theso-calledlapsed listenerproblem).  This risk is reduced by thelexical scoping of lambda bodies.

The move to lexical scoping introduces a complication in creating

self-referential lambda expressions; in some cases it is desirable to

create lambdas such as the following:

Timer timer = ...

timer.schedule(

#{

if (somethingHappened())

// cancel the timer task that this lambda represents

someRefToTimerTask.cancel();

else

System.out.println("foo");

});

We cannot refer to the TimerTask to which the lambda is being

converted, since it is anonymous.  Instead, we will update the

definitely assigned/unassigned rules to allow the above code to be

refactored as:

Timer timer = ...

final TimerTask t = #{

if (somethingHappened())

// cancel the timer task that this lambda represents

t.cancel();

else

System.out.println("foo");

});

timer.schedule(t);

Under current DA/DU rules, the reference to 't' in the lambda body

would be illegal.  However, the compiler can verify that the body

cannot be executed (and therefore 't' cannot be evaluated) until the

assignment to 't' completes.  The DA/DU rules will be updated to allow

this situation.  (The same issue would come up with lambda expressions

that wanted to recurse.)

8.  Method references

SAM conversion allows us to take an anonymous method body and treat it

as if it were a SAM type.  It is often desirable to do the same with

an existing method (such as when a class has multiple methods that are

signature-compatible with Comparable.compareTo().)

Method references are expressions which have the same treatment as

lambda expressions (i.e., they can only be SAM-converted), but instead

of providing a method body they refer to a method of an existing class

or object instance.

For example, consider a Person class that can be sorted by name or by

age:

class Person {

private final String name;

private final int age;

public static int compareByAge(Person a, Person b) { ... }

public static int compareByName(Person a, Person b) { ... }

}

Person[] people = ...

Arrays.sort(people, #Person.compareByAge);

Here, the expression #Person.compareByAge can be considered sugar for

a lambda expression whose formal argument list is copied from the

method Person.compareByAge, and whose body calls Person.compareByAge

(though the actual implementation is more efficient than this.)  This

lambda expression will then get SAM-converted to a Comparator.

If the method being referenced is overloaded, it can be disambiguated

by providing a list of argument types:

Arrays.sort(people, #Person.compareByAge(Person, Person));

Instance methods can be referenced as well, by providing a receiver

variable:

Arrays.sort(people, #comparatorHolder.comparePersonByAge);

In this case, the implicit lambda expression would capture a final

copy of the "comparatorHolder" reference and the body would invoke

the comparePersonByAge using that as the receiver.

We will likely restrict the forms that the receiver can take, rather

than allowing arbitrary object-valued expressions like

"#foo(bar).moo", when capturing instance method references.

One syntactic alternatives that is under consideration include placing

the '#' symbol in place of the dot (making '#' an infix rather than

prefix token).

9.  Extension methods

A separate document onvirtual extension methodsproposes our strategy for extending existing interfaces with virtualextension methods.

10.  Exception transparency

A separate document onexception transparencyproposes ourstrategy for amending generics to allow abstraction over thrownchecked exception types.

11.  Putting it together

The features described here are not an arbitrary combination of

features, but instead designed to work together.  Take the typical

example of sorting a collection.  Today we write:

Collections.sort(people, new Comparator() {

public int compare(Person x, Person y) {

return x.getLastName().compareTo(y.getLastName());

}

});

This is a very verbose way to write "sort people by last name"!

With lambda expressions, we can make this expression more concise:

Collections.sort(people,

#{ Person x, Person y -> x.getLastName().compareTo(y.getLastName()) });

However, while more concise, it is not any more abstract; it still

burdens the programmer with the need to do the actual comparison

(which is even worse when the sort key is a primitive.)  Small changes

to the libraries can help here, such as introducing a sortyBy method

which takes a function that extracts a (Comparable) sort key:

interface Extractor { public U extract(T t); }

public void>

sortBy(Collection collection, Extractor extractor);

Collections.sortBy(people, #{ Person p -> p.getLastName() });

Even this can be made less verbose, by allowing the compiler to infer the

type of the lambda formal:

Collections.sortBy(people, #{ p -> p.getLastName() });

The lambda in the above expression is simply a forwarder for the existing method

getLastName().  We can use method references to reuse the existing method rather

than creating a new lambda:

Collections.sortBy(people, #Person.getLastName);

The use of an anciallary method like Collections.sortBy() is

undesirable; not only is it more verbose, but it is less

object-oriented and undermines the value of the Collection

interface since users can't easily discover the static sortBy()

method when inspecting the documentation for Collection.

Extension methods address this problem:

people.sortBy(#Person.getLastName);

Which reads like the problem statement in the first place: sort the people collection

by last name.