Implementation of byname implicits with recursive dictionaries

This is an implementation of byname implicit parameters with recursive
dictionaries, intended as a language-level replacement for shapeless's
Lazy type. As of this commit, implicit arguments can be marked as byname
and at call sites recursive uses of implicit values are permitted if
they occur in an implicit byname argument.

Consider the following example,

  trait Foo { def next: Foo }
  object Foo {
    implicit def foo(implicit rec: Foo): Foo =
      new Foo { def next = rec }
  }
  val foo = implicitly[Foo]
  assert(foo eq foo.next)

In current Scala this would diverge due to the recursive implicit
argument rec of method foo. Under the scheme implemented in this commit
we can mark the recursive implicit parameter as byname,

  trait Foo { def next: Foo }
  object Foo {
    implicit def foo(implicit rec: => Foo): Foo =
      new Foo { def next = rec }
  }
  val foo = implicitly[Foo]
  assert(foo eq foo.next)

and recursive occurrences of this sort are extracted out as val members
of a local synthetic object as follows,

  val foo: Foo = scala.Predef.implicitly[Foo](
    {
      object LazyDefns$1 {
        val rec$1: Foo = Foo.foo(rec$1)
                         //      ^^^^^
                         // recursive knot tied here
      }
      LazyDefns$1.rec$1
    }
  )
  assert(foo eq foo.next)

and the example compiles with the assertion successful. Note that the
recursive use of rec$1 occurs within the byname argument of foo and is
consequently deferred. The desugaring matches what a programmer would do
to construct such a recursive value explicitly.

This general pattern is essential to the derivation of type class
instances for recursive data types, one of shapeless's most common
applications.

Byname implicits have a number of benefits over the macro implementation
of Lazy in shapeless,

+ the implementation of Lazy in shapeless is extremely delicate, relying
  on non-portable compiler internals. By contrast there is already an
  initial implementation of byname implicits in Dotty:
  scala/scala3#1998.
+ the shapeless implementation is unable to modify divergence checking,
  so to solve recursive instances it effectively disables divergence
  checking altogether ... this means that incautious use of Lazy can cause
  the typechecker to loop indefinitely. The byname implicits
  implementation is able to both solve recursive occurrences and check for
  divergence.
+ the implementation of Lazy interferes with the heuristics for solving
  inductive implicits in scala#6481 because the latter depends on being able to
  verify that induction steps strictly reduce the size of the types being
  solved for; the additional Lazy type constructors make the type appear
  be non-decreasing in size. Whilst this could be special-cased, doing so
  would require some knowledge of shapeless to be incorporated into the
  compiler. Being a language-level feature, byname implicits can be
  accommodated directly in the induction heuristics.
+ in common cases more implicit arguments would have to be marked as
  Lazy than would have to be marked as byname under this PR due to
  restrictions on what the Lazy macro is able to do. Given that there is a
  runtime cost associated with capturing the thunks required for both Lazy
  and byname arguments, any reduction in the number is beneficial.

Author: milessabin

spec/04-basic-declarations-and-definitions.md

@@ -693,9 +693,7 @@ function. That is, the argument is evaluated using _call-by-name_.
 
 The by-name modifier is disallowed for parameters of classes that
 carry a `val` or `var` prefix, including parameters of case
-classes for which a `val` prefix is implicitly generated. The
-by-name modifier is also disallowed for
-[implicit parameters](07-implicits.html#implicit-parameters).
+classes for which a `val` prefix is implicitly generated.
 
 ###### Example
 The declaration

spec/07-implicits.md

@@ -177,22 +177,91 @@ expansion:
 sort(arg)(x => magic(x)(x => magic(x)(x => ... )))
 ```
 
-To prevent such infinite expansions, the compiler keeps track of
-a stack of “open implicit types” for which implicit arguments are currently being
-searched. Whenever an implicit argument for type $T$ is searched, the
-“core type” of $T$ is added to the stack. Here, the _core type_
-of $T$ is $T$ with aliases expanded, top-level type [annotations](11-annotations.html#user-defined-annotations) and
-[refinements](03-types.html#compound-types) removed, and occurrences
-of top-level existentially bound variables replaced by their upper
-bounds. The core type is removed from the stack once the search for
-the implicit argument either definitely fails or succeeds. Everytime a
-core type is added to the stack, it is checked that this type does not
-dominate any of the other types in the set.
-
-Here, a core type $T$ _dominates_ a type $U$ if $T$ is
-[equivalent](03-types.html#equivalence)
-to $U$, or if the top-level type constructors of $T$ and $U$ have a
-common element and $T$ is more complex than $U$.
+Such infinite expansions should be detected and reported as errors, however to support the deliberate
+implicit construction of recursive values we allow implicit arguments to be marked as by-name. At call
+sites recursive uses of implicit values are permitted if they occur in an implicit by-name argument.
+
+Consider the following example,
+
+```scala
+trait Foo {
+  def next: Foo
+}
+
+object Foo {
+  implicit def foo(implicit rec: Foo): Foo =
+    new Foo { def next = rec }
+}
+
+val foo = implicitly[Foo]
+assert(foo eq foo.next)
+```
+
+As with the `magic` case above this diverges due to the recursive implicit argument `rec` of method
+`foo`. If we mark the implicit argument as by-name,
+
+```scala
+trait Foo {
+  def next: Foo
+}
+
+object Foo {
+  implicit def foo(implicit rec: => Foo): Foo =
+    new Foo { def next = rec }
+}
+
+val foo = implicitly[Foo]
+assert(foo eq foo.next)
+```
+
+the example compiles with the assertion successful.
+
+When compiled, recursive by-name implicit arguments of this sort are extracted out as val members of a
+local synthetic object at call sites as follows,
+
+```scala
+val foo: Foo = scala.Predef.implicitly[Foo](
+  {
+    object LazyDefns$1 {
+      val rec$1: Foo = Foo.foo(rec$1)
+                       //      ^^^^^
+                       // recursive knot tied here
+    }
+    LazyDefns$1.rec$1
+  }
+)
+assert(foo eq foo.next)
+```
+
+Note that the recursive use of `rec$1` occurs within the by-name argument of `foo` and is consequently
+deferred. The desugaring matches what a programmer would do to construct such a recursive value
+explicitly.
+
+To prevent infinite expansions, such as the `magic` example above, the compiler keeps track of a stack
+of “open implicit types” for which implicit arguments are currently being searched. Whenever an
+implicit argument for type $T$ is searched, $T$ is added to the stack paired with the implicit
+definition which produces it, and whether it was required to satisfy a by-name implicit argument or
+not. The type is removed from the stack once the search for the implicit argument either definitely
+fails or succeeds. Everytime a type is about to be added to the stack, it is checked against
+existing entries which were produced by the same implicit definition and then,
+
++ if it is equivalent to some type which is already on the stack and there is a by-name argument between
+  that entry and the top of the stack. In this case the search for that type succeeds immediately and
+  the implicit argument is compiled as a recursive reference to the found argument.  That argument is
+  added as an entry in the synthesized implicit dictionary if it has not already been added.
++ otherwise if the _core_ of the type _dominates_ the core of a type already on the stack, then the
+  implicit expansion is said to _diverge_ and the search for that type fails immediately.
++ otherwise it is added to the stack paired with the implicit definition which produces it.
+  Implicit resolution continues with the implicit arguments of that definition (if any).
+
+Here, the _core type_ of $T$ is $T$ with aliases expanded,
+top-level type [annotations](11-annotations.html#user-defined-annotations) and
+[refinements](03-types.html#compound-types) removed, and occurrences of top-level existentially bound
+variables replaced by their upper bounds.
+
+A core type $T$ _dominates_ a type $U$ if $T$ is [equivalent](03-types.html#equivalence) to $U$,
+or if the top-level type constructors of $T$ and $U$ have a common element and $T$ is more complex
+than $U$ and the _covering sets_ of $T$ and $U$ are equal.
 
 The set of _top-level type constructors_ $\mathit{ttcs}(T)$ of a type $T$ depends on the form of
 the type:
@@ -211,6 +280,21 @@ the type:
 - For any other singleton type, $\operatorname{complexity}(p.type) ~=~ 1 + \operatorname{complexity}(T)$, provided $p$ has type $T$;
 - For a compound type, `$\operatorname{complexity}(T_1$ with $\ldots$ with $T_n)$` $= \Sigma\operatorname{complexity}(T_i)$
 
+The _covering set_ $\mathit{cs}(T)$ of a type $T$ is the set of type designators mentioned in a type.
+For example, given the following,
+
+```scala
+type A = List[(Int, Int)]
+type B = List[(Int, (Int, Int))]
+type C = List[(Int, String)]
+```
+
+the corresponding covering sets are:
+
+- $\mathit{cs}(A)$: List, Tuple2, Int
+- $\mathit{cs}(B)$: List, Tuple2, Int
+- $\mathit{cs}(C)$: List, Tuple2, Int, String
+
 ###### Example
 When typing `sort(xs)` for some list `xs` of type `List[List[List[Int]]]`,
 the sequence of types for

src/compiler/scala/tools/nsc/ast/parser/Parsers.scala

@@ -2402,10 +2402,6 @@ self =>
                 in.offset,
                 (if (mods.isMutable) "`var'" else "`val'") +
                 " parameters may not be call-by-name", skipIt = false)
-            else if (implicitmod != 0)
-              syntaxError(
-                in.offset,
-                "implicit parameters may not be call-by-name", skipIt = false)
             else bynamemod = Flags.BYNAMEPARAM
           }
           paramType()

src/compiler/scala/tools/nsc/typechecker/Contexts.scala

@@ -17,8 +17,10 @@ import scala.reflect.internal.Reporter
  */
 trait Contexts { self: Analyzer =>
   import global._
-  import definitions.{ JavaLangPackage, ScalaPackage, PredefModule, ScalaXmlTopScope, ScalaXmlPackage }
+  import definitions.{ AnyRefClass, JavaLangPackage, ScalaPackage, PredefModule, ScalaXmlTopScope, ScalaXmlPackage }
   import ContextMode._
+  import scala.reflect.internal.Flags._
+
 
   protected def onTreeCheckerError(pos: Position, msg: String): Unit = ()
 
@@ -244,6 +246,117 @@ trait Contexts { self: Analyzer =>
       openImplicits.nonEmpty && openImplicits.exists(x => !x.isView)
     }
 
+    private type ImplicitDict = List[(Type, (Symbol, Tree))]
+    private var implicitDictionary: ImplicitDict = null
+
+    def implicitRootContext: Context = {
+      if(implicitDictionary != null) this
+      else if(outerIsNoContext || outer.openImplicits.isEmpty) {
+        implicitDictionary = Nil
+        this
+      } else outer.implicitRootContext
+    }
+
+    private def linkImpl(tpe: Type): Tree = {
+      val sym =
+        implicitDictionary.find(_._1 =:= tpe) match {
+          case Some((_, (sym, _))) => sym
+          case None =>
+            val vname = unit.freshTermName("rec$")
+            val vsym = owner.newValue(vname, newFlags = FINAL | SYNTHETIC) setInfo tpe
+            implicitDictionary +:= (tpe -> (vsym, EmptyTree))
+            vsym
+        }
+      gen.mkAttributedRef(sym) setType tpe
+    }
+
+    def linkByNameImplicit(tpe: Type): Tree = implicitRootContext.linkImpl(tpe)
+
+    private def refImpl(tpe: Type): Tree =
+      implicitDictionary.find(_._1 =:= tpe) match {
+        case Some((_, (sym, _))) =>
+          gen.mkAttributedRef(sym) setType tpe
+        case None =>
+          EmptyTree
+      }
+
+    def refByNameImplicit(tpe: Type): Tree = implicitRootContext.refImpl(tpe)
+
+    private def defineImpl(tpe: Type, result: SearchResult): SearchResult = {
+      @tailrec
+      def patch(d: ImplicitDict, acc: ImplicitDict): (ImplicitDict, SearchResult) = d match {
+        case Nil => (implicitDictionary, result)
+        case (tp, (sym, EmptyTree)) :: tl if tp =:= tpe =>
+          val ref = gen.mkAttributedRef(sym) setType tpe
+          val res = new SearchResult(ref, result.subst, result.undetparams)
+          (acc reverse_::: ((tpe, (sym, result.tree)) :: tl), res)
+        case hd :: tl =>
+          patch(tl, hd :: acc)
+      }
+
+      val (d, res) = patch(implicitDictionary, Nil)
+      implicitDictionary = d
+      res
+    }
+
+    def defineByNameImplicit(tpe: Type, result: SearchResult): SearchResult = implicitRootContext.defineImpl(tpe, result)
+
+    def emitImplicitDictionary(result: SearchResult): SearchResult =
+      if(implicitDictionary == null || implicitDictionary.isEmpty || result.tree == EmptyTree) result
+      else {
+        val typer = newTyper(this)
+
+        @tailrec
+        def prune(trees: List[Tree], pending: List[(Symbol, Tree)], acc: List[(Symbol, Tree)]): List[(Symbol, Tree)] = pending match {
+          case Nil => acc
+          case ps =>
+            val (in, out) = ps.partition { case (vsym, rhs) => trees.exists(_.exists(_.symbol == vsym)) }
+            if(in.isEmpty) acc
+            else prune(in.map(_._2) ++ trees, out, in ++ acc)
+        }
+
+        val pruned = prune(List(result.tree), implicitDictionary.map(_._2), List.empty[(Symbol, Tree)])
+        if(pruned.isEmpty) result
+        else {
+          val (vsyms, vdefs) = pruned.map { case (vsym, rhs) =>
+            (vsym, ValDef(Modifiers(FINAL | SYNTHETIC), vsym.name.toTermName, TypeTree(rhs.tpe), rhs))
+          }.unzip
+
+          val ctor = DefDef(NoMods, nme.CONSTRUCTOR, Nil, ListOfNil, TypeTree(), Block(List(pendingSuperCall), Literal(Constant(()))))
+          val mname = unit.freshTermName("LazyDefns$")
+          val mdef =
+            ModuleDef(Modifiers(SYNTHETIC), mname,
+              Template(List(Ident(AnyRefClass)), noSelfType, ctor :: vdefs.toList)
+            )
+
+          typer.namer.enterSym(mdef)
+
+          val mdef0 = typer.typed(mdef)
+          val msym0 = mdef0.symbol
+          val vsyms0 = vsyms.map { vsym =>
+            mdef0.symbol.moduleClass.asClass.toType.decl(vsym.name)
+          }
+
+          val vsymMap = (vsyms zip vsyms0).toMap
+
+          object patchRefs extends Transformer {
+            override def transform(tree: Tree): Tree = {
+              tree match {
+                case i: Ident if vsymMap.contains(i.symbol) =>
+                  gen.mkAttributedSelect(gen.mkAttributedRef(msym0), vsymMap(i.symbol)) setType i.tpe
+                case _ =>
+                  super.transform(tree)
+              }
+            }
+          }
+
+          val tree0 = patchRefs.transform(result.tree)
+          val tree1 = Block(mdef0, tree0).substituteSymbols(vsyms.toList, vsyms0.toList) setType tree.tpe
+
+          new SearchResult(tree1, result.subst, result.undetparams)
+        }
+      }
+
     var prefix: Type = NoPrefix
 
     def inSuperInit_=(value: Boolean)         = this(SuperInit) = value