Designing libraries for source and binary compatibility

Scala.js logo

Sébastien Doeraene @sjrdoeraene

April 17, 2018 -- Scala.sphere.it

LAMP, lamp.epfl.ch
École polytechnique fédérale de Lausanne
 

EPFL logo             Scala logo

Who am I ...

to lecture you about source and binary compatibility?

Answer: the author of Scala.js

Part Ⅰ

What is:

  • Source compatibility?
  • Binary compatibility?
  • Forward/backward compatibility?
  • Compatibility?

What is compatibility?

Scala.js logo

What is compatibility?

Scala.js logo

What is library compatibility?

Scala.js logo

(backward)

What is library compatibility?

Scala.js logo

(forward)

Source v binary compatibility

A v1


class Foo {
  def bar(x: Int): Int = x + 1
}

A v2


class Foo {
  def bar(x: Double): Double = x + 1
}

Source compat ✖ -- Binary compat ✖

A v1


class Foo {
  def bar(x: Int): Int = x + 1
}

A v2


class Foo {
  private val one: Int = 1

  def bar(x: Int): Int = x + one
}

Source compat ✔ -- Binary compat ✔

A v1


trait Foo {
  def bar(x: Int): Int = x + 1
}

A v2


trait Foo {
  private val one: Int = 1

  def bar(x: Int): Int = x + one
}

Source compat ✔ -- Binary compat ✖

A v1


class Foo {
  def bar(x: Int): Int = x + 1
}

A v2


class Foo {
  def bar(x: Int): Int = x + 1

  def foobar(y: Int): Int = y * 2
}

Source compat ✖ -- Binary compat ✔


class Foo {
  def bar(x: Int): Int = x + 1

  def foobar(y: Int): Int = y * 2
}
    

implicit class FooOps(val self: Foo) {
  def foobar(y: Int): Int = y / 3
}

Not a theoretical issue

When does compatibility matter?

Writing an application X

Scala.js logo

Neither source nor binary compat matter

Writing a library A used by an app X

Scala.js logo

Only source compatibility matters

Writing a library A used by other libraries

Scala.js logo

 

Writing a library A used by other libraries

Scala.js logo

Source compat matters for C; binary compat matters for X

Scala.js logo
Scala.js logo
Scala.js logo
Scala.js logo

Binary incompatibilities lock down the ecosystem

Source incompatibilities are only a one-time inconvenience to directly dependent projects

Summary so far

  • Binary compatibility:
    • hard to get right
    • when broken: disastrous for the ecosystem
  • Source compatibility:
    • virtually impossible to achieve
    • when broken: inconvenient for direct dependents

Versioning scheme

When the new version is:

  • Binary incompatible: major version bump
  • Source incompatible: minor version bump
  • Compatible: patch version bump

Part Ⅱ

How to avoid binary incompatibilities

Answer: the Migration Manager, aka MiMa

Sometimes needs filters
requires good knowledge about the compile scheme
(even scalac people get it wrong sometimes)

How to avoid source incompatibilities

Do not change your public/protected API

Use MiMa both ways (forward+backward) as an approximation

When MiMa is not enough

Learn the compilation scheme with -Xprint:mixin

Default parameters


class Test {
  def foo(x: Int, y: Int = 1, z: String = "hello"): Int = ???
}

test.foo(3, 4)
    

def foo(x: Int, y: Int, z: String): Int = scala.Predef.???();
<synthetic> def foo$default$2(): Int = 1;
<synthetic> def foo$default$3(): String = "hello";

test.foo(3, 4, test.foo$default$3());

How to recover?


class Test {
  @deprecated("...", "...")
  def foo(x: Int, y: Int = 1, z: String = "hello"): Int = ???

  def foo(x: Int, y: Int, z: String, w: Boolean): Int = ???
}

test.foo(3, 4) // deprecated
test.foo(3, 4, "hello", false)

Option bags


case class FooOptions(x: Int, y: Int = 1)
    

case class FooOptions extends Object with Product with Serializable {
    private[this] val x: Int = _;
      def x(): Int = ...;
    private[this] val y: Int = _;
      def y(): Int = ...;
   def copy(x: Int, y: Int): bincompat.FooOptions = ...;
   def copy$default$1(): Int = ...;
   def copy$default$2(): Int = ...;
  override  def productPrefix(): String = ...;
   def productArity(): Int = 2;
   def productElement(x$1: Int): Object = ...;
  override  def productIterator(): Iterator = ...;
   def canEqual(x$1: Object): Boolean = ...;
  override  def hashCode(): Int = ...;
  override  def toString(): String = ...;
  override  def equals(x$1: Object): Boolean = ...;
  def (x: Int, y: Int): bincompat.FooOptions = ...
};
 object FooOptions extends AbstractFunction2 with Serializable {
   def $default$2(): Int = ...;
  final override  def toString(): String = ...;
  case  def apply(x: Int, y: Int): bincompat.FooOptions = ...;
   def apply$default$2(): Int = ...;
  case  def unapply(x$0: bincompat.FooOptions): Option = if (x$0.==(null))
    scala.None
  else
    new Some(new Tuple2$mcII$sp(x$0.x(), x$0.y()));
   private def readResolve(): Object = ...;
  case    def apply(v1: Object, v2: Object): Object = ...;
  def (): bincompat.FooOptions.type = ...
};

How to recover?

Tough! Desugar it by hand.


final class FooOptions private (val x: Int, val y: Int) {
  private def this() = this(x = 0, y = 1)

  private def withX(x: Int): FooOptions = copy(x = x)
  private def withY(y: Int): FooOptions = copy(y = y)

  private def copy(x: Int = x, y: Int = y): FooOptions =
    new FooOptions(x, y)

  override def toString(): String = s"FooOptions($x, $y)"

  override def equals(that: Any): Boolean = that match {
    case that: FooOptions => this.x == that.y && this.y == y
    case _                => false
  }

  override def hashCode(): Int = {
    import scala.util.hashing.MurmurHash3._
    var acc = classOf[FooOptions].getName.##
    acc = mix(acc, x.##)
    acc = mixLast(acc, y.##)
    finalizeHash(acc, 2)
  }
}

object FooOptions {
  def apply(): FooOptions = new FooOptions()
}

What are case classes good for, then?

For pattern matching, in order words, if you intend to use them in case clauses!

Oh and ... make them final!

Open traits


trait Test {
  private var bar: Int = 5
  def foo(x: Int): Int = x + bar
}
class TestImpl extends Test
    

abstract trait Test extends Object {
  <accessor> <sub_synth> def bar(): Int;
  <accessor> <sub_synth> def bar_=(x$1: Int): Unit;
  def foo(x: Int): Int = x.+(Test.this.bar());
  def /*Test*/$init$(): Unit = {
    Test.this.bar_=((5: Int));
    ()
  }
};
class TestImpl extends Object with bincompat.Test {
  def foo(x: Int): Int = TestImpl.super.foo(x);
  override <accessor> def bar(): Int = this.bar;
  private[this] var bar: Int = _;
  override <accessor> def bar_=(x$1: Int): Unit =
    TestImpl.this.bar = (x$1: Int);
  def <init>(): bincompat.TestImpl = {
    TestImpl.super.<init>();
    TestImpl.super./*Test*/$init$();
    ()
  }
};

Open traits

  • Seal your traits
  • Or make sure they are completely abstract interfaces (Java-like) -- and never change them

Write-only ADTs


abstract sealed class Foo private ()

object Foo {
  private[lib] case class Bar(x: Int) extends Foo
  ...

  def bar(x: Int): Foo = Bar(x)
}

Construct in user-space; extract in library

Disappearing methods


final class Test {
  @deprecated("don't use this", "1.1.0")
  def foo(x: Int): Int = x + 1
}
    

final class Test {
  protected[Test] def foo(x: Int): Int = x + 1
}

Privatize

I have stopped using ...

  • Result type inference
  • Default parameters
  • Case classes (unless write-only)
  • Non-sealed classes and traits
  • Concrete members in open traits
  • Public APIs

What about Scala.js? Scala Native?

How long can we sustain this?

Outlast each Scala major version

Will TASTY solve this?

A bit, but mostly no
(it also makes some things worse)

Summary

  • Binary compat matters more than source compat
    -> reflect in version numbers
  • MiMa prevents you from creating an incompatible version
  • It won't help you if you've locked yourself down in a previous version
  • Learn the compilation scheme
  • Same reasoning and tools apply to Scala.js and Scala Native libraries
Scala.js logo

scala-js.org