Exploring type level programming in Scala
0 likes91 views
The document discusses type-level programming in Scala, emphasizing the significance of dependent types in enhancing type safety and minimizing runtime bugs, particularly in the context of Spark's DataFrame API. It provides examples of how libraries such as Frameless leverage Scala's type system to catch errors at compile-time instead of at runtime. The presentation concludes by highlighting various libraries that utilize type-level programming techniques, such as Shapeless, Refinement, and ZIO SQL.
![Concrete example: Spark with Frameless
// Error:
No column Symbol with
shapeless.tag.Tagged[Str
ing("username")]](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-18-320.jpg)
![Concrete example: Spark with Frameless
// Error:
overloaded method value
* with alternatives
[...] cannot be applied
to (Int)](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-20-320.jpg)
![Concrete example: Spark with Frameless
// Error:
overloaded method value
=== with alternatives
[...] cannot be applied
to (Boolean)](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-22-320.jpg)
![Concrete example: Spark with Frameless
// Error:
overloaded method value /
with alternatives [...]
cannot be applied to
(Int)](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-24-320.jpg)
![Example 1: Merging Files
import java.io.File
sealed trait MergeStrategy {
type Output
}
object MergeStrategy {
case object Single extends MergeStrategy { type Output = File }
case object Multiple extends MergeStrategy { type Output = List[File] }
case object None extends MergeStrategy { type Output = Unit }
}
def mergeFiles(files: List[File]): File = ???](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-35-320.jpg)
(f: mergeStrategy.Output => T): T =
mergeStrategy match {
case MergeStrategy.Single => f(mergeFiles(files))
case MergeStrategy.Multiple => f(files)
case MergeStrategy.None => f(())
}](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-36-320.jpg)
(f: O => T): T =
mergeStrategy match {
case MergeStrategy.Single => f(mergeFiles(files))
case MergeStrategy.Multiple => f(files)
case MergeStrategy.None => f(())
}](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-38-320.jpg)
![Example 1: Merging Files
val files: List[File] = ???
merge(files, MergeStrategy.Single) { file: File =>
// Do some processing
}
merge(files, MergeStrategy.Multiple) { files: List[File] =>
// Do some processing
}
merge(files, MergeStrategy.None) { _: Unit =>
// Do some processing
}](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-39-320.jpg)
![Example 2: Merging Elements
sealed trait MergeStrategy {
type Output[_]
}
object MergeStrategy {
case object Single extends MergeStrategy { type Output[A] = A }
case object Multiple extends MergeStrategy { type Output[A] = List[A] }
case object None extends MergeStrategy { type Output[_] = Unit }
}
def mergeElements[E](elements: List[E]): E = ???](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-41-320.jpg)
![Example 2: Merging Elements
def merge[E, O[_], T](
elements: List[E],
mergeStrategy: MergeStrategy { type Output[A] = O[A] }
)(f: O[E] => T): T =
mergeStrategy match {
case MergeStrategy.Single => f(mergeElements(elements))
case MergeStrategy.Multiple => f(elements)
case MergeStrategy.None => f(())
}](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-42-320.jpg)
![Example 2: Merging Elements
val messages: List[String] = ???
merge(messages, MergeStrategy.Single) { message: String =>
// Do some processing
}
merge(messages, MergeStrategy.Multiple) { messages: List[String] =>
// Do some processing
}
merge(messages, MergeStrategy.None) { _: Unit =>
// Do some processing
}](https://image.slidesharecdn.com/exploringtype-levelprogramminginscalav2-201208150451/85/Exploring-type-level-programming-in-Scala-43-320.jpg)
Exploring type level programming in Scala
- 4. Agenda ● Motivations around Type-Level Programming in Scala ● Background ○ Dependent types ○ Path-dependent types ○ Abstract type members ● Examples of libraries that use Type-Level Programming
- 6. Why Type-Level Programming? ● We turn to Scala for type-safety ● But… Sometimes it’s not enough!
- 7. Concrete example: Spark DataFrame API We have lots of potential for runtime bugs!
- 8. Concrete example: Spark DataFrame API df.select( $"username", $"tweet" )
- 9. Concrete example: Spark DataFrame API Error: org.apache.spark.sql. AnalysisException: cannot resolve '`username`'
- 10. Concrete example: Spark DataFrame API df.select( $"timestamp" * 10 )
- 11. Concrete example: Spark DataFrame API Error: org.apache.spark.sql. AnalysisException: cannot resolve '(`timestamp` * 10)' due to data type mismatch
- 12. Concrete example: Spark DataFrame API df.filter( $"text" === true ) // Does not fail
- 13. Concrete example: Spark DataFrame API df.select( $"text" / 1000 ) // Does not fail
- 15. Can we do better? Scala has a VERY powerful type system, why not use it?
- 16. Concrete example: Spark with Frameless Frameless helps us to eliminate a lot of bugs… At compile time!
- 17. Concrete example: Spark with Frameless tds.select( tds('username), tds('tweet) )
- 18. Concrete example: Spark with Frameless // Error: No column Symbol with shapeless.tag.Tagged[Str ing("username")]
- 19. Concrete example: Spark with Frameless tds.select( tds('timestamp) * 10 )
- 20. Concrete example: Spark with Frameless // Error: overloaded method value * with alternatives [...] cannot be applied to (Int)
- 21. Concrete example: Spark with Frameless tds.filter( tds('text) === true )
- 22. Concrete example: Spark with Frameless // Error: overloaded method value === with alternatives [...] cannot be applied to (Boolean)
- 23. Concrete example: Spark with Frameless tds.select( tds('text) / 1000 )
- 24. Concrete example: Spark with Frameless // Error: overloaded method value / with alternatives [...] cannot be applied to (Int)
- 25. In conclusion... Type-level Programming lets you eliminate bugs at compile-time
- 26. In conclusion...
- 27. Our focus today: Dependent types Dependent types are the heart of Type-Level Programming in Scala
- 29. What are Dependent Types? ● Dependent Types are types that depend on values. ● With this, we remove the usual separation between the type and value worlds.
- 30. What are Dependent Types? ● Scala is not a fully dependently typed language. ● However, it supports some form of Dependent Types, which is called Path Dependent Types.
- 31. How we define Path Dependent Types? ● In Scala, we can define nested components ● For example, a class inside a trait, a trait inside a class, etc.
- 32. How we define Path Dependent Types? sealed trait Foo { sealed trait Bar } val foo1 = new Foo {} val foo2 = new Foo {} val a: Foo#Bar = new foo1.Bar {} // OK val b: Foo#Bar = new foo2.Bar {} // OK val c: foo1.Bar = new foo1.Bar {} // OK val d: foo2.Bar = new foo1.Bar {} // Required: foo2.Bar, Found: foo1.Bar
- 33. How we define Path Dependent Types? ● Another useful tool is Abstract Type Members, which are types we don’t know yet and that we can define later trait Bar { type T }
- 34. Example 1: Merging Files Define a merge function, which should take: ● A list of files ● A merge strategy: Single/Multiple/None ● A callback function: Which should expect: ○ A single file if merge strategy is Single ○ A list of files if merge strategy is Multiple ○ A unit value if merge strategy is None
- 35. Example 1: Merging Files import java.io.File sealed trait MergeStrategy { type Output } object MergeStrategy { case object Single extends MergeStrategy { type Output = File } case object Multiple extends MergeStrategy { type Output = List[File] } case object None extends MergeStrategy { type Output = Unit } } def mergeFiles(files: List[File]): File = ???
- 36. Example 1: Merging Files def merge[T](files: List[File], mergeStrategy: MergeStrategy) (f: mergeStrategy.Output => T): T = mergeStrategy match { case MergeStrategy.Single => f(mergeFiles(files)) case MergeStrategy.Multiple => f(files) case MergeStrategy.None => f(()) }
- 37. Example 1: Merging Files
- 38. Example 1: Merging Files def merge[O, T]( files: List[File], mergeStrategy: MergeStrategy { type Output = O } )(f: O => T): T = mergeStrategy match { case MergeStrategy.Single => f(mergeFiles(files)) case MergeStrategy.Multiple => f(files) case MergeStrategy.None => f(()) }
- 39. Example 1: Merging Files val files: List[File] = ??? merge(files, MergeStrategy.Single) { file: File => // Do some processing } merge(files, MergeStrategy.Multiple) { files: List[File] => // Do some processing } merge(files, MergeStrategy.None) { _: Unit => // Do some processing }
- 40. Example 2: Merging Elements Define a merge function, which should take: ● A list of elements of any type ● A merge strategy: Single/Multiple/None ● A callback function: Which should expect: ○ A single element if merge strategy is Single ○ A list of elements if merge strategy is Multiple ○ A unit value if merge strategy is None
- 41. Example 2: Merging Elements sealed trait MergeStrategy { type Output[_] } object MergeStrategy { case object Single extends MergeStrategy { type Output[A] = A } case object Multiple extends MergeStrategy { type Output[A] = List[A] } case object None extends MergeStrategy { type Output[_] = Unit } } def mergeElements[E](elements: List[E]): E = ???
- 42. Example 2: Merging Elements def merge[E, O[_], T]( elements: List[E], mergeStrategy: MergeStrategy { type Output[A] = O[A] } )(f: O[E] => T): T = mergeStrategy match { case MergeStrategy.Single => f(mergeElements(elements)) case MergeStrategy.Multiple => f(elements) case MergeStrategy.None => f(()) }
- 43. Example 2: Merging Elements val messages: List[String] = ??? merge(messages, MergeStrategy.Single) { message: String => // Do some processing } merge(messages, MergeStrategy.Multiple) { messages: List[String] => // Do some processing } merge(messages, MergeStrategy.None) { _: Unit => // Do some processing }
- 44. In conclusion... ● Path-dependent types are the heart and soul of Scala's type system ● They help you to improve compile-time type safety
- 45. In conclusion... DOTTY = DOT Calculus = Path Dependent Types
- 47. Examples of libraries that use Type Level Programming ● Shapeless: Generic programming ○ Generic Product Type: HList ○ Generic Sum Type: Coproduct
- 48. Examples of libraries that use Type Level Programming ● Frameless: Expressive types for Spark
- 49. Examples of libraries that use Type Level Programming ● Refined: Refinement types
- 50. Examples of libraries that use Type Level Programming ● ZIO SQL: Type-safe SQL queries
- 51. References ● Dependent types in Scala, blog post by Yao Li ● Type Level Programming in Scala step by step, blog series by Luigi Antonini ● The Type Astronaut’s Guide to Shapeless Book, by Dave Gurnell ● Introduction to Apache Spark with Frameless, by Brian Clapper
- 52. Special thanks ● To micro sphere.it organizers for hosting this presentation ● To John De Goes for guidance and support
- 53. Thank You!