Practical Aggregate Programming in Scala

Practical Aggregate Programming in Scala
Roberto Casadei
PhD Student in CS&Eng
roby.casadei@unibo.it
Department of Computer Science and Engineering
University of Bologna
Student talk at Scala Symposium, Amsterdam 2016
Slides available at http://www.slideshare.net/RobertoCasadei/presentations
Sample code at https://bitbucket.org/metaphori/scafi-tutorial
R. Casadei (Università di Bologna) Practical Aggregate Programming in Scala Scala Symposium ’16 1 / 30

Outline
1 Aggregate Computing: The Basics
2 SCAFI: Practical Aggregate Programming in Scala
3 Conclusion

Aggregate Computing: The Basics
Outline
3 Conclusion

Problem: design/programming CASs
Collective/Complex Adaptive Systems (CASs)
Structure: Environment + (Mobile, Large-scale) Networks of { people + devices }
Global interpretation: embedded devices collectively form a “diffused” computational system

An approach to CAS development
Issues ⇒ approach
• Large-scale ⇒ decentralised coordination
• Situatedness + distributed autonomy ⇒ substantial unpredictability ⇒ self-*
• Complex collective behavior ⇒ good abstractions, layered approach, compositionality
Shifting the mindset: from local to global
• Declarativeness and the global viewpoint
• Crowd-aware services
• Failure recovery of enterprise services
• Distributed monitoring and reacting (e.g., temperature, ﬁre)
• Expected global behavior vs. traditional device-centric interface
⇒ Aggregate Programming [BPV15]: a paradigm for programming whole aggregates of devices.

Aggregate programming [BPV15]
From the local/device-centric viewpoint to the global/aggregate viewpoint
Aggregate programming: what
Goal: programming the collective behaviour of aggregates (of devices) ⇒ global-to-local
Aggregate programming: how
Prominent approach (generalizing over several prior approaches and strategies [BDU+
12])
founded on ﬁeld calculus and self-org patterns
• Computational ﬁelds as unifying abstraction of local/global viewpoints

Aggregate Programming Stack

Aggregate (computing) systems & Execution model
Structure ⇒ (network/graph)
• A set of devices (aka nodes/points/things).
• Each device is able to communicate with a subset of devices known as its neighbourhood.
Dynamics
Each device is given the same aggregate program and works at async / partially-sync rounds:
(1) Retrieve context
⇐ Messages from neighbours
⇐ Sensor values
(2) Aggregate program execution
⇒ export (a tree-like repr of computation) + output (result of last expr in body)
(3) Broadcast export to neighbourhood
(4) Execute actuators

SCAFI: Practical Aggregate Programming in Scala
Outline
3 Conclusion

SCAFI: Scala with Computational Fields
Goal: bring Aggregate Computing to the field of mainstream software development
What
SCAFI [CV16] is an integrated framework for building systems with aggregate programming.
• Scala-internal DSL for expressing aggregate computations.
• Linguistic support + execution support (interpreter/VM)
• Correct, complete, efficient impl. of the Higher-Order Field Calculus
semantics [DVPB15]
• Distributed platform for execution of aggregate systems.
• Support for multiple architectural styles and system configurations.
• Actor-based implementation (based on Akka).
Where
• https://bitbucket.org/scafiteam/scafi
libraryDependencies += "it.unibo.apice.scafiteam" % "scafi-core_2.11" % "0.1.0" // on Maven Central

Computational ﬁelds [DVB16]
• (Abstract interpretation) Mapping space-time to computational objects
• (Concrete interpretation) Mapping devices to values: φ : δ →
• “Distributed” data structure working as the global abstraction
• The bridge abstraction between local behavior and global behavior
Discrete systems as an approximation of
spacetime

Field/aggregate computations
Global viewpoint
• Aggregate interpretation
• Natural/denotational semantics
• Program: computation over whole ﬁelds
• Output (at a given time): system-wide
snapshot of a computational ﬁeld
• Geometric view: properties of collections
of points
Local viewpoint
• Device-centric interpretation
• Operational semantics
• Program: steps of a single device
• Output (at a given time): latest value
yielded by the device
• Geometric view: properties of a single
point

So, what is an aggregate program?
• The global program
– "Local programs" obtained via global-to-local mapping
• May take the form of field calculus programs (in the representation given by some PL)
– Actually, the field calculus is like FJ for Java, or the lambda calculus for Haskell
• An aggregate program consists of
1) A set of function definitions
2) A body of expressions.
• Example: an aggregate program in SCAFI
class MyProgram extends AggregateProgram with MyAPI {
def isSource = sense[Boolean]("source")
// Entry point for execution
override def main() = gradient(isSource)
}
– Each device of the aggregate system is given an instance of MyProgram.
– Each device repeatedly runs the main method at async rounds of execution.

Computing with fields
Expressing aggregate/field computations in SCAFI
trait Constructs {
def rep[A](init: A)(fun: (A) => A): A
def nbr[A](expr: => A): A
def foldhood[A](init: => A)(acc: (A,A)=>A)(expr: => A): A
def aggregate[A](f: => A): A
// Not primitive, but foundational
def sense[A](name: LSNS): A
def nbrvar[A](name: NSNS): A
def branch[A](cond: => Boolean)(th: => A)(el: => A): A
}
• Mechanisms for context-sensitiveness: nbr, nbrvar, sense
• Mechanisms for field evolution: rep
• Mechanisms for interaction: nbr
• Mechanisms for field domain restriction and partitioning: aggregate, branch
• Reference formal system: field calculus [DVB16, DVPB15]

Simple fields
0
(x)=>x+1
true t<0,1>
Constant, uniform field: 5
– Local view: evaluates to 5 in the context of a single device
– Global view: yields a uniform constant field that holds 5 at any point (i.e., at any device)
Constant, non-uniform field: mid()
– mid() is a built-in function that returns the ID of the running device

rep: dynamically evolving ﬁelds
rep
0
(x)=>x+1
t
v0
t
v1
..
rep(0){(x)=>x+1}
// Signature: def rep[A](init: A)(fun: (A) => A): A
// Initially 0; state is incremented at each round
rep(0){ _+1 }
– Notice: the frequency of computation can vary over time and from device to device
– In general, the resulting ﬁeld will be heterogeneous in time and space

nbr: interaction, communication, observation
nbr de
nbr{e}
φd=[d1→v1,..,dn→vn]
– Local view: nbr returns a field from neighbors to their corresponding value of the given expr e
– Global view: a field of fields
– Needs to be reduced using a *hood operation
– foldhood works by retrieving the value of expr for each neighbour and then folding over the
resulting structure as you’d expect from FP.
// Signature: def nbr[A](expr: => A): A
// Signature: def foldhood[A](init: => A)(acc: (A,A)=>A)(expr: => A): A
foldhood(0)(_+_){ nbr{1} }

Context-sensitiveness and sensors
Local context:
1) The export of the previous computation
2) Messages received from neighbours
3) Values perceived from the physical/software environment
Sensing
// Query a local sensor
sense[Double]("temperature")
// Compute the maximum distance from neighbours
foldhood(Double.MinValue)(max(_,_)){ // Also: maxHood {...}
nbrvar[Double](NBR_RANGE_NAME)
}
– nbr queries a local sensor
– nbrvar queries a "neighbouring sensor" (a sort of "environmental probe")

Field domain restriction
Alignment
• Aggregate computations can be represented as a trees
• Device exports are "paths" along these trees
• When two devices execute the same tree node, they are said to be aligned
• Interaction is possible only between aligned devices
Use cases for branch
• Partitioning the space into subspaces performing subcomputations
• Regulating admissible interactions (i.e., further restricting the neighbourhood)
branch(sense[Boolean]("flag")){
compute(...) // sub-computation
}{
Double.MaxValue // stable value (i.e., not computing)
}

Functions
Two kinds of functions in SCAFI:
1) "Normal" Scala functions: serve as units for encapsulating behavior/logic
def foldhoodMinus[A](init: => A)(acc: (A,A) => A)(ex: => A): A =
foldhood(init)(acc){ mux(mid()==nbr(mid())){ init }{ ex } }
def isSource = sense[Boolean]("source")
2) First-class "aggregate" functions [DVPB15] – which also work as units for alignment
def branch[A](cond: => Boolean)(th: => A)(el: => A): A =
mux(cond, ()=>aggregate{ th }, ()=>aggregate{ el })()

Example: the gradient [BBVT08]
def nbrRange = nbrvar[Double](NBR_RANGE_NAME)
def gradient(source: Boolean): Double =
rep(Double.PositiveInfinity){ distance =>
mux(source) {
0.0
}{
minHood { nbr{distance} + nbrRange }
}
}

Example: the channel I

Example: the channel II
Each device is given the same aggregate program:
class ChannelProgram extends AggregateProgram with ChannelAPI {
def main = channel(isSource, isDestination, width)
}
def channel(src: Boolean, dest: Boolean, width: Double) =
distanceTo(src) + distanceTo(dest) <= distBetween(src, dest) + width

Example: the channel III
trait ChannelAPI extends Language with Builtins {
def channel(src: Boolean, dest: Boolean, width: Double) =
distanceTo(src) + distanceTo(dest) <= distBetween(src, dest) + width
def G[V:OB](src: Boolean, field: V, acc: V=>V, metric: =>Double): V =
rep( (Double.MaxValue, field) ){ dv =>
mux(src) { (0.0, field) } {
minHoodMinus {
val (d, v) = nbr { (dv._1, dv._2) }
(d + metric, acc(v))
}
}
}._2
def broadcast[V:OB](source: Boolean, field: V): V =
G[V](source, field, x=>x, nbrRange())
def distanceTo(source: Boolean): Double =
G[Double](source, 0, _ + nbrRange(), nbrRange())
def distBetween(source: Boolean, target: Boolean): Double =
broadcast(source, distanceTo(target))
def nbrRange(): Double = nbrvar[Double](NBR_RANGE_NAME)
def isSource = sense[Boolean]("source"); def isDestination = sense[Boolean]("destination")
}

Scaling with complexity
General coordination operators [VBDP15]
• Gradient-cast: accumulates values “outward” along a gradient starting from source nodes.
def G[V:OB](src: Boolean, init: V,
acc: V=>V, metric: =>Double): V
• Converge-cast: collects data distributed across space “inward” by accumulating values from
edge nodes to sink nodes down a “potential” ﬁeld.
def C[V:OB](potential: V, acc: (V,V)=>V, local: V, Null: V): V
• Time-decay: supports information summarisation across time.
def T[V:Numeric](initial: V, floor: V, decay: V=>V): V
• Sparse-choice: supports creation of partitions and selection of sparse subsets of devices in
space
def S(grain: Double, metric: => Double): Boolean

A case study: crowd engineering [CPV16]
val (high,low,none) = (2,1,0) // crowd level
def crowdWarning(p: Double, r: Double, warn: Double, t: Double):
Boolean =
distanceTo(crowdTracking(p,r,t) == high) < warn
def crowdTracking(p: Double, r: Double, t: Double) = {
val crowdRgn = recentTrue(densityEst(p, r)>1.08, t)
branch(crowdRgn){ dangerousDensity(p, r) }{ none }
}
def dangerousDensity(p: Double, r: Double) = {
val mr = managementRegions(r*2, () => { nbrRange })
val danger = average(mr, densityEst(p, r)) > 2.17 &&
summarize(mr, (_:Double)+(_:Double), 1 / p, 0) > 300
mux(danger){ high }{ low }
}
// Auxiliary functions
def recentTrue(state: Boolean, memTime: Double): Boolean
def managementRegions(grain: Double,
metric: => Double): Boolean = S(gran,metric)
def densityEst(p: Double, range: Double): Double
def summarize(sink: Boolean, acc: (Double,Double)=>Double,
local: Double, Null: Double): Double
def average(sink: Boolean, value: Double): Double

Quick platform setup
// STEP 1: CHOOSE INCARNATION
import it.unibo.scafi.incarnations.{ BasicActorP2P => Platform }
// STEP 2: DEFINE AGGREGATE PROGRAM SCHEMA
class Demo_AggregateProgram extends Platform.AggregateProgram {
override def main(): Any = foldhood(0){_ + _}(1)
}
// STEP 3: DEFINE MAIN PROGRAM
object Demo_MainProgram extends Platform.CmdLineMain
1) Demo_MainProgram -h 127.0.0.1 -p 9000
-e 1:2,4,5;2;3 --subsystems 127.0.0.1:9500:4:5
--program "demos.Demo_AggregateProgram"
2) Demo_MainProgram -h 127.0.0.1 -p 9500
-e 4;5:4
--program "demos.Demo_AggregateProgram"

Manual node setup
// STEP 1: CHOOSE INCARNATION
import scafi.incarnations.{ BasicActorP2P => Platform }
import Platform.{AggregateProgram,Settings,PlatformConfig}
// STEP 2: DEFINE AGGREGATE PROGRAM SCHEMA
class Program extends AggregateProgram with CrowdAPI {
// Specify a "dangerous density" aggregate computation
override def main(): Any = crowdWarning(...)
}
// STEP 3: PLATFORM SETUP
val settings = Settings()
val platform = PlatformConfig.setupPlatform(settings)
// STEP 4: NODE SETUP
val sys = platform.newAggregateApplication()
val dm = sys.newDevice(id = Utils.newId(),
program = Program,
neighbours = Utils.discoverNbrs())
val devActor = dm.actorRef // get underlying actor

Conclusion
Outline
3 Conclusion

Conclusion
Summary: key ideas
Aggregate programming
• A "macro" (programmingengineering) approach to CASs, formally grounded in the Field
Calculus.
• Allows to compose “emergent” phenomena & deﬁnes layers of (self-stabilizing) building blocks.
SCAFI: a Scala framework for Aggregate Programming
• Provides an internal DSL for ﬁeld-based computations
• Provides an actor-based platform for building aggregate systems
Future work
• Evolve SCAFI to support scalable computations in cluster- and cloud-based systems.
• What does it take to set up a framework for adaptive execution strategies?

Conclusion
Question time
Questions?

Appendix References
References I
[BBVT08] Jacob Beal, Jonathan Bachrach, Dan Vickery, and Mark Tobenkin.
Fast self-healing gradients.
In Proceedings of the 2008 ACM symposium on Applied computing, pages 1969–1975.
ACM, 2008.
[BDU+
12] Jacob Beal, Stefan Dulman, Kyle Usbeck, Mirko Viroli, and Nikolaus Correll.
Organizing the aggregate: Languages for spatial computing.
CoRR, abs/1202.5509, 2012.
[BPV15] Jacob Beal, Danilo Pianini, and Mirko Viroli.
Aggregate Programming for the Internet of Things.
IEEE Computer, 2015.

Appendix References
References II
[CPV16] Roberto Casadei, Danilo Pianini, and Mirko Viroli.
Simulating large-scale aggregate mass with alchemist and scala.
In Maria Ganzha, Leszek Maciaszek, and Marcin Paprzycki, editors, Proceedings of the
Federated Conference on Computer Science and Information Systems (FedCSIS
2016), Gdansk, Poland, 11-14 September 2016. IEEE Computer Society Press.
To appear.
[CV16] Roberto Casadei and Mirko Viroli.
Towards aggregate programming in Scala.
In First Workshop on Programming Models and Languages for Distributed Computing,
PMLDC ’16, pages 5:1–5:7, New York, NY, USA, 2016. ACM.
[DVB16] Ferruccio Damiani, Mirko Viroli, and Jacob Beal.
A type-sound calculus of computational ﬁelds.
Science of Computer Programming, 117:17 – 44, 2016.

Appendix References
References III
[DVPB15] Ferruccio Damiani, Mirko Viroli, Danilo Pianini, and Jacob Beal.
Code mobility meets self-organisation: A higher-order calculus of computational fields.
volume 9039 of Lecture Notes in Computer Science, pages 113–128. Springer
International Publishing, 2015.
[VBDP15] Mirko Viroli, Jacob Beal, Ferruccio Damiani, and Danilo Pianini.
Efficient engineering of complex self-organising systems by self-stabilising fields.
2015.

Practical Aggregate Programming in Scala

Recommended

More Related Content

What's hot (18)

Viewers also liked (10)

Similar to Practical Aggregate Programming in Scala (20)

More from Roberto Casadei (20)

Recently uploaded (20)

Practical Aggregate Programming in Scala