Setup worker-worker connections lazily #42
Description
The default all_to_all topology connects all processes to each other. While this is fine for small clusters, the total number of TCP connections increases rapidly as (N^2)/2.
Considering that a large class of parallel problems only need master-worker connections we should change the default topology to all_to_all_lazy where worker-worker connections are setup only on the first request from a worker to another worker. And also introduce another topology master_routed which only connects master to workers, and in case of a worker-worker call, routes the request through the master.
To summarize, implement 2 new topologies:
-
all_to_all_lazywhere worker-worker connections are setup lazily, and is the default for addprocs and -
master_routedin which only the master connects to workers and worker-worker messages are routed via the master.