Support dynamically add/set worker threads.

[josefsachsconning]

Said Sachs:
Is there any prospect for setting jl_n_threads from within Julia? That would be more convenient than setting JULIA_NUM_THREADS before starting Julia.

Said Kelman:
Sounds like a reasonable feature request to me, to have a Threads.getnum and Threads.setnum (or equivalent names) API.

[kpamnany]

High performance fork-join constructs (broadcast, reduce-barrier) typically require some setup that is dependent on the number of threads. Adding/removing threads would require teardown and recreation, and while this is certainly not impossible, I haven't run into a situation where this would be helpful. Can you describe such a situation?

[josefsachsconning]

I have 2 use cases where this would be helpful.

1. I have some applications where I want a maximum of 1 thread, and other applications where I want as many threads as I have cores. It would be much more convenient for the logic implementing this choice to be in the Julia code, rather than in a wrapper script that sets the JULIA_NUM_THREADS environment variable and then invokes the Julia script.
2. When I benchmark the effect of multithreading with varying numbers of threads, I would prefer to do it in a single Julia process rather than a different Julia process for each number of threads that I am testing.

[StefanKarpinski]

Any reason this can't be done by allocating space for all potential threads up to the number of physical cores and then allowing the number of those actually used to be ramped up and down dynamically?

[kpamnany]

Any reason this can't be done by allocating space for all potential threads up to the number of physical cores and then allowing the number of those actually used to be ramped up and down dynamically?

Probably be too costly (remember KNL) this way, but like I said, it isn't that hard to do. At least in the threading infrastructure.

I'm not sure about the GC or anything in the runtime. @yuyichao, @vtjnash: can we realloc jl_all_heaps and jl_all_task_states? Would it be hard to do?

[yuyichao]

Runtime shouldn't have too much trouble if the scheduler can support this (what does it mean if a worker thread request itself to be deleted?). jl_all_heaps should be removed (merged into jl_tls_states_t) and then all access to jl_all_task_states (should mostly be in slow path like gc and signal handling anyway) needs to be protected with a lock.

[vchuravy]

@vtjnash Mentioned to me that the ability of dynamically launching threads + setting up ptls state would maybe enable calling (asynchronously) into Julia from C/C++ in a thread-safe manner.

[orenbenkiki]

A potential use case for this: Running multiple cooperating Julia HPC applications on a multi-core server.

For example: If an application runs by itself, it can use all the CPUs. If a second application is launched, the "polite" thing to do would be for the 1st to halve its number of threads (at the nearest "convenient point", i.e. some task end), and for the 2nd application to only request half the number of CPUs. When the 1st application ends, the 2nd application can increase its number of threads to take over all the CPUs.

In general, assume:

• All the applications that "matter" are run in Julia, and use the same run-time, that coordinates between them to adjust the threads and schedule tasks.
• The applications are implemented as a DAG of "not-too-long" tasks.
• Whenever an application starts (or ends), the run-time adjusts the (target) number of threads in the remaining applications.
• Adding threads allows immediately scheduling a task on each.
• Whenever a task ends, the run-time decides whether to schedule a new one on this thread, or to terminate the thread, if the target number of threads is above the current number of threads.
• When a thread is terminated, the run-time can launch a new thread in another "more deserving" application.

Such a system would maximize CPU utilization, while providing "fairness" between multiple applications. Both are desired features in a server used by multiple users. In fact, these properties are desired even when the server is used by a single user, but the workflow invokes different Julia applications for different processing steps, possibly in parallel.

This is not to say that the above behavior should be built into Julia (though that would be very nice, at least as an option ;-). Rather, the point is that if someone wanted to implement such a run-time, the ability to dynamically adjust the number of threads would be required or at least very helpful.

Currently the only way to implement such a system would be for each application to spawn the maximal number of threads and "pause" the excess threads. I'm not certain how to effectively "pause" a thread in Julia (and be able to later wake it up).

From OS perspective, the "good old way" would be to send that process a STOP signal, but I'm not certain how Julia would respond to stopping just one thread of a multi-process application. Another option would be for the process to use a pthread_mutex, I'm not certain this is accessible from within a single Julia thread w/o impacting the other threads.

Given that launching an application is a "heavy" operation, the cost of starting/stopping threads at such times is not an excessive overhead. Assuming it is possible to "pause" a thread, the question becomes whether the extra cost of maintaining such some "paused" threads for the total duration of the application execution is higher than the overhead or occasionally starting/stopping threads.

The simplest API would be exit_current_thread() and spawn_more_threads(n). This avoids the issue of one thread killing another in mid-work.