mdp.parallel.ThreadScheduler

Class ThreadScheduler

Thread based scheduler. Because of the GIL this only makes sense if most of the time is spend in numpy calculations (or some other external non-blocking C code) or for IO, but can be more efficient than ProcessScheduler because of the shared memory.

Instance Methods

[hide private]

__init__(self, result_container=None, verbose=False, n_threads=1, copy_callable=True)
Initialize the scheduler.

_process_task(self, data, task_callable, task_index)
Add a task, if possible without blocking.

_task_thread(self, data, task_callable, task_index)
Thread function which processes a single task.

Inherited from unreachable.newobject: __long__, __native__, __nonzero__, __unicode__, next

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Inherited from Scheduler

__enter__(self)
Return self.

__exit__(self, type, value, traceback)
Shutdown the scheduler.

_shutdown(self)
Hook method for shutdown to be used in custom schedulers.

_store_result(self, result, task_index)
Store a result in the internal result container.

add_task(self, data, task_callable=None)
Add a task to be executed.

get_results(self)
Get the accumulated results from the result container.

set_task_callable(self, task_callable)
Set the callable that will be used if no task_callable is given.

shutdown(self)
Controlled shutdown of the scheduler.

Properties

[hide private]

Inherited from object: __class__

Inherited from Scheduler

n_open_tasks
This property counts of submitted but unfinished tasks.

task_counter
This property counts the number of submitted tasks.

init(self, result_container=None, verbose=False, n_threads=1, copy_callable=True)
(Constructor)

Initialize the scheduler.

result_container -- ResultContainer used to store the results.
verbose -- Set to True to get progress reports from the scheduler
    (default value is False).
n_threads -- Number of threads used in parallel. If None (default)
    then the number of detected CPU cores is used.
copy_callable -- Use deep copies of the task callable in the threads.
    This is for example required if some nodes are stateful during
    execution (e.g., a BiNode using the coroutine decorator).

Overrides: object.__init__

_process_task(self, data, task_callable, task_index)

Add a task, if possible without blocking.

It blocks when the maximum number of threads is reached (given by
n_threads) or when the system is not able to start a new thread.

Overrides: Scheduler._process_task

_task_thread(self, data, task_callable, task_index)

Thread function which processes a single task.

Class ThreadScheduler

__init__(self, result_container=None, verbose=False, n_threads=1, copy_callable=True) (Constructor)

_process_task(self, data, task_callable, task_index)

_task_thread(self, data, task_callable, task_index)

init(self, result_container=None, verbose=False, n_threads=1, copy_callable=True)
(Constructor)