fix(ui): delay removal of jobs in the progress bar to avoid dead locks
Description
We often faced a deadlock in applications showing a progress bard, at the end of a job.
The deadlock occurred because of the following situation:
progress_bar::update_widgets() | progress_bar::show_job()::add_state_hook() |
---|---|
lock the progress bar | lock the job (higher in stack in ~jobs::observer) |
lock the job (get_state() ) |
lock the progress bar |
To overcome the issue, we delayed removing the job from the widgets and moved it to the first thread using a task.
How to test it?
It was quite hard to catch actually. Stress testing the affecting applications might be the only way.