Why do NOT process.join() return when the process terminates?
0
My code is following:
from multiprocessing import Process, Queue
task_queue = Queue(TASK_QUEUE_SIZE)
result_queue = Queue()
def worker_thread(worker_id, task_queue, result_queue):
# consume task_queue and put result into result_queue
# if task is None, it prints closed message and returns 0
...
workers = [Process(target=worker_thread, args=(i, task_queue, result_queue))
for i in range(NUM_OF_THREADS)]
for w in workers:
w.start()
# put task into task_queue, read result from result_queue...
# all done. try to exit.
print('waiting 120 seconds for workers...')
time.sleep(120) # wait long enough for workers return
for no, w in enumerate(workers):
print(f'joining worker #{no}')
w.join(1)
print(f'worker #{no} is_alive: {w.is_alive()} exitcode: {w.exitcode}')
This program generate log like this:
pending jobs: 1
worker #0 is started.
waiting 120 seconds for workers...
worker #0 got task id: 123
worker #0 got result for task id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: False exitcode: 0
If I use w.join(), it will wait forever. Very strange. Could you give me some clue? Thank you in advance.
====
UPDATE: I found my code sometimes wait forever. If logging is in the form of my original post, it will safe exit. However, if it like below [use join(1)], it will be stuck forever. It is strange that worker #0 prints closed message but does not exit.
waiting 120 seconds for workers...
worker #0 got result id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: True exitcode: None
def worker_thread(worker_id: int, task_queue, result_queue):
print(f'worker #{worker_id} is started.')
while 1:
try:
t = task_queue.get()
if t is None:
break
r = do_task(t)
if r:
result_queue.put((t, r))
except Exception as e:
print(f'[FATAL] worker #{worker_id} got unknown error on task {t}.nReason: {type(e)} {e}')
traceback.print_exc()
continue
print(f'worker #{worker_id} is closed.')
return 0
python-3.x python-multiprocessing
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
add a comment |
0
My code is following:
from multiprocessing import Process, Queue
task_queue = Queue(TASK_QUEUE_SIZE)
result_queue = Queue()
def worker_thread(worker_id, task_queue, result_queue):
# consume task_queue and put result into result_queue
# if task is None, it prints closed message and returns 0
...
workers = [Process(target=worker_thread, args=(i, task_queue, result_queue))
for i in range(NUM_OF_THREADS)]
for w in workers:
w.start()
# put task into task_queue, read result from result_queue...
# all done. try to exit.
print('waiting 120 seconds for workers...')
time.sleep(120) # wait long enough for workers return
for no, w in enumerate(workers):
print(f'joining worker #{no}')
w.join(1)
print(f'worker #{no} is_alive: {w.is_alive()} exitcode: {w.exitcode}')
This program generate log like this:
pending jobs: 1
worker #0 is started.
waiting 120 seconds for workers...
worker #0 got task id: 123
worker #0 got result for task id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: False exitcode: 0
If I use w.join(), it will wait forever. Very strange. Could you give me some clue? Thank you in advance.
====
UPDATE: I found my code sometimes wait forever. If logging is in the form of my original post, it will safe exit. However, if it like below [use join(1)], it will be stuck forever. It is strange that worker #0 prints closed message but does not exit.
waiting 120 seconds for workers...
worker #0 got result id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: True exitcode: None
def worker_thread(worker_id: int, task_queue, result_queue):
print(f'worker #{worker_id} is started.')
while 1:
try:
t = task_queue.get()
if t is None:
break
r = do_task(t)
if r:
result_queue.put((t, r))
except Exception as e:
print(f'[FATAL] worker #{worker_id} got unknown error on task {t}.nReason: {type(e)} {e}')
traceback.print_exc()
continue
print(f'worker #{worker_id} is closed.')
return 0
python-3.x python-multiprocessing
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
What is the implementation ofworker_thread? Usingjoin()without any arguments would give it no timeout, hence it will wait forever or until the process completed. Given that it is waiting forever, yourworker_threadprocess might never end because of its implementation
– Andreas
Nov 26 '18 at 7:03
@Andreas the logging shows worker_thread printed close message and returns 0. Therefore, I think, it exits before join().
– ShenLei
Nov 26 '18 at 7:07
add a comment |
0
0
0
My code is following:
from multiprocessing import Process, Queue
task_queue = Queue(TASK_QUEUE_SIZE)
result_queue = Queue()
def worker_thread(worker_id, task_queue, result_queue):
# consume task_queue and put result into result_queue
# if task is None, it prints closed message and returns 0
...
workers = [Process(target=worker_thread, args=(i, task_queue, result_queue))
for i in range(NUM_OF_THREADS)]
for w in workers:
w.start()
# put task into task_queue, read result from result_queue...
# all done. try to exit.
print('waiting 120 seconds for workers...')
time.sleep(120) # wait long enough for workers return
for no, w in enumerate(workers):
print(f'joining worker #{no}')
w.join(1)
print(f'worker #{no} is_alive: {w.is_alive()} exitcode: {w.exitcode}')
This program generate log like this:
pending jobs: 1
worker #0 is started.
waiting 120 seconds for workers...
worker #0 got task id: 123
worker #0 got result for task id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: False exitcode: 0
If I use w.join(), it will wait forever. Very strange. Could you give me some clue? Thank you in advance.
====
UPDATE: I found my code sometimes wait forever. If logging is in the form of my original post, it will safe exit. However, if it like below [use join(1)], it will be stuck forever. It is strange that worker #0 prints closed message but does not exit.
waiting 120 seconds for workers...
worker #0 got result id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: True exitcode: None
def worker_thread(worker_id: int, task_queue, result_queue):
print(f'worker #{worker_id} is started.')
while 1:
try:
t = task_queue.get()
if t is None:
break
r = do_task(t)
if r:
result_queue.put((t, r))
except Exception as e:
print(f'[FATAL] worker #{worker_id} got unknown error on task {t}.nReason: {type(e)} {e}')
traceback.print_exc()
continue
print(f'worker #{worker_id} is closed.')
return 0
python-3.x python-multiprocessing
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
My code is following:
from multiprocessing import Process, Queue
task_queue = Queue(TASK_QUEUE_SIZE)
result_queue = Queue()
def worker_thread(worker_id, task_queue, result_queue):
# consume task_queue and put result into result_queue
# if task is None, it prints closed message and returns 0
...
workers = [Process(target=worker_thread, args=(i, task_queue, result_queue))
for i in range(NUM_OF_THREADS)]
for w in workers:
w.start()
# put task into task_queue, read result from result_queue...
# all done. try to exit.
print('waiting 120 seconds for workers...')
time.sleep(120) # wait long enough for workers return
for no, w in enumerate(workers):
print(f'joining worker #{no}')
w.join(1)
print(f'worker #{no} is_alive: {w.is_alive()} exitcode: {w.exitcode}')
This program generate log like this:
pending jobs: 1
worker #0 is started.
waiting 120 seconds for workers...
worker #0 got task id: 123
worker #0 got result for task id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: False exitcode: 0
If I use w.join(), it will wait forever. Very strange. Could you give me some clue? Thank you in advance.
====
UPDATE: I found my code sometimes wait forever. If logging is in the form of my original post, it will safe exit. However, if it like below [use join(1)], it will be stuck forever. It is strange that worker #0 prints closed message but does not exit.
waiting 120 seconds for workers...
worker #0 got result id: 123
worker #0 is closed.
joining worker #0
worker #0 isalive: True exitcode: None
def worker_thread(worker_id: int, task_queue, result_queue):
print(f'worker #{worker_id} is started.')
while 1:
try:
t = task_queue.get()
if t is None:
break
r = do_task(t)
if r:
result_queue.put((t, r))
except Exception as e:
print(f'[FATAL] worker #{worker_id} got unknown error on task {t}.nReason: {type(e)} {e}')
traceback.print_exc()
continue
print(f'worker #{worker_id} is closed.')
return 0
python-3.x python-multiprocessing
python-3.x python-multiprocessing
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
edited Nov 26 '18 at 7:49
ShenLei
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
asked Nov 26 '18 at 6:56
ShenLeiShenLei
119129
119129
What is the implementation ofworker_thread? Usingjoin()without any arguments would give it no timeout, hence it will wait forever or until the process completed. Given that it is waiting forever, yourworker_threadprocess might never end because of its implementation
– Andreas
Nov 26 '18 at 7:03
@Andreas the logging shows worker_thread printed close message and returns 0. Therefore, I think, it exits before join().
– ShenLei
Nov 26 '18 at 7:07
add a comment |
What is the implementation ofworker_thread? Usingjoin()without any arguments would give it no timeout, hence it will wait forever or until the process completed. Given that it is waiting forever, yourworker_threadprocess might never end because of its implementation
– Andreas
Nov 26 '18 at 7:03
@Andreas the logging shows worker_thread printed close message and returns 0. Therefore, I think, it exits before join().
– ShenLei
Nov 26 '18 at 7:07
What is the implementation of
worker_thread? Using join() without any arguments would give it no timeout, hence it will wait forever or until the process completed. Given that it is waiting forever, your worker_thread process might never end because of its implementation– Andreas
Nov 26 '18 at 7:03
What is the implementation of
worker_thread? Using join() without any arguments would give it no timeout, hence it will wait forever or until the process completed. Given that it is waiting forever, your worker_thread process might never end because of its implementation– Andreas
Nov 26 '18 at 7:03
@Andreas the logging shows worker_thread printed close message and returns 0. Therefore, I think, it exits before join().
– ShenLei
Nov 26 '18 at 7:07
@Andreas the logging shows worker_thread printed close message and returns 0. Therefore, I think, it exits before join().
– ShenLei
Nov 26 '18 at 7:07
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53476065%2fwhy-do-not-process-join-return-when-the-process-terminates%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53476065%2fwhy-do-not-process-join-return-when-the-process-terminates%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What is the implementation of
worker_thread? Usingjoin()without any arguments would give it no timeout, hence it will wait forever or until the process completed. Given that it is waiting forever, yourworker_threadprocess might never end because of its implementation– Andreas
Nov 26 '18 at 7:03
@Andreas the logging shows worker_thread printed close message and returns 0. Therefore, I think, it exits before join().
– ShenLei
Nov 26 '18 at 7:07