Dynamic multithreading python
0
I am pretty new to python. I am trying to dynamically create some threads but am getting some errors. Can you help me please. If you don't like my code feel free to give me advices.
This is the code:
from multiprocessing import Process
...
for x in range(1, 6):
globals()['p%s' % x] = Process(target=findContent, args=(fileList[curr:curr + fileAmount],))
curr += fileAmount
globals()['p%s' % x].start()
globals()['p%s' % x].join()
And this is the error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 114, in _main
prepare(preparation_data)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 85, in _run_code
exec(code, run_globals)
File "E:test.scanCombolistsMultiThreading.py", line 207, in <module>
splitFileList()
File "E:test.scanCombolistsMultiThreading.py", line 92, in splitFileList
globals()['p%s' % x].start()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingprocess.py", line 112, in start
self._popen = self._Popen(self)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingpopen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
python multithreading process
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
add a comment |
0
I am pretty new to python. I am trying to dynamically create some threads but am getting some errors. Can you help me please. If you don't like my code feel free to give me advices.
This is the code:
from multiprocessing import Process
...
for x in range(1, 6):
globals()['p%s' % x] = Process(target=findContent, args=(fileList[curr:curr + fileAmount],))
curr += fileAmount
globals()['p%s' % x].start()
globals()['p%s' % x].join()
And this is the error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 114, in _main
prepare(preparation_data)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 85, in _run_code
exec(code, run_globals)
File "E:test.scanCombolistsMultiThreading.py", line 207, in <module>
splitFileList()
File "E:test.scanCombolistsMultiThreading.py", line 92, in splitFileList
globals()['p%s' % x].start()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingprocess.py", line 112, in start
self._popen = self._Popen(self)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingpopen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
python multithreading process
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
I assume a missingif __name__ == '__main__'around the start of the processes causes this (see also this Q&A). Your code will not run in parallel, because you wait for each process to complete before starting the next one (i.e. the<Process>.join()call should be outside the for loop). Not sure if altering the symbol table by creating/altering items on the return value ofglobals()can be considered a good practice :)
– shmee
Nov 26 '18 at 12:04
Possible duplicate of RuntimeError on windows trying python multiprocessing
– shmee
Nov 26 '18 at 12:05
add a comment |
0
0
0
0
I am pretty new to python. I am trying to dynamically create some threads but am getting some errors. Can you help me please. If you don't like my code feel free to give me advices.
This is the code:
from multiprocessing import Process
...
for x in range(1, 6):
globals()['p%s' % x] = Process(target=findContent, args=(fileList[curr:curr + fileAmount],))
curr += fileAmount
globals()['p%s' % x].start()
globals()['p%s' % x].join()
And this is the error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 114, in _main
prepare(preparation_data)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 85, in _run_code
exec(code, run_globals)
File "E:test.scanCombolistsMultiThreading.py", line 207, in <module>
splitFileList()
File "E:test.scanCombolistsMultiThreading.py", line 92, in splitFileList
globals()['p%s' % x].start()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingprocess.py", line 112, in start
self._popen = self._Popen(self)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingpopen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
python multithreading process
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
I am pretty new to python. I am trying to dynamically create some threads but am getting some errors. Can you help me please. If you don't like my code feel free to give me advices.
This is the code:
from multiprocessing import Process
...
for x in range(1, 6):
globals()['p%s' % x] = Process(target=findContent, args=(fileList[curr:curr + fileAmount],))
curr += fileAmount
globals()['p%s' % x].start()
globals()['p%s' % x].join()
And this is the error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 114, in _main
prepare(preparation_data)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32librunpy.py", line 85, in _run_code
exec(code, run_globals)
File "E:test.scanCombolistsMultiThreading.py", line 207, in <module>
splitFileList()
File "E:test.scanCombolistsMultiThreading.py", line 92, in splitFileList
globals()['p%s' % x].start()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingprocess.py", line 112, in start
self._popen = self._Popen(self)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingcontext.py", line 322, in _Popen
return Popen(process_obj)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingpopen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "C:UsersAdminAppDataLocalProgramsPythonPython37-32libmultiprocessingspawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
python multithreading process
python multithreading process
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
edited Nov 25 '18 at 23:36
NineThree
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
asked Nov 25 '18 at 23:23
NineThreeNineThree
11
11
I assume a missingif __name__ == '__main__'around the start of the processes causes this (see also this Q&A). Your code will not run in parallel, because you wait for each process to complete before starting the next one (i.e. the<Process>.join()call should be outside the for loop). Not sure if altering the symbol table by creating/altering items on the return value ofglobals()can be considered a good practice :)
– shmee
Nov 26 '18 at 12:04
Possible duplicate of RuntimeError on windows trying python multiprocessing
– shmee
Nov 26 '18 at 12:05
add a comment |
I assume a missingif __name__ == '__main__'around the start of the processes causes this (see also this Q&A). Your code will not run in parallel, because you wait for each process to complete before starting the next one (i.e. the<Process>.join()call should be outside the for loop). Not sure if altering the symbol table by creating/altering items on the return value ofglobals()can be considered a good practice :)
– shmee
Nov 26 '18 at 12:04
Possible duplicate of RuntimeError on windows trying python multiprocessing
– shmee
Nov 26 '18 at 12:05
I assume a missing
if __name__ == '__main__' around the start of the processes causes this (see also this Q&A). Your code will not run in parallel, because you wait for each process to complete before starting the next one (i.e. the <Process>.join() call should be outside the for loop). Not sure if altering the symbol table by creating/altering items on the return value of globals() can be considered a good practice :)– shmee
Nov 26 '18 at 12:04
I assume a missing
if __name__ == '__main__' around the start of the processes causes this (see also this Q&A). Your code will not run in parallel, because you wait for each process to complete before starting the next one (i.e. the <Process>.join() call should be outside the for loop). Not sure if altering the symbol table by creating/altering items on the return value of globals() can be considered a good practice :)– shmee
Nov 26 '18 at 12:04
Possible duplicate of RuntimeError on windows trying python multiprocessing
– shmee
Nov 26 '18 at 12:05
Possible duplicate of RuntimeError on windows trying python multiprocessing
– shmee
Nov 26 '18 at 12:05
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473024%2fdynamic-multithreading-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473024%2fdynamic-multithreading-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I assume a missing
if __name__ == '__main__'around the start of the processes causes this (see also this Q&A). Your code will not run in parallel, because you wait for each process to complete before starting the next one (i.e. the<Process>.join()call should be outside the for loop). Not sure if altering the symbol table by creating/altering items on the return value ofglobals()can be considered a good practice :)– shmee
Nov 26 '18 at 12:04
Possible duplicate of RuntimeError on windows trying python multiprocessing
– shmee
Nov 26 '18 at 12:05