Where to join in multithreaded directory search?
I've been working on a multithreaded directory search that creates a thread everytime it runs into a new directory or file. So far the file threads are parallel and works fine, but I'm not sure where to join the new directory threads since it is recursive. I've tried putting it on the bottom of the function but it just creates a loop (not infinite but definitely wrong)
My code is below, I also have a pseudocode for ease of understanding.
For ease of explaining and clearing up the messy code, here is a pseudocode:
recursive function{
if (!(dir = opendir(ca->SD))){{
return;
}
while ((ptr = readdir(dir)) != NULL) {
if (ptr->d_type == DT_DIR) {
if (strcmp(ptr->d_name, "..") == 0||strcmp(ptr->d_name, ".") == 0){
continue;
}
create thread/call recursive function
}else(if file){
create thread/call file handler function
}
}
thread join for any live threads;
}
I'm just confused as to where to properly put the join for any of the directory threads. I currently keep getting some sort of loop.
c multithreading recursion directory thread-safety
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
add a comment |
I've been working on a multithreaded directory search that creates a thread everytime it runs into a new directory or file. So far the file threads are parallel and works fine, but I'm not sure where to join the new directory threads since it is recursive. I've tried putting it on the bottom of the function but it just creates a loop (not infinite but definitely wrong)
My code is below, I also have a pseudocode for ease of understanding.
For ease of explaining and clearing up the messy code, here is a pseudocode:
recursive function{
if (!(dir = opendir(ca->SD))){{
return;
}
while ((ptr = readdir(dir)) != NULL) {
if (ptr->d_type == DT_DIR) {
if (strcmp(ptr->d_name, "..") == 0||strcmp(ptr->d_name, ".") == 0){
continue;
}
create thread/call recursive function
}else(if file){
create thread/call file handler function
}
}
thread join for any live threads;
}
I'm just confused as to where to properly put the join for any of the directory threads. I currently keep getting some sort of loop.
c multithreading recursion directory thread-safety
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
pthread_mutex_lock(&mutex);thenif (pthread_self() == initialtid){...with nopthread_mutex_unlock()after theifcloses but with multiple calls topthread_mutex_unlock()in theifstatement itself is a recipe for disaster.
– Andrew Henle
Nov 24 '18 at 1:59
add a comment |
I've been working on a multithreaded directory search that creates a thread everytime it runs into a new directory or file. So far the file threads are parallel and works fine, but I'm not sure where to join the new directory threads since it is recursive. I've tried putting it on the bottom of the function but it just creates a loop (not infinite but definitely wrong)
My code is below, I also have a pseudocode for ease of understanding.
For ease of explaining and clearing up the messy code, here is a pseudocode:
recursive function{
if (!(dir = opendir(ca->SD))){{
return;
}
while ((ptr = readdir(dir)) != NULL) {
if (ptr->d_type == DT_DIR) {
if (strcmp(ptr->d_name, "..") == 0||strcmp(ptr->d_name, ".") == 0){
continue;
}
create thread/call recursive function
}else(if file){
create thread/call file handler function
}
}
thread join for any live threads;
}
I'm just confused as to where to properly put the join for any of the directory threads. I currently keep getting some sort of loop.
c multithreading recursion directory thread-safety
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
I've been working on a multithreaded directory search that creates a thread everytime it runs into a new directory or file. So far the file threads are parallel and works fine, but I'm not sure where to join the new directory threads since it is recursive. I've tried putting it on the bottom of the function but it just creates a loop (not infinite but definitely wrong)
My code is below, I also have a pseudocode for ease of understanding.
For ease of explaining and clearing up the messy code, here is a pseudocode:
recursive function{
if (!(dir = opendir(ca->SD))){{
return;
}
while ((ptr = readdir(dir)) != NULL) {
if (ptr->d_type == DT_DIR) {
if (strcmp(ptr->d_name, "..") == 0||strcmp(ptr->d_name, ".") == 0){
continue;
}
create thread/call recursive function
}else(if file){
create thread/call file handler function
}
}
thread join for any live threads;
}
I'm just confused as to where to properly put the join for any of the directory threads. I currently keep getting some sort of loop.
c multithreading recursion directory thread-safety
c multithreading recursion directory thread-safety
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
edited Nov 25 '18 at 19:56
Andrew Hsu
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
asked Nov 24 '18 at 1:01
Andrew HsuAndrew Hsu
12
12
pthread_mutex_lock(&mutex);thenif (pthread_self() == initialtid){...with nopthread_mutex_unlock()after theifcloses but with multiple calls topthread_mutex_unlock()in theifstatement itself is a recipe for disaster.
– Andrew Henle
Nov 24 '18 at 1:59
add a comment |
pthread_mutex_lock(&mutex);thenif (pthread_self() == initialtid){...with nopthread_mutex_unlock()after theifcloses but with multiple calls topthread_mutex_unlock()in theifstatement itself is a recipe for disaster.
– Andrew Henle
Nov 24 '18 at 1:59
pthread_mutex_lock(&mutex); then if (pthread_self() == initialtid){... with no pthread_mutex_unlock() after the if closes but with multiple calls to pthread_mutex_unlock() in the if statement itself is a recipe for disaster.– Andrew Henle
Nov 24 '18 at 1:59
pthread_mutex_lock(&mutex); then if (pthread_self() == initialtid){... with no pthread_mutex_unlock() after the if closes but with multiple calls to pthread_mutex_unlock() in the if statement itself is a recipe for disaster.– Andrew Henle
Nov 24 '18 at 1:59
add a comment |
1 Answer
1
active
oldest
votes
I think you shoudn't try to create an unknown number of threads because It will have poorly performance. In this case it's better to use a pool of threads where you can control the maximum number of threads and you will not be creating and destroying threads for every directory entry.
Take a look on this link for information of what a pool of threads is.
In your example you have to wait for the threads termination at the end of each recursive call. I mean each call of searchdirectory should wait for the termination of the threads it created before returning.
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
Unfortunately the nature of the project requires a thread to be created for every directory, for an unknown amount of directories, so I am unable to do that.
– Andrew Hsu
Nov 24 '18 at 2:48
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53454314%2fwhere-to-join-in-multithreaded-directory-search%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I think you shoudn't try to create an unknown number of threads because It will have poorly performance. In this case it's better to use a pool of threads where you can control the maximum number of threads and you will not be creating and destroying threads for every directory entry.
Take a look on this link for information of what a pool of threads is.
In your example you have to wait for the threads termination at the end of each recursive call. I mean each call of searchdirectory should wait for the termination of the threads it created before returning.
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
Unfortunately the nature of the project requires a thread to be created for every directory, for an unknown amount of directories, so I am unable to do that.
– Andrew Hsu
Nov 24 '18 at 2:48
add a comment |
I think you shoudn't try to create an unknown number of threads because It will have poorly performance. In this case it's better to use a pool of threads where you can control the maximum number of threads and you will not be creating and destroying threads for every directory entry.
Take a look on this link for information of what a pool of threads is.
In your example you have to wait for the threads termination at the end of each recursive call. I mean each call of searchdirectory should wait for the termination of the threads it created before returning.
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
Unfortunately the nature of the project requires a thread to be created for every directory, for an unknown amount of directories, so I am unable to do that.
– Andrew Hsu
Nov 24 '18 at 2:48
add a comment |
I think you shoudn't try to create an unknown number of threads because It will have poorly performance. In this case it's better to use a pool of threads where you can control the maximum number of threads and you will not be creating and destroying threads for every directory entry.
Take a look on this link for information of what a pool of threads is.
In your example you have to wait for the threads termination at the end of each recursive call. I mean each call of searchdirectory should wait for the termination of the threads it created before returning.
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
I think you shoudn't try to create an unknown number of threads because It will have poorly performance. In this case it's better to use a pool of threads where you can control the maximum number of threads and you will not be creating and destroying threads for every directory entry.
Take a look on this link for information of what a pool of threads is.
In your example you have to wait for the threads termination at the end of each recursive call. I mean each call of searchdirectory should wait for the termination of the threads it created before returning.
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
answered Nov 24 '18 at 1:50
MarcelinoMarcelino
1277
1277
Unfortunately the nature of the project requires a thread to be created for every directory, for an unknown amount of directories, so I am unable to do that.
– Andrew Hsu
Nov 24 '18 at 2:48
add a comment |
Unfortunately the nature of the project requires a thread to be created for every directory, for an unknown amount of directories, so I am unable to do that.
– Andrew Hsu
Nov 24 '18 at 2:48
Unfortunately the nature of the project requires a thread to be created for every directory, for an unknown amount of directories, so I am unable to do that.
– Andrew Hsu
Nov 24 '18 at 2:48
Unfortunately the nature of the project requires a thread to be created for every directory, for an unknown amount of directories, so I am unable to do that.
– Andrew Hsu
Nov 24 '18 at 2:48
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53454314%2fwhere-to-join-in-multithreaded-directory-search%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
pthread_mutex_lock(&mutex);thenif (pthread_self() == initialtid){...with nopthread_mutex_unlock()after theifcloses but with multiple calls topthread_mutex_unlock()in theifstatement itself is a recipe for disaster.– Andrew Henle
Nov 24 '18 at 1:59