Scikit-Learn no logging output for transformers in FeatureUnion when n_jobs != 1
Using a FeatureUnion with some custom transformers I noticed that some output wasn't showing up in my terminal that I was expecting from logger.info() statements within the transformers. After digging around, I was able to track it down to the FeatureUnion's n_jobs parameter. If it is not set to n_jobs=1 output from loggers will not show up. However, output from regular print() statements does show up.
MWE
import logging
import numpy as np
from sklearn.pipeline import FeatureUnion
from sklearn.preprocessing import MinMaxScaler
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)
class MyMinMaxScaler(MinMaxScaler):
def fit(self, X, y=None):
logger.info("fitting MyMinMaxScaler")
print("fitting MyMinMaxScaler")
super().fit(X, y)
fu = FeatureUnion(
[("myscaler", MyMinMaxScaler()),
("regular_scaler", MinMaxScaler())],
n_jobs=-1
)
data = np.array([40, 41, 42]).reshape(-1, 1)
fu.fit(data)
logger.info(fu.transform(data))
Output n_jobs=-1
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
Output n_jobs=1
INFO:__main__:fitting MyMinMaxScaler
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
I wonder if this is the intended behavior. I was thinking this may be related to multiprocessing and output from child processes not showing up. But why would regular print() statements work as usual and logging output would not?
This behavior, at first, let me believe that my transformers weren't actually being fitted and thus not used at all. It's obviously not a big deal now that I know that the transformers are being used as expected. I would, however, appreciate some insight into why this logging behavior may occur the way it does, and if there is any way around it.
python logging scikit-learn
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
|
show 2 more comments
Using a FeatureUnion with some custom transformers I noticed that some output wasn't showing up in my terminal that I was expecting from logger.info() statements within the transformers. After digging around, I was able to track it down to the FeatureUnion's n_jobs parameter. If it is not set to n_jobs=1 output from loggers will not show up. However, output from regular print() statements does show up.
MWE
import logging
import numpy as np
from sklearn.pipeline import FeatureUnion
from sklearn.preprocessing import MinMaxScaler
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)
class MyMinMaxScaler(MinMaxScaler):
def fit(self, X, y=None):
logger.info("fitting MyMinMaxScaler")
print("fitting MyMinMaxScaler")
super().fit(X, y)
fu = FeatureUnion(
[("myscaler", MyMinMaxScaler()),
("regular_scaler", MinMaxScaler())],
n_jobs=-1
)
data = np.array([40, 41, 42]).reshape(-1, 1)
fu.fit(data)
logger.info(fu.transform(data))
Output n_jobs=-1
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
Output n_jobs=1
INFO:__main__:fitting MyMinMaxScaler
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
I wonder if this is the intended behavior. I was thinking this may be related to multiprocessing and output from child processes not showing up. But why would regular print() statements work as usual and logging output would not?
This behavior, at first, let me believe that my transformers weren't actually being fitted and thus not used at all. It's obviously not a big deal now that I know that the transformers are being used as expected. I would, however, appreciate some insight into why this logging behavior may occur the way it does, and if there is any way around it.
python logging scikit-learn
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
How are you executing this? From an IDE or in terminal? If from an IDE, you will not get the output. If that ide is in turn started through terminal, then you will get the output on that actual terminal, instead of output space of IDE.
– Vivek Kumar
Nov 26 '18 at 14:43
I execute this in a terminal. I use and IDE only as a code editor.
– pmlk
Nov 26 '18 at 17:14
Can't reproduce - I see the log output you're not seeing withn_jobs=-1. CPython 3.6.3, scikit-learn 0.19.0.
– user2357112
Dec 11 '18 at 18:23
Can reproduce on scikit-learn 0.20.1. Something must have changed between those versions.
– user2357112
Dec 11 '18 at 18:24
print(logger)shows that the logger's level isWARNINGinstead ofINFOinsidefit.
– user2357112
Dec 11 '18 at 18:29
|
show 2 more comments
Using a FeatureUnion with some custom transformers I noticed that some output wasn't showing up in my terminal that I was expecting from logger.info() statements within the transformers. After digging around, I was able to track it down to the FeatureUnion's n_jobs parameter. If it is not set to n_jobs=1 output from loggers will not show up. However, output from regular print() statements does show up.
MWE
import logging
import numpy as np
from sklearn.pipeline import FeatureUnion
from sklearn.preprocessing import MinMaxScaler
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)
class MyMinMaxScaler(MinMaxScaler):
def fit(self, X, y=None):
logger.info("fitting MyMinMaxScaler")
print("fitting MyMinMaxScaler")
super().fit(X, y)
fu = FeatureUnion(
[("myscaler", MyMinMaxScaler()),
("regular_scaler", MinMaxScaler())],
n_jobs=-1
)
data = np.array([40, 41, 42]).reshape(-1, 1)
fu.fit(data)
logger.info(fu.transform(data))
Output n_jobs=-1
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
Output n_jobs=1
INFO:__main__:fitting MyMinMaxScaler
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
I wonder if this is the intended behavior. I was thinking this may be related to multiprocessing and output from child processes not showing up. But why would regular print() statements work as usual and logging output would not?
This behavior, at first, let me believe that my transformers weren't actually being fitted and thus not used at all. It's obviously not a big deal now that I know that the transformers are being used as expected. I would, however, appreciate some insight into why this logging behavior may occur the way it does, and if there is any way around it.
python logging scikit-learn
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
Using a FeatureUnion with some custom transformers I noticed that some output wasn't showing up in my terminal that I was expecting from logger.info() statements within the transformers. After digging around, I was able to track it down to the FeatureUnion's n_jobs parameter. If it is not set to n_jobs=1 output from loggers will not show up. However, output from regular print() statements does show up.
MWE
import logging
import numpy as np
from sklearn.pipeline import FeatureUnion
from sklearn.preprocessing import MinMaxScaler
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)
class MyMinMaxScaler(MinMaxScaler):
def fit(self, X, y=None):
logger.info("fitting MyMinMaxScaler")
print("fitting MyMinMaxScaler")
super().fit(X, y)
fu = FeatureUnion(
[("myscaler", MyMinMaxScaler()),
("regular_scaler", MinMaxScaler())],
n_jobs=-1
)
data = np.array([40, 41, 42]).reshape(-1, 1)
fu.fit(data)
logger.info(fu.transform(data))
Output n_jobs=-1
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
Output n_jobs=1
INFO:__main__:fitting MyMinMaxScaler
fitting MyMinMaxScaler
INFO:__main__:[[0. ]
[0.5]
[1. ]]
I wonder if this is the intended behavior. I was thinking this may be related to multiprocessing and output from child processes not showing up. But why would regular print() statements work as usual and logging output would not?
This behavior, at first, let me believe that my transformers weren't actually being fitted and thus not used at all. It's obviously not a big deal now that I know that the transformers are being used as expected. I would, however, appreciate some insight into why this logging behavior may occur the way it does, and if there is any way around it.
python logging scikit-learn
python logging scikit-learn
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
edited Dec 11 '18 at 18:19
pmlk
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
asked Nov 23 '18 at 22:51
pmlkpmlk
10917
10917
How are you executing this? From an IDE or in terminal? If from an IDE, you will not get the output. If that ide is in turn started through terminal, then you will get the output on that actual terminal, instead of output space of IDE.
– Vivek Kumar
Nov 26 '18 at 14:43
I execute this in a terminal. I use and IDE only as a code editor.
– pmlk
Nov 26 '18 at 17:14
Can't reproduce - I see the log output you're not seeing withn_jobs=-1. CPython 3.6.3, scikit-learn 0.19.0.
– user2357112
Dec 11 '18 at 18:23
Can reproduce on scikit-learn 0.20.1. Something must have changed between those versions.
– user2357112
Dec 11 '18 at 18:24
print(logger)shows that the logger's level isWARNINGinstead ofINFOinsidefit.
– user2357112
Dec 11 '18 at 18:29
|
show 2 more comments
How are you executing this? From an IDE or in terminal? If from an IDE, you will not get the output. If that ide is in turn started through terminal, then you will get the output on that actual terminal, instead of output space of IDE.
– Vivek Kumar
Nov 26 '18 at 14:43
I execute this in a terminal. I use and IDE only as a code editor.
– pmlk
Nov 26 '18 at 17:14
Can't reproduce - I see the log output you're not seeing withn_jobs=-1. CPython 3.6.3, scikit-learn 0.19.0.
– user2357112
Dec 11 '18 at 18:23
Can reproduce on scikit-learn 0.20.1. Something must have changed between those versions.
– user2357112
Dec 11 '18 at 18:24
print(logger)shows that the logger's level isWARNINGinstead ofINFOinsidefit.
– user2357112
Dec 11 '18 at 18:29
How are you executing this? From an IDE or in terminal? If from an IDE, you will not get the output. If that ide is in turn started through terminal, then you will get the output on that actual terminal, instead of output space of IDE.
– Vivek Kumar
Nov 26 '18 at 14:43
How are you executing this? From an IDE or in terminal? If from an IDE, you will not get the output. If that ide is in turn started through terminal, then you will get the output on that actual terminal, instead of output space of IDE.
– Vivek Kumar
Nov 26 '18 at 14:43
I execute this in a terminal. I use and IDE only as a code editor.
– pmlk
Nov 26 '18 at 17:14
I execute this in a terminal. I use and IDE only as a code editor.
– pmlk
Nov 26 '18 at 17:14
Can't reproduce - I see the log output you're not seeing with
n_jobs=-1. CPython 3.6.3, scikit-learn 0.19.0.– user2357112
Dec 11 '18 at 18:23
Can't reproduce - I see the log output you're not seeing with
n_jobs=-1. CPython 3.6.3, scikit-learn 0.19.0.– user2357112
Dec 11 '18 at 18:23
Can reproduce on scikit-learn 0.20.1. Something must have changed between those versions.
– user2357112
Dec 11 '18 at 18:24
Can reproduce on scikit-learn 0.20.1. Something must have changed between those versions.
– user2357112
Dec 11 '18 at 18:24
print(logger) shows that the logger's level is WARNING instead of INFO inside fit.– user2357112
Dec 11 '18 at 18:29
print(logger) shows that the logger's level is WARNING instead of INFO inside fit.– user2357112
Dec 11 '18 at 18:29
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53453659%2fscikit-learn-no-logging-output-for-transformers-in-featureunion-when-n-jobs-1%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53453659%2fscikit-learn-no-logging-output-for-transformers-in-featureunion-when-n-jobs-1%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How are you executing this? From an IDE or in terminal? If from an IDE, you will not get the output. If that ide is in turn started through terminal, then you will get the output on that actual terminal, instead of output space of IDE.
– Vivek Kumar
Nov 26 '18 at 14:43
I execute this in a terminal. I use and IDE only as a code editor.
– pmlk
Nov 26 '18 at 17:14
Can't reproduce - I see the log output you're not seeing with
n_jobs=-1. CPython 3.6.3, scikit-learn 0.19.0.– user2357112
Dec 11 '18 at 18:23
Can reproduce on scikit-learn 0.20.1. Something must have changed between those versions.
– user2357112
Dec 11 '18 at 18:24
print(logger)shows that the logger's level isWARNINGinstead ofINFOinsidefit.– user2357112
Dec 11 '18 at 18:29