Checking that PYTHONIOENCODING is always “utf8”












0















I know unittests and use write them daily.



They get executed during development and CI.



Now I have test which I would like to ensure on the production system:




PYTHONIOENCODING must be "utf8"




Above I used the verb "test", this means I want to check the state. This question is not about how to do this.



AFAIK the unittest framework can't help me here, since it only gets executed during development and CI.



How to solve this in the python world withou re-inventing the wheel?



Above is only an example. There are several other things next to PYTHONIOENCODING which I would like to check.



Next use case for these checks: Some days ago we had an issue on the production sever. The command-line tool convert gets used and some versions are broken and create wrong results. I would like to write a simple check to ensure that the convert tool on the production server is not broken.










share|improve this question




















  • 1





    I would use an assert near the entry point of the program.

    – rfkortekaas
    Nov 28 '18 at 9:32











  • @rfkortekaas I updated the question. There are several other things which I want to check in the production environment. I am afraid that doing several assertions on every program start could slow it down.

    – guettli
    Nov 28 '18 at 11:34













  • @guettli No worries. These checks only takes minimal time and it shouldn't be a performance impact.

    – iBug
    Nov 28 '18 at 11:42






  • 1





    It looks like you're after infrastructure tests; we use pytest-testinfra for that. Tests are stored as a separate project in a separate repository and are invoked from CI server once the container images are built; we deploy built containers in an isolated production test env and run the tests against them.

    – hoefling
    Nov 28 '18 at 17:32






  • 1





    Option 3: Upgrade to Python 3.7 where UTF-8 is the default encoding :)

    – Dima Tisnek
    Jan 9 at 1:39
















0















I know unittests and use write them daily.



They get executed during development and CI.



Now I have test which I would like to ensure on the production system:




PYTHONIOENCODING must be "utf8"




Above I used the verb "test", this means I want to check the state. This question is not about how to do this.



AFAIK the unittest framework can't help me here, since it only gets executed during development and CI.



How to solve this in the python world withou re-inventing the wheel?



Above is only an example. There are several other things next to PYTHONIOENCODING which I would like to check.



Next use case for these checks: Some days ago we had an issue on the production sever. The command-line tool convert gets used and some versions are broken and create wrong results. I would like to write a simple check to ensure that the convert tool on the production server is not broken.










share|improve this question




















  • 1





    I would use an assert near the entry point of the program.

    – rfkortekaas
    Nov 28 '18 at 9:32











  • @rfkortekaas I updated the question. There are several other things which I want to check in the production environment. I am afraid that doing several assertions on every program start could slow it down.

    – guettli
    Nov 28 '18 at 11:34













  • @guettli No worries. These checks only takes minimal time and it shouldn't be a performance impact.

    – iBug
    Nov 28 '18 at 11:42






  • 1





    It looks like you're after infrastructure tests; we use pytest-testinfra for that. Tests are stored as a separate project in a separate repository and are invoked from CI server once the container images are built; we deploy built containers in an isolated production test env and run the tests against them.

    – hoefling
    Nov 28 '18 at 17:32






  • 1





    Option 3: Upgrade to Python 3.7 where UTF-8 is the default encoding :)

    – Dima Tisnek
    Jan 9 at 1:39














0












0








0








I know unittests and use write them daily.



They get executed during development and CI.



Now I have test which I would like to ensure on the production system:




PYTHONIOENCODING must be "utf8"




Above I used the verb "test", this means I want to check the state. This question is not about how to do this.



AFAIK the unittest framework can't help me here, since it only gets executed during development and CI.



How to solve this in the python world withou re-inventing the wheel?



Above is only an example. There are several other things next to PYTHONIOENCODING which I would like to check.



Next use case for these checks: Some days ago we had an issue on the production sever. The command-line tool convert gets used and some versions are broken and create wrong results. I would like to write a simple check to ensure that the convert tool on the production server is not broken.










share|improve this question
















I know unittests and use write them daily.



They get executed during development and CI.



Now I have test which I would like to ensure on the production system:




PYTHONIOENCODING must be "utf8"




Above I used the verb "test", this means I want to check the state. This question is not about how to do this.



AFAIK the unittest framework can't help me here, since it only gets executed during development and CI.



How to solve this in the python world withou re-inventing the wheel?



Above is only an example. There are several other things next to PYTHONIOENCODING which I would like to check.



Next use case for these checks: Some days ago we had an issue on the production sever. The command-line tool convert gets used and some versions are broken and create wrong results. I would like to write a simple check to ensure that the convert tool on the production server is not broken.







python unit-testing pytest






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 28 '18 at 12:27







guettli

















asked Nov 28 '18 at 9:26









guettliguettli

4,09124140284




4,09124140284








  • 1





    I would use an assert near the entry point of the program.

    – rfkortekaas
    Nov 28 '18 at 9:32











  • @rfkortekaas I updated the question. There are several other things which I want to check in the production environment. I am afraid that doing several assertions on every program start could slow it down.

    – guettli
    Nov 28 '18 at 11:34













  • @guettli No worries. These checks only takes minimal time and it shouldn't be a performance impact.

    – iBug
    Nov 28 '18 at 11:42






  • 1





    It looks like you're after infrastructure tests; we use pytest-testinfra for that. Tests are stored as a separate project in a separate repository and are invoked from CI server once the container images are built; we deploy built containers in an isolated production test env and run the tests against them.

    – hoefling
    Nov 28 '18 at 17:32






  • 1





    Option 3: Upgrade to Python 3.7 where UTF-8 is the default encoding :)

    – Dima Tisnek
    Jan 9 at 1:39














  • 1





    I would use an assert near the entry point of the program.

    – rfkortekaas
    Nov 28 '18 at 9:32











  • @rfkortekaas I updated the question. There are several other things which I want to check in the production environment. I am afraid that doing several assertions on every program start could slow it down.

    – guettli
    Nov 28 '18 at 11:34













  • @guettli No worries. These checks only takes minimal time and it shouldn't be a performance impact.

    – iBug
    Nov 28 '18 at 11:42






  • 1





    It looks like you're after infrastructure tests; we use pytest-testinfra for that. Tests are stored as a separate project in a separate repository and are invoked from CI server once the container images are built; we deploy built containers in an isolated production test env and run the tests against them.

    – hoefling
    Nov 28 '18 at 17:32






  • 1





    Option 3: Upgrade to Python 3.7 where UTF-8 is the default encoding :)

    – Dima Tisnek
    Jan 9 at 1:39








1




1





I would use an assert near the entry point of the program.

– rfkortekaas
Nov 28 '18 at 9:32





I would use an assert near the entry point of the program.

– rfkortekaas
Nov 28 '18 at 9:32













@rfkortekaas I updated the question. There are several other things which I want to check in the production environment. I am afraid that doing several assertions on every program start could slow it down.

– guettli
Nov 28 '18 at 11:34







@rfkortekaas I updated the question. There are several other things which I want to check in the production environment. I am afraid that doing several assertions on every program start could slow it down.

– guettli
Nov 28 '18 at 11:34















@guettli No worries. These checks only takes minimal time and it shouldn't be a performance impact.

– iBug
Nov 28 '18 at 11:42





@guettli No worries. These checks only takes minimal time and it shouldn't be a performance impact.

– iBug
Nov 28 '18 at 11:42




1




1





It looks like you're after infrastructure tests; we use pytest-testinfra for that. Tests are stored as a separate project in a separate repository and are invoked from CI server once the container images are built; we deploy built containers in an isolated production test env and run the tests against them.

– hoefling
Nov 28 '18 at 17:32





It looks like you're after infrastructure tests; we use pytest-testinfra for that. Tests are stored as a separate project in a separate repository and are invoked from CI server once the container images are built; we deploy built containers in an isolated production test env and run the tests against them.

– hoefling
Nov 28 '18 at 17:32




1




1





Option 3: Upgrade to Python 3.7 where UTF-8 is the default encoding :)

– Dima Tisnek
Jan 9 at 1:39





Option 3: Upgrade to Python 3.7 where UTF-8 is the default encoding :)

– Dima Tisnek
Jan 9 at 1:39












2 Answers
2






active

oldest

votes


















1





+50









testinfra



If you want to write and run tests against the deployment infrastructure, you can use the testinfra plugin for pytest. For example, test for a simple requirement of validating an environment variable on target machine could look like:



def test_env_var(host):
assert host.run_expect((0,), 'test "$PYTHONIOENCODING" == "utf8"')


This infrastructure test suite can be developed in a separate project and invoked before the actual deployment takes place (for example, we invoke the infra tests right after the docker image is built; if the tests fail, the image is not uploaded to our private image repository/deployed to prod etc).






share|improve this answer


























  • Yes, this answer sound good. Unfortunately this makes my question a question about a tool. I hope the stackoverflow police does not catch us.

    – guettli
    Jan 3 at 15:16






  • 1





    No worries, there are so many pytest plugins, you can't remember them all anyway. Also, maybe someone will provide a better answer now that the question is featured.

    – hoefling
    Jan 3 at 23:59



















2














Straightforward approach (Checking)



Put this near the start of the code:



import os

if 'PYTHONIOENCODING' not in os.environ or
os.environ['PYTHONIOENCODING'].lower() not in {'utf-8', 'utf8'}:
raise EnvironmentError("Environment variable $PYTHONIOENCODING must be set to 'utf8'")




Alternative solution (Ensuring)



In one of the projects I code for, there's a "startup script", so instead of running python3 main.py, we run this in production:



bash main.sh


whose content is rather simple:



#!/bin/bash

export PYTHONIOENCODING=utf8

exec /usr/bin/env python3 main.py





share|improve this answer


























  • export should be on a separate line if you really need this to be /bin/sh. The export variable=value syntax extension is supported by ksh and bash and some others, but not (portably) by sh. The portable way to write this is variable=value; export variable

    – tripleee
    Nov 28 '18 at 9:57













  • @tripleee but it's been working for years on Ubuntu (dash).

    – iBug
    Nov 28 '18 at 9:59











  • All I'm saying it isn't strictly portable. You can find environments where puke is a valid command, too.

    – tripleee
    Nov 28 '18 at 10:06











  • @tripleee I picked the easier choice: the shebang line is changed.

    – iBug
    Nov 28 '18 at 10:09











  • @iBug I am sorry. This question is about checking the final state, not about creating this state. I updated the question.

    – guettli
    Nov 28 '18 at 11:37











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53516107%2fchecking-that-pythonioencoding-is-always-utf8%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes









1





+50









testinfra



If you want to write and run tests against the deployment infrastructure, you can use the testinfra plugin for pytest. For example, test for a simple requirement of validating an environment variable on target machine could look like:



def test_env_var(host):
assert host.run_expect((0,), 'test "$PYTHONIOENCODING" == "utf8"')


This infrastructure test suite can be developed in a separate project and invoked before the actual deployment takes place (for example, we invoke the infra tests right after the docker image is built; if the tests fail, the image is not uploaded to our private image repository/deployed to prod etc).






share|improve this answer


























  • Yes, this answer sound good. Unfortunately this makes my question a question about a tool. I hope the stackoverflow police does not catch us.

    – guettli
    Jan 3 at 15:16






  • 1





    No worries, there are so many pytest plugins, you can't remember them all anyway. Also, maybe someone will provide a better answer now that the question is featured.

    – hoefling
    Jan 3 at 23:59
















1





+50









testinfra



If you want to write and run tests against the deployment infrastructure, you can use the testinfra plugin for pytest. For example, test for a simple requirement of validating an environment variable on target machine could look like:



def test_env_var(host):
assert host.run_expect((0,), 'test "$PYTHONIOENCODING" == "utf8"')


This infrastructure test suite can be developed in a separate project and invoked before the actual deployment takes place (for example, we invoke the infra tests right after the docker image is built; if the tests fail, the image is not uploaded to our private image repository/deployed to prod etc).






share|improve this answer


























  • Yes, this answer sound good. Unfortunately this makes my question a question about a tool. I hope the stackoverflow police does not catch us.

    – guettli
    Jan 3 at 15:16






  • 1





    No worries, there are so many pytest plugins, you can't remember them all anyway. Also, maybe someone will provide a better answer now that the question is featured.

    – hoefling
    Jan 3 at 23:59














1





+50







1





+50



1




+50





testinfra



If you want to write and run tests against the deployment infrastructure, you can use the testinfra plugin for pytest. For example, test for a simple requirement of validating an environment variable on target machine could look like:



def test_env_var(host):
assert host.run_expect((0,), 'test "$PYTHONIOENCODING" == "utf8"')


This infrastructure test suite can be developed in a separate project and invoked before the actual deployment takes place (for example, we invoke the infra tests right after the docker image is built; if the tests fail, the image is not uploaded to our private image repository/deployed to prod etc).






share|improve this answer















testinfra



If you want to write and run tests against the deployment infrastructure, you can use the testinfra plugin for pytest. For example, test for a simple requirement of validating an environment variable on target machine could look like:



def test_env_var(host):
assert host.run_expect((0,), 'test "$PYTHONIOENCODING" == "utf8"')


This infrastructure test suite can be developed in a separate project and invoked before the actual deployment takes place (for example, we invoke the infra tests right after the docker image is built; if the tests fail, the image is not uploaded to our private image repository/deployed to prod etc).







share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 29 '18 at 16:22

























answered Nov 29 '18 at 15:42









hoeflinghoefling

13.3k43467




13.3k43467













  • Yes, this answer sound good. Unfortunately this makes my question a question about a tool. I hope the stackoverflow police does not catch us.

    – guettli
    Jan 3 at 15:16






  • 1





    No worries, there are so many pytest plugins, you can't remember them all anyway. Also, maybe someone will provide a better answer now that the question is featured.

    – hoefling
    Jan 3 at 23:59



















  • Yes, this answer sound good. Unfortunately this makes my question a question about a tool. I hope the stackoverflow police does not catch us.

    – guettli
    Jan 3 at 15:16






  • 1





    No worries, there are so many pytest plugins, you can't remember them all anyway. Also, maybe someone will provide a better answer now that the question is featured.

    – hoefling
    Jan 3 at 23:59

















Yes, this answer sound good. Unfortunately this makes my question a question about a tool. I hope the stackoverflow police does not catch us.

– guettli
Jan 3 at 15:16





Yes, this answer sound good. Unfortunately this makes my question a question about a tool. I hope the stackoverflow police does not catch us.

– guettli
Jan 3 at 15:16




1




1





No worries, there are so many pytest plugins, you can't remember them all anyway. Also, maybe someone will provide a better answer now that the question is featured.

– hoefling
Jan 3 at 23:59





No worries, there are so many pytest plugins, you can't remember them all anyway. Also, maybe someone will provide a better answer now that the question is featured.

– hoefling
Jan 3 at 23:59













2














Straightforward approach (Checking)



Put this near the start of the code:



import os

if 'PYTHONIOENCODING' not in os.environ or
os.environ['PYTHONIOENCODING'].lower() not in {'utf-8', 'utf8'}:
raise EnvironmentError("Environment variable $PYTHONIOENCODING must be set to 'utf8'")




Alternative solution (Ensuring)



In one of the projects I code for, there's a "startup script", so instead of running python3 main.py, we run this in production:



bash main.sh


whose content is rather simple:



#!/bin/bash

export PYTHONIOENCODING=utf8

exec /usr/bin/env python3 main.py





share|improve this answer


























  • export should be on a separate line if you really need this to be /bin/sh. The export variable=value syntax extension is supported by ksh and bash and some others, but not (portably) by sh. The portable way to write this is variable=value; export variable

    – tripleee
    Nov 28 '18 at 9:57













  • @tripleee but it's been working for years on Ubuntu (dash).

    – iBug
    Nov 28 '18 at 9:59











  • All I'm saying it isn't strictly portable. You can find environments where puke is a valid command, too.

    – tripleee
    Nov 28 '18 at 10:06











  • @tripleee I picked the easier choice: the shebang line is changed.

    – iBug
    Nov 28 '18 at 10:09











  • @iBug I am sorry. This question is about checking the final state, not about creating this state. I updated the question.

    – guettli
    Nov 28 '18 at 11:37
















2














Straightforward approach (Checking)



Put this near the start of the code:



import os

if 'PYTHONIOENCODING' not in os.environ or
os.environ['PYTHONIOENCODING'].lower() not in {'utf-8', 'utf8'}:
raise EnvironmentError("Environment variable $PYTHONIOENCODING must be set to 'utf8'")




Alternative solution (Ensuring)



In one of the projects I code for, there's a "startup script", so instead of running python3 main.py, we run this in production:



bash main.sh


whose content is rather simple:



#!/bin/bash

export PYTHONIOENCODING=utf8

exec /usr/bin/env python3 main.py





share|improve this answer


























  • export should be on a separate line if you really need this to be /bin/sh. The export variable=value syntax extension is supported by ksh and bash and some others, but not (portably) by sh. The portable way to write this is variable=value; export variable

    – tripleee
    Nov 28 '18 at 9:57













  • @tripleee but it's been working for years on Ubuntu (dash).

    – iBug
    Nov 28 '18 at 9:59











  • All I'm saying it isn't strictly portable. You can find environments where puke is a valid command, too.

    – tripleee
    Nov 28 '18 at 10:06











  • @tripleee I picked the easier choice: the shebang line is changed.

    – iBug
    Nov 28 '18 at 10:09











  • @iBug I am sorry. This question is about checking the final state, not about creating this state. I updated the question.

    – guettli
    Nov 28 '18 at 11:37














2












2








2







Straightforward approach (Checking)



Put this near the start of the code:



import os

if 'PYTHONIOENCODING' not in os.environ or
os.environ['PYTHONIOENCODING'].lower() not in {'utf-8', 'utf8'}:
raise EnvironmentError("Environment variable $PYTHONIOENCODING must be set to 'utf8'")




Alternative solution (Ensuring)



In one of the projects I code for, there's a "startup script", so instead of running python3 main.py, we run this in production:



bash main.sh


whose content is rather simple:



#!/bin/bash

export PYTHONIOENCODING=utf8

exec /usr/bin/env python3 main.py





share|improve this answer















Straightforward approach (Checking)



Put this near the start of the code:



import os

if 'PYTHONIOENCODING' not in os.environ or
os.environ['PYTHONIOENCODING'].lower() not in {'utf-8', 'utf8'}:
raise EnvironmentError("Environment variable $PYTHONIOENCODING must be set to 'utf8'")




Alternative solution (Ensuring)



In one of the projects I code for, there's a "startup script", so instead of running python3 main.py, we run this in production:



bash main.sh


whose content is rather simple:



#!/bin/bash

export PYTHONIOENCODING=utf8

exec /usr/bin/env python3 main.py






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 28 '18 at 11:41

























answered Nov 28 '18 at 9:37









iBugiBug

21.4k64066




21.4k64066













  • export should be on a separate line if you really need this to be /bin/sh. The export variable=value syntax extension is supported by ksh and bash and some others, but not (portably) by sh. The portable way to write this is variable=value; export variable

    – tripleee
    Nov 28 '18 at 9:57













  • @tripleee but it's been working for years on Ubuntu (dash).

    – iBug
    Nov 28 '18 at 9:59











  • All I'm saying it isn't strictly portable. You can find environments where puke is a valid command, too.

    – tripleee
    Nov 28 '18 at 10:06











  • @tripleee I picked the easier choice: the shebang line is changed.

    – iBug
    Nov 28 '18 at 10:09











  • @iBug I am sorry. This question is about checking the final state, not about creating this state. I updated the question.

    – guettli
    Nov 28 '18 at 11:37



















  • export should be on a separate line if you really need this to be /bin/sh. The export variable=value syntax extension is supported by ksh and bash and some others, but not (portably) by sh. The portable way to write this is variable=value; export variable

    – tripleee
    Nov 28 '18 at 9:57













  • @tripleee but it's been working for years on Ubuntu (dash).

    – iBug
    Nov 28 '18 at 9:59











  • All I'm saying it isn't strictly portable. You can find environments where puke is a valid command, too.

    – tripleee
    Nov 28 '18 at 10:06











  • @tripleee I picked the easier choice: the shebang line is changed.

    – iBug
    Nov 28 '18 at 10:09











  • @iBug I am sorry. This question is about checking the final state, not about creating this state. I updated the question.

    – guettli
    Nov 28 '18 at 11:37

















export should be on a separate line if you really need this to be /bin/sh. The export variable=value syntax extension is supported by ksh and bash and some others, but not (portably) by sh. The portable way to write this is variable=value; export variable

– tripleee
Nov 28 '18 at 9:57







export should be on a separate line if you really need this to be /bin/sh. The export variable=value syntax extension is supported by ksh and bash and some others, but not (portably) by sh. The portable way to write this is variable=value; export variable

– tripleee
Nov 28 '18 at 9:57















@tripleee but it's been working for years on Ubuntu (dash).

– iBug
Nov 28 '18 at 9:59





@tripleee but it's been working for years on Ubuntu (dash).

– iBug
Nov 28 '18 at 9:59













All I'm saying it isn't strictly portable. You can find environments where puke is a valid command, too.

– tripleee
Nov 28 '18 at 10:06





All I'm saying it isn't strictly portable. You can find environments where puke is a valid command, too.

– tripleee
Nov 28 '18 at 10:06













@tripleee I picked the easier choice: the shebang line is changed.

– iBug
Nov 28 '18 at 10:09





@tripleee I picked the easier choice: the shebang line is changed.

– iBug
Nov 28 '18 at 10:09













@iBug I am sorry. This question is about checking the final state, not about creating this state. I updated the question.

– guettli
Nov 28 '18 at 11:37





@iBug I am sorry. This question is about checking the final state, not about creating this state. I updated the question.

– guettli
Nov 28 '18 at 11:37


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53516107%2fchecking-that-pythonioencoding-is-always-utf8%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

Calculate evaluation metrics using cross_val_predict sklearn

Insert data from modal to MySQL (multiple modal on website)