Is there a way to map error messages to the source file for generated python scripts?












0















I have a python script that reads some input, generates a python script, and then runs the generated script. What I would like is that if there are errors in the generated script, they are reported at the file/lineno of the input file, not in the generated script.



If python supported a #line directive as in C or perl, I'd just put those in the generated script, and that would be all I needed. It doesn't look like that is supported, however.



I tried having my generated script sit in a try block, with the idea that I could catch exceptions and use the traceback module to tweak how the stack trace is printed. The problem is that the most common type of error is a parse error, and those seem to happen and get reported outside my control (i.e., the exception is thrown before the try, so my exception handler is never invoked).



The best bet I've been able to come up with is to use compile() on my generated script and then try to catch errors. That looks like it would force me to slurp in my entire generated script as a string (which could be large), though. Also, what I really want is to be able to run the generated script in its own sandbox, so it's not affected at all by the calling script (if not for error reporting, I would invoke the script as a new python process). If I use compile/exec, it's unclear to me that would be the case. Would passing an empty dictionary for the globals argument of exec() do the trick?



Is there a better way to do this sort of thing?



Edit:



The generated script could be almost anything. The input file allows the user to insert arbitrary python code in the generated script. What I'd like to do is generate a script like:



temp.py:



#line 10 original_file
for i in range(0,3)::


and have the syntax error be reported on line 10 of "original_file" and not line 2 of temp.py (which would mean nothing to the user).



The best I have so far is to insert #line directives in the generated script and have have the original script (i.e., the one that generated temp.py) execute the generated script as:



import runpy

try:
runpy.run_path(generated_file)

except:
print_stack_trace(traceback.format_exc(), generated_file)


The "print_stack_trace()" function parses the traceback and uses the #line directives to translate back to the original source.



This seems to work okay for simple cases, but I'm not sure of the implications of running the generated script (whose contents are arbitrary) in the same python that is running my generator script.










share|improve this question

























  • Can you give us some code? I think seeing an example of a typical generated script would help, but seeing your main program's code (especially the part around which you see the error) would also be helpful.

    – Green Cloak Guy
    Nov 28 '18 at 21:05






  • 1





    Use the inspect package?

    – s3cur3
    Nov 28 '18 at 21:27











  • I see that the inspect package may help me do something more powerful regarding printing information once I've found an error. The difficulty I've had is finding the right place to catch the error in the first place. I haven't found a way to make the generated script catch its own syntax errors. Runtime errors are easy, but I don't know how to use a try block to catch parse-time errors, unless the generated script is executed in the context of the calling script, which I'm trying to avoid (or at least find a way to do safely).

    – user1806566
    Nov 29 '18 at 16:28
















0















I have a python script that reads some input, generates a python script, and then runs the generated script. What I would like is that if there are errors in the generated script, they are reported at the file/lineno of the input file, not in the generated script.



If python supported a #line directive as in C or perl, I'd just put those in the generated script, and that would be all I needed. It doesn't look like that is supported, however.



I tried having my generated script sit in a try block, with the idea that I could catch exceptions and use the traceback module to tweak how the stack trace is printed. The problem is that the most common type of error is a parse error, and those seem to happen and get reported outside my control (i.e., the exception is thrown before the try, so my exception handler is never invoked).



The best bet I've been able to come up with is to use compile() on my generated script and then try to catch errors. That looks like it would force me to slurp in my entire generated script as a string (which could be large), though. Also, what I really want is to be able to run the generated script in its own sandbox, so it's not affected at all by the calling script (if not for error reporting, I would invoke the script as a new python process). If I use compile/exec, it's unclear to me that would be the case. Would passing an empty dictionary for the globals argument of exec() do the trick?



Is there a better way to do this sort of thing?



Edit:



The generated script could be almost anything. The input file allows the user to insert arbitrary python code in the generated script. What I'd like to do is generate a script like:



temp.py:



#line 10 original_file
for i in range(0,3)::


and have the syntax error be reported on line 10 of "original_file" and not line 2 of temp.py (which would mean nothing to the user).



The best I have so far is to insert #line directives in the generated script and have have the original script (i.e., the one that generated temp.py) execute the generated script as:



import runpy

try:
runpy.run_path(generated_file)

except:
print_stack_trace(traceback.format_exc(), generated_file)


The "print_stack_trace()" function parses the traceback and uses the #line directives to translate back to the original source.



This seems to work okay for simple cases, but I'm not sure of the implications of running the generated script (whose contents are arbitrary) in the same python that is running my generator script.










share|improve this question

























  • Can you give us some code? I think seeing an example of a typical generated script would help, but seeing your main program's code (especially the part around which you see the error) would also be helpful.

    – Green Cloak Guy
    Nov 28 '18 at 21:05






  • 1





    Use the inspect package?

    – s3cur3
    Nov 28 '18 at 21:27











  • I see that the inspect package may help me do something more powerful regarding printing information once I've found an error. The difficulty I've had is finding the right place to catch the error in the first place. I haven't found a way to make the generated script catch its own syntax errors. Runtime errors are easy, but I don't know how to use a try block to catch parse-time errors, unless the generated script is executed in the context of the calling script, which I'm trying to avoid (or at least find a way to do safely).

    – user1806566
    Nov 29 '18 at 16:28














0












0








0








I have a python script that reads some input, generates a python script, and then runs the generated script. What I would like is that if there are errors in the generated script, they are reported at the file/lineno of the input file, not in the generated script.



If python supported a #line directive as in C or perl, I'd just put those in the generated script, and that would be all I needed. It doesn't look like that is supported, however.



I tried having my generated script sit in a try block, with the idea that I could catch exceptions and use the traceback module to tweak how the stack trace is printed. The problem is that the most common type of error is a parse error, and those seem to happen and get reported outside my control (i.e., the exception is thrown before the try, so my exception handler is never invoked).



The best bet I've been able to come up with is to use compile() on my generated script and then try to catch errors. That looks like it would force me to slurp in my entire generated script as a string (which could be large), though. Also, what I really want is to be able to run the generated script in its own sandbox, so it's not affected at all by the calling script (if not for error reporting, I would invoke the script as a new python process). If I use compile/exec, it's unclear to me that would be the case. Would passing an empty dictionary for the globals argument of exec() do the trick?



Is there a better way to do this sort of thing?



Edit:



The generated script could be almost anything. The input file allows the user to insert arbitrary python code in the generated script. What I'd like to do is generate a script like:



temp.py:



#line 10 original_file
for i in range(0,3)::


and have the syntax error be reported on line 10 of "original_file" and not line 2 of temp.py (which would mean nothing to the user).



The best I have so far is to insert #line directives in the generated script and have have the original script (i.e., the one that generated temp.py) execute the generated script as:



import runpy

try:
runpy.run_path(generated_file)

except:
print_stack_trace(traceback.format_exc(), generated_file)


The "print_stack_trace()" function parses the traceback and uses the #line directives to translate back to the original source.



This seems to work okay for simple cases, but I'm not sure of the implications of running the generated script (whose contents are arbitrary) in the same python that is running my generator script.










share|improve this question
















I have a python script that reads some input, generates a python script, and then runs the generated script. What I would like is that if there are errors in the generated script, they are reported at the file/lineno of the input file, not in the generated script.



If python supported a #line directive as in C or perl, I'd just put those in the generated script, and that would be all I needed. It doesn't look like that is supported, however.



I tried having my generated script sit in a try block, with the idea that I could catch exceptions and use the traceback module to tweak how the stack trace is printed. The problem is that the most common type of error is a parse error, and those seem to happen and get reported outside my control (i.e., the exception is thrown before the try, so my exception handler is never invoked).



The best bet I've been able to come up with is to use compile() on my generated script and then try to catch errors. That looks like it would force me to slurp in my entire generated script as a string (which could be large), though. Also, what I really want is to be able to run the generated script in its own sandbox, so it's not affected at all by the calling script (if not for error reporting, I would invoke the script as a new python process). If I use compile/exec, it's unclear to me that would be the case. Would passing an empty dictionary for the globals argument of exec() do the trick?



Is there a better way to do this sort of thing?



Edit:



The generated script could be almost anything. The input file allows the user to insert arbitrary python code in the generated script. What I'd like to do is generate a script like:



temp.py:



#line 10 original_file
for i in range(0,3)::


and have the syntax error be reported on line 10 of "original_file" and not line 2 of temp.py (which would mean nothing to the user).



The best I have so far is to insert #line directives in the generated script and have have the original script (i.e., the one that generated temp.py) execute the generated script as:



import runpy

try:
runpy.run_path(generated_file)

except:
print_stack_trace(traceback.format_exc(), generated_file)


The "print_stack_trace()" function parses the traceback and uses the #line directives to translate back to the original source.



This seems to work okay for simple cases, but I'm not sure of the implications of running the generated script (whose contents are arbitrary) in the same python that is running my generator script.







python






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 29 '18 at 16:23







user1806566

















asked Nov 28 '18 at 21:01









user1806566user1806566

203210




203210













  • Can you give us some code? I think seeing an example of a typical generated script would help, but seeing your main program's code (especially the part around which you see the error) would also be helpful.

    – Green Cloak Guy
    Nov 28 '18 at 21:05






  • 1





    Use the inspect package?

    – s3cur3
    Nov 28 '18 at 21:27











  • I see that the inspect package may help me do something more powerful regarding printing information once I've found an error. The difficulty I've had is finding the right place to catch the error in the first place. I haven't found a way to make the generated script catch its own syntax errors. Runtime errors are easy, but I don't know how to use a try block to catch parse-time errors, unless the generated script is executed in the context of the calling script, which I'm trying to avoid (or at least find a way to do safely).

    – user1806566
    Nov 29 '18 at 16:28



















  • Can you give us some code? I think seeing an example of a typical generated script would help, but seeing your main program's code (especially the part around which you see the error) would also be helpful.

    – Green Cloak Guy
    Nov 28 '18 at 21:05






  • 1





    Use the inspect package?

    – s3cur3
    Nov 28 '18 at 21:27











  • I see that the inspect package may help me do something more powerful regarding printing information once I've found an error. The difficulty I've had is finding the right place to catch the error in the first place. I haven't found a way to make the generated script catch its own syntax errors. Runtime errors are easy, but I don't know how to use a try block to catch parse-time errors, unless the generated script is executed in the context of the calling script, which I'm trying to avoid (or at least find a way to do safely).

    – user1806566
    Nov 29 '18 at 16:28

















Can you give us some code? I think seeing an example of a typical generated script would help, but seeing your main program's code (especially the part around which you see the error) would also be helpful.

– Green Cloak Guy
Nov 28 '18 at 21:05





Can you give us some code? I think seeing an example of a typical generated script would help, but seeing your main program's code (especially the part around which you see the error) would also be helpful.

– Green Cloak Guy
Nov 28 '18 at 21:05




1




1





Use the inspect package?

– s3cur3
Nov 28 '18 at 21:27





Use the inspect package?

– s3cur3
Nov 28 '18 at 21:27













I see that the inspect package may help me do something more powerful regarding printing information once I've found an error. The difficulty I've had is finding the right place to catch the error in the first place. I haven't found a way to make the generated script catch its own syntax errors. Runtime errors are easy, but I don't know how to use a try block to catch parse-time errors, unless the generated script is executed in the context of the calling script, which I'm trying to avoid (or at least find a way to do safely).

– user1806566
Nov 29 '18 at 16:28





I see that the inspect package may help me do something more powerful regarding printing information once I've found an error. The difficulty I've had is finding the right place to catch the error in the first place. I haven't found a way to make the generated script catch its own syntax errors. Runtime errors are easy, but I don't know how to use a try block to catch parse-time errors, unless the generated script is executed in the context of the calling script, which I'm trying to avoid (or at least find a way to do safely).

– user1806566
Nov 29 '18 at 16:28












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53528035%2fis-there-a-way-to-map-error-messages-to-the-source-file-for-generated-python-scr%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53528035%2fis-there-a-way-to-map-error-messages-to-the-source-file-for-generated-python-scr%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

Calculate evaluation metrics using cross_val_predict sklearn

Insert data from modal to MySQL (multiple modal on website)