Jersey: Close I/O resources after HTTP response
up vote
1
down vote
favorite
My Setup: I have created a REST service (Jersey/Dropwizard) that streams large content from a database. During a GET operation, the service taps into the database through a connection pool, wraps the data into a stream and performs some on-the-fly transformation to render the requested data in various encodings (CSV, JSON, ...). The life time of the database connection is tied to the life time of the stream and only when the stream is closed, the database connection is released.
The stream transformation is performed by an Encoder class that returns a StreamingOutput which is then passed to the Response object. The Encoder currently handles resource closing when the stream is fully consumed.
My Problem: Since StreamingOutput does not implement AutoCloseable, connection leaks may occur when the output is only partially consumed.
I sometimes observe that stale active connections are piling up in the connection pool, and I suspect that they arise from aborted HTTP connections. As you can see below, the current code handles exceptions that occur in the try block. What I cannot handle are Exceptions that occur after the return statement and I don't know how to attach any instructions for resource closing to the Response object.
My Question: How can I inform the Response object to close particular resources after the request has terminated (regularly or due to an error)? Or: Is there a better way to safely close any associated resources when the request context ends?
@GET
//@Produces(...)
public Response streamData(
@PathParam("key") String key,
// ... other params
) {
//decode and validate params
Stream<Pojo> ps = null;
try {
// connect to db and obtain data stream for <key>
ps = loadData(db, key);
// apply detailed encoding instrunctions and create a StreamingOutput
final StreamingOutput stream = Encoder.encodeData(ps, encodingArgs);
return Response.ok(stream).build();
} catch (Exception e) {
closeOnException(ps); // wrapper for ps.close();
throw e;
}
}
java jersey response autocloseable
|
show 2 more comments
up vote
1
down vote
favorite
My Setup: I have created a REST service (Jersey/Dropwizard) that streams large content from a database. During a GET operation, the service taps into the database through a connection pool, wraps the data into a stream and performs some on-the-fly transformation to render the requested data in various encodings (CSV, JSON, ...). The life time of the database connection is tied to the life time of the stream and only when the stream is closed, the database connection is released.
The stream transformation is performed by an Encoder class that returns a StreamingOutput which is then passed to the Response object. The Encoder currently handles resource closing when the stream is fully consumed.
My Problem: Since StreamingOutput does not implement AutoCloseable, connection leaks may occur when the output is only partially consumed.
I sometimes observe that stale active connections are piling up in the connection pool, and I suspect that they arise from aborted HTTP connections. As you can see below, the current code handles exceptions that occur in the try block. What I cannot handle are Exceptions that occur after the return statement and I don't know how to attach any instructions for resource closing to the Response object.
My Question: How can I inform the Response object to close particular resources after the request has terminated (regularly or due to an error)? Or: Is there a better way to safely close any associated resources when the request context ends?
@GET
//@Produces(...)
public Response streamData(
@PathParam("key") String key,
// ... other params
) {
//decode and validate params
Stream<Pojo> ps = null;
try {
// connect to db and obtain data stream for <key>
ps = loadData(db, key);
// apply detailed encoding instrunctions and create a StreamingOutput
final StreamingOutput stream = Encoder.encodeData(ps, encodingArgs);
return Response.ok(stream).build();
} catch (Exception e) {
closeOnException(ps); // wrapper for ps.close();
throw e;
}
}
java jersey response autocloseable
Just thinking, from my understanding of your code, the real problem seems the reference to StreamingOutput object inside the Encode class (I don't know this Encode class, perhaps you wrote it). An alternative solution, you can extend StreamingOutput and create a class that also contains the loaded data as well, and the Encode is only a builder that does not maintain references to created streams.
– Fabiano Tarlao
Nov 22 at 17:05
In this way, when something goes wrong (everywhere), the GC is able to free the object cause the only reference is in the Response obj, that is going to be freed by GC as well. Is this feasible in your case? In that case I'll convert comment into an answer.
– Fabiano Tarlao
Nov 22 at 17:06
@FabianoTarlao: True, and this is probably the most robust solution. However, that way forces me to construct a StreamingOutput around every block of application logic that streams from the DB and encodes.
– Matthias
Nov 22 at 18:39
@FabianoTarlao: You really want to close I/O resources eagerly and don't wait for the GC. There's simply no guarantee that the GC will clean up the response objects in the near future.
– Matthias
Nov 22 at 18:45
Got it. The other idea is to return an object that implements StreamingOutput interface, perhaps extending your previous class, but that also updates an internal long "last_used_time" variable that can be periodically checked by Encode class. But I don't understand your constraints, can you change/rewrite the internals of StreamingOutput and Encode class, or not? I think that this is my last useful tip, I have not other ideas :-) and all these solutions look too much convoluted ;-)
– Fabiano Tarlao
Nov 22 at 22:18
|
show 2 more comments
up vote
1
down vote
favorite
up vote
1
down vote
favorite
My Setup: I have created a REST service (Jersey/Dropwizard) that streams large content from a database. During a GET operation, the service taps into the database through a connection pool, wraps the data into a stream and performs some on-the-fly transformation to render the requested data in various encodings (CSV, JSON, ...). The life time of the database connection is tied to the life time of the stream and only when the stream is closed, the database connection is released.
The stream transformation is performed by an Encoder class that returns a StreamingOutput which is then passed to the Response object. The Encoder currently handles resource closing when the stream is fully consumed.
My Problem: Since StreamingOutput does not implement AutoCloseable, connection leaks may occur when the output is only partially consumed.
I sometimes observe that stale active connections are piling up in the connection pool, and I suspect that they arise from aborted HTTP connections. As you can see below, the current code handles exceptions that occur in the try block. What I cannot handle are Exceptions that occur after the return statement and I don't know how to attach any instructions for resource closing to the Response object.
My Question: How can I inform the Response object to close particular resources after the request has terminated (regularly or due to an error)? Or: Is there a better way to safely close any associated resources when the request context ends?
@GET
//@Produces(...)
public Response streamData(
@PathParam("key") String key,
// ... other params
) {
//decode and validate params
Stream<Pojo> ps = null;
try {
// connect to db and obtain data stream for <key>
ps = loadData(db, key);
// apply detailed encoding instrunctions and create a StreamingOutput
final StreamingOutput stream = Encoder.encodeData(ps, encodingArgs);
return Response.ok(stream).build();
} catch (Exception e) {
closeOnException(ps); // wrapper for ps.close();
throw e;
}
}
java jersey response autocloseable
My Setup: I have created a REST service (Jersey/Dropwizard) that streams large content from a database. During a GET operation, the service taps into the database through a connection pool, wraps the data into a stream and performs some on-the-fly transformation to render the requested data in various encodings (CSV, JSON, ...). The life time of the database connection is tied to the life time of the stream and only when the stream is closed, the database connection is released.
The stream transformation is performed by an Encoder class that returns a StreamingOutput which is then passed to the Response object. The Encoder currently handles resource closing when the stream is fully consumed.
My Problem: Since StreamingOutput does not implement AutoCloseable, connection leaks may occur when the output is only partially consumed.
I sometimes observe that stale active connections are piling up in the connection pool, and I suspect that they arise from aborted HTTP connections. As you can see below, the current code handles exceptions that occur in the try block. What I cannot handle are Exceptions that occur after the return statement and I don't know how to attach any instructions for resource closing to the Response object.
My Question: How can I inform the Response object to close particular resources after the request has terminated (regularly or due to an error)? Or: Is there a better way to safely close any associated resources when the request context ends?
@GET
//@Produces(...)
public Response streamData(
@PathParam("key") String key,
// ... other params
) {
//decode and validate params
Stream<Pojo> ps = null;
try {
// connect to db and obtain data stream for <key>
ps = loadData(db, key);
// apply detailed encoding instrunctions and create a StreamingOutput
final StreamingOutput stream = Encoder.encodeData(ps, encodingArgs);
return Response.ok(stream).build();
} catch (Exception e) {
closeOnException(ps); // wrapper for ps.close();
throw e;
}
}
java jersey response autocloseable
java jersey response autocloseable
asked Nov 22 at 14:32
Matthias
948
948
Just thinking, from my understanding of your code, the real problem seems the reference to StreamingOutput object inside the Encode class (I don't know this Encode class, perhaps you wrote it). An alternative solution, you can extend StreamingOutput and create a class that also contains the loaded data as well, and the Encode is only a builder that does not maintain references to created streams.
– Fabiano Tarlao
Nov 22 at 17:05
In this way, when something goes wrong (everywhere), the GC is able to free the object cause the only reference is in the Response obj, that is going to be freed by GC as well. Is this feasible in your case? In that case I'll convert comment into an answer.
– Fabiano Tarlao
Nov 22 at 17:06
@FabianoTarlao: True, and this is probably the most robust solution. However, that way forces me to construct a StreamingOutput around every block of application logic that streams from the DB and encodes.
– Matthias
Nov 22 at 18:39
@FabianoTarlao: You really want to close I/O resources eagerly and don't wait for the GC. There's simply no guarantee that the GC will clean up the response objects in the near future.
– Matthias
Nov 22 at 18:45
Got it. The other idea is to return an object that implements StreamingOutput interface, perhaps extending your previous class, but that also updates an internal long "last_used_time" variable that can be periodically checked by Encode class. But I don't understand your constraints, can you change/rewrite the internals of StreamingOutput and Encode class, or not? I think that this is my last useful tip, I have not other ideas :-) and all these solutions look too much convoluted ;-)
– Fabiano Tarlao
Nov 22 at 22:18
|
show 2 more comments
Just thinking, from my understanding of your code, the real problem seems the reference to StreamingOutput object inside the Encode class (I don't know this Encode class, perhaps you wrote it). An alternative solution, you can extend StreamingOutput and create a class that also contains the loaded data as well, and the Encode is only a builder that does not maintain references to created streams.
– Fabiano Tarlao
Nov 22 at 17:05
In this way, when something goes wrong (everywhere), the GC is able to free the object cause the only reference is in the Response obj, that is going to be freed by GC as well. Is this feasible in your case? In that case I'll convert comment into an answer.
– Fabiano Tarlao
Nov 22 at 17:06
@FabianoTarlao: True, and this is probably the most robust solution. However, that way forces me to construct a StreamingOutput around every block of application logic that streams from the DB and encodes.
– Matthias
Nov 22 at 18:39
@FabianoTarlao: You really want to close I/O resources eagerly and don't wait for the GC. There's simply no guarantee that the GC will clean up the response objects in the near future.
– Matthias
Nov 22 at 18:45
Got it. The other idea is to return an object that implements StreamingOutput interface, perhaps extending your previous class, but that also updates an internal long "last_used_time" variable that can be periodically checked by Encode class. But I don't understand your constraints, can you change/rewrite the internals of StreamingOutput and Encode class, or not? I think that this is my last useful tip, I have not other ideas :-) and all these solutions look too much convoluted ;-)
– Fabiano Tarlao
Nov 22 at 22:18
Just thinking, from my understanding of your code, the real problem seems the reference to StreamingOutput object inside the Encode class (I don't know this Encode class, perhaps you wrote it). An alternative solution, you can extend StreamingOutput and create a class that also contains the loaded data as well, and the Encode is only a builder that does not maintain references to created streams.
– Fabiano Tarlao
Nov 22 at 17:05
Just thinking, from my understanding of your code, the real problem seems the reference to StreamingOutput object inside the Encode class (I don't know this Encode class, perhaps you wrote it). An alternative solution, you can extend StreamingOutput and create a class that also contains the loaded data as well, and the Encode is only a builder that does not maintain references to created streams.
– Fabiano Tarlao
Nov 22 at 17:05
In this way, when something goes wrong (everywhere), the GC is able to free the object cause the only reference is in the Response obj, that is going to be freed by GC as well. Is this feasible in your case? In that case I'll convert comment into an answer.
– Fabiano Tarlao
Nov 22 at 17:06
In this way, when something goes wrong (everywhere), the GC is able to free the object cause the only reference is in the Response obj, that is going to be freed by GC as well. Is this feasible in your case? In that case I'll convert comment into an answer.
– Fabiano Tarlao
Nov 22 at 17:06
@FabianoTarlao: True, and this is probably the most robust solution. However, that way forces me to construct a StreamingOutput around every block of application logic that streams from the DB and encodes.
– Matthias
Nov 22 at 18:39
@FabianoTarlao: True, and this is probably the most robust solution. However, that way forces me to construct a StreamingOutput around every block of application logic that streams from the DB and encodes.
– Matthias
Nov 22 at 18:39
@FabianoTarlao: You really want to close I/O resources eagerly and don't wait for the GC. There's simply no guarantee that the GC will clean up the response objects in the near future.
– Matthias
Nov 22 at 18:45
@FabianoTarlao: You really want to close I/O resources eagerly and don't wait for the GC. There's simply no guarantee that the GC will clean up the response objects in the near future.
– Matthias
Nov 22 at 18:45
Got it. The other idea is to return an object that implements StreamingOutput interface, perhaps extending your previous class, but that also updates an internal long "last_used_time" variable that can be periodically checked by Encode class. But I don't understand your constraints, can you change/rewrite the internals of StreamingOutput and Encode class, or not? I think that this is my last useful tip, I have not other ideas :-) and all these solutions look too much convoluted ;-)
– Fabiano Tarlao
Nov 22 at 22:18
Got it. The other idea is to return an object that implements StreamingOutput interface, perhaps extending your previous class, but that also updates an internal long "last_used_time" variable that can be periodically checked by Encode class. But I don't understand your constraints, can you change/rewrite the internals of StreamingOutput and Encode class, or not? I think that this is my last useful tip, I have not other ideas :-) and all these solutions look too much convoluted ;-)
– Fabiano Tarlao
Nov 22 at 22:18
|
show 2 more comments
1 Answer
1
active
oldest
votes
up vote
0
down vote
You can add to your method HttpServletResponse:
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Object streamData(
@PathParam("key") String key,
@Context HttpServletResponse response,
// ... other params
) {
...
response.getOutputStream().write(....)
response.flushBuffer();
response.getOutputStream().close();
return null;
}
Hmm, I was hoping for some convenient hook (similar to Stream.onClose(Autocloseable)) that I could use to avoid boilerplate code. Do you know how Jersey treats the retuned object from the method signature? Is it just discarded?
– Matthias
Nov 22 at 14:59
may be this will be helpfull stackoverflow.com/questions/39572872/…
– xyz
Nov 22 at 15:24
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53433174%2fjersey-close-i-o-resources-after-http-response%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
You can add to your method HttpServletResponse:
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Object streamData(
@PathParam("key") String key,
@Context HttpServletResponse response,
// ... other params
) {
...
response.getOutputStream().write(....)
response.flushBuffer();
response.getOutputStream().close();
return null;
}
Hmm, I was hoping for some convenient hook (similar to Stream.onClose(Autocloseable)) that I could use to avoid boilerplate code. Do you know how Jersey treats the retuned object from the method signature? Is it just discarded?
– Matthias
Nov 22 at 14:59
may be this will be helpfull stackoverflow.com/questions/39572872/…
– xyz
Nov 22 at 15:24
add a comment |
up vote
0
down vote
You can add to your method HttpServletResponse:
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Object streamData(
@PathParam("key") String key,
@Context HttpServletResponse response,
// ... other params
) {
...
response.getOutputStream().write(....)
response.flushBuffer();
response.getOutputStream().close();
return null;
}
Hmm, I was hoping for some convenient hook (similar to Stream.onClose(Autocloseable)) that I could use to avoid boilerplate code. Do you know how Jersey treats the retuned object from the method signature? Is it just discarded?
– Matthias
Nov 22 at 14:59
may be this will be helpfull stackoverflow.com/questions/39572872/…
– xyz
Nov 22 at 15:24
add a comment |
up vote
0
down vote
up vote
0
down vote
You can add to your method HttpServletResponse:
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Object streamData(
@PathParam("key") String key,
@Context HttpServletResponse response,
// ... other params
) {
...
response.getOutputStream().write(....)
response.flushBuffer();
response.getOutputStream().close();
return null;
}
You can add to your method HttpServletResponse:
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Object streamData(
@PathParam("key") String key,
@Context HttpServletResponse response,
// ... other params
) {
...
response.getOutputStream().write(....)
response.flushBuffer();
response.getOutputStream().close();
return null;
}
answered Nov 22 at 14:44
xyz
133113
133113
Hmm, I was hoping for some convenient hook (similar to Stream.onClose(Autocloseable)) that I could use to avoid boilerplate code. Do you know how Jersey treats the retuned object from the method signature? Is it just discarded?
– Matthias
Nov 22 at 14:59
may be this will be helpfull stackoverflow.com/questions/39572872/…
– xyz
Nov 22 at 15:24
add a comment |
Hmm, I was hoping for some convenient hook (similar to Stream.onClose(Autocloseable)) that I could use to avoid boilerplate code. Do you know how Jersey treats the retuned object from the method signature? Is it just discarded?
– Matthias
Nov 22 at 14:59
may be this will be helpfull stackoverflow.com/questions/39572872/…
– xyz
Nov 22 at 15:24
Hmm, I was hoping for some convenient hook (similar to Stream.onClose(Autocloseable)) that I could use to avoid boilerplate code. Do you know how Jersey treats the retuned object from the method signature? Is it just discarded?
– Matthias
Nov 22 at 14:59
Hmm, I was hoping for some convenient hook (similar to Stream.onClose(Autocloseable)) that I could use to avoid boilerplate code. Do you know how Jersey treats the retuned object from the method signature? Is it just discarded?
– Matthias
Nov 22 at 14:59
may be this will be helpfull stackoverflow.com/questions/39572872/…
– xyz
Nov 22 at 15:24
may be this will be helpfull stackoverflow.com/questions/39572872/…
– xyz
Nov 22 at 15:24
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53433174%2fjersey-close-i-o-resources-after-http-response%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Just thinking, from my understanding of your code, the real problem seems the reference to StreamingOutput object inside the Encode class (I don't know this Encode class, perhaps you wrote it). An alternative solution, you can extend StreamingOutput and create a class that also contains the loaded data as well, and the Encode is only a builder that does not maintain references to created streams.
– Fabiano Tarlao
Nov 22 at 17:05
In this way, when something goes wrong (everywhere), the GC is able to free the object cause the only reference is in the Response obj, that is going to be freed by GC as well. Is this feasible in your case? In that case I'll convert comment into an answer.
– Fabiano Tarlao
Nov 22 at 17:06
@FabianoTarlao: True, and this is probably the most robust solution. However, that way forces me to construct a StreamingOutput around every block of application logic that streams from the DB and encodes.
– Matthias
Nov 22 at 18:39
@FabianoTarlao: You really want to close I/O resources eagerly and don't wait for the GC. There's simply no guarantee that the GC will clean up the response objects in the near future.
– Matthias
Nov 22 at 18:45
Got it. The other idea is to return an object that implements StreamingOutput interface, perhaps extending your previous class, but that also updates an internal long "last_used_time" variable that can be periodically checked by Encode class. But I don't understand your constraints, can you change/rewrite the internals of StreamingOutput and Encode class, or not? I think that this is my last useful tip, I have not other ideas :-) and all these solutions look too much convoluted ;-)
– Fabiano Tarlao
Nov 22 at 22:18