AWS API Gateway with Lambda proxy always produces base64 string response
I'm using API Gateway Lambda proxy integration and trying to return a binary application/protobuf response. No matter what I do, the response body is always a base64 encoded string
- I have
application/protobufsetup as abinary media typesin APIG - My client (javascript) is sending following headers in the
POST:
Accept: application/protobuf
Content-Type: application/protobuf
- My lambda is responing with
content-type: application/protobuf, and correctly setting theIsBase64EncodedLambda response totrue
How do you get APIG to base64 decode the string? I swear I had this working a few months ago when I 1st tried this.
Note: I've also tried */* as a binary media types
Some related posts to add background:
- https://github.com/twitchtv/twirp/issues/81
- https://github.com/awslabs/aws-serverless-express/issues/39#issuecomment-276019222
Update:
Turns out I can only get it working if binary media type is set to */*. The client Accept header has no impact once it is set to this.
Many bad side effects of using */* because every response is attempted to get decoded (even when IsBase64Encoded is false or not set)
I thought it wasn't decoding because Chrome network inspect tools will always show binary data as base64 encoded in the Preview tab. You can see the protobuf in the Response tab.
amazon-web-services aws-lambda aws-api-gateway
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
add a comment |
I'm using API Gateway Lambda proxy integration and trying to return a binary application/protobuf response. No matter what I do, the response body is always a base64 encoded string
- I have
application/protobufsetup as abinary media typesin APIG - My client (javascript) is sending following headers in the
POST:
Accept: application/protobuf
Content-Type: application/protobuf
- My lambda is responing with
content-type: application/protobuf, and correctly setting theIsBase64EncodedLambda response totrue
How do you get APIG to base64 decode the string? I swear I had this working a few months ago when I 1st tried this.
Note: I've also tried */* as a binary media types
Some related posts to add background:
- https://github.com/twitchtv/twirp/issues/81
- https://github.com/awslabs/aws-serverless-express/issues/39#issuecomment-276019222
Update:
Turns out I can only get it working if binary media type is set to */*. The client Accept header has no impact once it is set to this.
Many bad side effects of using */* because every response is attempted to get decoded (even when IsBase64Encoded is false or not set)
I thought it wasn't decoding because Chrome network inspect tools will always show binary data as base64 encoded in the Preview tab. You can see the protobuf in the Response tab.
amazon-web-services aws-lambda aws-api-gateway
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
You mentioned "many bad side effects of using /", but I don't see them. Can you help me understand how to observe these side effects. I have a simple repo at github.com/activescott/apig-lambda-proxy-binary-response that demonstrates this but I'm not seeing any side-effects (e.g. on the html function response for example).
– Scott Willeke
Dec 17 '18 at 3:32
I have to admit, I didn't test the*/*on responses that were not binary. I was just making an educated guess that if APIG was configured to this, it would try to base64 decode EVERY response. Now that you mention this, I'm wondering if APIG only does the decode ifIsBase64Encodedis true. However that doesn't make sense in my head. If this were the case why the need for APIGbinary media typein the 1st place.
– rynop
Dec 17 '18 at 17:12
Thanks for responding. I'm trying to reason through the same things here :) And I have to admit, I'm unclear on the need/purpose of needing bothisBase64EncodedandbinaryMediaTypes. I'll probably keep tinkering and update that repo with more details as I confirm them...
– Scott Willeke
Dec 18 '18 at 1:50
In case anyone might find it helpful, I have dedicated a repo at github.com/activescott/apigateway-lambda-binary-response to documenting and demonstrating the various details of binary responses with Amazon API Gateway + AWS Lambda.
– Scott Willeke
Dec 18 '18 at 7:22
add a comment |
I'm using API Gateway Lambda proxy integration and trying to return a binary application/protobuf response. No matter what I do, the response body is always a base64 encoded string
- I have
application/protobufsetup as abinary media typesin APIG - My client (javascript) is sending following headers in the
POST:
Accept: application/protobuf
Content-Type: application/protobuf
- My lambda is responing with
content-type: application/protobuf, and correctly setting theIsBase64EncodedLambda response totrue
How do you get APIG to base64 decode the string? I swear I had this working a few months ago when I 1st tried this.
Note: I've also tried */* as a binary media types
Some related posts to add background:
- https://github.com/twitchtv/twirp/issues/81
- https://github.com/awslabs/aws-serverless-express/issues/39#issuecomment-276019222
Update:
Turns out I can only get it working if binary media type is set to */*. The client Accept header has no impact once it is set to this.
Many bad side effects of using */* because every response is attempted to get decoded (even when IsBase64Encoded is false or not set)
I thought it wasn't decoding because Chrome network inspect tools will always show binary data as base64 encoded in the Preview tab. You can see the protobuf in the Response tab.
amazon-web-services aws-lambda aws-api-gateway
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
I'm using API Gateway Lambda proxy integration and trying to return a binary application/protobuf response. No matter what I do, the response body is always a base64 encoded string
- I have
application/protobufsetup as abinary media typesin APIG - My client (javascript) is sending following headers in the
POST:
Accept: application/protobuf
Content-Type: application/protobuf
- My lambda is responing with
content-type: application/protobuf, and correctly setting theIsBase64EncodedLambda response totrue
How do you get APIG to base64 decode the string? I swear I had this working a few months ago when I 1st tried this.
Note: I've also tried */* as a binary media types
Some related posts to add background:
- https://github.com/twitchtv/twirp/issues/81
- https://github.com/awslabs/aws-serverless-express/issues/39#issuecomment-276019222
Update:
Turns out I can only get it working if binary media type is set to */*. The client Accept header has no impact once it is set to this.
Many bad side effects of using */* because every response is attempted to get decoded (even when IsBase64Encoded is false or not set)
I thought it wasn't decoding because Chrome network inspect tools will always show binary data as base64 encoded in the Preview tab. You can see the protobuf in the Response tab.
amazon-web-services aws-lambda aws-api-gateway
amazon-web-services aws-lambda aws-api-gateway
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
edited Nov 28 '18 at 3:08
rynop
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
asked Nov 28 '18 at 0:23
rynoprynop
23.1k196768
23.1k196768
You mentioned "many bad side effects of using /", but I don't see them. Can you help me understand how to observe these side effects. I have a simple repo at github.com/activescott/apig-lambda-proxy-binary-response that demonstrates this but I'm not seeing any side-effects (e.g. on the html function response for example).
– Scott Willeke
Dec 17 '18 at 3:32
I have to admit, I didn't test the*/*on responses that were not binary. I was just making an educated guess that if APIG was configured to this, it would try to base64 decode EVERY response. Now that you mention this, I'm wondering if APIG only does the decode ifIsBase64Encodedis true. However that doesn't make sense in my head. If this were the case why the need for APIGbinary media typein the 1st place.
– rynop
Dec 17 '18 at 17:12
Thanks for responding. I'm trying to reason through the same things here :) And I have to admit, I'm unclear on the need/purpose of needing bothisBase64EncodedandbinaryMediaTypes. I'll probably keep tinkering and update that repo with more details as I confirm them...
– Scott Willeke
Dec 18 '18 at 1:50
In case anyone might find it helpful, I have dedicated a repo at github.com/activescott/apigateway-lambda-binary-response to documenting and demonstrating the various details of binary responses with Amazon API Gateway + AWS Lambda.
– Scott Willeke
Dec 18 '18 at 7:22
add a comment |
You mentioned "many bad side effects of using /", but I don't see them. Can you help me understand how to observe these side effects. I have a simple repo at github.com/activescott/apig-lambda-proxy-binary-response that demonstrates this but I'm not seeing any side-effects (e.g. on the html function response for example).
– Scott Willeke
Dec 17 '18 at 3:32
I have to admit, I didn't test the*/*on responses that were not binary. I was just making an educated guess that if APIG was configured to this, it would try to base64 decode EVERY response. Now that you mention this, I'm wondering if APIG only does the decode ifIsBase64Encodedis true. However that doesn't make sense in my head. If this were the case why the need for APIGbinary media typein the 1st place.
– rynop
Dec 17 '18 at 17:12
Thanks for responding. I'm trying to reason through the same things here :) And I have to admit, I'm unclear on the need/purpose of needing bothisBase64EncodedandbinaryMediaTypes. I'll probably keep tinkering and update that repo with more details as I confirm them...
– Scott Willeke
Dec 18 '18 at 1:50
In case anyone might find it helpful, I have dedicated a repo at github.com/activescott/apigateway-lambda-binary-response to documenting and demonstrating the various details of binary responses with Amazon API Gateway + AWS Lambda.
– Scott Willeke
Dec 18 '18 at 7:22
You mentioned "many bad side effects of using /", but I don't see them. Can you help me understand how to observe these side effects. I have a simple repo at github.com/activescott/apig-lambda-proxy-binary-response that demonstrates this but I'm not seeing any side-effects (e.g. on the html function response for example).
– Scott Willeke
Dec 17 '18 at 3:32
You mentioned "many bad side effects of using /", but I don't see them. Can you help me understand how to observe these side effects. I have a simple repo at github.com/activescott/apig-lambda-proxy-binary-response that demonstrates this but I'm not seeing any side-effects (e.g. on the html function response for example).
– Scott Willeke
Dec 17 '18 at 3:32
I have to admit, I didn't test the
*/* on responses that were not binary. I was just making an educated guess that if APIG was configured to this, it would try to base64 decode EVERY response. Now that you mention this, I'm wondering if APIG only does the decode if IsBase64Encoded is true. However that doesn't make sense in my head. If this were the case why the need for APIG binary media type in the 1st place.– rynop
Dec 17 '18 at 17:12
I have to admit, I didn't test the
*/* on responses that were not binary. I was just making an educated guess that if APIG was configured to this, it would try to base64 decode EVERY response. Now that you mention this, I'm wondering if APIG only does the decode if IsBase64Encoded is true. However that doesn't make sense in my head. If this were the case why the need for APIG binary media type in the 1st place.– rynop
Dec 17 '18 at 17:12
Thanks for responding. I'm trying to reason through the same things here :) And I have to admit, I'm unclear on the need/purpose of needing both
isBase64Encoded and binaryMediaTypes. I'll probably keep tinkering and update that repo with more details as I confirm them...– Scott Willeke
Dec 18 '18 at 1:50
Thanks for responding. I'm trying to reason through the same things here :) And I have to admit, I'm unclear on the need/purpose of needing both
isBase64Encoded and binaryMediaTypes. I'll probably keep tinkering and update that repo with more details as I confirm them...– Scott Willeke
Dec 18 '18 at 1:50
In case anyone might find it helpful, I have dedicated a repo at github.com/activescott/apigateway-lambda-binary-response to documenting and demonstrating the various details of binary responses with Amazon API Gateway + AWS Lambda.
– Scott Willeke
Dec 18 '18 at 7:22
In case anyone might find it helpful, I have dedicated a repo at github.com/activescott/apigateway-lambda-binary-response to documenting and demonstrating the various details of binary responses with Amazon API Gateway + AWS Lambda.
– Scott Willeke
Dec 18 '18 at 7:22
add a comment |
1 Answer
1
active
oldest
votes
The problem was I'm using CloudFront in front of API Gateway, and I was not passing the Accept header to the origin (APIG).
The docs on handling binary with Lambda proxy are not great, so here is a quick summary:
- Your client must send an
Acceptheader who's 1st media type matches what you have set as abinary media typesin API Gateway - Your Lambda, if serving a binary media type, must set
IsBase64Encodedto true AND thebodymust be base64 encoded
If the clients Accept header matches an entry in API Gateway's binary media types and these conditions are met, API Gateway will transform (base64 decode) before sending a response to the client.
This blog post walks you through step-by-step on how to get it working (without CloudFront).
This is a full blown aws-blueprint for getting a production grade ci/cd with CloudFront.
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53510286%2faws-api-gateway-with-lambda-proxy-always-produces-base64-string-response%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The problem was I'm using CloudFront in front of API Gateway, and I was not passing the Accept header to the origin (APIG).
The docs on handling binary with Lambda proxy are not great, so here is a quick summary:
- Your client must send an
Acceptheader who's 1st media type matches what you have set as abinary media typesin API Gateway - Your Lambda, if serving a binary media type, must set
IsBase64Encodedto true AND thebodymust be base64 encoded
If the clients Accept header matches an entry in API Gateway's binary media types and these conditions are met, API Gateway will transform (base64 decode) before sending a response to the client.
This blog post walks you through step-by-step on how to get it working (without CloudFront).
This is a full blown aws-blueprint for getting a production grade ci/cd with CloudFront.
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
add a comment |
The problem was I'm using CloudFront in front of API Gateway, and I was not passing the Accept header to the origin (APIG).
The docs on handling binary with Lambda proxy are not great, so here is a quick summary:
- Your client must send an
Acceptheader who's 1st media type matches what you have set as abinary media typesin API Gateway - Your Lambda, if serving a binary media type, must set
IsBase64Encodedto true AND thebodymust be base64 encoded
If the clients Accept header matches an entry in API Gateway's binary media types and these conditions are met, API Gateway will transform (base64 decode) before sending a response to the client.
This blog post walks you through step-by-step on how to get it working (without CloudFront).
This is a full blown aws-blueprint for getting a production grade ci/cd with CloudFront.
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
add a comment |
The problem was I'm using CloudFront in front of API Gateway, and I was not passing the Accept header to the origin (APIG).
The docs on handling binary with Lambda proxy are not great, so here is a quick summary:
- Your client must send an
Acceptheader who's 1st media type matches what you have set as abinary media typesin API Gateway - Your Lambda, if serving a binary media type, must set
IsBase64Encodedto true AND thebodymust be base64 encoded
If the clients Accept header matches an entry in API Gateway's binary media types and these conditions are met, API Gateway will transform (base64 decode) before sending a response to the client.
This blog post walks you through step-by-step on how to get it working (without CloudFront).
This is a full blown aws-blueprint for getting a production grade ci/cd with CloudFront.
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
The problem was I'm using CloudFront in front of API Gateway, and I was not passing the Accept header to the origin (APIG).
The docs on handling binary with Lambda proxy are not great, so here is a quick summary:
- Your client must send an
Acceptheader who's 1st media type matches what you have set as abinary media typesin API Gateway - Your Lambda, if serving a binary media type, must set
IsBase64Encodedto true AND thebodymust be base64 encoded
If the clients Accept header matches an entry in API Gateway's binary media types and these conditions are met, API Gateway will transform (base64 decode) before sending a response to the client.
This blog post walks you through step-by-step on how to get it working (without CloudFront).
This is a full blown aws-blueprint for getting a production grade ci/cd with CloudFront.
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
edited Nov 28 '18 at 6:03
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
answered Nov 28 '18 at 1:28
rynoprynop
23.1k196768
23.1k196768
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53510286%2faws-api-gateway-with-lambda-proxy-always-produces-base64-string-response%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You mentioned "many bad side effects of using /", but I don't see them. Can you help me understand how to observe these side effects. I have a simple repo at github.com/activescott/apig-lambda-proxy-binary-response that demonstrates this but I'm not seeing any side-effects (e.g. on the html function response for example).
– Scott Willeke
Dec 17 '18 at 3:32
I have to admit, I didn't test the
*/*on responses that were not binary. I was just making an educated guess that if APIG was configured to this, it would try to base64 decode EVERY response. Now that you mention this, I'm wondering if APIG only does the decode ifIsBase64Encodedis true. However that doesn't make sense in my head. If this were the case why the need for APIGbinary media typein the 1st place.– rynop
Dec 17 '18 at 17:12
Thanks for responding. I'm trying to reason through the same things here :) And I have to admit, I'm unclear on the need/purpose of needing both
isBase64EncodedandbinaryMediaTypes. I'll probably keep tinkering and update that repo with more details as I confirm them...– Scott Willeke
Dec 18 '18 at 1:50
In case anyone might find it helpful, I have dedicated a repo at github.com/activescott/apigateway-lambda-binary-response to documenting and demonstrating the various details of binary responses with Amazon API Gateway + AWS Lambda.
– Scott Willeke
Dec 18 '18 at 7:22