Is there a way to customize Google provided Cloud Dataflow templates?
I have a requirement wherein I need to create a dataflow job that will transfer data from Google PubSub topics to BigQuery tables by filtering messages based on some attribute which will be part of json data. Using the Google provided templates I can use Google provided "PubSub to BigQuery" template, but that doesn't allow filtering of data.
Is there a way to customize the template "PubSub to Bigquery"? If yes, any links or samples on how to do that would really help.
google-cloud-platform google-bigquery google-cloud-dataflow google-cloud-pubsub
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
add a comment |
I have a requirement wherein I need to create a dataflow job that will transfer data from Google PubSub topics to BigQuery tables by filtering messages based on some attribute which will be part of json data. Using the Google provided templates I can use Google provided "PubSub to BigQuery" template, but that doesn't allow filtering of data.
Is there a way to customize the template "PubSub to Bigquery"? If yes, any links or samples on how to do that would really help.
google-cloud-platform google-bigquery google-cloud-dataflow google-cloud-pubsub
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
add a comment |
I have a requirement wherein I need to create a dataflow job that will transfer data from Google PubSub topics to BigQuery tables by filtering messages based on some attribute which will be part of json data. Using the Google provided templates I can use Google provided "PubSub to BigQuery" template, but that doesn't allow filtering of data.
Is there a way to customize the template "PubSub to Bigquery"? If yes, any links or samples on how to do that would really help.
google-cloud-platform google-bigquery google-cloud-dataflow google-cloud-pubsub
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
I have a requirement wherein I need to create a dataflow job that will transfer data from Google PubSub topics to BigQuery tables by filtering messages based on some attribute which will be part of json data. Using the Google provided templates I can use Google provided "PubSub to BigQuery" template, but that doesn't allow filtering of data.
Is there a way to customize the template "PubSub to Bigquery"? If yes, any links or samples on how to do that would really help.
google-cloud-platform google-bigquery google-cloud-dataflow google-cloud-pubsub
google-cloud-platform google-bigquery google-cloud-dataflow google-cloud-pubsub
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
asked Nov 27 '18 at 5:02
Rahul AgrawalRahul Agrawal
857
857
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Everything should be published on GitHub - feel free to clone and modify.
Templates:
- https://github.com/GoogleCloudPlatform/DataflowTemplates
Pub/Sub to BQ:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java
Also note that the template supports JS UDFs that you could use to customize transformations and filtering without the need to modify the template.
For example, this one provides transformations and filtering:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/test/resources/JavascriptTextTransformerTest/transform.js
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53493048%2fis-there-a-way-to-customize-google-provided-cloud-dataflow-templates%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Everything should be published on GitHub - feel free to clone and modify.
Templates:
- https://github.com/GoogleCloudPlatform/DataflowTemplates
Pub/Sub to BQ:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java
Also note that the template supports JS UDFs that you could use to customize transformations and filtering without the need to modify the template.
For example, this one provides transformations and filtering:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/test/resources/JavascriptTextTransformerTest/transform.js
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
add a comment |
Everything should be published on GitHub - feel free to clone and modify.
Templates:
- https://github.com/GoogleCloudPlatform/DataflowTemplates
Pub/Sub to BQ:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java
Also note that the template supports JS UDFs that you could use to customize transformations and filtering without the need to modify the template.
For example, this one provides transformations and filtering:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/test/resources/JavascriptTextTransformerTest/transform.js
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
add a comment |
Everything should be published on GitHub - feel free to clone and modify.
Templates:
- https://github.com/GoogleCloudPlatform/DataflowTemplates
Pub/Sub to BQ:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java
Also note that the template supports JS UDFs that you could use to customize transformations and filtering without the need to modify the template.
For example, this one provides transformations and filtering:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/test/resources/JavascriptTextTransformerTest/transform.js
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
Everything should be published on GitHub - feel free to clone and modify.
Templates:
- https://github.com/GoogleCloudPlatform/DataflowTemplates
Pub/Sub to BQ:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java
Also note that the template supports JS UDFs that you could use to customize transformations and filtering without the need to modify the template.
For example, this one provides transformations and filtering:
- https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/test/resources/JavascriptTextTransformerTest/transform.js
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
edited Nov 27 '18 at 5:32
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
answered Nov 27 '18 at 5:25
Felipe HoffaFelipe Hoffa
21.7k252113
21.7k252113
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53493048%2fis-there-a-way-to-customize-google-provided-cloud-dataflow-templates%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
