Web api call in foreach loop - Timeout Error
I am using .NET Core Web API. I am working with some third party web api which I need to call in my Web API (so my .NET Core Web API is kind of wrapper on third party ones).
To get my result, I need to call more then one third party api using foreach loop. Details are as below:
First will be a Web API call which gives me result of around 4000 rows (each row is object of Id and value fields).
After that I need to loop through this 4000 rows and using each Id I need to call another API. On the result of this Web API I need to check some validation and return the valid ones.
I am able to make first Web API call successfully but when I do looping for another API call it gives me timeout error.
I have tried below things
1) making batches of 4000 rows and processing in batches.
2) Adding tasks in `foreach` loop and using `Task.WhenAll`
Example :
var batchSize = 50;
var returnData = new List<Order>();
foreach (var batchedItems in inventoriesList.Batch(batchSize)) //4000 rows
{
var tasks = new List<Task<Order>>();
foreach (var item in batchedItems)
{
tasks.Add(GetOrder(item.Value)); //call to another api
}
foreach (var task in await Task.WhenAll(tasks))
{
returnData.Add(task);
}
}
private async Task<Order> GetOrder(string id)
{
var order = await GetAsync<Order>(api-url);
if (order != null
&& order.IsAvailable == false
&& ValidateOrder(order)))
{
isValidOrder = true;
}
return isValidOrder == true ? order : null;
}
I have tried with LINQ as well rather then doing foreach loop for second API
call. Like below,
tasks = batchedInventories.Select(t => GetOrder(t.Value));
var result = await Task.WhenAll(tasks);
I have also tried with increasing KeepAliveTimeout of Kestrel. But no luck.
Could anybody suggest me correct and working way to do this?
c# .net-core async-await asp.net-core-webapi dotnet-httpclient
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
asked Nov 26 '18 at 1:20
vibsvibs
4719
|
show 5 more comments
I am using .NET Core Web API. I am working with some third party web api which I need to call in my Web API (so my .NET Core Web API is kind of wrapper on third party ones).
To get my result, I need to call more then one third party api using foreach loop. Details are as below:
First will be a Web API call which gives me result of around 4000 rows (each row is object of Id and value fields).
After that I need to loop through this 4000 rows and using each Id I need to call another API. On the result of this Web API I need to check some validation and return the valid ones.
I am able to make first Web API call successfully but when I do looping for another API call it gives me timeout error.
I have tried below things
1) making batches of 4000 rows and processing in batches.
2) Adding tasks in `foreach` loop and using `Task.WhenAll`
Example :
var batchSize = 50;
var returnData = new List<Order>();
foreach (var batchedItems in inventoriesList.Batch(batchSize)) //4000 rows
{
var tasks = new List<Task<Order>>();
foreach (var item in batchedItems)
{
tasks.Add(GetOrder(item.Value)); //call to another api
}
foreach (var task in await Task.WhenAll(tasks))
{
returnData.Add(task);
}
}
private async Task<Order> GetOrder(string id)
{
var order = await GetAsync<Order>(api-url);
if (order != null
&& order.IsAvailable == false
&& ValidateOrder(order)))
{
isValidOrder = true;
}
return isValidOrder == true ? order : null;
}
I have tried with LINQ as well rather then doing foreach loop for second API
call. Like below,
tasks = batchedInventories.Select(t => GetOrder(t.Value));
var result = await Task.WhenAll(tasks);
I have also tried with increasing KeepAliveTimeout of Kestrel. But no luck.
Could anybody suggest me correct and working way to do this?
c# .net-core async-await asp.net-core-webapi dotnet-httpclient
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
asked Nov 26 '18 at 1:20
vibsvibs
4719
3
I'd recommend showing yourGetAsynccode, and if my hunch is right, you might also need to read this. Alternatively the API is throttling your many requests, or you're simply overloading it with too many requests.
– John
Nov 26 '18 at 1:32
1
yes, I am using "using" statement in GetAsync and creating new HttpClient for each request. I will try sharing single httpclient instance.
– vibs
Nov 26 '18 at 1:46
Just a quick question, I have HttpClientHandler (with cookiecontainer) , how should I use that with static httpclient?
– vibs
Nov 26 '18 at 3:31
Can you provide the exact error message, and the line of code it is occurring on.
– Dale Burrell
Nov 26 '18 at 3:44
Before you start experimenting with code - did you actually estimated (or better yet tried with something like Fiddler) to send 4K requests to that service and see how long it takes? Sending so many requests in short amount of time will overwhelm almost all except the most powerful servers...
– Alexei Levenkov
Nov 26 '18 at 3:44
|
show 5 more comments
I am using .NET Core Web API. I am working with some third party web api which I need to call in my Web API (so my .NET Core Web API is kind of wrapper on third party ones).
To get my result, I need to call more then one third party api using foreach loop. Details are as below:
First will be a Web API call which gives me result of around 4000 rows (each row is object of Id and value fields).
After that I need to loop through this 4000 rows and using each Id I need to call another API. On the result of this Web API I need to check some validation and return the valid ones.
I am able to make first Web API call successfully but when I do looping for another API call it gives me timeout error.
I have tried below things
1) making batches of 4000 rows and processing in batches.
2) Adding tasks in `foreach` loop and using `Task.WhenAll`
Example :
var batchSize = 50;
var returnData = new List<Order>();
foreach (var batchedItems in inventoriesList.Batch(batchSize)) //4000 rows
{
var tasks = new List<Task<Order>>();
foreach (var item in batchedItems)
{
tasks.Add(GetOrder(item.Value)); //call to another api
}
foreach (var task in await Task.WhenAll(tasks))
{
returnData.Add(task);
}
}
private async Task<Order> GetOrder(string id)
{
var order = await GetAsync<Order>(api-url);
if (order != null
&& order.IsAvailable == false
&& ValidateOrder(order)))
{
isValidOrder = true;
}
return isValidOrder == true ? order : null;
}
I have tried with LINQ as well rather then doing foreach loop for second API
call. Like below,
tasks = batchedInventories.Select(t => GetOrder(t.Value));
var result = await Task.WhenAll(tasks);
I have also tried with increasing KeepAliveTimeout of Kestrel. But no luck.
Could anybody suggest me correct and working way to do this?
c# .net-core async-await asp.net-core-webapi dotnet-httpclient
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
asked Nov 26 '18 at 1:20
vibsvibs
4719
I am using .NET Core Web API. I am working with some third party web api which I need to call in my Web API (so my .NET Core Web API is kind of wrapper on third party ones).
To get my result, I need to call more then one third party api using foreach loop. Details are as below:
First will be a Web API call which gives me result of around 4000 rows (each row is object of Id and value fields).
After that I need to loop through this 4000 rows and using each Id I need to call another API. On the result of this Web API I need to check some validation and return the valid ones.
I am able to make first Web API call successfully but when I do looping for another API call it gives me timeout error.
I have tried below things
1) making batches of 4000 rows and processing in batches.
2) Adding tasks in `foreach` loop and using `Task.WhenAll`
Example :
var batchSize = 50;
var returnData = new List<Order>();
foreach (var batchedItems in inventoriesList.Batch(batchSize)) //4000 rows
{
var tasks = new List<Task<Order>>();
foreach (var item in batchedItems)
{
tasks.Add(GetOrder(item.Value)); //call to another api
}
foreach (var task in await Task.WhenAll(tasks))
{
returnData.Add(task);
}
}
private async Task<Order> GetOrder(string id)
{
var order = await GetAsync<Order>(api-url);
if (order != null
&& order.IsAvailable == false
&& ValidateOrder(order)))
{
isValidOrder = true;
}
return isValidOrder == true ? order : null;
}
I have tried with LINQ as well rather then doing foreach loop for second API
call. Like below,
tasks = batchedInventories.Select(t => GetOrder(t.Value));
var result = await Task.WhenAll(tasks);
I have also tried with increasing KeepAliveTimeout of Kestrel. But no luck.
Could anybody suggest me correct and working way to do this?
c# .net-core async-await asp.net-core-webapi dotnet-httpclient
c# .net-core async-await asp.net-core-webapi dotnet-httpclient
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
asked Nov 26 '18 at 1:20
vibsvibs
4719
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
asked Nov 26 '18 at 1:20
vibsvibs
4719
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
edited Nov 27 '18 at 1:28
Dale Burrell
3,17332451
3,17332451
asked Nov 26 '18 at 1:20
vibsvibs
4719
asked Nov 26 '18 at 1:20
vibsvibs
4719
asked Nov 26 '18 at 1:20
vibsvibs
4719
4719
3
I'd recommend showing yourGetAsynccode, and if my hunch is right, you might also need to read this. Alternatively the API is throttling your many requests, or you're simply overloading it with too many requests.
– John
Nov 26 '18 at 1:32
1
yes, I am using "using" statement in GetAsync and creating new HttpClient for each request. I will try sharing single httpclient instance.
– vibs
Nov 26 '18 at 1:46
Just a quick question, I have HttpClientHandler (with cookiecontainer) , how should I use that with static httpclient?
– vibs
Nov 26 '18 at 3:31
Can you provide the exact error message, and the line of code it is occurring on.
– Dale Burrell
Nov 26 '18 at 3:44
Before you start experimenting with code - did you actually estimated (or better yet tried with something like Fiddler) to send 4K requests to that service and see how long it takes? Sending so many requests in short amount of time will overwhelm almost all except the most powerful servers...
– Alexei Levenkov
Nov 26 '18 at 3:44
|
show 5 more comments
3
I'd recommend showing yourGetAsynccode, and if my hunch is right, you might also need to read this. Alternatively the API is throttling your many requests, or you're simply overloading it with too many requests.
– John
Nov 26 '18 at 1:32
1
yes, I am using "using" statement in GetAsync and creating new HttpClient for each request. I will try sharing single httpclient instance.
– vibs
Nov 26 '18 at 1:46
Just a quick question, I have HttpClientHandler (with cookiecontainer) , how should I use that with static httpclient?
– vibs
Nov 26 '18 at 3:31
Can you provide the exact error message, and the line of code it is occurring on.
– Dale Burrell
Nov 26 '18 at 3:44
Before you start experimenting with code - did you actually estimated (or better yet tried with something like Fiddler) to send 4K requests to that service and see how long it takes? Sending so many requests in short amount of time will overwhelm almost all except the most powerful servers...
– Alexei Levenkov
Nov 26 '18 at 3:44
3
3
I'd recommend showing your
GetAsync code, and if my hunch is right, you might also need to read this. Alternatively the API is throttling your many requests, or you're simply overloading it with too many requests.– John
Nov 26 '18 at 1:32
I'd recommend showing your
GetAsync code, and if my hunch is right, you might also need to read this. Alternatively the API is throttling your many requests, or you're simply overloading it with too many requests.– John
Nov 26 '18 at 1:32
1
1
yes, I am using "using" statement in GetAsync and creating new HttpClient for each request. I will try sharing single httpclient instance.
– vibs
Nov 26 '18 at 1:46
yes, I am using "using" statement in GetAsync and creating new HttpClient for each request. I will try sharing single httpclient instance.
– vibs
Nov 26 '18 at 1:46
Just a quick question, I have HttpClientHandler (with cookiecontainer) , how should I use that with static httpclient?
– vibs
Nov 26 '18 at 3:31
Just a quick question, I have HttpClientHandler (with cookiecontainer) , how should I use that with static httpclient?
– vibs
Nov 26 '18 at 3:31
Can you provide the exact error message, and the line of code it is occurring on.
– Dale Burrell
Nov 26 '18 at 3:44
Can you provide the exact error message, and the line of code it is occurring on.
– Dale Burrell
Nov 26 '18 at 3:44
Before you start experimenting with code - did you actually estimated (or better yet tried with something like Fiddler) to send 4K requests to that service and see how long it takes? Sending so many requests in short amount of time will overwhelm almost all except the most powerful servers...
– Alexei Levenkov
Nov 26 '18 at 3:44
Before you start experimenting with code - did you actually estimated (or better yet tried with something like Fiddler) to send 4K requests to that service and see how long it takes? Sending so many requests in short amount of time will overwhelm almost all except the most powerful servers...
– Alexei Levenkov
Nov 26 '18 at 3:44
|
show 5 more comments
1 Answer
1
active
oldest
votes
I managed to resolve this error by creating static HttpClient for GetAsync method rather then creating instance for each request. Thanks a ton to John !!
I initialized static HttpClient in constructor with cookiecontainer.
Reference : Using httpclient throughout methods without losing session and cookies
answered Nov 27 '18 at 1:21
vibsvibs
4719
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473650%2fweb-api-call-in-foreach-loop-timeout-error%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I managed to resolve this error by creating static HttpClient for GetAsync method rather then creating instance for each request. Thanks a ton to John !!
I initialized static HttpClient in constructor with cookiecontainer.
Reference : Using httpclient throughout methods without losing session and cookies
answered Nov 27 '18 at 1:21
vibsvibs
4719
add a comment |
I managed to resolve this error by creating static HttpClient for GetAsync method rather then creating instance for each request. Thanks a ton to John !!
I initialized static HttpClient in constructor with cookiecontainer.
Reference : Using httpclient throughout methods without losing session and cookies
answered Nov 27 '18 at 1:21
vibsvibs
4719
add a comment |
I managed to resolve this error by creating static HttpClient for GetAsync method rather then creating instance for each request. Thanks a ton to John !!
I initialized static HttpClient in constructor with cookiecontainer.
Reference : Using httpclient throughout methods without losing session and cookies
answered Nov 27 '18 at 1:21
vibsvibs
4719
I managed to resolve this error by creating static HttpClient for GetAsync method rather then creating instance for each request. Thanks a ton to John !!
I initialized static HttpClient in constructor with cookiecontainer.
Reference : Using httpclient throughout methods without losing session and cookies
answered Nov 27 '18 at 1:21
vibsvibs
4719
answered Nov 27 '18 at 1:21
vibsvibs
4719
answered Nov 27 '18 at 1:21
vibsvibs
4719
answered Nov 27 '18 at 1:21
vibsvibs
4719
4719
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473650%2fweb-api-call-in-foreach-loop-timeout-error%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
I'd recommend showing your
GetAsynccode, and if my hunch is right, you might also need to read this. Alternatively the API is throttling your many requests, or you're simply overloading it with too many requests.– John
Nov 26 '18 at 1:32
1
yes, I am using "using" statement in GetAsync and creating new HttpClient for each request. I will try sharing single httpclient instance.
– vibs
Nov 26 '18 at 1:46
Just a quick question, I have HttpClientHandler (with cookiecontainer) , how should I use that with static httpclient?
– vibs
Nov 26 '18 at 3:31
Can you provide the exact error message, and the line of code it is occurring on.
– Dale Burrell
Nov 26 '18 at 3:44
Before you start experimenting with code - did you actually estimated (or better yet tried with something like Fiddler) to send 4K requests to that service and see how long it takes? Sending so many requests in short amount of time will overwhelm almost all except the most powerful servers...
– Alexei Levenkov
Nov 26 '18 at 3:44