Updating page number on API call with recursion - recursion

I'm making API calls and the service I am using only returns a certain number of results per page(I need all the results, not just first page). Fortunately they have a key on the JSON object called "total_pages" that says how many pages there are. There's also a "current_page" attribute. You can add a pager to the end of the URL you are calling like so: "example.com/endpoint&page=2"
I realized that the only way to get all of the pages is to loop through the total_pages number and somehow create another call to the api with the new pager info.
This can be done with a for or while loop by incrementing the current_page attribute but I can't seem to figure out how to do it with recursion. I have come up with a way to do it with Agent that holds and updates the current_page state but it seems unnecessary.
Agent.start(fn -> 1 end, name: current_page)
Agent.update(:current_page, fn curr_page -> state + 1 end)
Is there a way to do this in Elixir, maybe with a cond statement that checks if the current_page == total_pages and if not it runs a function that calls the API again?
THanks

Yes, an Agent is overkill for this. A simple recursive function can solve this elegantly. Here's a basic implementation that should help you get started:
defmodule A do
def fetch(current_page) do
response = get("/?page=#{current_page}")
if response.total_pages == current_page do
:ok
else
fetch(current_page + 1)
end
end
# Sample function that acts like an HTTP GET request.
def get(url) do
IO.puts "GET #{url}"
%{total_pages: 5}
end
end
A.fetch(1)
Output:
GET /?page=1
GET /?page=2
GET /?page=3
GET /?page=4
GET /?page=5
We start with page 1. After fetching the contents of the page, we check if the current page equals the total_pages in the response. If it does, we stop, otherwise we recurse with current_page set to current_page + 1.

Related

Firebase + DialogFlow - Realtime database- Cloud function return query result only after second request

I'm writing a simple Google Action which will read the Firebase Realtime Database, and return result in the response. My problem is, the query result is being passed back in response to DialogFlow only after at least 2 attempts.
Below the screenshots showing the end result in the Simulator
First query screenshot
The first line of the response is returned from the Cloud Function, and contains values passed with the "Context". There is no second line in this response.
below is the screen showing the result after sending exactly the same request second time.
Second query screenshot
First line is the same as previously, but this time I also get the second line which contains the query result data.
It looks like my code is "working" (I get the correct data from the database), but for some reason it only works if I trigger it at least 2 times in quick succession.
Below is the code snipped which handle this request:
function googleAssistantHandler(agent) {
let conv = agent.conv();
let outCommandContext = agent.getContext('outcommand');
let outCharacterContext = agent.getContext('outcharacter');
let character = outCharacterContext.parameters.character;
let command = outCommandContext.parameters.command;
agent.add('<prosody rate="140%" pitch="0.4">' + character +' '+ command +'</prosody>');
var movesRef = admin.database().ref('characters/'+character.toLowerCase()+'/moves/');
movesRef.limitToFirst(1).orderByChild("notation")
.equalTo(command.toString()).on("child_added",function(snapshot){
agent.add(`record number is ` + snapshot.key);
});
}
I've tried using once() instead of on() (as it would make more sense in my case... i don't need to listen to changes on the database, i just want to retrieve data once)- but, I couldn't get it to work.
Can you guys help me out understanding why my query returns result only after the second trigger?
Thanks!
you are using a callback method to get the data from database so there is no guaranty that it will be called before your function is returned. to solve the issue, you need to use a Promise and return that Promise in your function so the last few lines of your function will look like this
return movesRef.limitToFirst(1).orderByChild("notation")
.equalTo(command.toString()).on("child_added").then(snapshot= > {
agent.add(`record number is ` + snapshot.key);
});
You need to always use promises when working with databases. Moreover, the first response that you see might be because of the failed function which timed out. If you see your console logs in firebase, you might see the errors. Also check your default response, if it has the text that User said $name or something similar, then that is what causes the issue in the first attempt.
If you still don't get it to work, try logging the returned data and post your logs here.

Meteorjs collection insert in loop gives wrong results without errors

I have an array of data that I want to fill some collection with. I have this collection as SomeCollection.
If I go through array like this
_.each(dataArray, function(d) {
var retId = SomeCollection.insert(d);
console.warn(retId);
});
Where dataArray has 720 unique items with unique _id's.
On loop execution I get all retIds returned and no errors.
If I write SomeCollection.count() after that, I get 720.
If I reload page after that, SomeCollection.count() gives some number less than 720 (each reload the same number, but number different after 'filling' script re-execution), it can be 320, 521, etc.
I do it on client with an 'admin' user who have whole SomeCollection published and subscribed.
Collection is clean before this loop; I remove all item from it explicitly.
Why is it happens?
Problem was I just reloaded page too fast. After loop ended, insertion process seems to be still active.

How can I delay execution of a part of script in Classic ASP?

I've a classic ASP page which call a en external webservice.
this is how the actual process works:
'[Part0 : Call the external webservice]
wsResponse = setConfirmation(...)
' [PART1: external webservice is responding]
if not wsResponse is Nothing then
'....Process the response from the webservice according to wsResponse
code =wsResponse.getCode
if code = 'C' then
'We confirm the transaction and call a storedprocedure in SqlServer
else
'if code is different from C, we assume, the transaction status is 'not confirmed' or 'cancelled'
'[PART2: no answer from external webservice]
Else
'so wsReponse is nothing..We don't get any answer from the external webservice
'transaction is not confirmed
'the transaction status is set to 'not confirmed' in database
So what I want to do is that in PART2 (when no answer is get from external webservice), wait 30 seconds before send 'not confirmed' status in database . So I will like to do PART0 again ie: Call again the external webservice at least 10 times and see if it responding or not. A kind of recursive process.
So I was thinking of 2 way of doing this:
In PART2, put ASP to sleep for 30s and to PART0 again (Call the webservice) and if still no response, write in DB, transaction not confirmed, but if response, then do PART1.
In PART2, Call repeat PART0 at least 10 times, if after 10 trials, there is no response, then write in DB, transaction is not confirmed.
So my question is: is there a better way to do this or which or 1 or 2 will be better? and also, for 1, how can we put ASP to sleep like in dotnet or PHP?
Thanks for your answers.
Regards
Here is a simple subroutine that will delay for as many seconds as you need
Sub MyDelay(NumberOfSeconds)
Dim DateTimeResume
DateTimeResume= DateAdd("s", NumberOfSeconds, Now())
Do Until (Now() > DateTimeResume)
Loop
End Sub
Just call this subroutine in Part 2
Call MyDelay(30)
I don't know any way to put ASP to sleep. Especially when you are on a hosted web. So what I would do is this: I would have a service like Pingdom.com to request an ASP page every say 10 seconds. The requested ASP page will then attend to any pending transactions (logged in a tabel).

"Throttled" async download in F#

I'm trying to download the 3000+ photos referenced from the xml backup of my blog. The problem I came across is that if just one of those photos is no longer available, the whole async gets blocked because AsyncGetResponse doesn't do timeouts.
ildjarn helped me to put together a version of AsyncGetResponse which does fail on timeout, but using that gives a lot more timeouts - as though requests that are just queued timeout. It seems like all the WebRequests are launched 'immediately', the only way to make it work is to set the timeout to the time required to download all of them combined: which isn't great because it means I have adjust the timeout depending on the number of images.
Have I reached the limits of vanilla async? Should I be looking at reactive extensions instead?
This is a bit embarassing, because I've already asked two questions here on this particular bit of code, and I still haven't got it working the way I want!
I think there must be a better way to find out that a file is not available than using a timeout. I'm not exactly sure, but is there some way to make it throw an exception if a file cannot be found? Then you could just wrap your async code inside try .. with and you should avoid most of the problems.
Anyway, if you want to write your own "concurrency manager" that runs certain number of requests in parallel and queues remaining pending requests, then the easiest option in F# is to use agents (the MailboxProcessor type). The following object encapsulates the behavior:
type ThrottlingAgentMessage =
| Completed
| Work of Async<unit>
/// Represents an agent that runs operations in concurrently. When the number
/// of concurrent operations exceeds 'limit', they are queued and processed later
type ThrottlingAgent(limit) =
let agent = MailboxProcessor.Start(fun agent ->
/// Represents a state when the agent is blocked
let rec waiting () =
// Use 'Scan' to wait for completion of some work
agent.Scan(function
| Completed -> Some(working (limit - 1))
| _ -> None)
/// Represents a state when the agent is working
and working count = async {
while true do
// Receive any message
let! msg = agent.Receive()
match msg with
| Completed ->
// Decrement the counter of work items
return! working (count - 1)
| Work work ->
// Start the work item & continue in blocked/working state
async { try do! work
finally agent.Post(Completed) }
|> Async.Start
if count < limit then return! working (count + 1)
else return! waiting () }
working 0)
/// Queue the specified asynchronous workflow for processing
member x.DoWork(work) = agent.Post(Work work)
Nothing is ever easy. :)
I think the issues you're hitting are intrinsic to the problem domain (as opposed to merely being issues with the async programming model, though they do interact somewhat).
Say you want to download 3000 pictures. First, in your .NET process, there is something like System.Net.ConnectionLimit or something I forget the name of, that will e.g. throttle the number of simultaneous HTTP connections your .NET process can run simultaneously (and the default is just '2' I think). So you could find that control and set it to a higher number, and it would help.
But then next, your machine and internet connection have finite bandwidth. So even if you could try to concurrently start 3000 HTTP connections, each individual connection would get slower based on the bandwidth pipe limitations. So this would also interact with timeouts. (And this doesn't even consider what kinds of throttles/limits are on the server. Maybe if you send 3000 requests it will think you are DoS attacking and blacklist your IP.)
So this is really a problem domain where a good solution requires some intelligent throttling and flow-control in order to manage how the underlying system resources are used.
As in the other answer, F# agents (MailboxProcessors) are a good programming model for authoring such throttling/flow-control logic.
(Even with all that, if most picture files are like 1MB but then there is a 1GB file mixed in there, that single file might trip a timeout.)
Anyway, this is not so much an answer to the question, as just pointing out how much intrinsic complexity there is in the problem domain itself. (Perhaps it's also suggestive of why UI 'download managers' are so popular.)
Here's a variation on Tomas's answer, because I needed an agent which could return results.
type ThrottleMessage<'a> =
| AddJob of (Async<'a>*AsyncReplyChannel<'a>)
| DoneJob of ('a*AsyncReplyChannel<'a>)
| Stop
/// This agent accumulates 'jobs' but limits the number which run concurrently.
type ThrottleAgent<'a>(limit) =
let agent = MailboxProcessor<ThrottleMessage<'a>>.Start(fun inbox ->
let rec loop(jobs, count) = async {
let! msg = inbox.Receive() //get next message
match msg with
| AddJob(job) ->
if count < limit then //if not at limit, we work, else loop
return! work(job::jobs, count)
else
return! loop(job::jobs, count)
| DoneJob(result, reply) ->
reply.Reply(result) //send back result to caller
return! work(jobs, count - 1) //no need to check limit here
| Stop -> return () }
and work(jobs, count) = async {
match jobs with
| [] -> return! loop(jobs, count) //if no jobs left, wait for more
| (job, reply)::jobs -> //run job, post Done when finished
async { let! result = job
inbox.Post(DoneJob(result, reply)) }
|> Async.Start
return! loop(jobs, count + 1) //job started, go back to waiting
}
loop([], 0)
)
member m.AddJob(job) = agent.PostAndAsyncReply(fun rep-> AddJob(job, rep))
member m.Stop() = agent.Post(Stop)
In my particular case, I just need to use it as a 'one shot' 'map', so I added a static function:
static member RunJobs limit jobs =
let agent = ThrottleAgent<'a>(limit)
let res = jobs |> Seq.map (fun job -> agent.AddJob(job))
|> Async.Parallel
|> Async.RunSynchronously
agent.Stop()
res
It seems to work ok...
Here's an out of the box solution:
FSharpx.Control offers an Async.ParallelWithThrottle function. I'm not sure if it is the best implementation as it uses SemaphoreSlim. But the ease of use is great and since my application doesn't need top performance it works well enough for me. Although since it is a library if someone knows how to make it better it is always a nice thing to make libraries top performers out of the box so the rest of us can just use the code that works and just get our work done!

Ada entry and when statement usage

I am a newbie in Ada programming language and I am working on concurrent programming, but I am having a problem with one implementation. This might be very dummy question. The code is:
type status is array(1..6) of boolean; --boolean values for each track
track_available :status:=(others=>true); --true if track is available
protected track_handler is
entry track_req(n:in track_part_type); --n is track number
entry track_rel(n:in track_part_type); --n is track number
end track_handler;
protected body track_handler is
--implement entries
entry track_req(n: in track_part_type) when track_available(n) is --here where the error occurs
begin
req(n);
end track_req;
entry track_rel(n: in track_part_type) when track_available(n) is
begin
rel(n);
end track_rel;
end track_handler;
procedure req(nr : track_part_type) is
begin
--null;
track_available(nr):=false;
end req;
procedure rel(nr : track_part_type) is
begin
--null;
track_available(nr):=true;
end rel;
Here I get a compilation error for "when track_available(n)" statement saying that "n is undefined". I think variable n is out of scope, but I also need to check if the n'th index of the array is true or false. How can I overcome this problem?
Thank you.
You can't actually use an entry's parameters in its own guard. You got that much, I gather.
The way guards work, all of them are evaluated before the wait starts, and only the ones that are active at that time will be available. They don't get periodicly re-evaluated or dynamicaly read or anything.
This means it will be tough to get the logic for your guards right, unless you write your code so that only other entries in your protected object modify the guards. If you want to use some data from outside of your protected object to control its behavior, you will probably need to use some mechanisim other than guards to do it. Like check just inside the entry and exit immediately or something.
There is one possibility for what you are trying to do though: Entry families. You should be able to use an entry family index in a guard.
The spec would change to:
entry track_req(track_part_type);
entry track_rel(track_part_type);
And the body would change to
entry track_req(for n in track_part_type) when track_available(n) is
begin
req(n);
end track_req;
entry track_rel(for n in track_part_type) when track_available(n) is
begin
rel(n);
end track_rel;
end track_handler;
In the code below you are trying to use track_available(n), before it has been fully defined by (n: in track_part_type).
entry track_req(n: in track_part_type) when track_available(n) is
See also http://en.wikibooks.org/wiki/Ada_Programming/Tasking#Protected_types
NWS

Resources