Using the Twitch API and vert.x - I'm looking to continuously send requests to Twitch's API using a WebClient and Twitch's cursor response to go page by page. However I'm not sure how to go back and keep doing queries until a condition is met due to vert.x's asynchronous nature.
Here's my code so far
public void getEntireStreamList(Handler<AsyncResult<JsonObject>> handler) {
JsonObject data = new JsonObject();
getLiveChannels(100, result -> {
if(result.succeeded()) {
JsonObject json = result.result();
String cursor = json.getJsonObject("pagination").getString("cursor");
data.put("data", json.getJsonArray("data"));
if(json.getJsonArray("data").size() < 100) { // IF NOT LAST PAGE
// GO BACK AND DO AGAIN WITH CURSOR IN REQUEST
}
handler.handle(Future.succeededFuture(data));
} else
handler.handle(Future.failedFuture(result.cause()));
});
}
Ideally I'd be able to call getLiveChannels with the cursor String from the previous request to continue the search.
You will need to use Future composition.
Here's my code for your problem:
public void getEntireStreamList(Handler<AsyncResult<JsonObject>> handler) {
JsonArray data = new JsonArray();
// create initial Future for first function call
Future<JsonObject> initFuture = Future.future();
// complete Future when getLiveChannels completes
// fail on exception
getLiveChannels(100, initFuture.completer());
// Create a callback that returns a Future
// for composition.
final AtomicReference<Function<JsonObject, Future<JsonObject>>> callback = new AtomicReference<>();
// Create Function that calls composition with itself.
// This is similar to recursion.
Function<JsonObject, Future<JsonObject>> cb = new Function<JsonObject, Future<JsonObject>>() {
#Override
public Future<JsonObject> apply(JsonObject json) {
// new Future to return
Future<JsonObject> f = Future.future();
// Do what you wanna do with the data
String cursor = json.getJsonObject("pagination").getString("cursor");
data.addAll(json.getJsonArray("data"));
// IF NOT LAST PAGE
if(json.getJsonArray("data").size() == 100) {
// get more live channels with cursor
getLiveChannels(100, cursor, f.completer());
// return composed Future
return f.compose(this);
}
// Otherwise return completed Future with results.
f.complete(new JsonObject().put("data", data));
return f;
}
};
Future<JsonObject> composite = initFuture.compose(cb);
// Set handler on composite Future (ALL composed futures together)
composite.setHandler(result -> handler.handle(result));
}
The code + comments should speak for themselves if you read the Vert.x docs on sequential Future composition.
Related
I have executeGetCapability method which is executed in different threads but these threads runs sequentially..meaning one is completed after the other
#Async("threadPoolCapabilitiesExecutor")
public CompletableFuture<CapabilityDTO> executeGetCapability(final String id, final LoggingContextData data){...}
and this method is called in following way:
public CapabilityResponseDTO getCapabilities(final List<String> ids) {
final CapabilityResponseDTO responseDTO = new CapabilityResponseDTO();
final List<CapabilityDTO> listOfCapabilityDTOS = new ArrayList<>();
try {
for (String id: ids) {
listOfCapabilityDTOS .add(
asyncProcessService.executeGetCapability(id, LoggingContext.getLoggingContextData()).get());
}
} catch (Exception e) {
....
}
responseDTO.setDTOS(listOfCapabilityDTOS );
return responseDTO;
}
How can i call executeGetCapability method using CompletableFuture so that thread runs in parallel without waiting for each other and then the result is combined ?? how can i use here CompletableFuture.supplyAsync and or .allOf methods ? Can someone explain me
Thanks
The reduce helper function from this answer converts a CompletableFuture<Stream<T>> to a Stream<CompletableFuture<T>>. You can use it to asynchronously combine the results for multiple calls to executeGetCapability:
// For each id create a future to asynchronously execute executeGetCapability
Stream<CompletableFuture<CapabilityDTO>> futures = ids.stream()
.map(id -> executeGetCapability(id, LoggingContext.getLoggingContextData()));
// Reduce the stream of futures to a future for a stream
// and convert the stream to list
CompletableFuture<List<CapabilityDTO>> capabilitiesFuture = reduce(futures)
.thenApply(Stream::toList);
responseDTO.setDTOS(capabilitiesFuture.get());
I subscribe to a stream with the listen method so I can process the data as it comes in. If I found what I am looking for, I want to cancel the stream and return the data from the stream subscription block.
I have a working implementation but I am using a Completer to block the function from returning immediate and it feels messy, so I want to know if there's a better way to achieve this. Here is my code snippet:
Future<String> _extractInfo(String url) async {
var data = <int>[];
var client = http.Client();
var request = http.Request('GET', Uri.parse(url));
var response = await client.send(request);
var process = Completer();
var interestingData = '';
StreamSubscription watch;
watch = response.stream.listen((value) async {
data.clear(); //clear the previous stale data from the list so searching can be faster
data.addAll(value);
if (
hasInterestingData(data) //search the bytelist for the data I'm looking for
){
interestingData = extractInterestingData(data) //extract the data if it's been found. This is what I want my function to return;
await watch.cancel(); //cancel the stream subscription if the data has been found. ie: Cancel the download
process.complete('done'); //complete the future, so the function can return interestingData.
}
});
watch.onDone(() {
//complete the future if the data was not found...so the function will return an empty string
if (!process.isCompleted) {
process.complete('done');
}
});
await process.future; //blocks the sync ops below so the function doesn't return immediately
client.close();
return interestingData;
}
You should use await for for things like this when you are in an async function.
Future<String> _extractInfo(String url) async {
var client = http.Client();
var request = http.Request('GET', Uri.parse(url));
var response = await client.send(request);
await for (var data in response.stream) {
if (hasInterestingData(data)) {
return extractInterestingData(data); // Exiting loop cancels the subscription.
}
}
return "done";
}
However, while this approach is equivalent to your code, it's probably equally flawed.
Unless you are looking for a single byte, you risk the thing you are looking for being split between consecutive data events. You like do need to retain some part of the previous data array, or some state summarizing what you have already seen, but how much depends on what you are actually looking for.
This is actually a 2-part question related directly to .net core 3.0 and specifically with PipeWriter: 1) How should I read in the HttpResponse body? 2) How can I update the HttpResponse? I'm asking both questions because I feel like the solution will likely involve the same understanding and code.
Below is how I got this working in .net core 2.2 - note that this is using streams instead of PipeWriter and other "ugly" things associated with streams - eg. MemoryStream, Seek, StreamReader, etc.
public class MyMiddleware
{
private RequestDelegate Next { get; }
public MyMiddleware(RequestDelegate next) => Next = next;
public async Task Invoke(HttpContext context)
{
var httpResponse = context.Response;
var originalBody = httpResponse.Body;
var newBody = new MemoryStream();
httpResponse.Body = newBody;
try
{
await Next(context);
}
catch (Exception)
{
// In this scenario, I would log out the actual error and am returning this "nice" error
httpResponse.StatusCode = StatusCodes.Status500InternalServerError;
httpResponse.ContentType = "application/json"; // I'm setting this because I might have a serialized object instead of a plain string
httpResponse.Body = originalBody;
await httpResponse.WriteAsync("We're sorry, but something went wrong with your request.");
return;
}
// If everything worked
newBody.Seek(0, SeekOrigin.Begin);
var response = new StreamReader(newBody).ReadToEnd(); // This is the only way to read the existing response body
httpResponse.Body = originalBody;
await context.Response.WriteAsync(response);
}
}
How would this work using PipeWriter? Eg. it seems that working with pipes instead of the underlying stream is preferable, but I can not yet find any examples on how to use this to replace my above code?
Is there a scenario where I need to wait for the stream/pipe to finish writing before I can read it back out and/or replace it with a new string? I've never personally done this, but looking at examples of PipeReader seems to indicate to read things in chunks and check for IsComplete.
To Update HttpRepsonse is
private async Task WriteDataToResponseBodyAsync(PipeWriter writer, string jsonValue)
{
// use an oversized size guess
Memory<byte> workspace = writer.GetMemory();
// write the data to the workspace
int bytes = Encoding.ASCII.GetBytes(
jsonValue, workspace.Span);
// tell the pipe how much of the workspace
// we actually want to commit
writer.Advance(bytes);
// this is **not** the same as Stream.Flush!
await writer.FlushAsync();
}
In general I want to export data from asp.net mvc application to Google Sheets for example list of people. I've already set up connection and authenticated app with my Google account (trough OAuth2) but now I'm trying to send my list of objects to api and then handle it in script (by putting all data in new file) and couldn't get my head around this.
Here is some sample code in my app that sends the request.
public async Task<ActionResult> SendTestData()
{
var result = new AuthorizationCodeMvcApp(this, new AppFlowMetadata()).
AuthorizeAsync(CancellationToken.None).Result;
if (result.Credential != null)
{
string scriptId = "MY_SCRIPT_ID";
var service = new ScriptService(new BaseClientService.Initializer
{
HttpClientInitializer = result.Credential,
ApplicationName = "Test"
});
IList<object> parameters = new List<object>();
var people= new List<Person>(); // next i'm selecting data from db.Person to this variable
parameters.Add(people);
ExecutionRequest req = new ExecutionRequest();
req.Function = "testFunction";
req.Parameters = parameters;
ScriptsResource.RunRequest runReq = service.Scripts.Run(req, scriptId);
try
{
Operation op = runReq.Execute();
if (op.Error != null)
{
// The API executed, but the script returned an error.
// Extract the first (and only) set of error details
// as a IDictionary. The values of this dictionary are
// the script's 'errorMessage' and 'errorType', and an
// array of stack trace elements. Casting the array as
// a JSON JArray allows the trace elements to be accessed
// directly.
IDictionary<string, object> error = op.Error.Details[0];
if (error["scriptStackTraceElements"] != null)
{
// There may not be a stacktrace if the script didn't
// start executing.
Newtonsoft.Json.Linq.JArray st =
(Newtonsoft.Json.Linq.JArray)error["scriptStackTraceElements"];
}
}
else
{
// The result provided by the API needs to be cast into
// the correct type, based upon what types the Apps
// Script function returns. Here, the function returns
// an Apps Script Object with String keys and values.
// It is most convenient to cast the return value as a JSON
// JObject (folderSet).
Newtonsoft.Json.Linq.JObject folderSet =
(Newtonsoft.Json.Linq.JObject)op.Response["result"];
}
}
catch (Google.GoogleApiException e)
{
// The API encountered a problem before the script
// started executing.
AddAlert(Severity.error, e.Message);
}
return RedirectToAction("Index", "Controller");
}
else
{
return new RedirectResult(result.RedirectUri);
}
}
The next is how to handle this data in scripts - are they serialized to JSON there?
The execution API calls are essentially REST calls so the payload should be serialized as per that. Stringified JSON is typically fine. Your GAS function should then parse that payload to consume the encoded lists
var data = JSON.parse(payload);
Is there a way, when using ASP.NET Web API, to return the response headers earlier?
An example:
Let's say I have an action in my controller which returns all companies for a filter
// GET api/companies/filter
public Companies Get(string someFilter)
{
// some long operation (10 seconds)
}
I would like to return the headers ASAP and while doing that, the long operation should take place, and then return the data of the long operation.
Is something like this possible?
You need to use PushStreamContent to do this
// GET api/companies/filter
public HttpResponseMessage Get(string someFilter)
{
// some long operation (10 seconds)
var pushContent = new PushStreamContent( (stream, content, ctx) =>
{
// Do long running thing here, writing to stream
});
return new HttpResponseMessage() {
Content = pushContent
}
}