Kotlin/Ktor: "closing chunk expected" when requesting a web page - http

I'm trying to request a simple HTTP resource in Kotlin using Ktor's client library (1.4.1):
//
// DomainSpecificObjectFactory.kt
//
object DomainSpecificObjectFactory {
private val client = HttpClient {
UserAgent("some user agent string")
}
suspend fun fromUrl(url: String): DomainSpecificObject = coroutineScope {
val pageHtml = client.get<String>(url)
val document = Jsoup.parse(pageHtml)
val objProps = getDomainSpecificProperties(document)
DomainSpecificObject(objProps)
}
}
//
// SomeOtherFile.kt
//
val obj = DomainSpecificObjectFactory.fromUrl("http://example.com/bla")
However, I get this exception:
org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
Should I be configuring the HTTP client any differently?

Related

Azure Function (v2, .NET Core): Deserialize Event Hub input coming from Stream Analytics Job

I have an Azure Stream Analytics job that outputs events as "Line separated" and encoded in UTF8.
I created an Azure Function (v2, .NET Core) that gets triggered by events on the Event Hub. In this context I process the events in batches:
public static async Task Run(
[EventHubTrigger(
"xxx",
Connection = "xxx",
ConsumerGroup = "xxx")
] EventData[] events,
ILogger log)
My goal is to deserialize the input into a POCO class based object. What I currently always get is an error of "Unexpected character...".
What is the correct way of deserialization in that context?
Finally the following approach worked:
private static List<a> DeserializeInput(EventData[] events, ILogger log)
{
try
{
List<a> inputsDeserializedAsList = new List<a>();
foreach (EventData eventData in events)
{
// Deserialize Input
string payload = Encoding.UTF8.GetString(eventData.Body);
log.LogInformation($"DeserializeInput: payload retrieved: {payload}...");
DateTime enqueuedTimeUtc = (DateTime)eventData.SystemProperties.EnqueuedTimeUtc;
using (StringReader reader = new StringReader(payload))
{
List<a> inputEventsAsList = new List<a>();
string line;
while ((line = reader.ReadLine()) != null)
{
a parsedData = JsonConvert.DeserializeObject<a>(line);
parsedData.EventEnqueuedUtcTime = enqueuedTimeUtc;
inputEventsAsList.Add(parsedData);
}
inputsDeserializedAsList.AddRange(inputEventsAsList);
}
}
return inputsDeserializedAsList;
}
catch (Exception e) {
throw new Exception("DeserializeInput: failed deserializing input...", e);
}
}

$$Error:Counterparty sent session rejection message at unexpected time with message class flow is not registered

I want to make external service call by using oraclizer. So I m trying to run the example file given below:
Example Oraclize
After running, I'm getting an error--
Counterparty sent session rejection message at unexpected time with message class it.oraclize.cordapi.flows.OraclizeQueryFlow is not registered
In the flow, OraclizeQueryAwaitFlow flow is called which in turn calls OraclizeQueryFlow. In this flow there is a sendAndReceive call which happens as follows:
#InitiatingFlow
#StartableByRPC
class OraclizeQueryFlow (val datasource: String, val query: Any, val proofType: Int = 0, val delay: Int = 0) : FlowLogic<String>() {
companion object {
object PROCESSING : ProgressTracker.Step("Submitting the query.")
#JvmStatic
fun tracker() = ProgressTracker(PROCESSING)
#JvmStatic
val console = loggerFor<OraclizeQueryFlow>()
}
override val progressTracker = tracker()
fun console(a: Any) = loggerFor<OraclizeQueryFlow>().info(a.toString())
// start OraclizeQueryFlow datasource: "URL", query: "json(https://min-api.cryptocompare.com/data/price?fsym=USD&tsyms=GBP).GBP", proofType: 16, delay: 0
// start OraclizeQueryFlow datasource: identity, query: hello, proofType: 0, delay: 0
#Suspendable
override fun call(): String {
val oraclize = serviceHub.identityService
.wellKnownPartyFromX500Name(OraclizeUtils.getNodeName()) as Party
progressTracker.currentStep = PROCESSING
val session = initiateFlow(oraclize)
val query = Query(datasource, query, delay, proofType)
val queryId = session.sendAndReceive<String>(query).unwrap { it }
console("Query id: $queryId")
return queryId
}
}
On checking logs I happened to find out that the sendAndReceive call is throwing this error.
This is Mauro from Oraclize, are you using the public testnet of Corda? If not, you should follow this guide to be able to use our service. If you are using it, then you should wait a couple of working days as we are still testing the service on the new testnet.

How can I get the URL and method of RetroFit request on onSubscribe of RxJava 2 custom operator?

So I'm trying to integrate Firebase performance for Http requests and add them manually as they show here (step 9).
I'm using Retrofit 2 and RxJava 2, so I had the idea of doing a custom operator, check code below:
Retrofit 2 Client
#GET("branch-{environment}/v2/branches")
fun getBranch(#Path("environment") environment: String, #Query("location") location: String, #Query("fulfilment_type") fulfilmentType: String): Single<Response<GetBranchResponse>>
RxJava Call to the Retrofit Client
private val client: BranchClient = clientFactory.create(urlProvider.apiUrl)
override fun getBranch(postCode: String, fulfilmentType: FulfilmentType): Single<GetBranchResponse> {
return client
.getBranch(environment, postCode.toUpperCase(), fulfilmentType.toString())
.lift(RxHttpPerformanceSingleOperator(URL?, METHOD?))
.map { it.body() }
.subscribeIO() //custom Kotlin extension
.observeMain() //custom Kotlin extension
...
}
RxJava 2 Custom Operator via lift:
class RxHttpPerformanceSingleOperator<T>(private val url: String, private val method: String) : SingleOperator<Response<T>, Response<T>> {
private lateinit var metric: HttpMetric
#Throws(Exception::class)
override fun apply(observer: SingleObserver<in Response<T>>): SingleObserver<in Response<T>> {
return object : SingleObserver<Response<T>> {
override fun onSubscribe(d: Disposable) {
metric = FirebasePerformance.getInstance().newHttpMetric(url,
method.toUpperCase())
metric.start()
observer.onSubscribe(d)
}
override fun onSuccess(t: Response<T>) {
observer.onSuccess(t)
//More info: https://firebase.google.com/docs/perf-mon/get-started-android
metric.setRequestPayloadSize(t.raw().body().contentLength())
metric.setHttpResponseCode(t.code())
metric.stop()
}
override fun onError(e: Throwable) {
observer.onError(e)
metric.stop()
}
}
}
So currently I'm not sure how it the proper way to get the URL and METHOD of the request (marked as URL? and METHOD? ) to send to the operator,
I need them on onSubscribe to start the metric.. and there I don't have the response with it...
Currently UGLYYYYYYYY my way to do it is:
Add to the Retrofit Client:
#GET("branch-{environment}/v2/branches")
fun getBranchURL(#Path("environment") environment: String, #Query("location") location: String, #Query("fulfilment_type") fulfilmentType: String): Call<JsonObject>
Add add the parameters as:
val request = client.getBranchURL(environment, postCode.toUpperCase(), fulfilmentType.toString()).request()
url = request.url().toString()
method = request.method()
This makes me have 2 entries on the Client for each request... which makes no sense.
Some helpful clues along the way:
- How to get the request url in retrofit 2.0 with rxjava?
Add a Retrofit Interceptor to your HttpClient.Builder with the FirebaseInstance and generate your HttpMetrics there:
class FirebasePerformanceInterceptor(val performanceInstance: FirebasePerformance) : Interceptor {
override fun intercept(chain: Interceptor.Chain): Response {
val request = chain.request()
//Get request values
val url = request.url().url()
val requestPayloadSize = request.body()?.contentLength() ?: 0L
val httpMethod = request.method()
//Initialize http trace
val trace = performanceInstance.newHttpMetric(url, httpMethod)
trace.setRequestPayloadSize(requestPayloadSize)
trace.start()
//Proceed
val response = chain.proceed(chain.request())
//Get response values
val responseCode = response.code()
val responsePayloadSize = response.body()?.contentLength() ?: 0L
//Add response values to trace and close it
trace.setHttpResponseCode(responseCode)
trace.setResponsePayloadSize(responsePayloadSize)
trace.stop()
return response
}
}
You can directly copy and paste this code and it will work.
Enjoy!
What I'm going to suggest doesn't necessarily goes to your approach, it's just a different way of thinking about what you're trying to accomplished.
I would suggest 2 different approaches:
Create your own observer (So create a class that extends Observer) that receives a Retrofit Call object and do your firebase logic in the subscribeActual method.
Use Aspectj to define an annotation that will be processed when the Retrofit call is about to be executed and you can do the firebase logic inside the Aspect. (I'm not sure how Aspectj and kotlin works tho)

How to make simultaneous https requests in Swift 3

I'm having problems to execute a https requests, if the request don't have any error i never get the message, this is a command line tool application and i have a plist to allow http requests, i always see the completion block.
typealias escHandler = ( URLResponse?, Data? ) -> Void
func getRequest(url : URL, _ handler : #escaping escHandler){
let session = URLSession.shared
var request = URLRequest(url:url)
request.cachePolicy = .reloadIgnoringLocalCacheData
request.httpMethod = "GET"
let task = session.dataTask(with: url ){ (data,response,error) in
handler(response,data)
}
task.resume()
}
func startOp(action : #escaping () -> Void) -> BlockOperation{
let exOp = BlockOperation(block: action)
exOp.completionBlock = {
print("Finished")
}
return exOp
}
for sUrl in textFile.components(separatedBy: "\n"){
let url = URL(string: sUrl)!
let queu = startOp {
getRequest(url: url){ response, data in
print("REACHED")
}
}
operationQueue.addOperation(queu)
operationQueue.waitUntilAllOperationsAreFinished()
One problem is that your operation is merely starting the request, but because the request is performed asynchronously, the operation is immediately completing, not actually waiting for the request to finish. You don't want to complete the operation until the asynchronous request is done.
If you want to do this with operation queues, the trick is that you must subclass Operation and perform the necessary KVO for isExecuting and isFinished. You then change isExecuting when you start the request and isFinished when you finish the request, with the associated KVO for both. This is all outlined in the Concurrency Programming Guide: Defining a Custom Operation Object, notably in the Configuring Operations for Concurrent Execution section. (Note, this guide is a little outdated (it refers to the isConcurrent property, which has been replaced is isAsynchronous; it's focusing on Objective-C; etc.), but it introduces you to the issues.
Anyway, This is an abstract class that I use to encapsulate all of this asynchronous operation silliness:
/// Asynchronous Operation base class
///
/// This class performs all of the necessary KVN of `isFinished` and
/// `isExecuting` for a concurrent `NSOperation` subclass. So, to developer
/// a concurrent NSOperation subclass, you instead subclass this class which:
///
/// - must override `main()` with the tasks that initiate the asynchronous task;
///
/// - must call `completeOperation()` function when the asynchronous task is done;
///
/// - optionally, periodically check `self.cancelled` status, performing any clean-up
/// necessary and then ensuring that `completeOperation()` is called; or
/// override `cancel` method, calling `super.cancel()` and then cleaning-up
/// and ensuring `completeOperation()` is called.
public class AsynchronousOperation : Operation {
override public var isAsynchronous: Bool { return true }
private let lock = NSLock()
private var _executing: Bool = false
override private(set) public var isExecuting: Bool {
get {
return lock.synchronize { _executing }
}
set {
willChangeValue(forKey: "isExecuting")
lock.synchronize { _executing = newValue }
didChangeValue(forKey: "isExecuting")
}
}
private var _finished: Bool = false
override private(set) public var isFinished: Bool {
get {
return lock.synchronize { _finished }
}
set {
willChangeValue(forKey: "isFinished")
lock.synchronize { _finished = newValue }
didChangeValue(forKey: "isFinished")
}
}
/// Complete the operation
///
/// This will result in the appropriate KVN of isFinished and isExecuting
public func completeOperation() {
if isExecuting {
isExecuting = false
isFinished = true
}
}
override public func start() {
if isCancelled {
isFinished = true
return
}
isExecuting = true
main()
}
}
And I use this Apple extension to NSLocking to make sure I synchronize the state changes in the above (theirs was an extension called withCriticalSection on NSLock, but this is a slightly more generalized rendition, working on anything that conforms to NSLocking and handles closures that throw errors):
extension NSLocking {
/// Perform closure within lock.
///
/// An extension to `NSLocking` to simplify executing critical code.
///
/// - parameter block: The closure to be performed.
func synchronize<T>(block: () throws -> T) rethrows -> T {
lock()
defer { unlock() }
return try block()
}
}
Then, I can create a NetworkOperation which uses that:
class NetworkOperation: AsynchronousOperation {
var task: URLSessionTask!
init(session: URLSession, url: URL, requestCompletionHandler: #escaping (Data?, URLResponse?, Error?) -> ()) {
super.init()
task = session.dataTask(with: url) { data, response, error in
requestCompletionHandler(data, response, error)
self.completeOperation()
}
}
override func main() {
task.resume()
}
override func cancel() {
task.cancel()
super.cancel()
}
}
Anyway, having done that, I can now create operations for network requests, e.g.:
let queue = OperationQueue()
queue.name = "com.domain.app.network"
let url = URL(string: "http://...")!
let operation = NetworkOperation(session: .shared, url: url) { data, response, error in
guard let data = data, error == nil else {
print("\(error)")
return
}
let string = String(data: data, encoding: .utf8)
print("\(string)")
// do something with `data` here
}
let operation2 = BlockOperation {
print("done")
}
operation2.addDependency(operation)
queue.addOperations([operation, operation2], waitUntilFinished: false) // if you're using command line app, you'd might use `true` for `waitUntilFinished`, but with standard Cocoa apps, you generally would not
Note, in the above example, I added a second operation that just printed something, making it dependent on the first operation, to illustrate that the first operation isn't completed until the network request is done.
Obviously, you would generally never use the waitUntilAllOperationsAreFinished of your original example, nor the waitUntilFinished option of addOperations in my example. But because you're dealing with a command line app that you don't want to exit until these requests are done, this pattern is acceptable. (I only mention this for the sake of future readers who are surprised by the free-wheeling use of waitUntilFinished, which is generally inadvisable.)

How to send List of objects to google execution api and handle it in google apps script

In general I want to export data from asp.net mvc application to Google Sheets for example list of people. I've already set up connection and authenticated app with my Google account (trough OAuth2) but now I'm trying to send my list of objects to api and then handle it in script (by putting all data in new file) and couldn't get my head around this.
Here is some sample code in my app that sends the request.
public async Task<ActionResult> SendTestData()
{
var result = new AuthorizationCodeMvcApp(this, new AppFlowMetadata()).
AuthorizeAsync(CancellationToken.None).Result;
if (result.Credential != null)
{
string scriptId = "MY_SCRIPT_ID";
var service = new ScriptService(new BaseClientService.Initializer
{
HttpClientInitializer = result.Credential,
ApplicationName = "Test"
});
IList<object> parameters = new List<object>();
var people= new List<Person>(); // next i'm selecting data from db.Person to this variable
parameters.Add(people);
ExecutionRequest req = new ExecutionRequest();
req.Function = "testFunction";
req.Parameters = parameters;
ScriptsResource.RunRequest runReq = service.Scripts.Run(req, scriptId);
try
{
Operation op = runReq.Execute();
if (op.Error != null)
{
// The API executed, but the script returned an error.
// Extract the first (and only) set of error details
// as a IDictionary. The values of this dictionary are
// the script's 'errorMessage' and 'errorType', and an
// array of stack trace elements. Casting the array as
// a JSON JArray allows the trace elements to be accessed
// directly.
IDictionary<string, object> error = op.Error.Details[0];
if (error["scriptStackTraceElements"] != null)
{
// There may not be a stacktrace if the script didn't
// start executing.
Newtonsoft.Json.Linq.JArray st =
(Newtonsoft.Json.Linq.JArray)error["scriptStackTraceElements"];
}
}
else
{
// The result provided by the API needs to be cast into
// the correct type, based upon what types the Apps
// Script function returns. Here, the function returns
// an Apps Script Object with String keys and values.
// It is most convenient to cast the return value as a JSON
// JObject (folderSet).
Newtonsoft.Json.Linq.JObject folderSet =
(Newtonsoft.Json.Linq.JObject)op.Response["result"];
}
}
catch (Google.GoogleApiException e)
{
// The API encountered a problem before the script
// started executing.
AddAlert(Severity.error, e.Message);
}
return RedirectToAction("Index", "Controller");
}
else
{
return new RedirectResult(result.RedirectUri);
}
}
The next is how to handle this data in scripts - are they serialized to JSON there?
The execution API calls are essentially REST calls so the payload should be serialized as per that. Stringified JSON is typically fine. Your GAS function should then parse that payload to consume the encoded lists
var data = JSON.parse(payload);

Resources