I know in KafkaJS we can consume message with the following code
const run = async () => {
await kafkaClient.consumer.subscribe({ topic: 'mytopic', fromBeginning: true })
await kafkaClient.consumer.run({
eachBatchAutoResolve: false,
eachBatch: async ({ batch, resolveOffset, heartbeat, isRunning, isStale }) => {
for (let message of batch.messages) {
if (!isRunning() || isStale()) break
processMessage(message)
resolveOffset(message.offset)
}
},
})
}
And the message format is like this
{
magicByte: 2,
attributes: 0,
timestamp: '540669',
offset: '601953',
key: <Buffer 39 63 37 23>,
value: <Buffer 7b 65 32 65 37 38 ... 555 more bytes>,
headers: {
'myheader': <Buffer 61 6f>,
},
‧‧‧‧‧‧
}
But now I'm using KafkaListener in spring boot as consumer.
#org.springframework.kafka.annotation.KafkaListener(
topics = "mykafkatopic",
groupId = "groupId"
)
void listener(WhatTypeShouldThisBe data){
}
}
In JS, the type of the message is object, so I'm confused about what type of object should be used in the java listener function to receive the message?
And the messages produced by KafkaJS are serialized, how to deal with the serialization in java consumer?
Update1:
Should I define a class like this
public class messageFormat{
int magicBytes;
‧‧‧
byte[] key;
selfDefinedHeaderClass header;
}
And then modify the ConsumerFactory function to this
public ConsumerFactory<String, messageFormat> consumerFactory() {
// ...
return new DefaultKafkaConsumerFactory<>(
props,
new StringDeserializer(),
new JsonDeserializer<>(messageFormat.class));
}
You need to define a Java class with the fields from the JSON and use the JsonDeserializer in the consumer configuration, configured to deserialize the JSON and create an instance of that class.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#json-serde
Related
I'm following thist tutorial : https://cloud.google.com/tasks/docs/tutorial-gcf
To create a Task that would call a cloud function.
I've done quite some tries and still get this error:
If I change the body encoding to something else, I get another error about serialisation method.
It's likely not a permission issues, as I got some before and got rid of it.
The object which is pass to the createTask() is the following :
task: {
httpRequest: {
url: "https://europe-west1-project_id.cloudfunctions.net/FunctionName"
httpMethod: "POST"
oidcToken: {
serviceAccountEmail: "cf-targetFunctionSA#project_id.gserviceaccount.com"
}
body: ""
headers: {
Content-Type: "application/json"
}
}
(or with body: base64 encoded json string.)
The code I use is the following :
'use strict';
const common = require('./common');
const {v2beta3} = require('#google-cloud/tasks');
const cloudTasksClient = new v2beta3.CloudTasksClient();
let projectName = common.getProjectName();
let location = "europe-west3";
let queue = "compute-stats-on-mysql";
const parent = cloudTasksClient.queuePath(projectName, location, queue);
async function createTask(url, serviceAccount, data)
{
const dataBuffer = Buffer.from(JSON.stringify(data)).toString('base64');
const task = {
httpRequest: {
httpMethod: 'POST',
url:url,
oidcToken: {
serviceAccountEmail: serviceAccount,
},
headers: {
'Content-Type': 'application/json',
},
body:dataBuffer,
},
};
try
{
// Send create task request.
common.logDebug(`Before creating task`, {parent:parent,task:task, data:data});
const [response] = await cloudTasksClient.createTask({parent, task});
common.logDebug(`Created task ${response.name}`, {parent:parent,task:task, response:response, data:data});
return response;
}
catch (error)
{
// Construct error for Stackdriver Error Reporting
console.error("error while creating tasks",error);
}
}
module.exports = {
createTask : createTask,
cloudTasksClient:cloudTasksClient
};
The lack of details in the error makes me hit a wall blind...
Any suggestions ?
My service account was missing a part...
it was
"cf-"+functionName+"#"+projectName+".gserviceaccount.com";
instead of
"cf-"+functionName+"#"+projectName+".iam.gserviceaccount.com";
I left out the ".iam" during my numerous test to make it work.
For sure there's room for improvement in the error messages.
I had same problem. In your case I think there is not property scheduleTime into task param.
To me, the scheduleTime.seconds was with a wrong value.
I try to modify a string using data in each node I retrieve from Firebase database, and then to write a file with the modifed string (called "content").
Here is what I tried :
// Retrieve initial content from Firebase storage
var data = await FirebaseStorage.instance.ref().child("...").getData(1048576);
var content = new String.fromCharCodes(data);
// Edit content with each node from Firebase database
final response = await FirebaseDatabase.instance.reference().child('...').once();
response.value.forEach((jsonString) async {
...
// cacheManager.getFile returns a Future<File>
cacheManager.getFile(signedurl).then((file){
// Modify content
content=content.replaceAll('test',file.path);
});
});
// Finally write the file with content
print("test");
final localfile = File('index.html');
await localfile.writeAsString(content);
Result :
"test" is shown before the forEach ends.
I found that we can do in Dart (https://groups.google.com/a/dartlang.org/forum/#!topic/misc/GHm2cKUxUDU) :
await Future.forEach
but in my case if I do : await Future.response.value.forEach (sounds a bit weird)
then I get :
Getter not found: 'response'.
await Future.response.value.forEach((jsonString) async {
How to wait that forEach ends (with "content" modified) before to write the file with new content?
Any idea?
If you use for(... in ) instead of forEach you can use async/await
Future someMethod() async {
...
for(final jsonString in response.value) {
...
// cacheManager.getFile returns a Future<File>
cacheManager.getFile(signedurl).then((file){
// Modify content
content=content.replaceAll('test',file.path);
});
});
}
With forEach the calls fire away for each jsonString immediately and inside it await works, but forEach has return type void and that can't be awaited, only a Future can.
You defined the callback for forEach as async, which means that it runs asynchronously. In other words: the code inside of that callback runs independently of the code outside of the callback. That is exactly why print("test"); runs before the code inside of the callback.
The simplest solution is to move all code that needs information from within the callback into the callback. But there might also be a way to await all of the asynchronous callbacks, similar to how you already await the once call above it.
Update I got working what I think you want to do. With this JSON:
{
"data" : {
"key1" : {
"created" : "20181221T072221",
"name" : "I am key1"
},
"key2" : {
"created" : "20181221T072258",
"name" : "I am key 2"
},
"key3" : {
"created" : "20181221T072310",
"name" : "I am key 3"
}
},
"index" : {
"key1" : true,
"key3" : true
}
}
I can read the index, and then join the data with:
final ref = FirebaseDatabase.instance.reference().child("/53886373");
final index = await ref.child("index").once();
List<Future<DataSnapshot>> futures = [];
index.value.entries.forEach((json) async {
print(json);
futures.add(ref.child("data").child(json.key).once());
});
Future.wait(futures).then((List<DataSnapshot> responses) {
responses.forEach((snapshot) {
print(snapshot.value["name"]);
});
});
Have you tried:
File file = await cacheManager.getFile(signedurl);
content = content.replaceAll('test', file.path);
instead of:
cacheManager.getFile(signedurl).then((file){ ...});
EDIT:
Here's a fuller example trying to replicate what you have. I use a for loop instead of the forEach() method:
void main () async {
List<String> str = await getFuture();
print(str);
var foo;
for (var s in str) {
var b = await Future(() => s);
foo = b;
}
print('finish $foo');
}
getFuture() => Future(() => ['1', '2']);
I have created a HTTP Interceptor in Angular 2. The code of the interceptor is below
export class HttpInterceptor extends Http {
private httpSubject = new Subject<Message>();
httpSubject$ = this.httpSubject.asObservable();
private block : boolean = true;
constructor(backend: ConnectionBackend, defaultOptions: RequestOptions, private _router: Router, private dataSharingService : DataSharingService) {
super(backend, defaultOptions);
}
post(url: string, body: any, options?: RequestOptionsArgs): Observable<Response> {
this.dataSharingService.beforeRequest.emit("beforeRequestEvent");
return this.intercept(super.post(url, body, this.getRequestOptionArgs(options)));
//return super.post(url, body, this.getRequestOptionArgs(options));
}
put(url: string, body: any, options?: RequestOptionsArgs): Observable<Response> {
this.dataSharingService.beforeRequest.emit("beforeRequestEvent");
return this.intercept(super.put(url, body, this.getRequestOptionArgs(options)));
}
getRequestOptionArgs(options?: RequestOptionsArgs) : RequestOptionsArgs {
if (options == null) {
options = new RequestOptions();
}
if (options.headers == null) {
options.headers = new Headers();
}
//options.headers.append('Content-Type', 'application/json');
return options;
}
intercept(observable: Observable<Response>): Observable<Response> {
observable.subscribe(
null,
error => this.dataSharingService.afterRequest.emit("afterRequestEvent"),
() => this.dataSharingService.afterRequest.emit("afterRequestEvent")
);
return observable;
}
}
In the intercept function if error callback is registered, browser makes same http requests twice, and when the error callback is removed then event is not fired (which is needed to hide loading indicator).
By error call back I mean this line
error => this.dataSharingService.afterRequest.emit("afterRequestEvent")
in the intercept method.
It seems the subject is registering subscribe() multiple times. Could you try unsubscribing the observable once the response through observable is returned back? Else, you may go for simpler way of creating a new observable during every intercept() call and return it instead to avoid multiple callbacks being registered.
my first question to the community out here!
i'm working on an app which does communicates to the API in the following way
step1: create request options, add request payload --> Post request to API
API responds with a request ID
Step2: update request options, send request ID as payload --> post request to API
final response: response.json
Now the final response can take a bit of time, depending on the data requested.
this can take from anywhere between 4 to 20 seconds on an average.
How do i chain these requests using observables, i've tried using switchmap and failed (as below) but not sure how do i add a interval?
Is polling every 4 second and unsubscribing on response a viable solution? how's this done in the above context?
Edit1:
End goal: i'm new to angular and learning observables, and i'm looking to understand what is the best way forward.. does chaining observable help in this context ? i.e after the initial response have some sort of interval and use flatMap
OR use polling with interval to check if report is ready.
Here's what i have so far
export class reportDataService {
constructor(private _http: Http) { }
headers: Headers;
requestoptions: RequestOptions;
payload: any;
currentMethod: string;
theCommonBits() {
//create the post request options
// headers, username, endpoint
this.requestoptions = new RequestOptions({
method: RequestMethod.Post,
url: url,
headers: newheaders,
body: JSON.stringify(this.payload)
})
return this.requestoptions;
}
// report data service
reportService(payload: any, method: string): Observable<any> {
this.payload = payload;
this.currentMethod = method;
this.theCommonBits();
// fetch data
return this._http.request(new Request(this.requestoptions))
.map(this.extractData)
.catch(this.handleError);
}
private extractData(res: Response) {
let body = res.json();
return body || {};
}
private handleError(error: any) {
let errMsg = (error.message) ? error.message :
error.status ? `${error.status} - ${error.statusText}` : 'Server error';
console.error(errMsg); // log to console instead
return Observable.throw(errMsg);
}
in my component
fetchData() {
this._reportService.reportService(this.payload, this.Method)
.switchMap(reportid => {
return this._reportService.reportService(reportid, this.newMethod)
}).subscribe(
data => {
this.finalData = data;
console.info('observable', this.finalData)
},
error => {
//console.error("Error fetcing data!");
return Observable.throw(error);
}
);
}
What about using Promise in your service instead of Observable, and the .then() method in the component. You can link as much .then() as you want to link actions between them.
I have a component which first need to call a service that POST something. Then in the same component I want to wait until the POST is done, to call another service which GETs data.
How can I make the GET call wait for the POST call to finish?
In new-version.component.ts:
private createNewVersion(value) {
...
// create new version, then call on all available versions
// POST call
this._newVersionService.createNewVersion(vnr);
// GET call
this._versionService.getAvailableVersions();
...
}
In new-version.service.ts:
export class NewVersionService {
response$: Subject<any>;
constructor(private _http: Http) {
this.response$ = new BehaviorSubject<any>(null);
}
public createNewVersion(versionNr) {
this._http.post('http://localhost:8080/services/' + versionNr, null, {
method: 'POST',
})
.subscribe(response => {
this.response$.next(response.status);
},
error => console.error(error));
}
Thanks!
When a call returns a Promise chain the calls with
someFunction() {
return returnsPromise()
.then(result => doSomethingNext())
.then(result => doSomethingAfterThat());
}
Ensure you have a return that returns the Promise of that chain so the caller of someFunc() also has a chance to time additional work to execute after doSomethingAfterThat() is completed.
When a call returns an Observable then use the complete callback
someFunction() {
return returnsObservable()
.subscribe(
event => doForEachEvent(),
error => handleError(),
() => doSomethingNext()
.then(result => doSomethingAfterThat());
}
doSomethingNext() is executed after the last event and doSomethingAfterThat() is again chained with then() to show how to mix observable and promise. doSomething().
You should be able to concat to achieve sequence, and reduce to collect the emitted values:
var a = this._newVersionService.createNewVersion(vnr);
var b = this._versionService.getAvailableVersions();
Rx.Observable.concat(a, b).reduce((acc:Array<any>, x:any) => {
acc.push(x); return acc;
}, []).subscribe(t=> {
var firstEmitted = t[0];
var secondEmitted = t[1];
});
You can do like this:
Change createNewVersion to:
public createNewVersion(versionNr) {
return this._http.post('http://localhost:8080/nod_inspection_plugin/services/' + versionNr, null, {
method: 'POST',
});
}
Then in your call:
this._newVersionService.createNewVersion(vnr).subscribe(response=> {
this._versionService.getAvailableVersions();
}, error => console.error(error));
Another way to do the same is to subscribe in the new-version.component.ts and call you GET request from within the POST request i.e check whether your POST request is done Correctly or not
if yes POST is done Properly then call you GET request. As below:
In new-version.component.ts:
private createNewVersion(value) {
...
// create new version, then call on all available versions
// POST call
this._newVersionService.createNewVersion(vnr)
.subscribe((res) => {
if(res){
console.log(res);
if (---Post request done properly check via status or something else here----{
CALL YOUR GET REQUEST HERE.....
// GET call
this._versionService.getAvailableVersions();
}
else {
DO something else whatever you want....
}
}
});
...
}
In new-version.service.ts:
export class NewVersionService {
response$: Subject<any>;
constructor(private _http: Http) {
this.response$ = new BehaviorSubject<any>(null);
}
public createNewVersion(versionNr) {
this._http.post('http://localhost:8080/nod_inspection_plugin/services/' + versionNr, null, {
method: 'POST',
})
.map(response => {
return [{status: response.status, json: response.json()}];
},
error => console.error(error));
}
for more info related to http request you can read here.
Better use switchMap() here.
const versions$ = this._newVersionService.createNewVersion(vnr)
.switchMap(response => this._versionService.getAvailableVersions());
versions$.subscribe(response2 => this.versions = response2)
But the problem will be if you make another POST request before first has been resolved, the previous request will get cancelled.