I am working on an app where I have a backend of NestJS GraphQL microservices TCP where I send files from the app. I have the type FileUpload and it arrives fine in my graph service but now I have to forward it to the FileService with the code:
return await this.client.send({ cmd: 'newfile' }, {
files: file,
}).toPromise();
file is then the FileUpload that comes in. On the FileService I get the following:
{
promise: {},
file: { mimetype: 'application/octet-stream', encoding: '7bit' }
}
I don't get a file to save it. So i try to get the stream in mu GraphService before sending it to the FileService.
const newFile = file.file;
const readStream = await newFile.createReadStream();
newFile.buffer = await this.stream2buffer(readStream);
async stream2buffer(stream: Stream): Promise<Buffer> {
return new Promise<Buffer>((resolve, reject) => {
const _buf = Array<any>();
stream.on("data", chunk => _buf.push(chunk));
stream.on("end", () => resolve(Buffer.concat(_buf)));
stream.on("error", err => reject(`error converting stream - ${err}`));
});
}
Because of this I send the buffer to the FileService but when you do this and you have multiple files it takes so long. How can I handle this better.
So how can I best create FileUpload with NestJS, GraphQL and microservices TCP?
Related
I'm wondering how I can get data from other servers and API with deno ? Everything in the documentation teach me about making http servers and read files from local source. But I can't find anything useful about reading something on the network.
How can read JSON data from the Stripe API ? Or if I want to read a HTML file with text inside ?
Thank you for your time!
I am just giving you an example of the GET request for fetching repositories of Github.
You can change the URL and Request Configuration as per your need.
In the code given below, I am calling another API of Github. By using the fetch() method you can do that.
fetch() method first takes the URL as the first parameter and the next parameter is RequestInit which takes the request method type, headers, body, etc and at the end returning JSON response of that API call.
const githubResponse = async (): Promise<any> => {
const response = await fetch("https://api.github.com/search/repositories?q=android", {
method: "GET",
headers: {
"Content-Type": "application/json",
},
});
return response.json(); // For JSON Response
// return response.text(); // For HTML or Text Response
}
console.log(await githubResponse());
I have written the above code in a ts file named Testing.ts . So, you can run the above code by the command given below:
deno run --allow-net Testing.ts
Next, I am giving you a sample POST request code:
const githubResponse = async (): Promise<any> => {
const body: URLSearchParams = new URLSearchParams({
q: "AvijitKarmakar",
});
const response = await fetch("https://api.github.com/search/repositories", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: body
});
return response.json();
// return response.text(); // For HTML or Text Response
}
console.log(await githubResponse());
You can see that I have created a body object and passed it in the RequestInit through the body parameter and also changed the request method type to POST.
You'll need to do a HTTP Request, for that in Deno you use fetch, the same Web API the browsers use.
To read JSON response:
const res = await fetch('https://api.stripe.com');
const data = await res.json();
If you want HTML:
const res = await fetch('https://example.com');
const html = await res.text();
// Now you can use some HTML parsing lib
fetch requires the --allow-net flag.
Deno strives to be as close to the existent browser API as possible.
That means, you can use fetch. Example:
// fetch-kitten.ts
fetch("https://placekitten.com/200/300").then(async (d) =>
Deno.writeFile("kitten.jpg", new Uint8Array(await d.arrayBuffer()))
);
CLI:
deno run --allow-net --allow-write fetch-kitten.ts
Reference
am playing around with downloading and serving mp3 files in Meteor.
I am trying to download an MP3 file (https://www.sample-videos.com/audio/mp3/crowd-cheering.mp3) on my MeteorJS Server side (to circumvent CORS issues) and then pass it back to the client to play it in a AUDIO tag.
In Meteor you use the Meteor.call function to call a server method. There is not much to configure, it's just a method call and a callback.
When I run the method I receive this:
content:
"ID3���#K `� )�<H� e0�)������1������J}��e����2L����������fȹ\�CO��ȹ'�����}$A�Lݓ����3D/����fijw��+�LF�$?��`R�l�YA:A��# �0��pq����4�.W"�P���2.Iƭ5��_I�d7d����L� �p0��0A��cA�xc��ٲR�BL8䝠4���T��..etc..", data:null,
headers: {
accept-ranges:"bytes",
connection:"close",
content-length:"443926",
content-type:"audio/mpeg",
date:"Mon, 20 Aug 2018 13:36:11 GMT",
last-modified:"Fri, 17 Jun 2016 18:16:53 GMT",
server:"Apache",
statusCode:200
which is the working Mp3 file (the content-length is exactly the same as the file I write to disk on the MeteorJS Server side, and it is playable).
However, my following code doesn't let me convert the response into a BLOB:
```
MeteorObservable.call( 'episode.download', episode.url.url ).subscribe( ( result: any )=> {
console.log( 'response', result);
let URL = window.URL;
let blob = new Blob([ result.content ], {type: 'audio/mpeg'} );
console.log('blob', blob);
let audioUrl = URL.createObjectURL(blob);
let audioElement:any = document.getElementsByTagName('audio')[0];
audioElement.setAttribute("src", audioUrl);
audioElement.play();
})
When I run the code, the Blob has the wrong size and is not playable
Blob(769806) {size: 769806, type: "audio/mpeg"}
size:769806
type:"audio/mpeg"
__proto__:Blob
Uncaught (in promise) DOMException: Failed to load because no supported source was found.
On the backend I just run a return HTTP.get( url ); in the method which is using import { HTTP } from 'meteor/http'.
I have been trying to use btoa or atob but that doesn't work and as far as I know it is already a base64 encoded file, right?
I am not sure why the Blob constructor creates a larger file then the source returned from the backend. And I am not sure why it is not playing.
Can anyone point me to the right direction?
Finally found a solution that uses request instead of Meteor's HTTP:
First you need to install request and request-promise-native in order to make it easy to return your result to clients.
$ meteor npm install --save request request-promise-native
Now you just return the promise of the request in a Meteor method:
server/request.js
import { Meteor } from 'meteor/meteor'
import request from 'request-promise-native'
Meteor.methods({
getAudio (url) {
return request.get({url, encoding: null})
}
})
Notice the encoding: null flag, which causes the result to be binary. I found this in a comment of an answer related to downloading binary data via node. This causes not to use string but binary representation of the data (I don't know how but maybe it is a fallback that uses Node Buffer).
Now it gets interesting. On your client you wont receive a complex result anymore but either an Error or a Uint8Array which makes sense because Meteor uses EJSON to send data over the wires with DDP and the representation of binary data is a Uint8Array as described in the documentation.
Because you can just pass in a Uint8Array into a Blob you can now easily create the blob like so:
const blob = new Blob([utf8Array], {type: 'audio/mpeg'})
Summarizing all this into a small template if could look like this:
client/fetch.html
<template name="fetch">
<button id="fetchbutton">Fetch Mp3</button>
{{#if source}}
<audio id="player" src={{source}} preload="none" content="audio/mpeg" controls></audio>
{{/if}}
</template>
client/fetch.js
import { Template } from 'meteor/templating'
import { ReactiveVar } from 'meteor/reactive-var'
import './fetch.html'
Template.fetch.onCreated(function helloOnCreated () {
// counter starts at 0
this.source = new ReactiveVar(null)
})
Template.fetch.helpers({
source () {
return Template.instance().source.get()
},
})
Template.fetch.events({
'click #fetchbutton' (event, instance) {
Meteor.call('getAudio', 'https://www.sample-videos.com/audio/mp3/crowd-cheering.mp3', (err, uint8Array) => {
const blob = new Blob([uint8Array], {type: 'audio/mpeg'})
instance.source.set(window.URL.createObjectURL(blob))
})
},
})
Alternative solution is adding a REST endpoint *using Express) to your Meteor backend.
Instead of HTTP we use request and request-progress to send the data chunked in case of large files.
On the frontend I catch the chunks using https://angular.io/guide/http#listening-to-progress-events to show a loader and deal with the response.
I could listen to the download via
this.http.get( 'the URL to a mp3', { responseType: 'arraybuffer'} ).subscribe( ( res:any ) => {
var blob = new Blob( [res], { type: 'audio/mpeg' });
var url= window.URL.createObjectURL(blob);
window.open(url);
} );
The above example doesn't show progress by the way, you need to implement the progress-events as explained in the angular article. Happy to update the example to my final code when finished.
The Express setup on the Meteor Server:
/*
Source:http://www.mhurwi.com/meteor-with-express/
## api.class.ts
*/
import { WebApp } from 'meteor/webapp';
const express = require('express');
const trackRoute = express.Router();
const request = require('request');
const progress = require('request-progress');
export function api() {
const app = express();
app.use(function(req, res, next) {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
next();
});
app.use('/episodes', trackRoute);
trackRoute.get('/:url', (req, res) => {
res.set('content-type', 'audio/mp3');
res.set('accept-ranges', 'bytes');
// The options argument is optional so you can omit it
progress(request(req.params.url ), {
// throttle: 2000, // Throttle the progress event to 2000ms, defaults to 1000ms
// delay: 1000, // Only start to emit after 1000ms delay, defaults to 0ms
// lengthHeader: 'x-transfer-length' // Length header to use, defaults to content-length
})
.on('progress', function (state) {
// The state is an object that looks like this:
// {
// percent: 0.5, // Overall percent (between 0 to 1)
// speed: 554732, // The download speed in bytes/sec
// size: {
// total: 90044871, // The total payload size in bytes
// transferred: 27610959 // The transferred payload size in bytes
// },
// time: {
// elapsed: 36.235, // The total elapsed seconds since the start (3 decimals)
// remaining: 81.403 // The remaining seconds to finish (3 decimals)
// }
// }
console.log('progress', state);
})
.on('error', function (err) {
// Do something with err
})
.on('end', function () {
console.log('DONE');
// Do something after request finishes
})
.pipe(res);
});
WebApp.connectHandlers.use(app);
}
and then add this to your meteor startup:
import { Meteor } from 'meteor/meteor';
import { api } from './imports/lib/api.class';
Meteor.startup( () => {
api();
});
Using Meteor HTTP I'm able to get a response from docusign and convert to a base64 buffer.
try {
const response = HTTP.get(`${baseUrl}/envelopes/${envelopeId}/documents/1`, {
headers: {
"Authorization": `bearer ${token}`,
"Content-Type": "application/json",
},
});
const buffer = new Buffer(response.content).toString('base64');
return buffer
} catch(e) {
console.log(e);
throw new Meteor.Error(e.reason);
}
Then on the client I'm using FileSaver.js to saveAs a blob created from an ArrayBuffer via this function
function _base64ToArrayBuffer(base64) {
const binary_string = window.atob(base64);
const len = binary_string.length;
const bytes = new Uint8Array( len );
for (let i = 0; i < len; i++) {
bytes[i] = binary_string.charCodeAt(i);
}
return bytes.buffer;
}
// template helper
'click [data-action="download"]'(e, tmpl){
const doc = this;
return Meteor.call('downloadPDF', doc, (err, pdf)=>{
if(err) {
return notify({
message: err,
timeout: 3000,
})
}
const pdfBuffer = pdf && _base64ToArrayBuffer(pdf);
console.log(pdfBuffer);
return saveAs(new Blob([pdfBuffer], {type: 'application/pdf'}), `docusign_pdf.pdf`);
});
},
The PDF is downloading with the correct size and page length, but all the pages are blank. Should I be encoding the buffer differently? Is there something else I'm missing?
When uploading documents into DocuSign you can either send the raw document bytes as part of a multipart/form-data request or you can send the document as a base64 encoded file in the document node in your request body.
However once the envelope is complete the DocuSign platform converts the doc into a PDF (if it wasn't one already) and provides that raw file. As such, you shouldn't have to base64 decode the doc so I would try removing that part of your code.
So I have this method in my component
uploadCallback (file) {
// TODO: Integrate dropbox with its SDK
// TODO: Pass the link to the editor
return new Promise(
(resolve, reject) => {
console.log('uploadCallback promise')
console.log('file', file)
const dataObject = {
file,
resolve,
reject
}
console.log('dataObject', dataObject)
Meteor.call('uploadToDropbox', dataObject, function (error, result) {
console.log('uploadToDropbox callback')
if (error) {
console.log('error', error)
}
if (result) {
console.log('result', result)
}
})
}
)
}
In my dataObject I am getting everything as needed. Here is what the console logs
uploadCallback promise
file File {name: "nodejs-2560x1440.png", lastModified: 1485410804857, lastModifiedDate: Thu Jan 26 2017 10:06:44 GMT+0400 (+04), webkitRelativePath: "", size: 1699460…}
dataObject Object {file: File}
uploadToDropbox callback
So everything seems to be ok here.
And here is my server code
import { Meteor } from 'meteor/meteor'
import Dropbox from 'dropbox'
console.log('dropbox settings', Meteor.settings.dropbox)
const dbx = new Dropbox({accessToken: Meteor.settings.dropbox.accessToken})
Meteor.methods({
'uploadToDropbox': function (dataObject) {
console.log('dataObject', dataObject)
const { file } = dataObject
console.log('file', file)
const { resolve, reject } = dataObject
console.log('resolve', resolve)
console.log('reject', reject)
dbx.filesUpload({path: '/' + file.name, contents: file})
.then(function (response) {
console.log(response)
resolve({ data: { link: 'http://dummy_image_src.com' } })
})
.catch(function (error) {
console.error(error)
reject('some error')
})
return false
}
})
The problem is here. dataObject is being passed almost empty
This is what the server logs
I20170217-11:44:36.141(4)? dataObject { file: {} }
I20170217-11:44:36.143(4)? file {}
I20170217-11:44:36.143(4)? resolve undefined
I20170217-11:44:36.144(4)? reject undefined
W20170217-11:44:36.371(4)? (STDERR) [TypeError: first argument must be a string or Buffer]
So why is this happening?
i suspect that File you're trying to pass to the method is a file handle. if true, then that's not going to work: even if the server did get that info, it has no access to your local filesystem to grab those bytes.
your solution is going to take 1 of 2 forms:
client uploads to dropbox
client reads bytes from file system into memory
client uploads bytes to dropbox
client receives some dropbox metadata about the uploaded file (e.g. location)
client calls server with that metadata info
server saves that info to db
server uploads to dropbox
client reads bytes from file system into memory
client formats that data into something that can be handled by JSON
client calls server with that JSON object
server uploads bytes to dropbox
server receives some dropbox metadata about the uploaded file (e.g. location)
server saves that info to db
which to do? it depends on which dropbox package/solution you're using and how you want to structure your app.
You are returning a promises not data, you have to wait for result and then return data.
I'm trying to write a simple http web server, that (among other features), can send the client a requested file.
Sending a regular text file/html file works as a charm. The problem is with sending image files.
Here is a part of my code (after parsing the MIME TYPE, and including fs node.js module):
if (MIMEtype == "image") {
console.log('IMAGE');
fs.readFile(path, "binary", function(err,data) {
console.log("Sending to user: ");
console.log('read the file!');
response.body = data;
response.end();
});
} else {
fs.readFile(path, "utf8", function(err,data) {
response.body = data ;
response.end() ;
});
}
Why all I'm getting is a blank page, upon opening http://localhost:<serverPort>/test.jpg?
Here's a complete example on how to send an image with Node.js in the simplest possible way (my example is a gif file, but it can be used with other file/images types):
var http = require('http'),
fs = require('fs'),
util = require('util'),
file_path = __dirname + '/web.gif';
// the file is in the same folder with our app
// create server on port 4000
http.createServer(function(request, response) {
fs.stat(file_path, function(error, stat) {
var rs;
// We specify the content-type and the content-length headers
// important!
response.writeHead(200, {
'Content-Type' : 'image/gif',
'Content-Length' : stat.size
});
rs = fs.createReadStream(file_path);
// pump the file to the response
util.pump(rs, response, function(err) {
if(err) {
throw err;
}
});
});
}).listen(4000);
console.log('Listening on port 4000.');
UPDATE:
util.pump has been deprecated for a while now and you can just use streams to acomplish this:
fs.createReadStream(filePath).pipe(req);