Does anyone know how the minio etag is generated when you PUT an object?
Is it a hash of the file and can we use it to prevent uploading the same file twice?
Many thanks!
etag is just a md5 hex string. You can simple test it like this:
MinioClient client = new MinioClient("your endpoint","your accesskey","your secretkey");
ObjectStat objectStat = client.statObject("test", "XW02.jpg");
System.out.println(objectStat);
InputStream inputStream = client.getObject("test", "XW02.jpg");
String md5 = DigestUtils.md5DigestAsHex(inputStream);
System.out.println(md5);
Related
I have upload attachment endpoint that do the following:
convert the file to bytes
convert the bytes to stringBase64
encrypting the string (using AES with Key)
saving the Encrypted string as txt in the System file
app.UseStaticFiles(new StaticFileOptions
{
FileProvider = new PhysicalFileProvider(Path.Combine(env.ContentRootPath,"Attachments")),
RequestPath = "/Attachments"
});
the above code is how I'm serving the files.
What I want is ?
Is there any way to run any code (code for decrypting the txt file with encrypted string) on the Attachment before serving it to be ready for immediate use.
I'm using netcoreapp3.1
Problem
I have a controller with the signature
#RequestMapping(path = "/upload/{imageTitle}", method = POST, produces = {APPLICATION_JSON_VALUE})
public ResponseEntity<EnrollFaceResult> uploadImage(#PathVariable String imageTitle, ServletInputStream imageDataStream) {
// handle upload
}
Now I want to test the logic using RestAssured:
given()
.pathParam("imageTitle", "title")
.multiPart(new File(getClass().getResource("image.jpg").toURI()))
.when()
.post("upload/{imageTitle}")
.then()
.statusCode(HttpServletResponse.SC_OK)
The problem is that there is no data in the imageDataStream.
The stream contains the data when I do a PostMan request however.
The difference is that PostMan makes a chunked data upload and RestAssured creates a multipart counterpart.
Question
Is there a way to make a chunked upload request with RestAssured instead of a multipart one?
I will answer my own question.
The solution is in RequestSpecification.body(File). I must have overlooked the method before.
The test snippet should be corrected by replacing multipart(File) with body(File):
given()
.pathParam("imageTitle", "title")
.body(getClass().getResourceAsStream("image.jpg"))
.when()
.post("upload/{imageTitle}")
.then()
.statusCode(HttpServletResponse.SC_OK)
I m using Soap UI free version for some rest mocking.
I need to persist my HTTP POST request (request received already compressed gzip) to a gzip file.
I have tried different ways to do that, however after to execute the code, when I try to decompress manually the file I have the following error: "The archive is either in unknown format or damaged".
The HTTP POST request has the following header:
Host : 127.0.0.1:8091
Content-Length : 636
User-Agent : Java/1.7.0_07
Connection : keep-alive
Content-Type : application/octet-stream
Accept : text/plain, application/json, application/*+json, */*
Pragma : no-cache
Cache-Control : no-cache
Below the solutions that I have tried:
Solution#1:
byte[] buffer = new byte[1024];
byte[] data = mockRequest.getRequestContent().getBytes();
def path="myfile.gz";
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(path));
bos.write(data);
bos.flush();
bos.close();
Solution#2
byte[] buffer = new byte[1024];
byte[] data = mockRequest.getRawRequestData();
def path="myfile.gz";
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(path));
bos.write(data);
bos.flush();
bos.close();
Can someone please help and let me know why I cannot decompress the gzip file and how I can do that?
Thanks,
This is Groovy, so you don't need all this Java clutter.
Here's some code that might work:
new File( path) << mockRequest.rawRequestData
EDIT
Ok, based on your comments, for zip files to be copied correctly, you probably need something a little different:
import java.nio.file.*
Files.copy(new ByteArrayInputStream(mockRequest.requestContent.bytes),
Paths.get( 'destination.zip' ) )
Tested this with an actual zip file's byte[] as source and it worked. If it does not work for you, then the byte array you're getting from requestContent.bytes just isn't a zip file.
An image gets corrupted while being retrieved (through HTTP) and then sent (through HTTP) to a database. Image's raw data is handled in String form.
The service sends a GET for an image file, receives response with the raw image data (response's body) and the Content-Type. Then, a PUT request is sent with the aforementioned request's body and Content-Type header. (The PUT request is constructed by providing the body in String) This PUT request is sent to a RESTful database (CouchDB), creating an attachment (for those unfamiliar with CouchDB an attachment acts like a static file).
Now I have the original image, which my service GETs and PUTs to a database, and this 'copy' of the original image, that I can now GET from the database. If I then `curl --head -v "[copy's url]" it has the Content-Type of the original image, but Content-Length has changed, went from 200kb to about 400kb. If I GET the 'copy' image with a browser, it is not rendered, whereas, the original renders fine. It is corrupted.
What might be the cause? My guess is that while handling the raw data as a string, my framework guesses the encoding wrong and corrupts it. I have not been able to confirm or deny this. How could I handle this raw data/request body in a safe manner, or how could I properly handle the encoding (if that proves to be the problem)?
Details: Play2 Framework's HTTP client, Scala. Below a test to reproduce:
"able to copy an image" in {
def waitFor[T](future:Future[T]):T = { // to bypass futures
Await.result(future, Duration(10000, "millis"))
}
val originalImageUrl = "http://laughingsquid.com/wp-content/uploads/grumpy-cat.jpg"
val couchdbUrl = "http://admin:admin#localhost:5984/testdb"
val getOriginal:ws.Response = waitFor(WS.url(originalImageUrl).get)
getOriginal.status mustEqual 200
val rawImage:String = getOriginal.body
val originalContentType = getOriginal.header("Content-Type").get
// need an empty doc to have something to attach the attachment to
val emptyDocUrl = couchdbUrl + "/empty_doc"
val putEmptyDoc:ws.Response = waitFor(WS.url(emptyDocUrl).put("{}"))
putEmptyDoc.status mustEqual 201
//uploading an attachment will require the doc's revision
val emptyDocRev = (putEmptyDoc.json \ "rev").as[String]
// create actual attachment/static file
val attachmentUrl = emptyDocUrl + "/0"
val putAttachment:ws.Response = waitFor(WS.url(attachmentUrl)
.withHeaders(("If-Match", emptyDocRev), ("Content-Type", originalContentType))
.put(rawImage))
putAttachment.status mustEqual 201
// retrieve attachment
val getAttachment:ws.Response = waitFor(WS.url(attachmentUrl).get)
getAttachment.status mustEqual 200
val attachmentContentType = getAttachment.header("Content-Type").get
originalContentType mustEqual attachmentContentType
val originalAndCopyMatch = getOriginal.body == getAttachment.body
originalAndCopyMatch aka "original matches copy" must beTrue // << false
}
Fails at the last 'must':
[error] x able to copy an image
[error] original matches copy is false (ApplicationSpec.scala:112)
The conversion to String is definitely going to cause problems. You need to work with the bytes as Daniel mentioned.
Looking at the source it looks like ws.Response is just a wrapper. If you get to the underlying class then there are some methods that may help you. On the Java side, someone made a commit on GitHub to expose more ways of getting the response data other than a String.
I'm not familiar with scala but something like this may work:
getOriginal.getAHCResponse.getResponseBodyAsBytes
// instead of getOriginal.body
WS.scala
https://github.com/playframework/playframework/blob/master/framework/src/play/src/main/scala/play/api/libs/ws/WS.scala
WS.java
Here you can see that Response has some new methods, getBodyAsStream() and asByteArray.
https://github.com/playframework/playframework/blob/master/framework/src/play-java/src/main/java/play/libs/WS.java
I'm writing a flex application that polls an xml file on the server to check for updated data every few seconds, and I'm having trouble preventing it from caching the data and failing to respond to it being updated.
I've attempted to set headers using the IIS control panel to use the following, without any luck:
CacheControl: no-cache
Pragma: no-cache
I've also attempted adding a random HTTP GET parameter to the end of the request URL, but that seems like it's stripped off by the HttpService class before the request is made. Here's the code to implement it:
http.url = "test.xml?time=" + new Date().getMilliseconds();
And here's the debug log that makes me think it failed:
(mx.messaging.messages::HTTPRequestMessage)#0
body = (Object)#1
clientId = (null)
contentType = "application/x-www-form-urlencoded"
destination = "DefaultHTTP"
headers = (Object)#2
httpHeaders = (Object)#3
messageId = "AAB04A17-8CB3-4175-7976-36C347B558BE"
method = "GET"
recordHeaders = false
timestamp = 0
timeToLive = 0
url = "test.xml"
Has anyone dealt with this problem?
The cache control HTTP header is "Cache-Control" ... note the hyphen! It should do the trick. If you leave out the hyphen, it is not likely to work.
I used the getTime() to make the date into a numeric string that did the trick. I also changed GET to POST. There were some issues with different file extensions being cached differently. For instance, a standard dynamic extension like .php or .jsp might not be cached by the browser and
private var myDate:Date = new Date();
[Bindable]
private var fileURLString:String = "http://www.mysite.com/data.txt?" + myDate.getTime();
Hopefully this helps someone.
I also threw a ton of the header parameters at it but they never fully did the trick. Examples:
// HTTPService called service
service.headers["Pragma"] = "no-cache"; // no caching of the file
service.headers["Cache-Control"] = "no-cache";