I need to send a POST request to some API which accepts only file as multipart/form-data. But I have the data as []byte. Now what I can do is write this []byte data to a temporary file and then send that file. After some googling I found this code to upload file:
fileDir, _ := os.Getwd()
fileName := "upload-file.txt"
filePath := path.Join(fileDir, fileName)
file, _ := os.Open(filePath)
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, _ := writer.CreateFormFile("file", filepath.Base(file.Name()))
io.Copy(part, file)
writer.Close()
r, _ := http.NewRequest("POST", "http://example.com", body)
r.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
client.Do(r)
After some more googling I learned this. Seems to me for sending files we only need file name and content (maybe size). All these data I can provide without creating a temporary file, writing to that file and then again reading back from that file.
Is it possible to do so? Can I send []bytes as a file somehow? A working example is much appreciated.
Write the []byte directly to the part. Use this code to write the slice content to the part and post the form:
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, _ := writer.CreateFormFile("file", "insert-name-here")
part.Write(content) // <-- content is the []byte
writer.Close()
r, _ := http.NewRequest("POST", "http://example.com", body)
r.Header.Add("Content-Type", writer.FormDataContentType())
err := http.DefaultClient.Do(r)
if err != nil {
// handle error
}
Related
I'm able to get an HTTP GET request to work like so:
resp, err := http.Get("https://services.nvd.nist.gov/rest/json/cves/1.0/?modStartDate=2021-10-29T12%3A00%3A00%3A000%20UTC-00%3A00&modEndDate=2021-10-30T00%3A00%3A00%3A000%20UTC-00%3A00&resultsPerPage=5000")
I wanted to have an easier way to construct the query parameters so I created this:
req, err := http.NewRequest("GET", "https://services.nvd.nist.gov/rest/json/cves/1.0/", nil)
if err != nil {
fmt.Printf("Error: %v\n", err)
os.Exit(1)
}
q := req.URL.Query()
q.Set("modStartDate", "2021-10-29T12:00:00:000 UTC-00:00")
q.Set("modEndDate", "2021-10-30T00:00:000 UTC-00:00")
q.Set("resultsPerPage", "5000")
req.URL.RawQuery = q.Encode()
client := http.Client{}
resp, err := client.Do(req)
The response status is a 404. It's not clear to me what I'm missing. What is the first GET request doing that I'm missing in the second one?
For reference, the API I'm working with:
https://nvd.nist.gov/developers/vulnerabilities
As #JimB noted, comparing your original raw query with your generate query shows the formatting issue:
origURL := "https://services.nvd.nist.gov/rest/json/cves/1.0/?modStartDate=2021-10-29T12%3A00%3A00%3A000%20UTC-00%3A00&modEndDate=2021-10-30T00%3A00%3A00%3A000%20UTC-00%3A00&resultsPerPage=5000"
u, _ := url.Parse(origURL)
q, _ := url.ParseQuery(u.RawQuery)
q2 := url.Values{}
q2.Set("modStartDate", "2021-10-29T12:00:00:000 UTC-00:00")
q2.Set("modEndDate", "2021-10-30T00:00:000 UTC-00:00")
q2.Set("resultsPerPage", "5000")
fmt.Println(q) // map[modEndDate:[2021-10-30T00:00:00:000 UTC-00:00] modStartDate:[2021-10-29T12:00:00:000 UTC-00:00] resultsPerPage:[5000]]
fmt.Println(q2) // map[modEndDate:[2021-10-30T00:00:000 UTC-00:00] modStartDate:[2021-10-29T12:00:00:000 UTC-00:00] resultsPerPage:[5000]]
https://play.golang.org/p/36RNIb7Micu
So add the extra :00 to your time format:
q.Set("modStartDate", "2021-10-29T12:00:00:00:000 UTC-00:00")
q.Set("modEndDate", "2021-10-30T00:00:00:000 UTC-00:00")
I have set up a basic project on Github: https://github.com/kounelios13/range-download.
Essentially this project tries to download a file using HTTP Range requests, assemble it, and save it back to disk. I am trying to follow this article (apart from the goroutines for the time being). When I try to download the file using range requests the final size, after all requests data are combined, is bigger than the original size I would get have and the final file is corrupted.
Here is the code responsible for downloading the file
type Manager struct{
limit int
}
func NewManager(limit int) *Manager{
return &Manager{
limit: limit,
}
}
func (m *Manager) DownloadBody(url string ) ([]byte ,error){
// First we need to determine the filesize
body := make([]byte ,0)
response , err := http.Head(url) // We perform a Head request to get header information
if response.StatusCode != http.StatusOK{
return nil ,fmt.Errorf("received code %d",response.StatusCode)
}
if err != nil{
return nil , err
}
maxConnections := m.limit // Number of maximum concurrent co routines
bodySize , _ := strconv.Atoi(response.Header.Get("Content-Length"))
bufferSize :=(bodySize) / (maxConnections)
diff := bodySize % maxConnections
read := 0
for i:=0;i<maxConnections;i++{
min := bufferSize * i
max := bufferSize * (i+1)
if i==maxConnections-1{
max+=diff // Check to see if we have any leftover data to retrieve for the last request
}
req , _ := http.NewRequest("GET" , url, nil)
req.Header.Add("Range" ,fmt.Sprintf("bytes=%d-%d",min,max))
res , e := http.DefaultClient.Do(req)
if e != nil{
return body , e
}
log.Printf("Index:%d . Range:bytes=%d-%d",i,min,max)
data , e :=ioutil.ReadAll(res.Body)
res.Body.Close()
if e != nil{
return body,e
}
log.Println("Data for request: ",len(data))
read = read + len(data)
body = append(body, data...)
}
log.Println("File size:",bodySize , "Downloaded size:",len(body)," Actual read:",read)
return body, nil
}
Also I noticed that the bigger the limit I set the more the difference between the original file content length and the actual size of all request bodies combined is.
Here is my main.go
func main() {
imgUrl := "https://media.wired.com/photos/5a593a7ff11e325008172bc2/16:9/w_2400,h_1350,c_limit/pulsar-831502910.jpg"
maxConnections := 4
manager := lib.NewManager(maxConnections)
data , e:= manager.DownloadBody(imgUrl)
if e!= nil{
log.Fatalln(e)
}
ioutil.WriteFile("foo.jpg" , data,0777)
}
Note: for the time being I am not interested in making the code concurrent.
Any ideas what I could be missing?
Note: I have confirmed that server returns a 206 partial content using the curl command below:
curl -I https://media.wired.com/photos/5a593a7ff11e325008172bc2/16:9/w_2400,h_1350,c_limit/pulsar-831502910.jpg
Thanks to #mh-cbon I managed to write a simple test that helped me find the solution . Here is the fixed code
for i:=0;i<maxConnections;i++{
min := bufferSize * i
if i != 0{
min++
}
max := bufferSize * (i+1)
if i==maxConnections-1{
max+=diff // Check to see if we have any leftover data to retrieve for the last request
}
req , _ := http.NewRequest("GET" , url, nil)
req.Header.Add("Range" ,fmt.Sprintf("bytes=%d-%d",min,max))
res , e := http.DefaultClient.Do(req)
if e != nil{
return body , e
}
log.Printf("Index:%d . Range:bytes=%d-%d",i,min,max)
data , e :=ioutil.ReadAll(res.Body)
res.Body.Close()
if e != nil{
return body,e
}
log.Println("Data for request: ",len(data))
read = read + len(data)
body = append(body, data...)
}
The problem was that I didn't have a correct min value to begin with . So lets say I have the following ranges to download :
0-100
101 - 200
My code would download bytes from 0-100 and then again from 100-200 instead of 101-200
So I made sure on every iteration (except the first one) to increment the min by 1 so as not to overlap with the previous range
Here is a simple test I managed to fix from the docs provided as comments:
func TestManager_DownloadBody(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
http.ServeContent(writer,request ,"hey" ,time.Now() ,bytes.NewReader([]byte(`hello world!!!!`)))
}))
defer ts.Close()
m := NewManager(4)
data , err := m.DownloadBody(ts.URL)
if err != nil{
t.Errorf("%s",err)
}
if string(data) != "hello world!!!!"{
t.Errorf("Expected hello world!!!! . received : [%s]",data)
}
}
Sure there are more tests to be written but it is a good start
Labstack Echo v4 has recently been updated to include ModifyResponse hook (ala golang's httputil ReverseProxy).
Most of the examples of using this seem to leverage ioutil.ReadAll().
For example:
func UpdateResponse(r *http.Response) error {
b, _ := ioutil.ReadAll(r.Body)
buf := bytes.NewBufferString("Monkey")
buf.Write(b)
r.Body = ioutil.NopCloser(buf)
r.Header["Content-Length"] = []string{fmt.Sprint(buf.Len())}
return nil
}
What I am looking to do is to avoid waiting for the entire response (from ReadAll) and monitor the stream for certain content (i.e. class='blue') then replace it with different text (i.e. class='blue-green')
How can this be done using streams efficiently and with as little allocations as possible?
I'm new to Go and I'm learning how to work with goroutines.
I have a function that downloads images:
func imageDownloader(uri string, filename string) {
fmt.Println("starting download for ", uri)
outFile, err := os.Create(filename)
defer outFile.Close()
if err != nil {
os.Exit(1)
}
client := &http.Client{}
req, err := http.NewRequest("GET", uri, nil)
resp, err := client.Do(req)
defer resp.Body.Close()
if err != nil {
panic(err)
}
header := resp.ContentLength
bar := pb.New(int(header))
rd := bar.NewProxyReader(resp.Body)
// and copy from reader
io.Copy(outFile, rd)
}
When I call by itself as part of another function, it downloads images completely and there is no truncated data.
However, when I try to modify it to make it a goroutine, images are often truncated or zero length files.
func imageDownloader(uri string, filename string, wg *sync.WaitGroup) {
...
io.Copy(outFile, rd)
wg.Done()
}
func main() {
var wg sync.WaitGroup
wg.Add(1)
go imageDownloader(url, file, &wg)
wg.Wait()
}
Am I using WaitGroups incorrectly? What could cause this and how can I fix it?
Update:
Solved it. I had placed the wg.add() function outside of a loop. :(
While I'm not sure exactly what's causing your issue, here's two options for how to get it back into working order.
First, looking to the example of how to use waitgroups from the sync library, try calling defer wg.Done() at the beginning of your function to ensure that even if the goroutine ends unexpectedly, that the waitgroup is properly decremented.
Second, io.Copy returns an error that you're not checking. That's not great practice anyway, but in your particular case it's preventing you from seeing if there is indeed an error in the copying routine. Check it and deal with it appropriately. It also returns the number of bytes written, which might help you as well.
Your example doesn't have anything obviously wrong with its use of WaitGroups. As long as you are calling wg.Add() with the same number as the number of goroutines you launch, or incrementing it by 1 every time you start a new goroutine, that should be correct.
However you call os.Exit and panic for certain errors conditions in the goroutine, so if you have more than one of these running, a failure in any one of them will terminate all of them, regardless of the use of WaitGroups. If it's failing without a panic message, I would take a look at the os.Exit(1) line.
It would also, be good practice in go to use defer wg.Done() at the start of your function, so that even if an error occurs, the goroutine still decrements its counter. That way your main thread won't hang on completion if one of the goroutines returns an error.
One change I would make in your example is leverage defer when you are Done. I think this defer ws.Done() should be the first statement in your function.
I like WaitGroup's simplicity. However, I do not like that we need to pass the reference to the goroutine because that would mean that the concurrency logic would be mixed with your business logic.
So I came up with this generic function to solve this problem for me:
// Parallelize parallelizes the function calls
func Parallelize(functions ...func()) {
var waitGroup sync.WaitGroup
waitGroup.Add(len(functions))
defer waitGroup.Wait()
for _, function := range functions {
go func(copy func()) {
defer waitGroup.Done()
copy()
}(function)
}
}
So your example could be solved this way:
func imageDownloader(uri string, filename string) {
...
io.Copy(outFile, rd)
}
func main() {
functions := []func(){}
list := make([]Object, 5)
for _, object := range list {
function := func(obj Object){
imageDownloader(object.uri, object.filename)
}(object)
functions = append(functions, function)
}
Parallelize(functions...)
fmt.Println("Done")
}
If you would like to use it, you can find it here https://github.com/shomali11/util
How can I get a file inode in Go?
I already can print it like this:
file := "/tmp/system.log"
fileinfo, _ := os.Stat(file)
fmt.Println(fileinfo.Sys())
fmt.Println(fileinfo)
Looking at Go implementation it was obvious looking for some stat method, but I still did not manage to find the structure definition for a Unix system.
How can I get the inode value directly?
Which file/s in the source code define the structure of Sys()?
You can use a type assertion to get the underlying syscall.Stat_t from the fileinfo like this
package main
import (
"fmt"
"os"
"syscall"
)
func main() {
file := "/etc/passwd"
fileinfo, _ := os.Stat(file)
fmt.Printf("fileinfo.Sys() = %#v\n", fileinfo.Sys())
fmt.Printf("fileinfo = %#v\n", fileinfo)
stat, ok := fileinfo.Sys().(*syscall.Stat_t)
if !ok {
fmt.Printf("Not a syscall.Stat_t")
return
}
fmt.Printf("stat = %#v\n", stat)
fmt.Printf("stat.Ino = %#v\n", stat.Ino)
}
You can do the following:
file := "/tmp/system.log"
var stat syscall.Stat_t
if err := syscall.Stat(file, &stat); err != nil {
panic(err)
}
fmt.Println(stat.Ino)
Where stat.Ino is the inode you are looking for.
Package syscall is now deprecated. See https://pkg.go.dev/golang.org/x/sys instead.