Multiple threads Go for HTTP get - http

I'm working with Go in an API for the bus frequency of my city, but i'm a little bit stuck on the threads when i try to make HTTP Get to many urls.
Without concurrency, the programs takes over 16 minutes to complete the 1500 url calls to take the HTTP status code, and i was trying to use the concurrency, but after reading many posts i don't understand how goroutines work...
The idea is to make ONE function and change the number of requests, like here:
go getBusPostStatus(600, 800)
But i'm completely stucked on that...
Here is the code:
package main
import (
"fmt"
"net/http"
"strconv"
"time"
)
var i int = 0
var convStr string
var message = make(chan string)
/*func main(){
for i = 0; i < 1500; i++ {
z = strconv.Itoa(i)
url := "http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=" + z
resp, err := http.Get(url)
if err != nil {
fmt.Println("Houston, we've got problems")
}else{
if resp.StatusCode == 200{
fmt.Println("OK: El poste "+z+" existe")
}else{
fmt.Println("WARN: El poste "+z+" NO existe")
}
}
}
}*/
//Return 2 houndred posts
func returnTH(c chan string){
for i = 0; i < 200; i++ {
convStr = strconv.Itoa(i)
url := "http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=" + convStr
resp, err := http.Get(url)
if err != nil {
fmt.Println("Houston, we've got problems")
}else{
if resp.StatusCode == 200{
//fmt.Println("OK: El poste "+z+" existe")
c <- "OK: The bus post "+convStr+" exists"
}else{
//fmt.Println("WARN: El poste "+z+" NO existe")
c <- "WARN: The bus post "+convStr+" does not exist"
}
}
}
}
func returnFH(z chan string){
for i = 201; i < 400; i++ {
convStr = strconv.Itoa(i)
url := "http://www.urbanosdezaragoza.es/frm_esquemaparadatime.php?poste=" + convStr
resp, err := http.Get(url)
if err != nil {
fmt.Println("Houston, we've got problems")
}else{
if resp.StatusCode == 200{
//fmt.Println("OK: El poste "+z+" existe")
z <- "OK: The bus post "+convStr+" exists"
}else{
//fmt.Println("WARN: El poste "+z+" NO existe")
z <- "WARN: The bus post "+convStr+" does not exist"
}
}
}
}
func threadPrint(c, z chan string){
for {
threadOne := <- c
threadTwo := <- z
fmt.Println(threadOne)
fmt.Println(threadTwo)
}
}
func main(){
start := time.Now()
var c chan string = make(chan string)
var z chan string = make(chan string)
//for i = 0; i < 1500; i++{
go returnTH(c)
go returnFH(z)
go threadPrint(c,z)
/*go getBusPostStatus(400, 600)
go getBusPostStatus(600, 800)
go getBusPostStatus(800, 1000)
go getBusPostStatus(1000, 1200)
go getBusPostStatus(1200, 1400)
go getBusPostStatus(1400, 1500)*/
//}
timeExec:= time.Since(start)
fmt.Println("Time to exec code = ", timeExec)
/*var input string
fmt.Scanln(&input)
fmt.Println("done")*/
}
Many thanks in advance!!

Following is a simplified example code which requests 100 times concurrently and prints results, using goroutine and channel. Hope this code helps.
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
rep := 100
results := make(chan string)
// Use goroutine to send multiple time-consuming jobs to the channel.
for i := 0; i < rep; i++ {
go func(num int) {
results <- mockHTTPRequest(num)
}(i)
}
// Receive results from the channel and use them.
for i := 0; i < rep; i++ {
fmt.Println(<-results)
}
}
func mockHTTPRequest(num int) string {
timeDelay := rand.Intn(5000)
time.Sleep(time.Duration(timeDelay) * time.Millisecond)
if timeDelay%2 == 0 {
return fmt.Sprintf("OK: The bus post %v exists", num)
}
return fmt.Sprintf("WARN: The bus post %v does not exist", num)
}
You can run this code on https://play.golang.org/p/RR34roRIl4 .

Related

Can't use socat to process HTTP request via serial port

I'm trying to use socat in order to process incoming HTTP requests and send them via serial port to a board which will process it and send the response back. The problem is that concurrent requests will be discarded, and it will process only the first one. I tried running socat -d -d -d -d TCP-LISTEN:8081,fork,tcp-defer-accept=1 FILE:/dev/ttyACM0,b115200,raw,lock in order to achieve that, but only processes the first request. I implemented a bridge in go which does what I described, but the ideal approach would be using socat. Currently the handler of the socket, which aslo parses the request looks like this:
var mutex sync.Mutex
func Handler(conn net.Conn, s *serial.Port) {
mutex.Lock()
defer mutex.Unlock()
defer conn.Close()
buf := make([]byte, 65535)
l := 0
cl := 0
readingBody := false
var index int
var index2 int
for {
n, err := conn.Read(buf[l:])
if err != nil {
break
}
l += n
if !readingBody {
if strings.Contains(string(buf), "\r\n\r\n") {
if !strings.Contains(string(buf), "Content-Length:") {
break
} else {
index = strings.Index(string(buf), "Content-Length:")
index2 = strings.Index(string(buf), "\r\n\r\n")
indexcl := strings.Index(string(buf[index:]), "\r\n")
cl, err = strconv.Atoi(string(buf[index+16 : index+indexcl]))
if cl == 0 {
break
}
readingBody = true
if l-cl == index2+4 {
break
}
}
}
} else {
if l-cl == index2+4 {
break
}
}
}
fmt.Println("REQUEST")
if v {
fmt.Println(string(buf) + "\r\n")
}
n, err := s.Write(buf[:l])
if err != nil {
s.Close()
log.Fatal(err)
}
_ = n
scanner := bufio.NewScanner(s)
scanner.Split(bufio.ScanRunes)
var b bytes.Buffer
l = 0
cl = 0
readingBody = false
for scanner.Scan() {
b.Write([]byte(scanner.Text()))
l++
if !readingBody {
if strings.Contains(b.String(), "\r\n\r\n") {
if !strings.Contains(b.String(), "Content-Length:") {
break
} else {
index = strings.Index(b.String(), "Content-Length:")
index2 = strings.Index(b.String(), "\r\n\r\n")
cl, err = strconv.Atoi(string(b.Bytes()[index+16 : index2]))
if cl == 0 {
break
}
readingBody = true
if l-cl == index2+4 {
break
}
}
}
} else {
if l-cl == index2+4 {
break
}
}
}
fmt.Println("RESPONSE")
if v {
fmt.Println(string(b.Bytes()) + "\r\n")
}
n, err = conn.Write(b.Bytes())
}
Is there any way to use socat so that the processing is made in a similar way queueing requests until response for the previous are given?

Cannot reuse a single variable in for loop

I want to create a pointer and use it in every iteration but in the and I get all the same values. So I found the solution using creating the pointer in every iteration.
But this approach seems wasting a lot of memory.
Is there a way to reuse the same variable in Go?
Here is my code
func GetSchedules(start, end time.Time, usr *user.User) ([]*Scheduler, error) {
queryStr := []string{"SELECT account_id,link,text,time,image_links from", Table, "where user_id=?"}
var results = make([]*Scheduler, 0)
rows, err := DB.MYSQL_DB.Query(strings.Join(queryStr, " "), usr.Id.Hex())
if nil != err {
return results, err
}
defer rows.Close()
a := new(Scheduler)
for rows.Next() {
cols := []interface{}{&a.AccountId, &a.Link, &a.Text, &a.Time, &a.Images}
fmt.Println(rows.Scan(cols...))
results = append(results, a)
}
return results, nil
}
hereThe problem is with the variable called a although I am scanning new stuff into it in every iteration it just keeps showing the last one and the results slice contains just the last item multiple times
Don't worry about GC until you have proof that it is a bottleneck in your application. When reading data from DB, GC will never be a bottleneck for you. This is a simple benchmark:
func BenchmarkReallocate(b *testing.B) {
for i := 0; i < b.N; i++ {
results := make([]*foo, 0)
for i := 0; i < 100; i++ {
f := new(foo)
f.bar = "baz"
results = append(results, f)
}
}
}
func BenchmarkReuse(b *testing.B) {
for i := 0; i < b.N; i++ {
results := make([]*foo, 0)
var f *foo
for i := 0; i < 100; i++ {
f = new(foo)
f.bar = "baz"
results = append(results, f)
}
}
}
Results of running go test -bench . -benchmem:
BenchmarkReallocate-8 300000 4416 ns/op 3640 B/op 108 allocs/op
BenchmarkReuse-8 300000 4359 ns/op 3640 B/op 108 allocs/op
PASS
So in the end both ways result in exactly the same amount of allocations.
And the typical reading from DB into struct looks like this:
rows, err := db.Query("SELECT * FROM foo")
if nil != err {
return nil, err
}
defer rows.Close()
results := make([]*Scheduler, 0)
for rows.Next() {
var s Scheduler
if err := rows.Scan(&s.Id, &s.Name, &s.Something); err != nil {
return nil, err
}
results = append(results, &s)
}
if err := rows.Err(); err != nil { // Don't forget to check rows.Err()
return nil, err
}
return results, nil

Calling winapi function in golang with struct union

I am trying to call the TransmitPackets function on windows using GO.
The goal is to be able to send multiple packets with one syscall (can't be achieved with WSASend [it'll send fragmented IP packets]).
My code panics
panic: write udp 192.168.1.26:51817->8.8.8.8:8000: transmitpackets: An invalid argument was supplied.
goroutine 1 [running]:
main.main()
c:/Users/amit/dev/go/src/rio/main.go:26 +0x210
exit status 2
Process exiting with code: 1
Here's my test code
package main
import (
"math/rand"
"net"
)
func main() {
raddr, err := net.ResolveUDPAddr("udp", "8.8.8.8:8000")
if err != nil {
panic(err)
}
con, err := net.DialUDP("udp", nil, raddr)
if err != nil {
panic(err)
}
packets := make(net.Buffers, 10)
for i := 0; i < len(packets); i++ {
packets[i] = make([]byte, 1400)
rand.Read(packets[i])
}
_, err = con.WriteMultiple(packets)
if err != nil {
panic(err)
}
}
and Here's my call to TransmitPackets:
type TransmitPacketsElement struct {
dwElFlags uint32
cLength uint32
pBuffer unsafe.Pointer
nFileOffset uint64
hFile uintptr
}
func transmitPackets(s Handle, bufs [][]byte, overlapped *Overlapped) (err error) {
var maxPacketLen = 0
tpElements := make([]TransmitPacketsElement, len(bufs))
for i, tpElement := range tpElements {
buffer := bufs[i]
if len(buffer) > maxPacketLen {
maxPacketLen = len(buffer)
}
tpElement.cLength = uint32(len(buffer))
tpElement.dwElFlags = uint32(uint32(TP_ELEMENT_MEMORY) | uint32(TP_ELEMENT_EOP))
tpElement.pBuffer = unsafe.Pointer(&buffer[0])
}
r1, _, e1 := Syscall6(transmitPacketsFunc.addr, 6, uintptr(s), uintptr(unsafe.Pointer(&tpElements[0])), uintptr(uint32(len(tpElements))), uintptr(uint32(maxPacketLen)), uintptr(unsafe.Pointer(overlapped)), 0)
if r1 == 0 {
if e1 != 0 {
err = error(e1)
} else {
err = EINVAL
}
}
return
}
You can see the full implementation in my modified go 1.8.3 source on github

Asynchronous messages golang

I have a golang server doing something like this:
package main
func main() {
for {
c := listener.Accept()
go handle(c)
}
}
...
func handle(c net.Conn) {
m := readMessage(c) // func(net.Conn)Message
r := processMessage(m) //func(Message)Result
sendResult(c, r) // func(net.Conn,Result)
}
Which reads and writes messages synchronously. What I need now is to send messages asynchronously through a given open connection, I know a channel can be used by I'm kind of lost.
This is my idea:
...
func someWhereElese(c chan Result) {
// generate a message and a result
r := createResultFromSomewhere()
c <- r // send the result through the channel
}
And modify my handle to use that same channel instead
func handle(c net.Conn, rc chan Result) {
m := readMessage(c) // func(net.Conn)Message
r := processMessage(m) //func(Message)Result
//sendResult(c, r) // func(net.Conn,Result)
rc <- r
}
And here's where my confusion lies.
The result channel should be created and it should have a connection where to send whatever it receives
func doSend(c net.Con, rc chan Result) {
r := rc // got a result channel
sendResult(c, r) // send it through the wire
}
But where should that channel be created? In the main loop?
func main() {
...
for {
c := l.Accept()
rc := make(chan Result)
go doSend(c, rc)
}
}
What about the read? Should it go in it's own channel/gorutine?
If I need to broadcast to n clients, should I keep a slice of result channels? a slice of connections?
I'm kind of confused here, but I feel I'm close.
This program seems to solve my immediate question
package main
import (
"bytes"
"encoding/binary"
"log"
"net"
)
var rcs []chan int = make([]chan int,0)
func main() {
a, e := net.ResolveTCPAddr("tcp", ":8082")
if e != nil {
log.Fatal(e)
}
l, e := net.ListenTCP("tcp", a)
for {
c, e := l.Accept()
if e != nil {
log.Fatal(e)
}
rc := make(chan int)
go read(c, rc)
go write(c, rc)
rcs = append(rcs, rc)
// simulate broacast
log.Println(len(rcs))
if len(rcs) > 5 {
func() {
for _, v := range rcs {
log.Println("sending")
select {
case v <- 34:
log.Println("done sending")
default:
log.Println("didn't send")
}
}
}()
}
}
}
func read(c net.Conn, rc chan int) {
h := make([]byte, 2)
for {
_, err := c.Read(h)
if err != nil {
rc <- -1
}
var v int16
binary.Read(bytes.NewReader(h[:2]), binary.BigEndian, &v)
rc <- int(v)
}
}
func write(c net.Conn, rc chan int) {
for {
r := <-rc
o := []byte{byte(r * 2)}
c.Write(o)
}
}

How to cast reflect.Value to its type?

How to cast reflect.Value to its type?
type Cat struct {
Age int
}
cat := reflect.ValueOf(obj)
fmt.Println(cat.Type()) // Cat
fmt.Println(Cat(cat).Age) // doesn't compile
fmt.Println((cat.(Cat)).Age) // same
Thanks!
concreteCat,_ := reflect.ValueOf(cat).Interface().(Cat)
see http://golang.org/doc/articles/laws_of_reflection.html
fox example
type MyInt int
var x MyInt = 7
v := reflect.ValueOf(x)
y := v.Interface().(float64) // y will have type float64.
fmt.Println(y)
Ok, I found it
reflect.Value has a function Interface() that converts it to interface{}
This func auto-converts types as needed. It loads a config file values into a simple struct based on struct name and fields:
import (
"fmt"
toml "github.com/pelletier/go-toml"
"log"
"os"
"reflect"
)
func LoadConfig(configFileName string, configStruct interface{}) {
defer func() {
if r := recover(); r != nil {
fmt.Println("LoadConfig.Recovered: ", r)
}
}()
conf, err := toml.LoadFile(configFileName)
if err == nil {
v := reflect.ValueOf(configStruct)
typeOfS := v.Elem().Type()
sectionName := getTypeName(configStruct)
for i := 0; i < v.Elem().NumField(); i++ {
if v.Elem().Field(i).CanInterface() {
kName := conf.Get(sectionName + "." + typeOfS.Field(i).Name)
kValue := reflect.ValueOf(kName)
if (kValue.IsValid()) {
v.Elem().Field(i).Set(kValue.Convert(typeOfS.Field(i).Type))
}
}
}
} else {
fmt.Println("LoadConfig.Error: " + err.Error())
}
}
Seems the only way would be to do a switch statement similar to (code below) (also, something like the commented line would've-been nice though doesn't work (:()):
func valuesFromStruct (rawV interface{}) []interface{} {
v := reflect.ValueOf(rawV)
out := make([]interface{}, 0)
for i := 0; i < v.NumField(); i += 1 {
field := v.Field(i)
fieldType := field.Type()
// out = append(out, field.Interface().(reflect.PtrTo(fieldType)))
switch (fieldType.Name()) {
case "int64":
out = append(out, field.Interface().(int64))
break`enter code here`
case "float64":
out = append(out, field.Interface().(float64))
break
case "string":
out = append(out, field.Interface().(string))
break
// And all your other types (here) ...
default:
out = append(out, field.Interface())
break
}
}
return out
}
Cheers!

Resources