In golang, how to write a pipeline stage that introduces a delay for the next stage? - asynchronous

Question

I am following the https://blog.golang.org/pipelines article to implement a few stages.
I need one of the stages to introduce a delay of a few seconds before the events are passed on the next stage in the pipeline.
My concern with the code below is that it will yield an unbounded number of go routines that time.Sleep() before passing the events along. Are there better ways to do this?
Thanks!
func fooStage(inChan <- chan *Bar) (<- chan *Bar) {
out := make(chan *Bar, 10000)
go func() {
defer close(out)
wg := sync.WaitGroup{}
for {
select {
case event, ok := <-inChan:
if !ok {
// inChan closed
break
}
wg.Add(1)
go func() {
time.Sleep(5 * time.Second)
out <- event
wg.Done()
}()
}
}
wg.Wait()
}()
return out
}

Answer 1

You could use another channel to limit the number of active goroutines your loop is able to create.
const numRoutines = 10
func fooStage(inChan <-chan *Bar) <-chan *Bar {
out := make(chan *Bar, 10000)
routines := make(chan struct{}, numRoutines)
go func() {
defer close(out)
wg := sync.WaitGroup{}
for {
select {
case event, ok := <-inChan:
if !ok {
// inChan closed
break
}
wg.Add(1)
routines <- struct{}{}
go func() {
time.Sleep(5 * time.Second)
out <- event
wg.Done()
<-routines
}()
}
}
wg.Wait()
}()
return out
}

Answer 2

You may fix the number of goroutines manually - starting only number you need.
func sleepStage(in <-chan *Bar) (out <-chan *Bar) {
out = make(<-chan *Bar)
wg := sync.WaitGroup
for i:=0; i < N; i++ { // Number of goroutines in parallel
wg.Add(1)
go func(){
defer wg.Done()
for e := range in {
time.Sleep(5*time.Seconds)
out <- e
}
}()
}
go func(){}
wg.Wait()
close(out)
}()
return out
}

Answer 3

You can use time.Ticker:
func fooStage(inChan <- chan *Bar) (<- chan *Bar) {
//... some code
ticker := time.NewTicker(5 * time.Second)
<-ticker // the delay, probably need to call twice
ticker.Stop()
close(ticker.C)
//... rest code
}

Answer 4

This is what you should use for a pipeline application. The context allows for faster tear down.
Whatever is responsible for managing your in channel must close it during tear down. Always close your channels.
// Delay delays each `interface{}` coming in through `in` by `duration`.
// If the context is canceled, `in` will be flushed until it closes.
// Delay is very useful for throtteling back CPU usage in your pipelines.
func Delay(ctx context.Context, duration time.Duration, in <-chan interface{}) <-chan interface{} {
out := make(chan interface{})
go func() {
// Correct memory management
defer close(out)
// Keep reading from in until its closed
for i := range in {
// Take one element from in and pass it to out
out <- i
select {
// Wait duration before reading from in again
case <-time.After(duration):
// Don't wait if the context is canceled
case <-ctx.Done():
}
}
}()
return out
}

Answer 5

I have already solved such a problem with my pipeline library, like that:
import "github.com/nazar256/parapipe"
//...
pipeline := parapipe.NewPipeline(10).
Pipe(func(msg interface{}) interface{} {
//some code
}).
Pipe(func(msg interface{}) interface{} {
time.Sleep(3*time.Second)
return msg
}).
Pipe(func(msg interface{}) interface{} {
//some other code
})