Stream Pipeline
About
A stream pipeline is a pipeline that has a supplier operation with a processing type set to stream.
Data Processing
A stream pipeline will:
- send the data resource down stream immediately or at interval (push-interval)
- executes:
- stream intermediate operation immediately
- batch intermediate operation at interval (window-interval)
Poll
- The polling happens when:
- there is no data resource anymore in the queue
- the interval determined by the poll-interval argument is exhausted
- The number of polling is stored in the poll counter metrics
Arguments
Every stream pipeline accepts the following arguments:
| Name | Default | Description |
|---|---|---|
| poll-interval | 1s | The interval between poll (a poll is a request for data resources) |
| push-interval | The interval between push (a push is the sending of a data resource into the pipeline) | |
| window-interval | 2s | The interval between execution of batch intermediate operation (generally known as the window) The first execution will be at pipeline start time + window interval |
End
A stream is not meant to end but you can use the duration control argument in your tests.