What Is Pipeline? #
A pipeline is a function combination used for processing tasks offline. It uses the pipeline design pattern, just as online request filters do. A processor is the basic unit of a pipeline. Each processing component focuses on one task and the components can be flexibly assembled, and plugged and removed as required.
Pipeline Definition #
A typical pipeline service is defined as follows:
pipeline: - name: request_logging_index auto_start: true keep_running: true processor: - json_indexing: index_name: "gateway_requests" elasticsearch: "dev" input_queue: "request_logging" idle_timeout_in_seconds: 1 worker_size: 1 bulk_size_in_mb: 10 #in MB
In the above configuration, a processing pipeline named
request_logging_index is defined, and the
processor parameter defines several processing units for the pipeline, which are executed in sequence.
Parameter Description #
Parameters related to pipeline definition are described as follows:
|name||string||Name of a pipeline, which must be unique|
|auto_start||bool||Whether the pipeline automatically starts with the gateway startup, that is, whether the task is executed immediately|
|keep_running||bool||Whether the gateway starts executing the task again after completing the execution|
|retry_delay_in_ms||int||Minimum waiting time for the task re-execution, which is set to |
|processor||array||List of processors to be executed by the pipeline in sequence|