queue_consumer

queue_consumer #

Description #

The queue_consumer processor is used to asynchronously consume requests in a queue and send the requests to Elasticsearch.

Configuration Example #

A simple example is as follows:

pipeline:
- name: bulk_request_ingest
  auto_start: true
  keep_running: true
  processor:
    - queue_consumer:
        input_queue: "backup"
        elasticsearch: "backup"
        waiting_after: [ "backup_failure_requests"]
        worker_size: 20
        when:
          cluster_available: [ "backup" ]

Parameter Description #

NameTypeDescription
input_queueintName of a subscribed queue
worker_sizeintNumber of threads that concurrently execute consumption tasks, which is set to 1 by default.
idle_timeout_in_secondsintTimeout duration of the consumption queue, which is set to 1 by default.
elasticsearchstringName of a target cluster, to which requests are saved.
waiting_afterarrayData in the main queue can be consumed only after data in a specified queue is consumed.
failure_queuestringRequest that fails to be executed because of a back-end failure. The default value is %input_queue%-failure.
invalid_queuestringRequest, for which the returned status code is 4xx. The default value is %input_queue%-invalid.
compressboolWhether to compress requests. The default value is false.
safety_parseboolWhether to enable secure parsing, that is, no buffer is used and memory usage is higher. The default value is true.
doc_buffer_sizeboolMaximum document buffer size for the processing of a single request. You are advised to set it to be greater than the maximum size of a single document. The default value is 256*1024.