json_indexing #

Description #

The json_indexing processor is used to consume pure JSON documents in queues and store them to a specified Elasticsearch server.

Configuration Example #

A simple example is as follows:

- name: request_logging_index
  auto_start: true
  keep_running: true
    - json_indexing:
        index_name: "gateway_requests"
        elasticsearch: "dev"
        input_queue: "request_logging"
        idle_timeout_in_seconds: 1
        worker_size: 1
        bulk_size_in_mb: 10

Parameter Description #

input_queueintName of a subscribed queue
worker_sizeintNumber of threads that concurrently execute consumption tasks, which is set to 1 by default.
idle_timeout_in_secondsintTimeout duration of the consumption queue, in seconds. The default value is 5.
bulk_size_in_kbintSize of a bulk request, in KB.
bulk_size_in_mbintSize of a bulk request, in MB.
elasticsearchstringName of a target cluster, to which requests are saved.
index_namestringName of the index stored to the target cluster.
type_namestringName of the index type stored to the target cluster. It is set based on the cluster version. The value is doc for Elasticsearch versions earlier than v7 and _doc for versions later than v7.