elasticsearch

elasticsearch #

Description #

The elasticsearch filter is used to forward requests to back-end Elasticsearch clusters.

Configuration Example #

Before using the elasticsearch filter, define one Elasticsearch cluster configuration node as follows:

elasticsearch:
- name: prod
  enabled: true
  endpoint: http://192.168.3.201:9200

The following shows a flow configuration example.

flow:
  - name: cache_first
    filter:
      - elasticsearch:
          elasticsearch: prod  

The preceding example forwards requests to the prod cluster.

Automatic Update #

For a large cluster that contains many nodes, it is almost impossible to configure all back-end nodes individually. Instead, you only need to enable auto-discovery of back-end nodes on the Elasticsearch module. See the following example.

elasticsearch:
- name: prod
  enabled: true
  endpoint: http://192.168.3.201:9200
  discovery:
    enabled: true
    refresh:
      enabled: true
  basic_auth:
    username: elastic
    password: pass

Then, enable automatic configuration refresh on the filter. Now, all back-end nodes can be accessed and the status of online and offline nodes is automatically updated. See the following example.

flow:
  - name: cache_first
    filter:
      - elasticsearch:
          elasticsearch: prod  
          refresh: 
            enabled: true
            interval: 30s

Setting the Weight #

If there are many back-end clusters, INFINI Gateway allows you to set different access weights for different nodes. See the following configuration example.

flow:
  - name: cache_first
    filter:
      - elasticsearch:
          elasticsearch: prod  
          balancer: weight
          refresh: 
            enabled: true
            interval: 30s
          weights:
            - host: 192.168.3.201:9200 
              weight: 10
            - host: 192.168.3.202:9200
              weight: 20
            - host: 192.168.3.203:9200
              weight: 30

In the above example, the traffic destined for an Elasticsearch cluster is distributed to the 203, 202, and 201 nodes at a ratio of 3:2:1.

Filtering Node #

INFINI Gateway can also filter requests based on node IP address, label, or role to avoid sending requests to specific nodes, such as the master and cold nodes. See the following configuration example.

flow:
  - name: cache_first
    filter:
      - elasticsearch:
          elasticsearch: prod  
          balancer: weight
          refresh: 
            enabled: true
            interval: 30s
          filter:
            hosts:
              exclude:
                - 192.168.3.201:9200
              include:
                - 192.168.3.202:9200
                - 192.168.3.203:9200
            tags:
              exclude:
                - temp: cold
              include:
                - disk: ssd
            roles:
              exclude:
                - master
              include:
                - data
                - ingest

Parameter Description #

NameTypeDescription
elasticsearchstringName of an Elasticsearch cluster
max_connection_per_nodeintMaximum number of TCP connections that are allowed to access each node of an Elasticsearch cluster. The default value is 5000.
max_response_sizeintMaximum size of the message body returned in response to an Elasticsearch request. The default value is 100*1024*1024.
max_conn_wait_timeoutintTimeout duration for Elasticsearch to wait for an idle connection. The default value is 10s.
max_idle_conn_durationintIdle duration of an Elasticsearch connection. The default value is 0s.
max_conn_durationintDuration of an Elasticsearch connection. The default value is 0s.
read_timeoutintRead timeout duration of an Elasticsearch request. The default value is 0s.
write_timeoutintWrite timeout duration of an Elasticsearch request. The default value is 0s.
read_buffer_sizeintRead cache size for an Elasticsearch request. The default value is 4096*4.
write_buffer_sizeintWrite cache size for an Elasticsearch request. The default value is 4096*4.
tls_insecure_skip_verifyboolWhether to ignore TLS certificate verification of an Elasticsearch cluster. The default value is true.
balancerstringLoad balancing algorithm of a back-end Elasticsearch node. Currently, only the weight weight-based algorithm is available.
refresh.enableboolWhether to enable automatic refresh of node status changes, to perceive changes in the back-end Elasticsearch topology
refresh.intervalintInterval of the node status refresh
weightsarrayPriority of a back-end node. A node with a larger weight is assigned a higher proportion of request forwarding.
filterobjectFiltering rules for back-end Elasticsearch nodes. Rules can be set to forward requests to a specific node.
filter.hostsobjectFiltering based on the access address of Elasticsearch
filter.tagsobjectFiltering based on the label of Elasticsearch
filter.rolesobjectFiltering based on the role of Elasticsearch
filter.*.excludearrayConditions for excluding. Any matched node is denied handling requests as a proxy.
filter.*.includearrayElasticsearch nodes that meet conditions are allowed to handle requests as a proxy. When the exclude parameter is not configured but include is configured, any condition in include must be met. Otherwise, the node is not allowed to handle requests as a proxy.