Release Notes

Release Notes #

Information about release notes of INFINI Gateway is provided here.

1.9.0 #

Breaking changes #

  • Refactoring config for ip access control
  • Disable metadata refresh and node availability check by default
  • Update default config path from configs to config

Features #

  • Support listen on IPv6 address
  • Add general health api
  • Add request_ip to context
  • Add badger filter plugin
  • Allow to split produce and consume messages from s3
  • Add bulk_request_throttle filter
  • Support access request context and more output options in echo filter
  • Add body_json to response context
  • Add cert config to API module, support mTLS
  • Add api to clear scroll context
  • Floating_ip support stick by priority

Bug fix #

  • Fix user was removed in logging filter
  • Fix incorrect message size issue, reload when files changed in disk_queue
  • Fix issue that index_diff could not finished automatically
  • Fix hostname was not well updated in filter set_request_header or set_hostname
  • Fix to check consumer’s lag instead of queue’s lag in flow_runner processor

Improvements #

  • Remove newline in indexing_merge and json_indexing processor
  • Improve instance check, add config to disable
  • Add option skip_insecure_verify to s3 module
  • Improve instance check, enable config to disable
  • Update the way to get ctx process info, optimize memory usage
  • Improve indexing performance for bulk_indexing processor
  • Refactoring disk_queue, speedup message consumption
  • Enable segment compress for disk_queue by default
  • Skip download s3 files when s3 was not enabled
  • Add option to log warning messages for throttle filters
  • Optimize hash performance for getting primary shardID and partitionID
  • Add cache for get index routing table
  • Optimize performance for bulk response processing
  • Refactoring bulk_processor, pass meta info to payload func
  • Don’t call payload func for delete action
  • Improve queue consumer’s lag check

1.8.1 #

Bug fix #

  • Remove newline in document for processor es_scroll and dump_hash

1.8.0 #

Breaking changes #

  • Remove config compress_on_message_payload from disk_queue
  • Rename parameter consumer to name in consumer_has_lag condition
  • Remove redundancy prefix name of the disk_queue files

Features #

  • Add segment level based disk_queue file compression

Bug fix #

  • Fix nil host in bulk_indexing processor
  • Fix nil body in bulk_response_process filter
  • Fix sliced consume in bulk_indexing processor

Improvements #

  • Handle bulk stats to bulk_response_process and used in logging filter

1.7.0 #

Breaking changes #

Features #

  • Add prometheus format to stats API
  • Add redirect filter
  • Add context_flow to flow filter
  • Add permitted_client_ip_list to router
  • Add Centos based docker image

Bug fix #

  • Fix date_range_precision filter failed to parse on specify field
  • Fix disk usage status in windows platform

Improvements #

  • Merge events during config change, prevent unnecessary reload
  • Handle templates when loading config
  • Add cache to ldap_auth filter

1.6.0 #

Breaking changes #

  • Update disk_queue folder structure, use UUID as folder name instead of the queue name
  • Parameter mode was removed from bulk_reshuffle filter, only async was supported
  • Rename filter bulk_response_validate to bulk_response_process

Features #

  • Add metadata to queue
  • Support subscribe queue by specify labels
  • Support concurrent worker control for bulk_indexing processor
  • Auto detect new queues for bulk_indexing processor
  • Allow to consume queue messages over disk queue
  • Auto sync disk_queue files to remote s3 in background
  • Add api to operate gateway entry
  • Support plugin auto discovery
  • Add API to operate gateway entities
  • Filter bulk_request_mutate support remove _type in bulk requests for es v8.0+
  • Add elasticsearch adapter for version 8.0+
  • Add http filter for general reverse proxy usage, like proxy Kibana
  • Add consumer_has_lag condition to check queue status
  • Add record filter to play requests easier
  • Add zstd compress to disk_queue, disabled by default
  • Add disorder_bulk_indexing processor
  • Add javascript filter
  • Add prefix and suffix to when conditions
  • Add indexing_merge processor

Bug fix #

  • Fix date_range_precision_tuning filter for complex range query
  • Fix node availability initially check
  • Fix basic_auth filter not asking user to input auth info in browser
  • Fix null id not fixed in filter bulk_request_mutate and bulk_reshuffle
  • Fix switch filter not forwarding when remove_prefix was disabled
  • Fix buffer was not proper reset in flow_runner processor
  • Fix entry not loading the pre-defined TLS certificates
  • Fix set_basic_auth not proper reset previous user information
  • Fix elapsed in request logging not correct
  • Fix switch filter, use strings.TrimPrefix instead of strings.TrimLeft
  • Fix the last query_string args can’t be deleted, parameter no_cache in get_cache filter fixed
  • Fix s3 downloaded file corrupted

Improvements #

  • Handle http public address, remove prefix if that exists
  • Refactor bulk_reshuffle filter and bulk_indexing processor
  • Should not fetch nodes info when elasticsearch discovery disabled
  • Seamless consume queue message across files
  • Persist consumer offset to local store
  • Add API to reset consumer offset
  • Refactoring ORM framework
  • Expose error of mapping put
  • Refactoring pipeline framework
  • Improve multi-instance check, multi-instance disabled by default
  • Add CPU and memory metrics to stats api
  • Seamless fetch queue files from s3 server
  • Proper handle 409 version conflicts in bulk requests
  • Allow memory queue to retry 1s when it is full
  • Proper handle the cluster available check
  • Proper handle the cluster partial failure check
  • Exit bulk worker while no new messages returned

1.5.0 #

Breaking changes #

Features #

  • Add API to scroll messages from disk queue
  • Prevent out of space, disk usage reserved for disk_queue
  • Add context_filter and context_limiter for general purpose
  • Add bulk_request_mutate filter
  • Add basic_auth filter
  • Add set_context filter
  • Add context_regex_replace filter
  • Add to_string property to request and response context

Bug fix #

  • Fix bulk response validate incorrectly caused by jsonParser
  • Fix nil exception in request_path_limiter caused by refactoring
  • Fix big size document out of order caused by bulk buffer

Improvements #

  • Fix TCP not keepalived in some case
  • Add closing progress bar to pipeline module
  • Add retry_delay_in_ms config to pipeline module
  • Handle partial failure in bulk requests
  • Optimize scroll performance of dump_hash processor
  • Improve API directory

1.4.0 #

Breaking changes #

  • Rename flow config filter_v2 to filter, only support new syntax
  • Rename pipeline config pipelines_v2 to pipeline, processors to processor, only support new syntax
  • Rename filter request_logging to logging
  • Merge dump filters to dump filter
  • Response headers renamed, dashboard may broken
  • Remove filter request_body_truncate and response_body_truncate

Features #

  • Add option to disable file logging output
  • Add option compress to queue_consumer processor

Bug fix #

  • Fix invalid host header setting in elasticsearch reverse proxy
  • Fix cluster available health check
  • Fix gzip encoding issue for requests forwarding

Improvements #

  • Support string type in in condition

1.3.0 #

Breaking changes #

  • Switch to use pipelines_v2 syntax only
  • Rename filter disk_enqueue to queue
  • Rename processor disk_queue_consumer to queue_consumer
  • Rename filter redis to redis_pubsub

Features #

  • Refactoring pipeline framework, support DAG based task schedule
  • Add dump_hash and index_diffs processor
  • Add redis output and redis queue adapter
  • Add set_request_query_args filter
  • Add ldap_auth filter
  • Add retry_limiter filter
  • Add request_body_json_set and request_body_json_del filter
  • Add stats filter
  • Add health_check config to elastic module
  • Add API to pipeline framework, support _start and _stop pipelines

Bug fix #

  • Fix data race issue in bulk_reshuffle
  • Fix fix_null_id always executed in bulk_reshuffle
  • Auto handle big sized documents in bulk requests

Improvements #

  • Refactoring flow runner to service pipeline
  • Optimize CPU and Memory usage
  • Optimize index diff service, speedup and cross version compatibility
  • Set the default max file size of queue files to 1 GB
  • Proper handle elasticsearch failure during startup
  • Support custom depth check to queue_has_lag condition
  • Support multi hosts for elasticsearch configuration
  • Add parameter auto_start to prevent pipeline running on start
  • Add keep_running parameter to pipeline config
  • Safety shutdown pipeline and entry service
  • Support more complex routing pattern rules

1.2.0 #

Features #

  • Support alias in bulk_reshuffle filter.
  • Support truncate in request_logging filter.
  • Handle 429 retry in json_indexing service.
  • Add forcemerge service.
  • Add response_body_regex_replace filter.
  • Add request_body_regex_replace filter.
  • Add sleep filter.
  • Add option to log slow requests only.
  • Add cluster and bulk status to request logging.
  • Add filter_v2 and support _ctx to access request context.
  • Add dump_context filter.
  • Add translog filter, support rotation and compression.
  • Add set_response filter.
  • Add set_request_header filter.
  • Add set_hostname filter.
  • Add set_basic_auth filter.
  • Add set_response_header filter.
  • Add elasticsearch_health_check filter.
  • Add drop filter.

Bug fix #

  • Fix truncate body filter, correctly resize the body bytes.
  • Fix cache filter.
  • Fix floating_ip module.
  • Fix dirty write in diskqueue.
  • Fix compression enabled requests.
  • Fix date_range_precision_tuning filter.
  • Fix invalid indices status on closed indices #23.
  • Fix document hash for elasticsearch 6.x.
  • Fix floating_ip feature run with daemon mode.
  • Fix async bulk to work with beats.

Improvements #

  • Optimize memory usage, fix memory leak.

Acknowledgement #

Thanks to the following enterprises and teams #

  • China Everbright Bank, China Citic Bank, BSG, Yogoo

Thanks to the following individual contributors #

  • MaQianghua, YangFan, Tanzi, FangLi

1.1.0 #

  • Request Logging and Dashboard.
  • Support ARM Platform [armv5\v6\v7\v8(arm64)].
  • Fix Elasticsearch Nodes Auto Discovery.
  • Add Request Header Filter.
  • Add Request Method Filter.
  • Add Sample Filter.
  • Request Logging Performance Optimized (100x speedup).
  • Add Request Path Filter.
  • Add Debug Filter.
  • Add User Info to Logging Message.
  • Support Routing Partial Traffic to Specify Processing Flow (by Ratio).
  • Support Traffic Clone, Support Dual-Write or 1:N Write.
  • Elasticsearch topology auto discovery, support filter by nodes,tags,roles.
  • Backend failure auto detection, auto retry and select another available endpoint.
  • Floating IP feature ready to use.
  • Add bulk_reshuffle filter.

1.0.0 #

  • Rewritten for performance
  • Index level request throttle
  • Request caching
  • Kibana MAGIC speedup
  • Upstream auto discovery
  • Weighted upstream selections
  • Max connection limit per upstream
  • Languages English
©INFINI.LTD, All Rights Reserved.