Read the value from a file for a variable being used in elasticsearch plugin in input of logstash config file

Elastic Stack Logstash
1

Hello!

We are working on a use case where we are trying to read delta load from an elasticsearch index using the elasticsearch plugin in the logstash configuration file.
We are using this elasticsearch plugin in the input of the configuration file as based on the query results of this index, we will be filtering the other indices in the filter plugin.
The date of the lastmodified is stored in a file and the value keeps changing, and hence needs to be read from that file every time the pipeline is run.
Can we read the value from the file dynamically in the input using any plugin and be able to access that value read in the input of the config file itself, so that the elasticsearch plugin in input plugin is able to pick that value.

Attaching the current input for reference.

input {
  elasticsearch {
    hosts => "<host>"
    index => "<index_name>"
query=>'{"query":{"range":{"lastmodified":{"format": "--- yyyy-MM-dd HH:mm:ss.SSSSSSSSS Z","gte": "${LAST_RUN}"}}}}'
      scroll => "5m"
    docinfo => true
  }
}

${LAST_RUN} => the value for LAST_RUN needs to be read from a file;
contents of the file is going to be in the format: --- yyyy-MM-dd HH:mm:ss.SSSSSSSSS Z
Please guide.

2

The elasticsearch input does not support persisting state like a jdbc input. There is an open issue requesting this.

If you want to reference an environment variable using ${LAST_RUN} then you would need to read the file and set the environment variable in the script that starts logstash.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.