How to keep one exec plugin waiting for another?

Elastic Stack Logstash
1

Given the following configuration:

input {
  exec {
    command => "python3 ${LOGSTASH_HOME}/script/firmware/backend/download_script.py"
    schedule => "${SCHEDULER}"
    type => "Backend"
  }
  exec {
    command => "python2 ${LOGSTASH_HOME}/script/firmware/error_parser/parsers/log_parser/parser.py -pp true -lf ${LOGSTASH_HOME}/firmware_data"
    schedule => "${FIRMWARE_SCHEDULER}"
    type => "Firmware"
  }
  file {
	 path => "${LOGSTASH_HOME}/firmware_data/errorcode_diagnostics/*.diag"
	 codec => json
	 start_position => "beginning"
	 sincedb_path => "${LOGSTASH_HOME}/file/.sincedb*"
	 type => "S3"
  }
}

Question:
Is there any way I can tell the exec input plugin to wait until command => "python3 ${LOGSTASH_HOME}/script/..." is done before it starts executing command => "python2 ${LOGSTASH_HOME}/script/firmware/error_parser/..."?

Note: The second exec command requires the first exec command to be finished first!

Possible solution 1: Set the time interval for the second command anywhere from 5-10m after the first command to ensure it is processed successfully before it starts.

Problem: This approach can sometimes fail if there is a lot of data being processed by the first command, so while the first command is still in process, the second command can kick in and fail.

Possible solution 2: Run the second command in the filter plugin.

Problem: This approach doesn't generate any extra fields to use after in the filter plugin or output plugin, only running the script in the input plugin does this.

2

There nothing in logstash that will help you do that. You would have to solve it in the Python scripts.

3

Yes, but I'm not the owner of the second script, so I'm not allowed to make a change.

Solution 2 that I suggested can work, but it doesn't generate any fields such as [message]. Do you know any way that I can get those fields as a result of running the second script in the filter plugin?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.