Single execution of logstash pipeline on demand

Elastic Stack Logstash
1

Hello,

i have a logstash 8.2 instance running with multiple scheduled pipelines.

What is the best way to start another logstash pipeline on demand for a single execution on the same logstash instance without restarting it?

I want to provide a button for a user that he can click to start the pipeline. He needs to do it once a month with varying date.

I read that i can add and remove pipelines with REST. (PUT _logstash/pipeline/<pipeline_id>)

  • What happens if i add a new pipeline without a schedule? Will it be executed once and that's it?
  • What happens if he wants to run it again the next month? Do i have to remove it first? Is it possible to automatically remove the pipeline after execution?
  • What happens if he tries to run it again, while it is already running? Can i check in an easy way if the pipeline is already running and only do the PUT if it is not?

Or would it be easier to add a scheduled pipeline that runs at night that day which is cleaned up by some scheduled job that runs a few hours later?

Best regards
Jonas

2

Any recommendations?

3

There is nothing native in Logstash to do things like that, but depending on what this on-demand pipeline will do, there are some tricks that you can do to try to emulate this behavior, but as I said, this depends on the pipeline and what you want to do.

Also, the endpoint you shared, _logstash/pipeline/<pipeline_id> is to be used when you have central pipeline management configured, which is a licensed feature, if you are on the basic license you won't be able to use it as it is not free.

if you add a pipeline using that endpoint, it will keep running until you delete it.

One thing that you cand do is have autoreload configured in logstash.yml and use the pipelines.yml, then when you change the pipelines.yml to add or remove pipelines, logstash will trigger a reload after the configured time.

You would need to build something that would add that on-deman pipeline to the file, or comment and uncomment it as needed.

What does your pipeline looks like?

1 Like
4

Thank you, i did not know that it is a subscription feature. We are thinking of getting a licence anyways, so this might be another reason for it.

What does running until you delete it mean? All my pipelines have a jdbc input plugin and an elasticsearch output plugin an thats it. The already existing ones execute a sql statement with a schedule and create an elastic index to write the results there.
The new ones should do the same but not with a schedule but only if a user initiates it.

So far i thought that a pipeline is stopped after its statement is executed and the result is written to elastic. But if you write it will keep running until you delete it, that sounds like it works different. Would that mean that a pipeline with jdbc input and elastic output keeps running even after the statement results are written to elastic and just do nothing? I don't think that the statement will be executed another time, right?

So if i comment in a pipeline (jdbc->elastic) with autoreload enabled will it execute the statement once, write the results, and keep running until it is commented out again while doing nothing?

And what would happen if a pipeline is commented out with autoreload enabled, while it is running? Will it finish it's task before being removed?

Best regards
Jonas

5

Since you didn't provide any information about your inputs, I assumed that you were using a long running input like tcp, beats, file and many others.

For long running inputs, and also for the jdbc input with a schedule option, they will keep running until they are removed from pipelines.yml or deleted using the API.

If you have an input that can run once Logstash will terminate the pipeline after the execution is finished and this pipeline will only run again if you trigger a reload of its configuration.

To trigger a reload you would need to remove the pipeline from pipelines.yml, wait for logstash to reload and then add it again to pipelines.yml, changing the configuration file of the pipeline would also trigger a reload. I do not use the central management API, but I think that you would need to Delete and Add the pipeline to run it again.

When you have autoreload enabled and you remove a running pipeline, Logstash will finish processing the current events (in-flight events) of this pipeline before terminating it, depending on the number of events this can take some time

1 Like
6

Thank you very much!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.