Run Logstash manually without interrupting logstash service and logstash Scheduler

Elastic Stack Logstash
1

Hello,

I have a configuration file with http_poller plugins, where I have some scheduler which on the given schedule pull the data and enter it into my elastic db.

But to manually run lagstash instantly, What I have to do is reschdule the scheduler to the current time in my config file, then stop the auto service and then run the logstash file from command prompt.

What I want is, is there a way that without interrupting the config file schedule and without stopping the auto service I can run the configuration instantly???

My config below:

input {
    http_poller {
	id => "test-plugin"
    urls => {
	test_api => {
        method => "POST"
        url => "api url"
		headers => {
                    "Content-Type" => "application/json"
                }
      }
	}	
	request_timeout => 120
	schedule => {cron => "8 5 * * * UTC"}
	codec => "json"
	tags => ["test-api"]
  }
}
filter { 
	mutate {
		remove_field => [ "[message]" ]
		remove_field => [ "[@version]" ]
		remove_field => [ "[event]" ]
	}
}
output {
	if "test-api" in [tags]
	{
		elasticsearch {
			id => "test-output"
			hosts => ["host1"]
			user => "user"
			password => "password"
			index => "my-index"
			document_id => "%{ID}"
			doc_as_upsert => true
			action => "update"
		 }
	}
}
2

You can run multiple copies of logstash provided they use different data directories. If you want to run the task immediately then instead of using a cronline use schedule => "at now".

3

Thank You Badger.. But I cannot run this file manually without stopping the service. I do not want to stop the service.. Is there any way to do so???

4

Explicitly set --path.data to a different directory.

5

Hello Badger,

schedule => "at now" is not working.. giving an error....

 input {
    http_poller {
      # This setting must be a hash
      # This field must contain an even number of items, got 1
      schedule => "at now"
      ...
    }
  }

[2023-02-27T12:37:02,556][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.",
6

This thread shows a user trying schedule=> { "in" => 0 } so I would suggest schedule => { "at" => "now" }.

7

Hello badger,

This solution is not working.....

error:

 Plugin: <LogStash::Inputs::HTTP_Poller schedule=>{"at"=>"now"}, codec=><LogStash::Codecs::JSON id=>"json_8dcb0866-d4d0-48a7-9e81-f33de83cd040", enable_metric=>true, charset=>"UTF-8">, urls=>{"sector_api"=>{"headers"=>{"Content-Type"=>"application/json"}, "method"=>"POST", "body"=>"{\"SecretKey\":\"T8xmLetfdt3dlUiVayHwZK0q3nUsQ8zrS\"}", "url"=>"https://staging.stockedge.com/Api/GlobalSearchApi/GetSectorList"}}, request_timeout=>120, id=>"se-sector-input", tags=>["se-api-sector"], enable_metric=>true, socket_timeout=>10, connect_timeout=>10, follow_redirects=>true, pool_max=>50, pool_max_per_route=>25, keepalive=>true, automatic_retries=>1, retry_non_idempotent=>false, validate_after_inactivity=>200, keystore_type=>"JKS", ssl_verification_mode=>"full", truststore_type=>"JKS", cookies=>true, metadata_target=>"@metadata">
  Error: couldn't parse "now"
  Exception: ArgumentError
8

You are right, rufus does not support "at now". You will have to use "in 0". I have tested

input { http_poller { urls => { "first" => "http://www.google.com" } schedule => { in => 0 } codec => plain } }

and it works.

2 Likes
9

yes it is working..

Thank You badger.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.