Take input from elk index and need to send notification over email against invalid inputs

I m a newbie in elk and i have been trying the below code from quite sometime.

This is the configuration file i have

input {
        elasticsearch {
    hosts => 'localhost'
    index => 'filebeat-*'


output {

email {
                        to => 'jerry@gmail.com'
                        from => 'localhost@otrs.com'
                        subject => 'critical event spotted by ELK from '


This is the error i m getting.

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logsash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Usingdefault config which logs errors to the console
[WARN ] 2021-01-05 16:36:23.128 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' fie because modules or command line options are specified
[INFO ] 2021-01-05 16:36:23.149 [LogStash::Runner] runner - Starting Logstash {"logstash.versio"=>"7.8.1", "jruby.version"=>"jruby (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Serve VM 25.131-b12 on 1.8.0_131-b12 +indy +jit [linux-x86_64]"}
[INFO ] 2021-01-05 16:36:27.281 [Converge PipelineAction::Create<main>] Reflections - Reflectios took 69 ms to scan 1 urls, producing 21 keys and 41 values
[INFO ] 2021-01-05 16:36:30.243 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pieline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50 "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/email.conf"], :thread>"#<Thread:0x9e9f462 run>"}
[INFO ] 2021-01-05 16:36:33.440 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipline.id"=>"main"}
[INFO ] 2021-01-05 16:36:33.676 [Agent thread] agent - Pipelines running {:count=>1, :running_ppelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-01-05 16:36:35.269 [Api Webserver] agent - Successfully started Logstash API endpont {:port=>9600}
[ERROR] 2021-01-05 16:39:54.464 [[main]<elasticsearch] elasticsearch - Scroll request error, abrting scroll {:error=>"#<Elasticsearch::Transport::Transport::Errors::NotFound: [404] {\"error\:{\"root_cause\":[{\"type\":\"search_context_missing_exception\",\"reason\":\"No search contextfound for id [3690]\"},{\"type\":\"search_context_missing_exception\",\"reason\":\"No search cotext found for id [3689]\"},{\"type\":\"search_context_missing_exception\",\"reason\":\"No searh context found for id [3691]\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"al shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":-1,\"index\:null,\"reason\":{\"type\":\"search_context_missing_exception\",\"reason\":\"No search context ound for id [3690]\"}},{\"shard\":-1,\"index\":null,\"reason\":{\"type\":\"search_context_missig_exception\",\"reason\":\"No search context found for id [3689]\"}},{\"shard\":-1,\"index\":nul,\"reason\":{\"type\":\"search_context_missing_exception\",\"reason\":\"No search context foun for id [3691]\"}}],\"caused_by\":{\"type\":\"search_context_missing_exception\",\"reason\":\"N search context found for id [3691]\"}},\"status\":404}>"}
[WARN ] 2021-01-05 16:39:54.665 [[main]<elasticsearch] elasticsearch - Ignoring clear_scroll exeption {:message=>"[404] {\"succeeded\":true,\"num_freed\":0}"}
[INFO ] 2021-01-05 16:41:25.769 [LogStash::Runner] runner - Logstash shut down.

request your assistance in this regards

This looks exactly the same as Send a notification mail for invalid inputs made to the index (filebeat)?

The reason is that was my friend's query, but was removed as it was considered a spam. So posted it from my account.

You probably want a query in there to reduce the response size that you get.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.