Send a notification mail for invalid inputs made to the index (filebeat)


I have been trying to work on the code to send an email to the support team for any invalid inputs such as (Invalid Credentials) into the system. A case being, when I try to use Putty and invalid credentials are entered by the user, an alert mail needs to be sent to the support team.

My Config file is as below

input {
		elasticsearch {
			hosts => 'localhost'
			index => 'filebeat-*'
output {
	email {
		to => ''
		from => ''
		subject => 'critical event spotted by ELK from '

However after multiple tries, I am unable to get this working. The error data is as below


WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logsash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/ Usingdefault config which logs errors to the console
[WARN ] 2021-01-05 16:36:23.128 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' fie because modules or command line options are specified
[INFO ] 2021-01-05 16:36:23.149 [LogStash::Runner] runner - Starting Logstash {"logstash.versio"=>"7.8.1", "jruby.version"=>"jruby (2.5.7) 2020-03-25 b1f55b1a40 OpenJDK 64-Bit Serve VM 25.131-b12 on 1.8.0_131-b12 +indy +jit [linux-x86_64]"}
[INFO ] 2021-01-05 16:36:27.281 [Converge PipelineAction::Create<main>] Reflections - Reflectios took 69 ms to scan 1 urls, producing 21 keys and 41 values
[INFO ] 2021-01-05 16:36:30.243 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pieline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50 "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/email.conf"], :thread>"#<Thread:0x9e9f462 run>"}
[INFO ] 2021-01-05 16:36:33.440 [[main]-pipeline-manager] javapipeline - Pipeline started {""=>"main"}
[INFO ] 2021-01-05 16:36:33.676 [Agent thread] agent - Pipelines running {:count=>1, :running_ppelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2021-01-05 16:36:35.269 [Api Webserver] agent - Successfully started Logstash API endpont {:port=>9600}
[ERROR] 2021-01-05 16:39:54.464 [[main]<elasticsearch] elasticsearch - Scroll request error, abrting scroll {:error=>"#<Elasticsearch::Transport::Transport::Errors::NotFound: [404] {\"error\:{\"root_cause\":[{\"type\":\"search_context_missing_exception\",\"reason\":\"No search contextfound for id [3690]\"},{\"type\":\"search_context_missing_exception\",\"reason\":\"No search cotext found for id [3689]\"},{\"type\":\"search_context_missing_exception\",\"reason\":\"No searh context found for id [3691]\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"al shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":-1,\"index\:null,\"reason\":{\"type\":\"search_context_missing_exception\",\"reason\":\"No search context ound for id [3690]\"}},{\"shard\":-1,\"index\":null,\"reason\":{\"type\":\"search_context_missig_exception\",\"reason\":\"No search context found for id [3689]\"}},{\"shard\":-1,\"index\":nul,\"reason\":{\"type\":\"search_context_missing_exception\",\"reason\":\"No search context foun for id [3691]\"}}],\"caused_by\":{\"type\":\"search_context_missing_exception\",\"reason\":\"N search context found for id [3691]\"}},\"status\":404}>"}
[WARN ] 2021-01-05 16:39:54.665 [[main]<elasticsearch] elasticsearch - Ignoring clear_scroll exeption {:message=>"[404] {\"succeeded\":true,\"num_freed\":0}"}
[INFO ] 2021-01-05 16:41:25.769 [LogStash::Runner] runner - Logstash shut down.

Request you help in this regard


Let's continue the discussion in this topic Take input from elk index and need to send notification over email against invalid inputs