Logger.logstash.outputs.elasticsearch not printing enough logs

Hi

I saw "logger.logstash.outputs.elasticsearch" : "DEBUG"
is printing debug level logs only during logstash start/stop.

But logstash not printing any logs when it parses an event and stores in elasticsearch

My output config looks like

output 
{
	if[eventID]=="002E000A"   
	{
  	 if[update_event]=="true"
           {
            elasticsearch 
            { 
                hosts => ["localhost:9200"]
               document_id => "%{authsid}"
               index =>  "dashboard_write"
               script => "ctx._source.loginCount= params.event.get('loginCount');
                       ctx._source.contractName= params.event.get('contractName');
                        ctx._source.userName= params.event.get('userName');
                         ctx._source.sessionID= params.event.get('sessionID');
                         ctx._source.eventID= params.event.get('eventID');"
              doc_as_upsert => "true"
               action => "update"        
            }
                 
        # stdout { codec => line {format => "Inserted Event Successfully"}}  
        # stdout { codec => rubydebug }
            }
          else
          {
  elasticsearch 
            {
  hosts => ["localhost:9200"]
               document_id => "%{sessionID}"
               index =>  "dashboard_write"
}

}
  	}

I would prefer some log messages once it inserted into index.
Am I missing something here?

I feel the document_id might be causing the problem. Try without it and put to stdout codec and see if you can print into console. (Do without any if conditions first and see)

Nope. It didnt help. Currently my output is changed as below

output
{

elasticsearch
{
hosts => ["localhost:9200"]
index => "dashboard_write"
}

}

Logs contain only below

[2020-08-03T08:06:25,649][DEBUG][logstash.filters.grok ][events] Event now: {:event=>#LogStash::Event:0x433e783e}
[2020-08-03T08:06:25,761][DEBUG][logstash.filters.grok ][events] Running grok filter {:event=>#LogStash::Event:0x433e783e}
[2020-08-03T08:06:25,769][DEBUG][logstash.filters.grok ][events] filters/LogStash::Filters::Grok: removing field {:field=>"accessManager"}
[2020-08-03T08:06:25,770][DEBUG][logstash.filters.grok ][events] Event now: {:event=>#LogStash::Event:0x433e783e}
[2020-08-03T08:06:25,784][DEBUG][logstash.filters.json ][events] Running json filter {:event=>#LogStash::Event:0x433e783e}
[2020-08-03T08:06:25,787][DEBUG][logstash.filters.json ][events] Event after json filter {:event=>#LogStash::Event:0x433e783e}
[2020-08-03T08:06:25,818][DEBUG][logstash.filters.grok ][events] Running grok filter {:event=>#LogStash::Event:0x433e783e}
[2020-08-03T08:06:25,822][DEBUG][logstash.filters.grok ][events] Event now: {:event=>#LogStash::Event:0x433e783e}
[2020-08-03T08:06:25,826][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::GeoIP: adding value to field {"field"=>"countryCode", "value"=>["%{[geoIP][country_code2]}"]}
[2020-08-03T08:06:25,830][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"loginCount", "value"=>[1]}
[2020-08-03T08:06:25,831][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"deviceName", "value"=>["%{[os_name][0]}"]}
[2020-08-03T08:06:25,832][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"deviceID", "value"=>["idp%{B}"]}
[2020-08-03T08:06:25,833][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"sessionID", "value"=>["%{[Y][0]}"]}
[2020-08-03T08:06:25,834][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"authsid", "value"=>["%{[Y][1]}"]}
[2020-08-03T08:06:25,835][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"risklevel", "value"=>["null"]}
[2020-08-03T08:06:25,836][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"failedCount", "value"=>["null"]}
[2020-08-03T08:06:25,837][DEBUG][logstash.util.decorators ][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"eventType", "value"=>["null"]}

I dont see any elasticsearch output related.

I have my input/filters/output config separate files with pipelines pointing to that directory. My pipeline.xml

  • pipeline.id: events
    path.config: "/etc/logstash/conf.d/events/"

Will that be an issue? Im using ELK 7.4.2