# Logger.logstash.outputs.elasticsearch not printing enough logs **URL:** \ **Category:** Logstash\ **Created:** [August 1, 2020, 10:13am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420 "2020-08-01T10:13:04Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 1, 2020, 10:13am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/1 "2020-08-01T10:13:04Z")
Hi I saw **"logger.logstash.outputs.elasticsearch" : "DEBUG"** is printing debug level logs only during logstash start/stop. But logstash not printing any logs when it parses an event and stores in elasticsearch My output config looks like ``` output { if[eventID]=="002E000A" { if[update_event]=="true" { elasticsearch { hosts => ["localhost:9200"] document_id => "%{authsid}" index => "dashboard_write" script => "ctx._source.loginCount= params.event.get('loginCount'); ctx._source.contractName= params.event.get('contractName'); ctx._source.userName= params.event.get('userName'); ctx._source.sessionID= params.event.get('sessionID'); ctx._source.eventID= params.event.get('eventID');" doc_as_upsert => "true" action => "update" } # stdout { codec => line {format => "Inserted Event Successfully"}} # stdout { codec => rubydebug } } else { elasticsearch { hosts => ["localhost:9200"] document_id => "%{sessionID}" index => "dashboard_write" } } } ``` I would prefer some log messages once it inserted into index. Am I missing something here? ---
**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\ **Post date:** [August 1, 2020, 6:26pm UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/2 "2020-08-01T18:26:11Z")
> [@RAM\_NATHAN](#): > > `document_id` I feel the `document_id` might be causing the problem. Try without it and put to stdout codec and see if you can print into console. (Do without any if conditions first and see) ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 3, 2020, 2:39am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/3 "2020-08-03T02:39:30Z")
Nope. It didnt help. Currently my output is changed as below output { elasticsearch { hosts =\> ["localhost:9200"] index =\> "dashboard\_write" } } Logs contain only below [2020-08-03T08:06:25,649][DEBUG][logstash.filters.grok][events] Event now: {:event=\>#LogStash::Event:0x433e783e} [2020-08-03T08:06:25,761][DEBUG][logstash.filters.grok][events] Running grok filter {:event=\>#LogStash::Event:0x433e783e} [2020-08-03T08:06:25,769][DEBUG][logstash.filters.grok][events] filters/LogStash::Filters::Grok: removing field {:field=\>"accessManager"} [2020-08-03T08:06:25,770][DEBUG][logstash.filters.grok][events] Event now: {:event=\>#LogStash::Event:0x433e783e} [2020-08-03T08:06:25,784][DEBUG][logstash.filters.json][events] Running json filter {:event=\>#LogStash::Event:0x433e783e} [2020-08-03T08:06:25,787][DEBUG][logstash.filters.json][events] Event after json filter {:event=\>#LogStash::Event:0x433e783e} [2020-08-03T08:06:25,818][DEBUG][logstash.filters.grok][events] Running grok filter {:event=\>#LogStash::Event:0x433e783e} [2020-08-03T08:06:25,822][DEBUG][logstash.filters.grok][events] Event now: {:event=\>#LogStash::Event:0x433e783e} [2020-08-03T08:06:25,826][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::GeoIP: adding value to field {"field"=\>"countryCode", "value"=\>["%{[geoIP][country\_code2]}"]} [2020-08-03T08:06:25,830][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"loginCount", "value"=\>[1]} [2020-08-03T08:06:25,831][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"deviceName", "value"=\>["%{[os\_name][0]}"]} [2020-08-03T08:06:25,832][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"deviceID", "value"=\>["idp%{B}"]} [2020-08-03T08:06:25,833][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"sessionID", "value"=\>["%{[Y][0]}"]} [2020-08-03T08:06:25,834][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"authsid", "value"=\>["%{[Y][1]}"]} [2020-08-03T08:06:25,835][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"risklevel", "value"=\>["null"]} [2020-08-03T08:06:25,836][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"failedCount", "value"=\>["null"]} [2020-08-03T08:06:25,837][DEBUG][logstash.util.decorators][events] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"eventType", "value"=\>["null"]} I dont see any elasticsearch output related. I have my input/filters/output config separate files with pipelines pointing to that directory. My pipeline.xml - pipeline.id: events path.config: "/etc/logstash/conf.d/events/" Will that be an issue? Im using ELK 7.4.2 ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 5, 2020, 5:08am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/4 "2020-08-05T05:08:27Z")
Can someone reply please? ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 7, 2020, 6:08am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/5 "2020-08-07T06:08:47Z")
Can someone help here please? ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 10, 2020, 2:19am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/6 "2020-08-10T02:19:48Z")
We are blocked. It will be really helpful if someone throw some light here ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2020, 1:31pm UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/7 "2020-08-10T13:31:11Z")
If you enable ``` stdout { codec => rubydebug } ``` are you seeing events on stdout? Are you running with the single unconditional elasticsearch output now? If so, are there any documents in the index? ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 11, 2020, 2:42am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/8 "2020-08-11T02:42:03Z")
Yes If I enable stdout, Im able to see events in logstash log. Right now I removed all conditional statements DELETE dashboard\_write I deleted index first and I generated the events. Im able to see that in index as below > { > "\_index" : "dashboard\_write", > "\_type" : "\_doc", > "\_id" : "gL9j23MBZwpRwXjSGiON", > "\_score" : 1.0, > "\_source" : { > "deviceName" : "Windows", > "eventID" : "\*\*\*_", > "browserName" : "Chrome", > "sourceIP" : "**", > "authsid" : "89999", > "contractName" : "kkkk", > "eventType" : "null", > "userName" : "admin", > "createDate" : "2020-08-11T02:39:31.944Z", > "deviceID" : "vyn", > "sessionID" : "**_", > "countryCode" : "US", > "loginCount" : 1, > "failedCount" : 0 > } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 11, 2020, 3:18pm UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/9 "2020-08-11T15:18:51Z")
OK, so documents are being written to the index. Is your issue just that there are not any DEBUG logs? If so, well, that is true. The [manticore\_adapter](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb) is passed the logger, but never uses it. [pool.rb](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/http_client/pool.rb) only logs when the pool of connections changes. The [output](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/http_client.rb) itself does not log anything when processing events normally. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 11, 2020, 4:35pm UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/10 "2020-08-11T16:35:10Z")
Thanks for the update. My requirement is to have logs even in case of successful indexing. We need that for troubleshooting. I thought **logger.logstash.outputs.elasticsearch" : "DEBUG** setting would help here. Is there any other way? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 11, 2020, 5:05pm UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/11 "2020-08-11T17:05:56Z")
There is no per event logging in the elasticsearch output. It does maintain some [metrics](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-enable_metric) around the number of response to bulk request, but I cannot tell you anything about them or how to access them. The elasticsearch output is built on top of cheald/manticore, and that is built on top of the Apache httpClient. I believe httpClient has [logging](http://hc.apache.org/httpcomponents-client-ga/logging.html) but that is low level. (I do not know what version of httpClient the output is built upon.) ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [August 12, 2020, 3:14am UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/12 "2020-08-12T03:14:28Z")
Ok . My overall requirement somehow I need to know logstash has received/processed events. Im happy even I will be able to print the event once it received in input or filter, even if not in output. Whole exercise is to make sure events are reaching logstash. I tried to set logstash.inputs.syslog to DEBUG. I thought it will print event/log once it recieved. But it didn't Is there any other way? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 12, 2020, 2:54pm UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/13 "2020-08-12T14:54:50Z")
> [@RAM\_NATHAN](#): > > My overall requirement somehow I need to know logstash has received/processed events. Take a look at the [pipeline stats](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats) API. That will tell you whether events are flowing through the pipelines and outputs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 9, 2020, 2:54pm UTC](https://discuss.elastic.co/t/logger-logstash-outputs-elasticsearch-not-printing-enough-logs/243420/14 "2020-09-09T14:54:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.