Hi,
I'm fairly new working with logstash (and actually the entire ELK components), but am trying to determine why some logs are not being ingested and indexed.
While I was investigating this, I noticed that on the machine where logstash is running, there are large number of files in the /var/log/logstash directory.
These files are logstash-plain-YYYY-MM-DD.log.gz and logstash-deprecation-YYYY-MM-DD-log.gz. It looks like there are one of each file being created each day.
I was looking at the contents of these files and it appears that the contents of these files parallel the logs that we believe are not being ingested.
Here's some of the lines from one of the logstash-plain .gz files:
./logstash-plain-2023-02-14-17.log.gz:[2023-02-14T12:23:39,645][DEBUG][logstash.filters.grok ][main][f56af23d4a177f04b6716dd44a03a631809face949522f99274125b028490828] Running grok filter {:event=>{"ecs"=>{"version"=>"1.12.0"}, "log"=>{"file"=>{"path"=>"/apps/common/logs/foo/wls_foo4/wls_foo4-servlet.log"}, "offset"=>20969649}, "agent"=>{"version"=>"7.16.3", "hostname"=>"server04", "ephemeral_id"=>"27222079-9a34-4e20-b897-7547d5bfe42b", "id"=>"776866d3-ce15-4f38-80e0-031c81a076a0", "name"=>"server04", "type"=>"filebeat"}, "@version"=>"1", "input"=>{"type"=>"log"}, "event"=>{"timezone"=>"+00:00"}, "@timestamp"=>2023-02-14T12:22:45.069Z, "tags"=>["beats_input_codec_plain_applied"], "fields"=>{"sourceEnv"=>"server04", "log_type"=>"servlet_log"}, "host"=>{"name"=>"server04"}, "message"=>"2023-02-08 17:30:18,673 [[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'] TRACE [fooservlet.audit.Auditor:logEvent] - ECID:df1effd9-e184-4f70-9dfa-c777ae33aee5-00000076|SID:hC0yFAQ0En4Pk4p3vGWXiOC2SP5rUGH9YjvL1ZOQA0R-zWclltgr!1191411166!1961774563!1675877418036|EID:6|ENAME:Authentication|STATUS:true|br://fooServlet/protected|wh:fooServlet|rh:SOAP_ATN_SERVICE|ru:%2Fprotected|rq:-|rr:-|oc:-|ai:-|zi:-|ip:-|si:20a10559-6eb4-477d-a1d1-86ee002cdd26|g1MisxKtKobNuNLE325DilB+8szvY4SYlxqObNu8zUc=|sd:cn=ntc.xxx.com,dc=foo,dc=com|su:ntc.xxx.com|is:LDAP|ca_1:USA|ca_2:LEVL"}}
./logstash-plain-2023-02-14-17.log.gz:[2023-02-14T12:23:39,645][DEBUG][logstash.filters.grok ][main][f56af23d4a177f04b6716dd44a03a631809face949522f99274125b028490828] Event now: {:event=>{"SSOSessionID"=>"20a10559-6eb4-477d-a1d1-86ee002cdd26|g1MisxKtKobNuNLE325DilB+8szvY4SYlxqObNu8zUc=", "second"=>"18", "Consumer"=>"fooServlet", "IdentityStore"=>"LDAP", "ResourceID"=>"//fooServlet/protected", "event"=>{"timezone"=>"+00:00"}, "@timestamp"=>2023-02-14T12:22:45.069Z, "nationalityextended"=>"USA", "minute"=>"30", "Username"=>"ntc.xxx.com", "tags"=>["beats_input_codec_plain_applied"], "ResourcePath"=>"%2Fprotected", "year"=>"2023", "ECID"=>"df1effd9-e184-4f70-9dfa-c777ae33aee5-00000076", "Timestamp"=>"2023-02-08 17:30:18,673", "ecs"=>{"version"=>"1.12.0"}, "log"=>{"file"=>{"path"=>"/apps/common/logs/foo/wls_foo4/wls_foo4-servlet.log"}, "offset"=>20969649}, "SessionID"=>"hC0yFAQ0En4Pk4p3vGWXiOC2SP5rUGH9YjvL1ZOQA0R-zWclltgr!1191411166!1961774563!1675877418036", "hour"=>"17", "agent"=>{"version"=>"7.16.3", "hostname"=>"server04", "ephemeral_id"=>"27222079-9a34-4e20-b897-7547d5bfe42b", "id"=>"776866d3-ce15-4f38-80e0-031c81a076a0", "name"=>"server04", "type"=>"filebeat"}, "EventType"=>"Authentication", "privilege"=>"LEVL", "@version"=>"1", "UserDN"=>"cn=ntc.xxx.com,dc=foo,dc=com", "input"=>{"type"=>"log"}, "day"=>"08", "EventStatus"=>"true", "month"=>"02", "fields"=>{"sourceEnv"=>"server04", "log_type"=>"servlet_log"}, "host"=>{"name"=>"server04"}, "message"=>"2023-02-08 17:30:18,673 [[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'] TRACE [fooservlet.audit.Auditor:logEvent] - ECID:df1effd9-e184-4f70-9dfa-c777ae33aee5-00000076|SID:hC0yFAQ0En4Pk4p3vGWXiOC2SP5rUGH9YjvL1ZOQA0R-zWclltgr!1191411166!1961774563!1675877418036|EID:6|ENAME:Authentication|STATUS:true|br://fooServlet/protected|wh:fooServlet|rh:SOAP_ATN_SERVICE|ru:%2Fprotected|rq:-|rr:-|oc:-|ai:-|zi:-|ip:-|si:20a10559-6eb4-477d-a1d1-86ee002cdd26|g1MisxKtKobNuNLE325DilB+8szvY4SYlxqObNu8zUc=|sd:cn=ntc.xxx.com,dc=foo,dc=com|su:ntc.xxx.com|is:LDAP|ca_1:USA|ca_2:LEVL"}}
Can someone explain to me what is causing these files to be created? Also, are these corresponding to the logs that appear to not being ingested, and if so, what do we need to do so these get ingested instead?
Thanks,
Jim
P.S. Apologies in advance if my terminology is not correct...as I said, I am pretty new with this.