What are logstash-plain-YYYY-MM-DD.log.gz and logstash-deprecation-YYYY-MM-DD.log.gz?

Elastic Stack Logstash
1

Hi,

I'm fairly new working with logstash (and actually the entire ELK components), but am trying to determine why some logs are not being ingested and indexed.

While I was investigating this, I noticed that on the machine where logstash is running, there are large number of files in the /var/log/logstash directory.

These files are logstash-plain-YYYY-MM-DD.log.gz and logstash-deprecation-YYYY-MM-DD-log.gz. It looks like there are one of each file being created each day.

I was looking at the contents of these files and it appears that the contents of these files parallel the logs that we believe are not being ingested.

Here's some of the lines from one of the logstash-plain .gz files:

./logstash-plain-2023-02-14-17.log.gz:[2023-02-14T12:23:39,645][DEBUG][logstash.filters.grok    ][main][f56af23d4a177f04b6716dd44a03a631809face949522f99274125b028490828] Running grok filter {:event=>{"ecs"=>{"version"=>"1.12.0"}, "log"=>{"file"=>{"path"=>"/apps/common/logs/foo/wls_foo4/wls_foo4-servlet.log"}, "offset"=>20969649}, "agent"=>{"version"=>"7.16.3", "hostname"=>"server04", "ephemeral_id"=>"27222079-9a34-4e20-b897-7547d5bfe42b", "id"=>"776866d3-ce15-4f38-80e0-031c81a076a0", "name"=>"server04", "type"=>"filebeat"}, "@version"=>"1", "input"=>{"type"=>"log"}, "event"=>{"timezone"=>"+00:00"}, "@timestamp"=>2023-02-14T12:22:45.069Z, "tags"=>["beats_input_codec_plain_applied"], "fields"=>{"sourceEnv"=>"server04", "log_type"=>"servlet_log"}, "host"=>{"name"=>"server04"}, "message"=>"2023-02-08 17:30:18,673 [[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'] TRACE [fooservlet.audit.Auditor:logEvent] - ECID:df1effd9-e184-4f70-9dfa-c777ae33aee5-00000076|SID:hC0yFAQ0En4Pk4p3vGWXiOC2SP5rUGH9YjvL1ZOQA0R-zWclltgr!1191411166!1961774563!1675877418036|EID:6|ENAME:Authentication|STATUS:true|br://fooServlet/protected|wh:fooServlet|rh:SOAP_ATN_SERVICE|ru:%2Fprotected|rq:-|rr:-|oc:-|ai:-|zi:-|ip:-|si:20a10559-6eb4-477d-a1d1-86ee002cdd26|g1MisxKtKobNuNLE325DilB+8szvY4SYlxqObNu8zUc=|sd:cn=ntc.xxx.com,dc=foo,dc=com|su:ntc.xxx.com|is:LDAP|ca_1:USA|ca_2:LEVL"}}

./logstash-plain-2023-02-14-17.log.gz:[2023-02-14T12:23:39,645][DEBUG][logstash.filters.grok    ][main][f56af23d4a177f04b6716dd44a03a631809face949522f99274125b028490828] Event now:  {:event=>{"SSOSessionID"=>"20a10559-6eb4-477d-a1d1-86ee002cdd26|g1MisxKtKobNuNLE325DilB+8szvY4SYlxqObNu8zUc=", "second"=>"18", "Consumer"=>"fooServlet", "IdentityStore"=>"LDAP", "ResourceID"=>"//fooServlet/protected", "event"=>{"timezone"=>"+00:00"}, "@timestamp"=>2023-02-14T12:22:45.069Z, "nationalityextended"=>"USA", "minute"=>"30", "Username"=>"ntc.xxx.com", "tags"=>["beats_input_codec_plain_applied"], "ResourcePath"=>"%2Fprotected", "year"=>"2023", "ECID"=>"df1effd9-e184-4f70-9dfa-c777ae33aee5-00000076", "Timestamp"=>"2023-02-08 17:30:18,673", "ecs"=>{"version"=>"1.12.0"}, "log"=>{"file"=>{"path"=>"/apps/common/logs/foo/wls_foo4/wls_foo4-servlet.log"}, "offset"=>20969649}, "SessionID"=>"hC0yFAQ0En4Pk4p3vGWXiOC2SP5rUGH9YjvL1ZOQA0R-zWclltgr!1191411166!1961774563!1675877418036", "hour"=>"17", "agent"=>{"version"=>"7.16.3", "hostname"=>"server04", "ephemeral_id"=>"27222079-9a34-4e20-b897-7547d5bfe42b", "id"=>"776866d3-ce15-4f38-80e0-031c81a076a0", "name"=>"server04", "type"=>"filebeat"}, "EventType"=>"Authentication", "privilege"=>"LEVL", "@version"=>"1", "UserDN"=>"cn=ntc.xxx.com,dc=foo,dc=com", "input"=>{"type"=>"log"}, "day"=>"08", "EventStatus"=>"true", "month"=>"02", "fields"=>{"sourceEnv"=>"server04", "log_type"=>"servlet_log"}, "host"=>{"name"=>"server04"}, "message"=>"2023-02-08 17:30:18,673 [[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'] TRACE [fooservlet.audit.Auditor:logEvent] - ECID:df1effd9-e184-4f70-9dfa-c777ae33aee5-00000076|SID:hC0yFAQ0En4Pk4p3vGWXiOC2SP5rUGH9YjvL1ZOQA0R-zWclltgr!1191411166!1961774563!1675877418036|EID:6|ENAME:Authentication|STATUS:true|br://fooServlet/protected|wh:fooServlet|rh:SOAP_ATN_SERVICE|ru:%2Fprotected|rq:-|rr:-|oc:-|ai:-|zi:-|ip:-|si:20a10559-6eb4-477d-a1d1-86ee002cdd26|g1MisxKtKobNuNLE325DilB+8szvY4SYlxqObNu8zUc=|sd:cn=ntc.xxx.com,dc=foo,dc=com|su:ntc.xxx.com|is:LDAP|ca_1:USA|ca_2:LEVL"}}

Can someone explain to me what is causing these files to be created? Also, are these corresponding to the logs that appear to not being ingested, and if so, what do we need to do so these get ingested instead?

Thanks,
Jim

P.S. Apologies in advance if my terminology is not correct...as I said, I am pretty new with this.

2

Those are just the normal logs where Logstash will write stuff about its own operation (like start and stop, connection problems, etc). The name of the file(s), their format, and when they are rotated, gzipped (compressed) and deleted is controlled by a file called log4j2.properties (a text file) on your disk. Could be in /etc/logstash, could be somewhere in /usr/share/logstash or wherever you installed the software.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.