Logstash never picked up any logs. I mean it never parse any logs except 2 log lines for the entire day

Elastic Stack Logstash
1

I have using logstash 8.10.0 for windows and linux downloaded from elastic.co downloads.
I could hardly see 2 events ingested by logstash.
logstash-plain.log does not have any error after pipeline_running.

image

configuration

input {
  file {
    path => "C:/setup/log/abbmq1*"
    type => "abbmq1"
    start_position => "beginning"
  }
  filter {
  if [type] == "abbmq1" {
    grok {
      match => {"message" => "%{DATA:logtimeStamp} %{WORD:operation} %{GREEDYDATA:message}"}
      overwrite => [ "message"]
    }
    mutate {
      remove_field => [ "logtimeStamp" ]
    }
  }
  }
  output {
  stdout {}
  }

please help.

2

What does the message on those files looks like? Each file has many lines or just one line per file?

3

C:/setup/log/abbmq1.log the file. It has many log lines. I have checked it manually.

4

And how are you running logstash? Which user? Does the user have permissions on the folder path?

What do you have in Logstash logs?

I do not use windows, so I'm not sure which user logstash uses if you run it as a service, but if the paht is correct and the permissions are also correct, it should've read the files.

Can you restart logstash and share fresh logs?

5

@leandrojmp thanks for looking into this one.
I would like to provide linux config details.

[root@hostname]# cat .sincedb_2801e934a5da4805e0d8082e0c062138
10783905 0 66305 14164630 1695906657.855361 /opt/tomcat.log

input {
  file {
    path => "/opt/tomcat/logs/cs.log"
    type => "cs"
    start_position => "beginning"
  }
 }
filter {
if [type] == "cs"{
    grok {
            match => {
                    "message" => [
                            "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{DATA:thread}\] %{SPACE}\[\] %{SPACE}\[%{URIPATH:uri_path}\]%{SPACE}\[\] %{GREEDYDATA:message}",
                            "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{DATA:thread}\] %{SPACE}\[\] %{SPACE}\[\]%{SPACE}\[\] %{SPACE}\[\]%{GREEDYDATA:message}",
                            "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{GREEDYDATA:thread}\] %{SPACE}\[\] %{SPACE}\[\]%{SPACE} \[\] %{SPACE}\[\]%{SPACE}%{DATA:logger}%{SPACE}\[\] - %{GREEDYDATA:message}",
                            "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{SPACE}\[%{DATA:thread}\]%{SPACE}\[%{DATA:tenant}\]%{SPACE}\[\]%{SPACE}\[%{DATA:class}\]%{SPACE}\[\] es.CSUtil%{SPACE}\[\]%{SPACE}- This condition should not occur for the tenant:%{SPACE}%{GREEDYDATA:tenant_value}"
                    ]
            }
            overwrite => ["message"]
    }
}
}

I run logstash as service 


[root@hostname file]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash service

[Service]
Type=simple
User=root
Group=root
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/logstash-8.10.0/bin/logstash "-f" "/usr/share/logstash/logstash-8.10.0/conf.d"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

When never I restart one log line getting indexed. But when I run in debug mode it does emit all the logs.

6

You need to share logs.

I do not see anything wrong in your linux configuration, just the fact that you are running logstash as the root user in the service, which is not recommended and it may be a security issue.

Not sure what you mean with that, can you explain?

Also, if logstash already read lines of a file, it will not read those lines again, only new lines.

7
  1. Set your own sincedb or use /dev/null temporarily in sincedb_path
input {
  file {
    path => "/opt/tomcat/logs/cs.log"
    type => "cs"
    start_position => "beginning"
    sincedb_path => "/path/since.db"
  }
 }
  1. Remove if [type] == "abbmq1" {, just for test and I don't see the point if is only one grok.
  2. Are paths OK?
  • 10783905 0 66305 14164630 1695906657.855361 /opt/tomcat.log
  • path => "/opt/tomcat/logs/cs.log"
  1. Check rights on LS somefile.conf, does LS can read. You will see is it picked up in debug mode.
  2. Set debug mode log.level: debug in logstash.yml
8

Thanks @leandrojmp I use latest logstash 8.10.0.
When I stdout the logs I could see many log lines ( I mean event/document)
But in ES I could see only one event
How do I verify if the logs are been sent from logstash-> ELK

9

@Rios In debug mode, I could stdout the logs. Is there any other ways to see if logstash is sending the logs. Like sincedb file.

10

Is this when you run in the service mode or the process?

When you run as the command, LS run under that the logged in user. The service mode is under the logstash user. If is this case, that means your the name.conf file doesn't have proper rights.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.