Elasticsearch's Docuements count dosen't match the exact number of input log lines

Elastic Stack Elasticsearch
1

have a log file containing 2044 lines, but after indexing into Elasticsearch using Logstash, I find only 2043 documents. I suspect that an empty line in the log file might have been skipped by Logstash during indexing.

I'm seeking advice on how to verify if this empty line was indeed ignored by Logstash during processing.

I've checked Logstash logs, but I haven't found specific indications regarding this empty line. Additionally, I've attempted to start Logstash in debug mode, but I'm uncertain what to look for in the logs.

How can I check if an empty line is being skipped by Logstash during indexing into Elasticsearch?

2

Hi @Dokh_Ahmed,

Do you have a rough idea which entry might be being skipped to look in Elasticsearch? Otherwise you could check for messages referencing skipping in the debug logs as you suggest.

4

Hi @carly
The last log in my file is not indexed as a document!

5

What does you Logstash input configuration look like? Does the last line end in a newline?

6

Hi @Christian and thanks for your interaction.
Here's my logstash config file

input {
  file {
    start_position => "beginning"
    path => "D:/estbo-esb.log"
    sincedb_path => "NUL"
  }
}

filter {
  grok {
    pattern_definitions => {
      "Extract_log_line" => "%{TIMESTAMP_ISO8601:timestamp} \|%{SPACE}*%{LOGLEVEL:log_level}%{SPACE}*\|%{SPACE}\[%{DATA:thread}\]%{SPACE}---%{SPACE}%{NUMBER:process_id}%{SPACE}\|%{SPACE}%{DATA:class}%{SPACE}\| \|%{SPACE}%{GREEDYDATA:log_message}(\\r|\\n)? "
    }
    match => {
      "message" => [
        "%{Extract_log_line}"
      ]
    }
  }
   if [log_message] =~ /^Started (SEND|GENERATE|route)/ {
      mutate {
        add_field => { "Started_Routes" => "true" }
      }
    }

   if [log_message] =~ /^Disabled/ {
       mutate {
         add_field => { "Disabled_Routes" => "true" }
       }
     }
  mutate {
    remove_field => ["@timestamp", "@version"]
  }
}




output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => "http://localhost:9200"
    user => "elastic"
    password => "_2k4SDOkGjpJgS*BIyIc"
    index => "estbo_logs"
  }
}
7

OK, so you are running the file input plugin in tail mode, which is the default value. This requires, as shown in the docs I linked to, that each line ends in a newline. Does the last line in the log file end in a newline?

8

no, the last line of my log file does not end with a newline

9

Then that is why it is missing. If you have a static file that is not being appended to you can change to read mode, which should fix this issue. If you are appending you need to make sure every line ends with a newline to avoid this issue.

10

Can i just modify the grok pattern ?

11

Thank youuuuu @Christian_Dahlqvist my problem was solved

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.