Logs not moved from Logstash to Elasticsearch

Elastic Stack Logstash
#1

Dear All,

I have installed ELK on single server. Now I am just trying to read a sample log file in my tmp folder andoutput it to elasticsearch and display in Kibana .

Below is my logstash configuration file:

input {
file {
path => "\tmp\Test.log"
start_position => "beginning"
}
}

output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

But in Kibana I am not able to view the contents.

I checked Elasticsearch data directory :/var/lib/elasticsearch/elasticsearch/nodes and found 0 files in it. I understand logs not moved from logstash to elasticsearch. Please advice.

#2

Have a read of https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files

#3

Mark Thanks for reply. I verified the link and did --configtest the configuration file and It seems to be OK.

[root@ip-10-32-71-115 bin]# ./logstash --configtest -f /etc/logstash/conf.d/syslog-elasticsearch.conf
Picked up _JAVA_OPTIONS: -Xmx2048m -Xms1024m
Configuration OK

Please suggest what else could be the other reasons.

#4

That is not what the link was about. Read about sincedb.

#5

I have updated config file with sincedb details. But still not seeing the logs moved to elasticsearch. Please advice.

input {
file {
path => "/tmp/Test.log"
type => "log"
start_position => "beginning"
sincedb_path => "/tmp/loc"
sincedb_write_interval => 10
}
}

#6

@manojvenkat

  • What version of LS are you using?
  • Is the last modified date of the file /tmp/Test.log more than 24 hours ago? if yes then either touch it or set ignore_older to zero in the file config.
#7

Hi Guyboertje, Thanks for reply.

  1. Logstash Version is 2.4
    2.I created Test.log file few hours back.
#8