Data sending to ES without using Filebeat

Elastic Stack Logstash
1

HI,
I want to send data from logstash to ES without filebeat. I use docker command to start the logstash service and log is JSON. pLease find the below code and tell me where im commiting mistake,
input{
file{
path => "/root/logstash/logs/*.log"
#type => "json"
}
}
filter {
grok {
match => [ "message", "%{GREEDYDATA:msg}" ]
}
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["xxxxxx:9200"]
index => "report-sample-%{+YYYY.MM.dd}"
workers => 1
}
}

2

There's nothing obviously wrong with your configuration, but:

  • Your grok filter is rather useless.
  • You have configured the file input to tail the input files. If you want the files to be read from the top you'll have to make some adjustments. See the file input documentation and the countless previous threads on this topic.
  • Are the log files actually available in /root/logstash/logs inside the container? Does the user that Logstash runs as have access to those files?
3
  • Your grok filter is rather useless.
    Removed this one
  • You have configured the file input to tail the input files. If you want the files to be read from the top you'll have to make some adjustments. See the file input documentation and the countless previous threads on this topic.
    added starting_postition => "beginning"
  • Are the log files actually available in /root/logstash/logs inside the container ? Does the user that Logstash runs as have access to those files?
    yes i have provide that chmod 775
4

added starting_postition => "beginning"

Okay, but make sure you clear the current sincedb state (if any). start_position only matters the first time Logstash discovers a file.

yes i have provide that chmod 775

If you increase Logstash's loglevel it'll tell you what the filename patterns expand to. That's a simple way of checking if the problem is that Logstash can't find or read the input files.

5

how to increase the logstash's loglevel?

6

There's a command line options for that, or you can set it via logstash.yml. Both methods are described in the documentation.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.