Filebeat log Not Coming to Logstash

[sumitsahay (Sumit)]

Hi All,

I am having fliebeat install on my client machine and elk masternode on different vm. I Configured everything though logs are not coming to Kibana index and index is not getting created.

[dadoonet (David Pilato)]

Welcome!

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

[sumitsahay (Sumit)]

Thanks a lot for your reply, I 'll do that here as suggested by you, I am new to this forum.
indent preformatted text by 4 spaces
`input {
beats {
port => 5044
type => syslog
ssl_certificate => "/etc/ssl/logstash_frwrd.crt"
ssl_key => "/etc/ssl/logstash-forwarder.key"
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
#index => ["waf"]
index => ["waf-(date '+%Y-%m-%d_%H-%M-%S')"]
}
stdout { codec => rubydebug }
}`
filebeat.inputs:

• input_type: log
  paths:

  • /var/log/*.log

  • /opt/test.log
    output.logstash:
    #output:
    #The Logstash hosts
    logstash:
    enabled: true
    hosts: ["172.31.192.3:5044"]
    index: "waf-(date '+%Y-%m-%d_%H-%M-%S')"
    tls:
    ssl.certificate: ["/etc/ssl/logstash_frwrd.crt"]
    ssl.key: "/etc/ssl/logstash-forwarder.key"

    `

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

[sumitsahay (Sumit)]

Is that ok.
<

input {
  beats {
    port => 5044
    type => syslog
    ssl_certificate => "/etc/ssl/logstash_frwrd.crt"
    ssl_key => "/etc/ssl/logstash-forwarder.key"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    #index => ["waf"]
    index => ["waf-(date '+%Y-%m-%d_%H-%M-%S')"]
  }
  stdout { codec => rubydebug }
}

/>

filebeat.yml
<

output.logstash:
logstash:
  enabled: true
  hosts: ["172.31.192.3:5044"]
  index: "waf-(date '+%Y-%m-%d_%H-%M-%S')"
  tls:
    ssl.certificate: ["/etc/ssl/logstash_frwrd.crt"]
    ssl.key: "/etc/ssl/logstash-forwarder.key"

/>

[dadoonet (David Pilato)]

Thank you.

As you have:

stdout { codec => rubydebug }

Can you see anything in the Logstash console/logs?
You should see the events coming.

That said, although I'm not a Logstash expert, I'm unsure about this:

index => ["waf-(date '+%Y-%m-%d_%H-%M-%S')"]

The doc says:

• Value type is string
• Default value is "logstash-%{+yyyy.MM.dd}"

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}. LS uses Joda to format the index pattern from event timestamp. Joda formats are defined here.

So I'm not sure where this (date '+%Y-%m-%d_%H-%M-%S') is coming from...