Send Logstash logs to Kibana remote server

[Sai_Avinash_Duddupud (avinash)]

I am new to ELK Stack and trying to view logs on kibana which is hosted on different server. Following are my configurations for Filebeat and logstash in my localhost pc and logstash is succesfully recieving logs from filbeat.

I am facing a lot of confusion in creating an index pattern in kibana. how do i know the index variable parameters [@metadata][beat] and [@metadata][version] present in output node in logstash.conf so that i will create an index pattern and access the same in kibana discover page

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/sai_avinash/Documents/refactor/unityapp/unity/media/*.log

output.logstash:
  enabled: true
  hosts: ["localhost:5044"]

logstash.conf

input {
  beats {
    port => 5044
    ssl => false
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL:loglevel}\|%{GREEDYDATA:module}\|%{GREEDYDATA:content}" }
  }
  date {
    locale => "en"
    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss"]
    target => "@timestamp"
    timezone => "America/New_York"
  }
}

output {
  elasticsearch {
    hosts => "elk_server_ip:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
  }
  stdout { codec => rubydebug { metadata => true } }
}

[warkolm (Mark Walkom)]

Welcome to our community!

@Sai_Avinash_Duddupud

how do i know the index variable parameters [@metadata][beat] and [@metadata][version] present in output node in logstash.conf so that i will create an index pattern and access the same in kibana discover page

You don't need to know what they are, but they will usually be something like;

• [@metadata][beat] - filebeat, metricbeat, etc
• [@metadata][version] - 7.8.0 or 7.7.1 etc

When you create the pattern in Kibana, just use filebeat-* or metricbeat-*.

Also if you are only doing simple grok and timestamp matching via Logstash, you can also look at using the Ingest API to reduce some of your complexity.

[Sai_Avinash_Duddupud (avinash)]

thanks @warkolm i found it in kibana indices list but since there are so many listed, is it possible to write a custom index name instead of the entire pattern like following?

can i replace
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

with
index => "avinash*"

now will avinash* get created on elk_server_ip:9200/_cat/indices?

[warkolm (Mark Walkom)]

You can, yes.

[Sai_Avinash_Duddupud (avinash)]

@warkolm I have created the following index under output node in logstash.conf...its been more than 30 min, still blend_test doesn't reflect in the kibana indices server

elasticsearch {
    hosts => "elk_server_ip:9200"
    manage_template => false
    index => "blend_test*" 
  }

Please suggest if am doing something wrong....FYI, I have also restarted filebeat and logstash as well

[Sai_Avinash_Duddupud (avinash)]

@warkolm did I make any mistake? Please suggest a workaround if so.