Not able to create /see any indexs in elasticsearch

Elastic Stack Beats
1

I was not able to see any data or any indexs in elasticsearch . I was using filebeat --> logstash --> elasticsearch

filebeat.yml

filebeat.inputs:

-> type: log

paths:

  • /var/log/hadoop/hadoop/hive/gc.log*
    document_type: hivemetastore
  • type: logpaths:
    • /var/log/hadoop/hadoop/hdfs/gc.log*
      document_type: hdfs

logstash.conf

input {
beats {
port => 5044
}
}

output{

if [document_type] == "hdfs" {
elasticsearch {
hosts => ["endpoint:443"]
action => "index"
index => "hdfs"
}
}

if [document_type] == "hivemetastore" {
elasticsearch {
hosts => ["endpoint:443"]
action => "index"
index => "hive"
}
}

}

I'm not sure where i was doing wrong. without if condition i was able to see logs.

2

I don't think filebeat has any field named document_type. That's why the conditions fail. If you insist on document_type, you can configure filebeat this way:

paths:
- /var/log/hadoop/hadoop/hive/gc.log*
fields.document_type: hivemetastore
fields_under_root: true
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.