Logstash not creating correct index for filebeat and packet beat

Elastic Stack Logstash
1

I have set up my elk stack like this. I am trying to ship logs and top data through topbeat and filebeat with custom index name.

Although, Logstash is not creating any index for the data i am passing with custom index name.

Logstash Conf

input{
    beats{
      port => 27080
      congestion_threshold => 1500
    }
    jmx {
      path => "file://Machine01/Users/username/projects/Logstash/logstash/bin/jmx"
      polling_frequency => 15
      type => "jmx"
      nb_thread => 4
 }
}
filter {
    if [type] == "Type1"{
        grok{
          break_on_match => false
          patterns_dir => ["C:\Users\users\projects\Logstash\logstash\bin\patterns"]
          match => { "message" => "%{YEAR:Year}%{MONTHNUM:Month}%{MONTHDAY:Day} %{HOUR:Hour}%{MINUTE:Minute}%{SECOND:Second} %{LogLevel:LogVerbosity} %{MODULE:MODULENAME}%{SPACE}%{MESSAGEID:MESSAGEID} %{SUBMODULE:SUBMODULE} %{MESSAGE:MESSAGE}"}
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
          add_tag => ["Groked"]
        }



 if "_grokparsefailure" in [tags] {
              drop { }
    }

   if [type] == "jmx" {
   if ("OperatingSystem.ProcessCpuLoad" in [metric_path] or "OperatingSystem.SystemCpuLoad" in [metric_path]) {
     ruby {
     code => "event['cpuLoad'] = event['metric_value_number'] * 100"
     add_tag => [ "cpuLoad" ]
     } 
   }
 }
  }
}

output {  
    if [type] == "jmx" {
        elasticsearch {  
            hosts => ["http://localhost:9200"]  
            index => "jmx-%{+YYYY.MM.dd}"   
        }
    } else {
        elasticsearch {  
            hosts => ["http://localhost:9200"] 
            manage_template => true
            index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
            document_type => "%{[@metadata][type]}"
        }

if [type] == "dbtable" {
    elasticsearch {  
        hosts => ["http://localhost:9200"]  
        index => "dbtable-%{+YYYY.MM.dd}"  
}
   } 
 }
}

filebeat conf:

filebeat:
  prospectors:
    - paths:
        - test.log
      input_type: log
      tail_files: false
      scan_frequency: 3s
      backoff: 20s
      backoff_factor: 1
      document_type: custom
      registry: 
      fields:
        type: custom
  spool_size: 10000
  idle_timeout: 2s
output:
  logstash:
    index: custom
    hosts: ["valid hostname"]
logging:
  to_files: true
  files:
    path: ./
    name: filebeat.log
    rotateeverybytes: 10485760
    level: debug

I am expecting , when i set index: custom , it (else part of logstash output) should create index in elasticsearch as "custom-YYYY.MM.dd"

But its just creating index in elasticsearch as "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

and if i comment #index: custom (in Filebeat ) it is creating index in elasticsearch as filebeat-YYYY.MM.dd

Where i am going wrong, why its not working for custom index pattern

2

The index value does nothing for filebeat, see https://github.com/elastic/beats/issues/574