I have set up my elk stack like this. I am trying to ship logs and top data through topbeat and filebeat with custom index name.
Although, Logstash is not creating any index for the data i am passing with custom index name.
Logstash Conf
input{
beats{
port => 27080
congestion_threshold => 1500
}
jmx {
path => "file://Machine01/Users/username/projects/Logstash/logstash/bin/jmx"
polling_frequency => 15
type => "jmx"
nb_thread => 4
}
}
filter {
if [type] == "Type1"{
grok{
break_on_match => false
patterns_dir => ["C:\Users\users\projects\Logstash\logstash\bin\patterns"]
match => { "message" => "%{YEAR:Year}%{MONTHNUM:Month}%{MONTHDAY:Day} %{HOUR:Hour}%{MINUTE:Minute}%{SECOND:Second} %{LogLevel:LogVerbosity} %{MODULE:MODULENAME}%{SPACE}%{MESSAGEID:MESSAGEID} %{SUBMODULE:SUBMODULE} %{MESSAGE:MESSAGE}"}
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
add_tag => ["Groked"]
}
if "_grokparsefailure" in [tags] {
drop { }
}
if [type] == "jmx" {
if ("OperatingSystem.ProcessCpuLoad" in [metric_path] or "OperatingSystem.SystemCpuLoad" in [metric_path]) {
ruby {
code => "event['cpuLoad'] = event['metric_value_number'] * 100"
add_tag => [ "cpuLoad" ]
}
}
}
}
}
output {
if [type] == "jmx" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "jmx-%{+YYYY.MM.dd}"
}
} else {
elasticsearch {
hosts => ["http://localhost:9200"]
manage_template => true
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
if [type] == "dbtable" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "dbtable-%{+YYYY.MM.dd}"
}
}
}
}
filebeat conf:
filebeat:
prospectors:
- paths:
- test.log
input_type: log
tail_files: false
scan_frequency: 3s
backoff: 20s
backoff_factor: 1
document_type: custom
registry:
fields:
type: custom
spool_size: 10000
idle_timeout: 2s
output:
logstash:
index: custom
hosts: ["valid hostname"]
logging:
to_files: true
files:
path: ./
name: filebeat.log
rotateeverybytes: 10485760
level: debug
I am expecting , when i set index: custom , it (else part of logstash output) should create index in elasticsearch as "custom-YYYY.MM.dd"
But its just creating index in elasticsearch as "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
and if i comment #index: custom (in Filebeat ) it is creating index in elasticsearch as filebeat-YYYY.MM.dd
Where i am going wrong, why its not working for custom index pattern