Problem with creation indexes based on custom field

Elastic Stack Logstash
1

Hi,
I am using ELK 5.5.2.
I have a problem with creation indexes.
On filebeat config I have a following setup

  • input_type: log
    document_type: prod
    paths:

    • c:\somepath*.log

    fields:
    my_log_type: app

On logstash config I have
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[my_log_type]}-%{+YYYY.MM.dd}"
}

On 2 instances of ELK 5.5.2 this config is working fine, generating index with name app-2017-09-06
On 1 instance this config is generating index - %{[my_log_type]}-2017-09-06

What is wrong with this configuration?

Thank you.

2

On 1 instance this config is generating index - %{[my_log_type]}-2017-09-06

That indicates that the events ending up in that index don't have a my_log_type field.

3

Hi.
But I have only one filebeat prospector on this ELK instance and I see this field in the logstash log message.

Thank you.

4

Please show a raw event from that index. Copy/paste from Kibana's JSON tab.

5

{
"_index": "%{[my_log_type]}-2017.09.07",
"_type": "log",
"_id": "AV5bTV6bnAhJx1DJGWW-",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2017-09-07T07:45:45.352Z",
"offset": 408517,
"@version": "1",
"input_type": "log",
"beat": {
"hostname": "XXXX",
"name": "XXXX",
"version": "5.5.2"
},
"host": "XXXX",
"source": "d:\somepath\Service-2017-09-07.log",
"type": "log",
"fields": {
"my_log_type": "app"
},
"tags": [
"beats_input_codec_plain_applied"
]
},
"fields": {
"@timestamp": [
1504770345352
]
},
"sort": [
1504770345352
]
}

6

But you don't have a my_log_type field (only [fields][my_log_type]). Look into Filebeat's fields_under_root option.

7

Thank you very much!
It is working now.

Best regards,
Alex.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.