Filebeat - json imports

Elastic Stack Beats
1

Need some help and hopefully its something simple I have overlooked. I am trying to import a JSON file, single line records and when I go into Kibana, to view the data in discover, I dont see any of the fields if I use a custom index. However, if I use the default (filebeat) format, the fields are displayed. however, if I go into visualizations and create a date table, I see the fields. Here is my filebeat yml file. I only included my changes I made. Everything else is default or commented out.

  • type: log

    Change to true to enable this input configuration.

    enabled: true
    json.keys_under_root: true
    json.add_error_key: true

setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: true
setup.ilm.enabled: false

output.elasticsearch:

Array of hosts to connect to.

enabled: true
hosts: ["servername.domain:9200"]
index: "indexname-%{+yyyy.MM.dd}"

setup.template.name: "indexname"
setup.template.pattern: "indexname-*"

processors:
#- add_host_metadata: ~
#- add_cloud_metadata: ~
- timestamp:
field: EventDate
layouts:
- '2006-01-02T15:04:05'
- '2006-01-02T15:04:05.999'
- drop_fields:
fields: EventDate

2

Figured out the problem. Was using the wrong settings. Needed to use the following settings.
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "AliasName"
setup.ilm.pattern: "{now/d}"

I also created a index policy in Kibana to manage the data.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.