Filebeats Sending Data to the Wrong index (ignoring .yml config and using default index)

Elastic Stack Beats
1

Hi,

I'm testing a filebeats indexing of JSON content created with wireshark.
I have the filebeat.yml configured as below, but the logs keep going to the default filebeats index, (filebeat-7.5.2-2020.01.27-000001).

Can someone point me on the right direction?

Thank you!

filebeat.inputs:

  • type: log
    paths:
    • "c:/sample/packets6.json"
      document_type: "pcap_file"
      json.keys_under_root: true

processors:

  • drop_event:
    when:
    equals:
    index._type: "pcap_file"

output.elasticsearch:
index: "packets-%{+yyyy-MM-dd}"
hosts: ["10.1.30.104:9200"]
path: "packets-test"

setup.template.name: "packets-"
setup.template.pattern: "packets-"
setup.template.enabled: false

Debug logging config.

logging.level: debug
logging.selectors: ['*']

2

Bringing the thread up to try to get some help

3

Can you validate your yaml here => http://www.yamllint.com/

4

I pasted it over and it said it was "valid"

5

Did you manually load the template named "packets-"? coz if its not there, by default, filebeat template will be used.

6

Yes, I manually created a template that applies to the pattern packets-*.
But in any case the data should be getting to the right index right? just using the wrong template even if I had not crated the template?

7

Could u please include ur template? Bcoz I can see u put “packet-“ as template name in filebeat config

8

{
"packets" : {
"order" : 0,
"index_patterns" : [
"packets-*"
],
"settings" : { },
"mappings" : {
"dynamic" : "false",
"properties" : {
"layers" : {
"properties" : {
"udp" : {
"properties" : {
"udp_udp_srcport" : {
"type" : "integer"
},
"udp_udp_dstport" : {
"type" : "integer"
}
}
},
"ip" : {
"properties" : {
"ip_ip_src" : {
"type" : "ip"
},
"ip_ip_dst" : {
"type" : "ip"
}
}
},
"frame" : {
"properties" : {
"frame_frame_len" : {
"type" : "long"
},
"frame_frame_protocols" : {
"type" : "keyword"
}
}
}
}
},
"timestamp" : {
"type" : "date"
}
}
},
"aliases" : { }
}
}

9

hi @guibarati,

in your filebeat config, you have this line:

setup.template.name: "packets-"

but your template is named "packets" :smiley:

10

Hi @inhinyera16, I've changed the setup.template.name to "packets" but it didn't have an effect.
I changed my index output to "packets-test" on my yml file and started receiving the following message:

Failed to connect to backoff(elasticsearch(http://10.1.30.104:9200/packets-test)): 404 Not Found: {"error":{"root_cause":[{"type":"index_not_found_exception"

Then I manually created the index, but got another error and the index is not being populated with data:

elasticsearch/client.go:771 GET http://10.1.30.104:9200/packets-test/_xpack?human=false
licenser/elastic_fetcher.go:105 Received 'Method Not allowed' (405) response from server, fallback to OSS license

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.