Output.elasticsearch.indices matching not working in 7.0.1

Elastic Stack Beats
1

Just upgraded from Elastic Stack 6.7.1 to 7.0.1 and filebeat configuration I had to dynamically change the index name appears to have stopped working.

My simple config:

filebeat.inputs:
- type: log
  paths:
    - /syslogs/f5.log
  pipeline: f5-requestlog-pipeline
  fields:
    service: f5-requestlogs

output.elasticsearch:
  indices:
    - default: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
    - index: "filebeat-14day-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.equals:
        fields.service: "f5-requestlogs"

Data is being put in filebeat-7.0.1-2019.05.23 rather than the expected filebeat-14day-7.0.1-2019.05.23

2

Hello looking,

At first I thought it was maybe because of the move to ECS in 7.0 but since the fields you are using in the conditionals are customs I would be surprised.

Can you go into Elasticsearch and add an JSON format of one of the event?

3

Much obfuscation later:

{
  "_index": "filebeat-7.0.1-2019.05.23",
  "_type": "_doc",
  "_id": "b4cq5moBJei9cPnCrClb",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "hostname": "filebeat.example.com",
      "id": "e39e30ea-0918-48fd-91f8-bc2eafec23f2",
      "type": "filebeat",
      "ephemeral_id": "1484f00e-abc1-40d4-98bb-62d5b136df69",
      "version": "7.0.1"
    },
    "log": {
      "file": {
        "path": "/syslogs/f5.log"
      },
      "offset": 103867050
    },
    "f5": {
      "server": "10.11.12.13",
      "request": "/example-request",
      "response_code": 200,
      "agent": "Thing",
      "verb": "GET",
      "virtual_ip": "10.20.30.40",
      "referrer": "https://web.example.com/",
      "bytes": 30211,
      "syslog_timestamp": "May 23 20:28:35",
      "clientip": "1.2.3.4",
      "appliance_hostname": "f5.example.com",
      "virtual_pool_name": "",
      "httpversion": "1.1",
      "server_port": "80",
      "virtual_name": "/Common/vip-80_sitea_virtual",
      "response_ms": 119,
      "timestamp": "23/May/2019:20:28:35 +0100"
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2019-05-23T19:28:36.466Z",
    "ecs": {
      "version": "1.0.0"
    },
    "host": {
      "hostname": "filebeat.example.com",
      "os": {
        "kernel": "3.10.0-957.12.1.el7.x86_64",
        "codename": "Core",
        "name": "CentOS Linux",
        "family": "redhat",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": true,
      "name": "filebeat.example.com",
      "id": "d7619efec8b24acf9ac7093f6f203d48",
      "architecture": "x86_64"
    },
    "fields": {
      "service": "f5-requestlogs"
    }
  },
  "fields": {
    "suricata.eve.timestamp": [
      "2019-05-23T19:28:36.466Z"
    ],
    "@timestamp": [
      "2019-05-23T19:28:36.466Z"
    ]
  },
  "highlight": {
    "fields.service": [
      "@kibana-highlighted-field@f5-requestlogs@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1558639716466
  ]
}
4

Should have mentioned I also set the following, to turn off ILM:

setup.ilm.enabled: false
5

Also ponder updating my pipeline to use ECS fields...

6

OK, the structure of the JSON document looks right, so the conditions should work on that field, I know that we have changed a few things for the ILM support I wonder if we have a regression/bug here.

7

Should I raise a bug on Github?

8

I can confirm its actually a bug, can you raise the issue I will comment on it.

9 
filebeat.inputs:
- type: log
  paths:
    - /tmp/hello.log
  fields:
    service: "hello"

setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  indices:
    - index: "filebeat-14day-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.equals:
        fields.service: "hello"

I have used the above config and was able to get different results, in 6.7 and 7.0. I know we did some changes in the code path to add support to ILM I presume something broke along the way.

10
11

Hi,

Based on the config you posted, the behavior you get is what I would expect from filebeat 7.
I don’t see a bug but a misconfiguration, lack of adaptation to version 7 breaking changes.

The field beat.version is now agent.version, so everywhere you ref beat.version in your format strings the rule will fail. If no rule match it uses the “index” setting which by default is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}".

Which is what you get for your index.

Also one of your rules under “indices” only has “default” defined and its misused. That makes no sense, that setting is used for rules which contains a “mapping” and it’s supposed to be a string which will be used if the mapping doesn’t match. See the doc, I think your misunderstanding this part. It’s not a default format strings for the index when no rule match. If no rule match the normal “index” setting gets used.

So I recommend you change beat.version for agent.version, that was documented in the breaking changes and release notes.
Also I recommend you remove the rule with “default”. Then I would expect you to get what you expect. And if the rule doesn’t match you’ll get the default filebeat index name.

12

OK, thank you - that does fix the index naming.

I'd challenge that the "obvious" meaning of the 'default' keyword is valid...the configuration offers an "if" syntax with no "else" without my understanding of the 'default' keyword...

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.