Document_type is being ignored

Elastic Stack Beats
1

I am new to file beat. I am using filebeat to forward logs to logstash, from where it gets forwarded to elastic search.
I am using version 6.0 for all of them.

The problem I am having is that even if I set document_type, the value that ends up in elastic for _type is doc.
I know I am doing something wrong, but can't figure what.

This is how my filebeat.yml looks like

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

  # Change to true to enable this prospector configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - E:\someLoc\IISLogs\inetpub\logs\LogFiles\W3SVC1\*.log
  document_type: iis_log

This is how my logstash config looks like

input {
    beats {
     port =>"5043"
    }
  
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "%{[@metadata][beat]}"
    document_type => "%{[@metadata][type]}"
  }
}

This is a sample of what ends up in elastic search

 {
                "_index": "filebeat",
                "_type": "doc",
                "_id": "IjYyCmABLUTmLp82mEUK",
                "_score": 1,
                "_source": {
                    "@timestamp": "2017-11-29T23:52:53.452Z",
                    "offset": 1168146,
                    "@version": "1",
                    "beat": {
                        "name": "XXXX",
                        "hostname": "XXXX",
                        "version": "6.0.0"
                    },
                    "host": "XXXX",
                    "prospector": {
                        "type": "log"
                    },
                    "source": "ZZZZ\IISLogs\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex171116.log",
                    "message": "Blah",
                    "tags": [
                        "beats_input_codec_plain_applied"
                    ]
                }
            }
2

docuement_type was deprecated in 5.5 and completely removed in 6.0 because _type is being removed from Elasticsearch.

You can add your own type field (not _type) with fields.

filebeat.prospectors:
- paths:
    - 'E:\someLoc\IISLogs\inetpub\logs\LogFiles\W3SVC1\*.log'
  fields_under_root: true
  fields:
    type: iis_log
2 Likes
3

Thank you Andrew!

How do I filter in logstash using this field? In the old way the log stash config would had looked like

    filter {
       

        if [type] == "iis_log" {
           

            grok {
                match => { xxx}
            } 

        }
    }

How would I do that if its in a field now? Also can i add a column with this field in logstash?

4

It's the same way. What you have should work fine if you are using the config I gave.

1 Like
5

That worked for me! Thank you!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.