# Specifying \_type with Filebeat **URL:** \ **Category:** Beats\ **Created:** [July 28, 2017, 12:59pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957 "2017-07-28T12:59:31Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![spur01](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@spur01](https://discuss.elastic.co/u/spur01)\ **Post date:** [July 28, 2017, 12:59pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/1 "2017-07-28T12:59:31Z")
I'm at a bit of a loss on how to do this correctly. I have a Filebeat pushing to a pipeline which targets an index that has dynamic mapping set to false and a type that enforces strict mapping. The type I'm using is not the Filebeat default and I have not loaded the Filebeat template. When trying to ingest, nothing makes it way into Elasticsearch. I've tried the following settings in filebeat.yml setting document\_type under the prospector setting document\_type under the fields setting \_type under the fields setting type under the fields I imagine I'm doing something obvious incorrectly but I'm unsure how to work my way around this. Before I indulge in some ugly kludge I was hoping someone might be able to point me in the right direction. Thanks! ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 28, 2017, 2:02pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/2 "2017-07-28T14:02:58Z")
which filebeat version are you using? Normally one sets the type in: ```auto filebeat.prospectors: - ... document_type: ... ``` but as Elasticsearch will remove support for types in the future, newer releases of any beats default the type to `docs` I think. ---
**Author:** ![spur01](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@spur01](https://discuss.elastic.co/u/spur01)\ **Post date:** [July 28, 2017, 2:15pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/3 "2017-07-28T14:15:20Z")
I'm using Filebeat 5.5. I tried that but it's still trying to set the type to docs according to the filebeat verbose logs. Is there any option to control this from within Filebeat? If not, I'll go see if I can manage it from within the pipeline definition. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 28, 2017, 4:07pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/4 "2017-07-28T16:07:10Z")
Starting with 5.5, the `_type` field is hard coded to "docs". The `document_type` still overwrites the "type" field. Note, the "type" field is beat specific, but the `_type` field is somewhat Elasticsearch specific and will be removed in future ES versions (as internally `_type` always used to be merged/treated like a normal field). If you really need to set `_type`, you have to use filebeat 5.4. But we'd rather recommend using the "type" field. ---
**Author:** ![spur01](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@spur01](https://discuss.elastic.co/u/spur01)\ **Post date:** [July 28, 2017, 6:46pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/5 "2017-07-28T18:46:40Z")
I apologize if I'm being obtuse but I'm a little confused. I've set up template in Elasticsearch which is associated with an index and a mapping for the type (which I presumed corresponds to \_type in Elasticsearch) in that mapping. I just went and tried to set ``` fields: logsource: mylogsource type: mytype ``` And I'm still receiving a > "type": "type\_missing\_exception", > "reason": "type[doc] missing", > "caused by": > "illegal\_state\_exception", > "reason":"trying to auto create mapping, but dynamic mapping is disabled" I'm still a bit of a newbie with Elasticsearch and even more so with Filebeat but I can't seem to find any indication in the documentation as to what I'm doing wrong. Is it possible to have Filebeat push to a custom mapping that exists in Elasticsearch? I'd like to use Filebeat and a pipeline. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 28, 2017, 7:02pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/6 "2017-07-28T19:02:31Z")
I had this problem when I changed to Filebeat 5.5, I was able to solve it using the following configuration: ```auto fields: document_type: your-type ``` This way Filebeat will override the default type the document\_type you specified. ---
**Author:** ![spur01](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@spur01](https://discuss.elastic.co/u/spur01)\ **Post date:** [July 28, 2017, 7:18pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/7 "2017-07-28T19:18:42Z")
I've tried that, that's what I tried first based on the documentation (and just now again in case I was crazy). I'm wondering if there is some other setting that is the issue here. I'm running it via `.\filebeat -once -v` to verify that it's working before I install it as a service and move it to production. Here's a sample prospector, I have a bunch of them in my filebeat.yml. Is it fields\_under\_root that's causing the issue? ``` filebeat.prospectors: - input_type: log multiline.match: after multiline.pattern: "^2" multiline.negate: true paths: - "mypath" fields: document_type: mytype logsource: mylogsource fields_under_root: true close_eof: true ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 28, 2017, 7:29pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/8 "2017-07-28T19:29:27Z")
I don't think so, I'm also using fields\_under\_root and it's working. Here is my configuration: ```auto filebeat: prospectors: - input_type: log paths: - C:\inetpub\logs\LogFiles\W3SVC1\*.log fields: document_type: my-type exclude_lines: ["^#"] exclude_files: [".zip"] fields_under_root: true output: logstash: hosts: ["x.x.x.x:5001"] ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 28, 2017, 7:36pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/9 "2017-07-28T19:36:30Z")
So you're having filebeat-\>logstash-\>elasticsearch? Can you also share the output section of logstash? When sending via logstash, it's logstash setting the actual document type. What does the mapping template look like? ---
**Author:** ![spur01](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@spur01](https://discuss.elastic.co/u/spur01)\ **Post date:** [July 28, 2017, 7:58pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/10 "2017-07-28T19:58:37Z")
I'm going direct from Filebeat to Elasticsearch. Here's my filebeat.yml. ``` name: "MyName" filebeat.prospectors: - input_type: log multiline.match: after multiline.pattern: "^2" multiline.negate: true paths: - "samplelogpath" fields: document_type: mytype logsource: mylogsource fields_under_root: true close_eof: true - input_type: log multiline.match: after multiline.pattern: "^2" multiline.negate: true paths: - "samplelogpath" fields: document_type: mytype logsource: mylogsource fields_under_root: true close_eof: true logging.level: debug output.elasticsearch: hosts: - "myeshost" index: "mylogs-%{+yyyy.MM.dd}" pipeline: mypipeline template.enabled: false processors: - drop_fields: fields: ["beat.version","input_type","offset"] ``` ---
**Author:** ![spur01](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@spur01](https://discuss.elastic.co/u/spur01)\ **Post date:** [July 28, 2017, 7:59pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/11 "2017-07-28T19:59:48Z")
Here's the Elasticsearch template ``` { "template": "mylogs*", "settings": { "number_of_shards": 2, "mapper": { "dynamic": "false" } }, "mappings": { "mytype": { "dynamic": "strict", "properties": { "beat": { "type": "nested", "properties": { "hostname": { "type": "keyword" }, "name": { "type": "keyword" } } }, "log_level" : { "type": "keyword" }, "logsource" : { "type": "keyword" }, "logtimestamp" : { "type": "date" }, "message": { "type": "text", "fields" : { "raw": { "type": "keyword" } } }, "method" : { "type": "keyword" }, "source" : { "type": "text" } } } } } ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 28, 2017, 8:16pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/12 "2017-07-28T20:16:23Z")
@spur01, oh, sorry, I misunderstood. The document\_type in the filebeat configuration will set the **type** field, not the **\_type**. However I am able to set the **\_type** field if I use logstash to send the filebeat events to elasticsearch. My Logstash output block configuration for this pipeline is: ```auto output { if [type] == "my-type" { elasticsearch { hosts => ["localhost:9200"] index => "my-index-%{+YYYY.MM.dd}" document_type => "my-type" template_name => "my-template" } } } ``` The document\_type in the output block from Logstash will set the **\_type** field. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 28, 2017, 8:26pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/13 "2017-07-28T20:26:37Z")
Thanks. As the `_type` field is hardcoded to `docs`, the most simple fix would be to use `docs` in the mapping. As support for `_type` will be removed in Elasticsearch in the future, I would not recommend having multiple values for `_type` anyways. The way you use `document_type` with fields, you might end up with a document like: ```auto { "_type": "docs", " type": "log", "document_type": "mytype" } ``` You will get ```auto { "_type": "doc", "type": "mytype", } ``` by changing the configuration to: ```auto name: "MyName" filebeat.prospectors: - input_type: log document_type: mytype multiline.match: after multiline.pattern: "^2" multiline.negate: true paths: - "samplelogpath" fields: logsource: mylogsource fields_under_root: true close_eof: true - input_type: log document_type: mytype multiline.match: after multiline.pattern: "^2" multiline.negate: true paths: - "samplelogpath" fields: logsource: mylogsource fields_under_root: true close_eof: true logging.level: debug output.elasticsearch: hosts: - "myeshost" index: "mylogs-%{+yyyy.MM.dd}" pipeline: mypipeline template.enabled: false processors: - drop_fields: fields: ["beat.version","input_type","offset"] ``` If you really insist on renaming `docs` to `mytype`, you can do so in the Ingest Pipeline via the [set](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html) and [remove](https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html) or [rename](https://www.elastic.co/guide/en/elasticsearch/reference/current/rename-processor.html) processors: ```auto { "rename" : { "field": "type", "target_field": "_type" } } ``` Special fields like `_index` and `_type` [can be changed](https://www.elastic.co/guide/en/elasticsearch/reference/current/accessing-data-in-pipelines.html#accessing-metadata-fields) from within the ingest pipeline (e.g. for routing to another index on failure). Using the [simulate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html), you can easily test modifications to events. ---
**Author:** ![spur01](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@spur01](https://discuss.elastic.co/u/spur01)\ **Post date:** [July 28, 2017, 8:47pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/14 "2017-07-28T20:47:36Z")
Awesome, thank you very much, that's a killer answer. I'm interested to see what an Elastisearch without types looks like especially in terms of mappings and the like! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 18, 2017, 12:59pm UTC](https://discuss.elastic.co/t/specifying--type-with-filebeat/94957/15 "2017-08-18T12:59:35Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.