# Logstash Configuration when removing specified type mapping

**URL:** <https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148>\
**Category:** Logstash\
**Created:** [November 10, 2018, 3:59pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148 "2018-11-10T15:59:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Josh\_cullum](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@Josh\_cullum](https://discuss.elastic.co/u/Josh_cullum)\
**Post date:** [November 10, 2018, 3:59pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148/1 "2018-11-10T15:59:56Z")

</div>

Hi,

There is probably a simple fix to this. We've recently upgrade our elastic stack from 5.3 to 6.4, and we're having the mapping issues that many other people are having but we seem to be using it in a slightly different way and therefore, we're a little lost.

We use filebeat to set the document\_type based on the log file prospector - for example:

```
prospectors:
    -
      paths:
        - /var/log/messages
        - /var/log/syslog
      document_type: syslog

```

This allows us in Logstash to apply the following output:

```
else {
elasticsearch {
   hosts => ["**"]
   index => "%{type}-%{+YYYY.MM.dd}"
   template_overwrite => true
}

```

}

For other document\_types we create other indexes, with templates etc.

Given that document\_type is now not working - what would be the best solution for creating an index based on the document\_type?

---

<div class="post-metadata">

**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\
**Post date:** [November 10, 2018, 4:27pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148/2 "2018-11-10T16:27:52Z")

</div>

@Josh_cullum

Probably you can achieve it by adding `type => syslog` in the logstash input section ??

---

<div class="post-metadata">

**Author:** ![Josh\_cullum](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@Josh\_cullum](https://discuss.elastic.co/u/Josh_cullum)\
**Post date:** [November 10, 2018, 4:39pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148/3 "2018-11-10T16:39:09Z")

</div>

Hi @Makra, how do you mean?

Our input configuration looks like this:

```
input {
  beats {
    port => 6782
  }
  lumberjack {
    port => 5782
    ssl_certificate => "/etc/logstash/ssl/logstash-forwarder.crt"
    ssl_key => "/etc/logstash/ssl/logstash-forwarder.key"
  }
  tcp {
    codec => "json"
    type => "curator"
    port => "28778"
  }
  file {
    path => ["/var/log/logstash/logstash-plain.log"]
    type => "logstash"

```

The logic of separating out to various doc\_types is done on the filebeat configuration:

```
-
      paths:
        - /var/log/exim/main.log
      document_type: exim

    -
      paths:
        - "/var/log/httpd/*-access_log"
      document_type: httpd

```

and then the logstash.elasticsearch output plugin has this:

```
else if [type] == "delivery_report" {
    elasticsearch {
      hosts => ["**"]
      index => "%{type}-%{_year}.%{_month}"
      action => "update"
      document_id => "%{delivery_report_id}"
      doc_as_upsert => true
    }
  }
}
  else if [type] == "pmta_acct" {
    elasticsearch {
       hosts => ["**"]
       index => "pmta_acct-%{+YYYY.MM.dd}"
       manage_template => true
       template => "/etc/logstash/templates/pmta_acct.json"
       template_name => "pmta_acct"
       template_overwrite => true
    }
  }

```

etc etc. So the need to be able to keep the a 1:1 relationship between a document\_type and an Index is essential without causing massive security holes in the firewalls etc etc.

---

<div class="post-metadata">

**Author:** ![Josh\_cullum](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@Josh\_cullum](https://discuss.elastic.co/u/Josh_cullum)\
**Post date:** [November 10, 2018, 4:57pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148/4 "2018-11-10T16:57:49Z")

</div>

@Makra I'm assuming that something like this: [https://stackoverflow.com/questions/45974963/elasticsearch-filebeat-document-type-deprecated-issue](https://stackoverflow.com/questions/45974963/elasticsearch-filebeat-document-type-deprecated-issue) using custom fields would allow us to specify the same thing but using a custom field instead of document\_type?

---

<div class="post-metadata">

**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\
**Post date:** [November 10, 2018, 5:42pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148/5 "2018-11-10T17:42:11Z")

</div>

Your answer is here

> [@How to tag log files in filebeat for logstash ingestion?](https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/6):
>
> Digging deeper, it seems like I should be able to say: type =\> "%{[@metadata][log\_type]}" or would it be type =\>"%{[log\_type]}" Or if I were trying to use the source field, using: type =\> "%{[@metadata][source]}" Does that look right? Is that how I would access a field within logstash, so that I can appropriately type the data within the input?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2018, 5:42pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-removing-specified-type-mapping/156148/6 "2018-12-08T17:42:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.