# Seperate indexes for each log path field error

**URL:** <https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235>\
**Category:** Logstash\
**Created:** [December 17, 2019, 10:50pm UTC](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235 "2019-12-17T22:50:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [December 17, 2019, 10:50pm UTC](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235/1 "2019-12-17T22:50:12Z")

</div>

I am trying to tag every log file path in filebeat and in logstash, use that tag to create separate index for each log file. But only one index is being created. Why does document\_type not work here?

Here is my filebeat.yml-

```auto
filebeat.inputs:
    
    -
      paths:
         - E:\DemoSetup\DispatcherApp\logs\dispatcher-scheduler.log
      input_type: log
      document_type: DispatcherApp
           
    -  
      paths:
         - E:\DemoSetup\Incident Agent\Logs\Trace.log
      input_type: log
      document_type: IncidentAgent
           
     
output:
  logstash:
    hosts: ["localhost:5044"]	

```

And my logstash.conf-

```auto
input {
  file{
     path: E:\DemoSetup\DispatcherApp\logs\dispatcher-scheduler.log
     type => "DispatcherApp"
  }
   file{
    path: E:\DemoSetup\Incident Agent\Logs\Trace.log
    type => "IncidentAgent"
  }
  beats {
    port => 5044
  }
}
filter {
    if[type] =="DispatcherApp"{
		grok {
			match => {"message" => "%{COMBINEDAPACHELOG}"}
        }
	} else [type] == "IncidentAgent" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
    }
	  
  }

    output {
      elasticsearch {
        hosts => ["http://localhost:9200"]
      sniffing => true
      manage_template => false
      index => "%{[@metadata][beat]-%{+YYYY.MM.dd}"
      
      }
    }
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 18, 2019, 6:50am UTC](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235/2 "2019-12-18T06:50:11Z")

</div>

> [@Mehak\_Bhargava](#):
>
> index =\> "%{[@metadata][beat]-%{+YYYY.MM.dd}"

This defines the name of the index created and does not contain the type you specified.

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [December 18, 2019, 6:47pm UTC](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235/3 "2019-12-18T18:47:00Z")

</div>

Hi @Christian_Dahlqvist, I made the change and my updated logstash.conf is-

The filter I had in old file was stopping the filebeat from running.

> [@Mehak\_Bhargava](#):
>
> filter { if[type] =="DispatcherApp"{ grok { match =\> {"message" =\> "%{COMBINEDAPACHELOG}"} } } else [type] == "IncidentAgent" { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}" } } } }

Does this part not qualify as proper filters?

```auto
input {
 file{
     path: E:\DemoSetup\DispatcherApp\logs\dispatcher-scheduler.log
     type => "access"
  }
 file{
    path: E:\DemoSetup\Incident Agent\Logs\Trace.log
    type => "error"
  }
 beats {
  port => "5044"
   }
}

filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
     match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }
}
 
output {
    elasticsearch {
    hosts => ["localhost:9200"]
   sniffing => true
   manage_template => false
   index => "%{type}-%{+YYYY.MM.dd}"  
   }
   stdout {
    codec => rubydebug
}

```

Is the index mentioned in output correct now? In Kibana, I still see all logs collected under one index only!

---

<div class="post-metadata">

**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\
**Post date:** [December 18, 2019, 6:54pm UTC](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235/4 "2019-12-18T18:54:44Z")

</div>

Hi @Christian_Dahlqvist, thanks for responding! Should I add the index line of code twice to get two indexes?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 15, 2020, 6:59pm UTC](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235/5 "2020-01-15T18:59:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.