# Logstash Conditional Indexes

**URL:** <https://discuss.elastic.co/t/logstash-conditional-indexes/170080>\
**Category:** Logstash\
**Created:** [February 26, 2019, 11:44pm UTC](https://discuss.elastic.co/t/logstash-conditional-indexes/170080 "2019-02-26T23:44:46Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![luke14](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@luke14](https://discuss.elastic.co/u/luke14)\
**Post date:** [February 26, 2019, 11:44pm UTC](https://discuss.elastic.co/t/logstash-conditional-indexes/170080/1 "2019-02-26T23:44:47Z")

</div>

I am having great difficulty parsing my Filebeat input into separate Indices dependant on some custom fields.

My IIS server has filebeat installed and the filebeat.yml looks like this:

```
filebeat.inputs:
  enabled: true
  fields: {log_type: iis}

```

My beats output on my logstash server looks like this:

```
output {
  if "iis" in [fields][log_type] {
    elasticsearch {
      hosts => elasticsearch
      index => "logstash-beats-iis-%{+YYYY.MM.dd}"      
      template_name => "logstash-beats"
      template => "/beats-template.json"
      template_overwrite => true
    }
  }
  else {
    elasticsearch {
      hosts => elasticsearch
      index => "logstash-beats-%{+YYYY.MM.dd}"
      template_name => "logstash-beats"
      template => "/beats-template.json"
      template_overwrite => true
    }
  }
}

```

No logs are coming through at all, I'm not sure if the conditional logic or the setup. I would appreciate if anyone can help.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [February 27, 2019, 11:14am UTC](https://discuss.elastic.co/t/logstash-conditional-indexes/170080/2 "2019-02-27T11:14:49Z")

</div>

Hi Luke,

There are two things that need to be considered.

> [@luke14](#):
>
> fields: {log\_type: iis}

I am not sure if this is the correct syntax to add a field. I believe it is:

```auto
fields:
  log_type: "iis"

```

If both the configurations are same, please let me know. I am not quite sure.

Secondly, is there a field called `[fields][log_type]` in Elasticsearch? I believe it would not be a nested field and should just be reference as `log_type`.

In that scenario, the logstash condition would be

```auto
if [log_type] == "iis" {
.....

```

Let me know if it works.

---

<div class="post-metadata">

**Author:** ![luke14](https://avatars.discourse-cdn.com/v4/letter/l/65b543/32.png) [@luke14](https://discuss.elastic.co/u/luke14)\
**Post date:** [February 27, 2019, 3:02pm UTC](https://discuss.elastic.co/t/logstash-conditional-indexes/170080/3 "2019-02-27T15:02:16Z")

</div>

Thanks for the reply.

I got this to work but only from a property on the log, I used the pre-defined

```
if [fileset][module] == "iis" {
....

```

The filebeat.yml was reading the config okay with that syntax. I noticed there wasn't any `[fields][log_type]` in elastic search so therefore my filebeat.yml config was not working.

I suppose my question is what is the best approach to tag or label filebeat inputs from within the filebeat.yml? What is the best practice?

Perhaps,

```
fields:
  log_type: "iis"

```

Would work, I will have to test.

Thanks

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [February 27, 2019, 3:09pm UTC](https://discuss.elastic.co/t/logstash-conditional-indexes/170080/4 "2019-02-27T15:09:39Z")

</div>

I have tried to stay away from `if` so I solved a similar situations like this

Filebeat

```
---
- type: log
  paths:
    - /var/log/foo.log
  encoding: plain
  fields:
    log_prefix: bar
    log_idx: logs
  fields_under_root: false

```

In Logstash filter section

```
# Adding @metadata needed for index sharding to Filebeat logs
mutate {
  copy => {
   "[fields][log_prefix]" => "[@metadata][log_prefix]"
   "[fields][log_idx]" => "[@metadata][index]"
  }
}

```

Logstash output

```
output {

  elasticsearch {
        hosts => ["10.1.1.1:9200", "10.1.1.2:9201"]
        index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"
  }

}

```

The logs form `/var/log/foo.log` would go into an index called `bar-logs-2019.02.27`

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [March 1, 2019, 3:57am UTC](https://discuss.elastic.co/t/logstash-conditional-indexes/170080/5 "2019-03-01T03:57:25Z")

</div>

> [@luke14](#):
>
> I suppose my question is what is the best approach to tag or label filebeat inputs from within the filebeat.yml? What is the best practice

Both the approaches are perfectly fine. I don't think there is any such best practice associated with this.

However, using the if statement does add one check. What we have done is added a field called index name in our inputs. We use the index name field to decide where it gets indexed.

```auto
    elasticsearch {
        action => "index"
        hosts => ["https://xyz:9200"]
        index => "operations-%{indice}"
    }

```

Something similar to what @A_B has mentioned.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 29, 2019, 3:57am UTC](https://discuss.elastic.co/t/logstash-conditional-indexes/170080/6 "2019-03-29T03:57:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.