# How to route logs to different Indexs

**URL:** <https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852>\
**Category:** Logstash\
**Created:** [June 18, 2017, 2:27pm UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852 "2017-06-18T14:27:25Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [June 18, 2017, 2:27pm UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/1 "2017-06-18T14:27:26Z")

</div>

Hi,

I am trying to route certain logs to index 1 and other logs to index 2. I know that this not the best practice but this will solve us many problems.

One log looks like:  
/opt/mod/a.log  
The other looks like:  
/opt/mod/b.log

in log type a.log I have a term called: "apiMedTimeSent", There for after looking at the link that I attached at the bottom I tried to do the following configuration:

```
output {
     if "apiMedTimeSent" in (What to put here) [
         index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
     ]
  }
}

```

But as you see I don't know how to continue this. Also is there a better way to do this?

Many thanks,  
Tomer

[https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html)

---

<div class="post-metadata">

**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\
**Post date:** [June 18, 2017, 5:09pm UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/2 "2017-06-18T17:09:03Z")

</div>

I am not sure if you can do it (what to put there) in the output section.  
Having said that, you can write a small ruby code fetch file "path" value in filter section. Then set the type of log based on the path value.

```
filter {
 ruby {
             " if event.get(path) is equal to '/opt/mod/a.log' // pseudocode
                   event.set('type', 'a')
              else
                  event.set('type', 'a')"
 }
 }

```

Now you can check the type in the output section -

```
if [type] == "a" {
                                 elasticsearch {

   }
}
else
          {
                                elasticsearch {

                                 }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 18, 2017, 7:01pm UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/3 "2017-06-18T19:01:09Z")

</div>

> in log type a.log I have a term called: "apiMedTimeSent"

What does this mean, exactly? That the string "apiMedTimeSent" occurs in the log message? If so:

```
if "apiMedTimeSent" in [message] {

```

How do these logs end up in Logstash? A file input in Logstash? Filebeat? Would it be possible to classify them at the source? In other words, would it be an option to configure the input so that it'll tag the messages in a way so that later stages in the pipeline knows what to do about them?

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [June 20, 2017, 7:40am UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/4 "2017-06-20T07:40:33Z")

</div>

> [@magnusbaeck](#):
>
> n option to configure the input so that it'll tag the messages in a way so that later stages in the pipeline knows what to do about

Hi they come in by filebeat, but I dont know how to tag multiple file in the filebeat. Also if I tag them how can I send the different tagged files to different indexs?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 20, 2017, 8:08am UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/5 "2017-06-20T08:08:51Z")

</div>

> Hi they come in by filebeat, but I dont know how to tag multiple file in the filebeat.

> **[Configure inputs | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_tags)**

> **[Configure inputs | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#configuration-fields)**

> Also if I tag them how can I send the different tagged files to different indexs?

You're _already_ using conditionals to send different events to different indexes so I don't know what's unclear.

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [June 27, 2017, 8:12am UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/6 "2017-06-27T08:12:40Z")

</div>

> [@magnusbaeck](#):
>
> if "apiMedTimeSent" in [message] {

Hi,

Thanks for the help until now. Sorry on the late response, I am just stuck on few issues.

Well I tried:

```
output {
  elasticsearch {
    hosts => ["192.168.1.116:9200"]
    manage_template => false
    if "apiMedTimeSent" in [message] {
        index => "%{[@metadata_traffic][beat]}-%{+YYYY.MM.dd}"
    }
    index => "%{[@metadata_else][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

but this gives me in the logs of logstash an error:

> [2017-06-27T11:11:05,012][ERROR][logstash.agent] fetched an invalid config {:config=\>"input {\n beats {\n port =\> 5044\n }\n}\n\nfilter {\n json{\n source =\> "message"\n }\n date {\n match =\> ["msgSubmissionTime", "UNIX\_MS"]\n target =\> "msgSubmissionTime"\n }\n date {\n match =\> ["msgDeliveryTime", "UNIX\_MS"]\n target =\> "msgDeliveryTime"\n }\n date {\n match =\> ["eventTs", "UNIX\_MS"]\n target =\> "eventTs"\n }\n\n\n mutate {\n convert =\> { \n\t"concatenated" =\> "boolean" \n\t"msgLength" =\> "integer"\n }\n }\n\n}\n\n\noutput {\n elasticsearch {\n hosts =\> ["192.168.1.116:9200"]\n manage\_template =\> false\n if "apiMedTimeSent" in [message] {\n index =\> "%{[@metadata\_traffic][beat]}-%{+YYYY.MM.dd}"\n }\n index =\> "%{[@metadata\_else][beat]}-%{+YYYY.MM.dd}"\n document\_type =\> "%{[@metadata][type]}"\n }\n}\n\n\ninput {\n beats {\n port =\> 5044\n }\n}\n\nfilter {\n json{\n source =\> "message"\n }\n date {\n match =\> ["msgSubmissionTime", "UNIX\_MS"]\n target =\> "msgSubmissionTime"\n }\n date {\n match =\> ["msgDeliveryTime", "UNIX\_MS"]\n target =\> "msgDeliveryTime"\n }\n date {\n match =\> ["eventTs", "UNIX\_MS"]\n target =\> "eventTs"\n }\n\n\n mutate {\n convert =\> { \n\t"concatenated" =\> "boolean" \n\t"msgLength" =\> "integer"\n }\n }\n\n}\n\n\noutput {\n elasticsearch {\n hosts =\> ["192.168.1.116:9200"]\n manage\_template =\> false\n index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"\n document\_type =\> "%{[@metadata][type]}"\n }\n}\n\n\n", :reason=\>"Expected one of #, =\> at line 39, column 8 (byte 561) after output {\n elasticsearch {\n hosts =\> ["192.168.1.116:9200"]\n manage\_template =\> false\n if "}

Any idea why is this?

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [June 27, 2017, 8:28am UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/7 "2017-06-27T08:28:27Z")

</div>

> [@magnusbaeck](#):
>
> it be an option to configure the input so that it'll tag the messages in a way so that later stages in the pipeline knows what to do about them?

I also tried to tag the different logs by doing the following:

```
- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /opt/modulo/log/smsc.full.*.log
    - /opt/modulo/smsc/cdr/*.cdr
    tags: ["cdr"]
    - /opt/modulo/smsc/cdr/traffic*
    tags: ["traffic"]
    #- c:\programdata\elasticsearch\logs\*

```

but this resulted in:

`Exiting: error loading config file: yaml: line 23: did not find expected '-' indicator`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 27, 2017, 8:35am UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/8 "2017-06-27T08:35:48Z")

</div>

You can't have conditionals _inside_ the elasticsearch output. You need this:

```nohighlight
if ... {
  elasticsearch {
    index => "x"
    ...
  }
} else {
  elasticsearch {
    index => "y"
    ...
  }
}

```

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [June 27, 2017, 9:12am UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/9 "2017-06-27T09:12:27Z")

</div>

Understood 🙂  
Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2017, 9:12am UTC](https://discuss.elastic.co/t/how-to-route-logs-to-different-indexs/89852/10 "2017-07-25T09:12:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.