# Creating index depends on source

**URL:** <https://discuss.elastic.co/t/creating-index-depends-on-source/167061>\
**Category:** Logstash\
**Created:** [February 5, 2019, 7:15am UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061 "2019-02-05T07:15:09Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kanthasamyraja](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@Kanthasamyraja](https://discuss.elastic.co/u/Kanthasamyraja)\
**Post date:** [February 5, 2019, 7:15am UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/1 "2019-02-05T07:15:10Z")

</div>

I am new to Elastic Stack.Using version :6.5.4

Data flow like : Filebeat -\> Logstash -\> ES-\> Kibana

Configured Filebeat on SIT and UAT environment to common Log server.

Would like to create separate index for SIT and UAT.

have written 30-elasticsearch-output.conf in Logstash like below  
output {  
if [@fields][env] == "SIT" {  
elasticsearch {  
hosts =\> ["\*\*\*\*\*\*:9200"]  
manage\_template =\> false  
index =\> "sit-%{+YYYY.MM.dd}"  
}  
} else if [@fields][env] == "UAT" {  
elasticsearch {  
hosts =\> ["\*\*\*\*\*\*:9200"]  
manage\_template =\> false  
index =\> "uat-%{+YYYY.MM.dd}"  
}  
} else {  
elasticsearch {  
hosts =\> ["\*\*\*\*\*\*:9200"]  
manage\_template =\> false  
index =\> "my-%{+YYYY.MM.dd}"  
}  
}  
stdout { codec =\> rubydebug }  
}

Logstash output

"message" =\> "04-Feb-2019 12:59:03.954 INFO [Thread-6] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-9999"]",  
"beat" =\> {  
"name" =\> "deala0",  
"hostname" =\> "deala0",  
"version" =\> "6.5.4"  
},  
"fields" =\> {  
"env" =\> "SIT"  
}  
}

Always going to else part and creating index my.

Would be helpful if anyone can help on this.

Thanks

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [February 5, 2019, 8:35am UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/2 "2019-02-05T08:35:10Z")

</div>

Hi @Kanthasamyraja,  
To achieve this I have done the following.

In Filebeat I set some fields like

```
- type: log
  paths:
    - /var/log/sensu/*.log
  encoding: plain
  fields:
    log_prefix: dc
    log_idx: sensu-logs
  fields_under_root: false

```

In Logstash I create @metadata fields based on these.

```
# Adding @metadata needed for index sharding to Filebeat logs
mutate {
  copy => {
   "[fields][log_prefix]" => "[@metadata][log_prefix]"
   "[fields][log_idx]" => "[@metadata][index]"
  }
}

```

And my Elasticsearch output if Logstash can then be just

```
  elasticsearch {
        hosts => ["10.1.1.1:9200", "10.1.1.2:9200", "10.1.1.3:9200"]
        index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"
  }

```

You can drop one of the @metadata fields if you just need the one "switch".

Hope that helps,  
AB

---

<div class="post-metadata">

**Author:** ![Kanthasamyraja](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@Kanthasamyraja](https://discuss.elastic.co/u/Kanthasamyraja)\
**Post date:** [February 5, 2019, 10:31am UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/3 "2019-02-05T10:31:23Z")

</div>

it has worked. Thanks @A_B

---

<div class="post-metadata">

**Author:** ![Aashish\_Chugh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aashish_chugh/32/40129_2.png) [@Aashish\_Chugh](https://discuss.elastic.co/u/Aashish_Chugh)\
**Post date:** [February 28, 2019, 6:18am UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/4 "2019-02-28T06:18:08Z")

</div>

I also applied same but i am getting %{[@metadata][log\_prefix]}-%{[@metadata][index]}-2019.02.28 as a index.

i am using filebeat 6.5.4 version.

Any help ?

---

<div class="post-metadata">

**Author:** ![Aashish\_Chugh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aashish_chugh/32/40129_2.png) [@Aashish\_Chugh](https://discuss.elastic.co/u/Aashish_Chugh)\
**Post date:** [February 28, 2019, 6:21am UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/5 "2019-02-28T06:21:12Z")

</div>

I am using below logstash configuration .

input {  
beats {  
port =\> 5044  
host =\> "0.0.0.0"  
}  
}  
filter {  
if [fileset][module] == "mysql" {  
if [fileset][name] == "error" {  
grok {  
match =\> { "message" =\> ["%{LOCALDATETIME:[mysql][error][timestamp]} ([%{DATA:[mysql][error][level]}] )?%{GREEDYDATA:[mysql][error][message]}",  
"%{TIMESTAMP\_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread\_id]} [%{DATA:[mysql][error][level]}] %{GREEDYDATA:[mysql][error][message1]}",  
"%{GREEDYDATA:[mysql][error][message2]}"] }  
pattern\_definitions =\> {  
"LOCALDATETIME" =\> "[0-9]+ %{TIME}"  
}  
remove\_field =\> "message"  
}  
mutate {  
rename =\> { "[mysql][error][message1]" =\> "[mysql][error][message]" }  
}  
mutate {  
rename =\> { "[mysql][error][message2]" =\> "[mysql][error][message]" }  
}  
date {  
match =\> ["[mysql][error][timestamp]", "ISO8601", "YYMMdd H:m:s" ]  
remove\_field =\> "[mysql][error][time]"  
}  
}  
else if [fileset][name] == "slowlog" {  
grok {  
match =\> { "message" =\> ["^# User@Host: %{USER:[mysql][slowlog][user]}([[^]]+])? @ %{HOSTNAME:[mysql][slowlog][host]} [(IP:[mysql][slowlog][ip])?](\s_Id:\s_ %{NUMBER:[mysql][slowlog][id]})?\n# Query\_time: %{NUMBER:[mysql][slowlog][query\_time][sec]}\s\* Lock\_time: %{NUMBER:[mysql][slowlog][lock\_time][sec]}\s\* Rows\_sent: %{NUMBER:[mysql][slowlog][rows\_sent]}\s\* Rows\_examined: %{NUMBER:[mysql][slowlog][rows\_examined]}\n(SET timestamp=%{NUMBER:[mysql][slowlog][timestamp]};\n)?%{GREEDYMULTILINE:[mysql][slowlog][query]}"] }  
pattern\_definitions =\> {  
"GREEDYMULTILINE" =\> "(.|\n)\*"  
}  
remove\_field =\> "message"  
}  
date {  
match =\> ["[mysql][slowlog][timestamp]", "UNIX" ]  
}  
mutate {  
copy =\> {  
"[fields][log\_prefix]" =\> "[@metadata][log\_prefix]"  
"[fields][log\_idx]" =\> "[@metadata][index]"  
}  
}  
}  
}  
}  
output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "%{[@metadata][log\_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> rubydebug }  
}

and this is my filebeat.yml

# Paths that should be crawled and fetched. Glob based paths.

paths:  
- /home/centos/\*.log  
document\_type: db\_log  
fields:  
log\_prefix: dc  
log\_idx: sensu-logs  
fields\_under\_root: true

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [February 28, 2019, 11:54am UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/6 "2019-02-28T11:54:56Z")

</div>

It is very difficult to read your config when it is not properly formatted. Could you try to select the config parts and click `</>` from the tools please.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 28, 2019, 2:15pm UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/7 "2019-02-28T14:15:21Z")

</div>

> [@Aashish\_Chugh](#):
>
> fields:  
> log\_prefix: dc  
> log\_idx: sensu-logs  
> fields\_under\_root: true

If you set fields\_under\_root then you will not have a field called "[fields][log\_prefix]", it will just be [log\_prefix].

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2019, 2:15pm UTC](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/8 "2019-03-28T14:15:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.