# Not getting dynamic index in elasticsearch using logstash

**URL:** <https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514>\
**Category:** Logstash\
**Created:** [April 16, 2017, 7:18pm UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514 "2017-04-16T19:18:07Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![NITIN-BHAISARE](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@NITIN-BHAISARE](https://discuss.elastic.co/u/NITIN-BHAISARE)\
**Post date:** [April 16, 2017, 7:18pm UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/1 "2017-04-16T19:18:07Z")

</div>

Hello,

I am facing issues when trying to create dynamic indexes using logstash. I have different log events coming from a single file,so i wanted to make them identifiable in the filter section using the identifier present in the log event. Also as per the identifier, I want to create index.  
i tried to make it possible with below given configuration but logstash stops and no error message is given out. I am stuck at this point. Could somebody please help me with this.  
Below is my configuration

> input  
> {  
> file  
> {  
> path =\> ["/var/log/bulk/sample\_audit.json"]  
> sincedb\_path =\> "/dev/null"  
> start\_position =\> "beginning"  
> }  
> }

> filter {  
> json { source =\> message }  
> ruby {  
> code =\> "event['logtype'] = event['preinfo']['task']['log']jobname"  
> }  
> split {  
> field =\> "preinfo[task][log][logmessage]"  
> target =\> "output"  
> }  
> }

> output  
> {  
> elasticsearch {  
> hosts =\> ["10.201.181.204:9200"]  
> sniffing =\> false  
> manage\_template =\> false  
> index =\> "%{logtype}-%{+YYYY.MM.dd}"  
> }  
> }

Could some one please help me out with this??

Thanks  
Nitin

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 17, 2017, 7:45am UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/2 "2017-04-17T07:45:20Z")

</div>

> [@NITIN-BHAISARE](#):
>
> i tried to make it possible with below given configuration but logstash stops and no error message is given out.

Can you explain this more please, what do you mean, what do you see, can you post logs or any output?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 17, 2017, 11:23am UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/3 "2017-04-17T11:23:38Z")

</div>

Do all the events have the field `logtype` defined? If this is not the case I suspect the index name could end up being invalid;id, but that should show up in the logs. have you tried outputting them to stdout using a rubydebug codec?

---

<div class="post-metadata">

**Author:** ![NITIN-BHAISARE](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@NITIN-BHAISARE](https://discuss.elastic.co/u/NITIN-BHAISARE)\
**Post date:** [April 17, 2017, 11:41am UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/4 "2017-04-17T11:41:15Z")

</div>

Thanks @Christian_Dahlqvist @warkolm i have changed my configuration. Below is my configuration

> ```
> input
> {
> file
> {
> path => ["/var/log/bulk/sample_audit.json"]
> sincedb_path => "/dev/null"
> start_position => "beginning"
> }
> }
> 
> ```

> ```
> filter {
> json {	source => message }
> if ["preinfo[task]jobname"] == "sudoers:linux"{
> split {
> field => "preinfo[task][log][logmessage]"
> target => "output"
> }
> }
> }
> output 
> {
> elasticsearch {
> hosts => ["10.201.181.204:9200"]
> sniffing => false
> manage_template => false
> index => "%{[preinfo[task]jobname]-%{+YYYY.MM.dd}"
> }
> } 
> 
> ```

With this config i can create the desired index as per the jobname but now the problem is I cannot split nested arrays present in the log event and intresting part is i can get my desired result after removing if conditions.

What i am doing wrong?  
Is that the right way to use conditions in logstash??

---

<div class="post-metadata">

**Author:** ![NITIN-BHAISARE](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@NITIN-BHAISARE](https://discuss.elastic.co/u/NITIN-BHAISARE)\
**Post date:** [April 17, 2017, 1:01pm UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/5 "2017-04-17T13:01:28Z")

</div>

can someone please help me on this ??

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 20, 2017, 5:06am UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/6 "2017-04-20T05:06:16Z")

</div>

> ```
> if ["preinfo[task]jobname"] == "sudoers:linux"{
> 
> ```

Do this instead:

```
if [preinfo][task][jobname] == "sudoers:linux" {

```

> ```
> index => "%{[preinfo[task]jobname]-%{+YYYY.MM.dd}"
> 
> ```

Do this instead:

```
 index => "%{[preinfo][task][jobname]}-%{+YYYY.MM.dd}"

```

Use a `stdout { codec => rubydebug }` output until you've verified that your events look like you expect them to. Only then is it useful to enable an elasticsearch output.

---

<div class="post-metadata">

**Author:** ![NITIN-BHAISARE](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@NITIN-BHAISARE](https://discuss.elastic.co/u/NITIN-BHAISARE)\
**Post date:** [April 20, 2017, 12:26pm UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/7 "2017-04-20T12:26:49Z")

</div>

Thanks @magnusbaeck ...i made the changes..but now this time its not showing anything. not even in the logs. I am using logstash 2.4 so i thought that might be a bug in 2.4 but its not wotking in 5.2 also. Anything else i need to do ??

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 20, 2017, 1:11pm UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/8 "2017-04-20T13:11:38Z")

</div>

It sounds quite unlikely that the changes I suggested would choke everything completely. Did you switch to a stdout output as I suggested? Have you increased the Logstash log level to get more clues in the log output?

---

<div class="post-metadata">

**Author:** ![NITIN-BHAISARE](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@NITIN-BHAISARE](https://discuss.elastic.co/u/NITIN-BHAISARE)\
**Post date:** [April 20, 2017, 4:01pm UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/9 "2017-04-20T16:01:19Z")

</div>

@magnusbaeck Thanks.. i messed up the stdout part. I made it right and now it works like a charm.  
Thanks a lot @magnusbaeck again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2017, 4:06pm UTC](https://discuss.elastic.co/t/not-getting-dynamic-index-in-elasticsearch-using-logstash/82514/10 "2017-05-18T16:06:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.