# Logstash dynamic id

**URL:** <https://discuss.elastic.co/t/logstash-dynamic-id/122627>\
**Category:** Logstash\
**Created:** [March 6, 2018, 2:32am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627 "2018-03-06T02:32:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![1243user](https://avatars.discourse-cdn.com/v4/letter/1/e99b99/32.png) [@1243user](https://discuss.elastic.co/u/1243user)\
**Post date:** [March 6, 2018, 2:32am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/1 "2018-03-06T02:32:45Z")

</div>

i am using elasticsearch + logstash + kibana .  
i have i question , i get log data like this :  
2018-03-01 18:03:13.504 INFO 30613 --- [nio-8040-exec-3] com.gh.controller.LogController : log/SearchIndex requestdata：|index:oodkfke|type:user|id=1|data={"data1":"data11","data2":"data22"}|

i want to use oodkfke as index, user as type , 1 as id ,then save the data into elasticsearch to use this index, type, id

you can look at logstash my config file , i donnot know where is the error ?  
by the way ,i am not good at gork .

input {  
file {  
type =\> "tomcatlog"  
path =\> ["/usr/local/tomcat8/logs/catalina.out"]  
discover\_interval =\> 1  
#start\_position =\> "beginning"  
sincedb\_path =\> "/usr/local/logstash/config/sincedb\_order.txt"  
sincedb\_write\_interval =\> 1   
codec =\> multiline {  
charset =\> "UTF-8"  
pattern =\> "^%{DATESTAMP\_CN}"  
negate =\> true  
what =\> "next"  
}  
}  
file {  
type =\> "demon"  
path =\> ["/usr/local/elasticsearch/log/demon.log"]  
discover\_interval =\> 1  
sincedb\_path =\> "/usr/local/logstash/config/sincedb\_demon.txt"  
sincedb\_write\_interval =\> 1   
codec =\> multiline {  
charset =\> "UTF-8"  
pattern =\> "^%{DATESTAMP\_CN}"  
negate =\> true  
what =\> "next"  
}  
}  
}

# 输出到 elasticsearch

filter {  
if [type] == "tomcatlog"{  
grok {  
match =\> { "message" =\> "%{DATESTAMP\_CN:[@metadata][logdate]} .\* | %{WORD:index}|%{WORD:type}|id:%{DATA:id}|%{WORD:data}|" }  
}  
ruby {  
code =\> " event.set('indexid', '%{id}');  
event.set('typeid', '%{id}');  
event.set('idid', '%{id}');  
event.set('dataid', '%{id}')  
"  
}  
}else if [type] == "demon" {

```
}    

```

}  
output {  
if "\_grokparsefailure" not in [tags] and "\_dateparsefailure" not in [tags] {  
elasticsearch {  
index =\> "oodkfkd"  
document\_type =\> "syslog"  
document\_id =\> "%{idid}"  
hosts =\> ["192.168.198.132:9200"]  
manage\_template =\> true  
template\_overwrite =\> true  
template\_name =\> "orderservice"  
template =\> "/usr/local/my\_logstash\_template/orderservice\_template.json"  
}  
}else{  
elasticsearch {  
hosts =\> ["192.168.198.132:9200"]  
index =\> "err"  
document\_type =\> "err"  
}  
}  
stdout {  
# 1 把采集的数据输出到elasticsearch里面 2 以JSON格式输出  
codec =\> rubydebug  
#codec =\> json\_lines  
}  
}

---

<div class="post-metadata">

**Author:** ![1243user](https://avatars.discourse-cdn.com/v4/letter/1/e99b99/32.png) [@1243user](https://discuss.elastic.co/u/1243user)\
**Post date:** [March 6, 2018, 2:45am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/2 "2018-03-06T02:45:42Z")

</div>

i have seen this , i donnot know the reason why his server is ok .

> <https://stackoverflow.com/questions/25876212/elasticsearch-index-limitations>

input {  
tcp{  
port=\>3362  
type="mf\_data"  
codec=\>"json\_lines"  
}

}

filter{  
json{source=\>"message"}  
grok{match=\>"message","documentID:%{DATA:documentID}"]}  
}  
output{  
elasticsearch{  
host=\>"localhost"  
index\_type=\>"customType"  
index=\>"event\_%{documentID}"  
}  
}  
Input is {"domain":"[test.com](http://test.com)","documentID":"cAmii"}

---

<div class="post-metadata">

**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\
**Post date:** [March 6, 2018, 5:38am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/3 "2018-03-06T05:38:48Z")

</div>

Your question seems incomplete. what went wrong with your conf and what exactly is your requirement ? If u got any errors ping them over here.

---

<div class="post-metadata">

**Author:** ![1243user](https://avatars.discourse-cdn.com/v4/letter/1/e99b99/32.png) [@1243user](https://discuss.elastic.co/u/1243user)\
**Post date:** [March 6, 2018, 6:00am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/4 "2018-03-06T06:00:24Z")

</div>

my bad , not error ,  
this is the log : 2018-03-01 18:03:13.504 INFO 30613 --- [nio-8040-exec-3] com.gh.controller.LogController : log/SearchIndex 请求数据数据：|index:oodkfke|type:user|id=1|data={"data1":"data11","data2":"data22"}|  
i want to set the value of elasticsearch \_id to the value 1 ,and set elasticsearch index to oodkfke , and set elasticsearch type to user .  
but i cannot get the value from the log data .

> > tag\_v  
> > March 6 |

Your question seems incomplete. what went wrong with your conf and what exactly is your requirement ? If u got any errors ping them over here.

Visit Topic or reply to this email to respond.

In Reply To

> > 1243userGh  
> > March 6 |  
> > i have seen this , i donnot know the reason why his server is ok . input { tcp{ port=\>3362 type="mf\_data" codec=\>"json\_lines" } } filter{ json{source=\>"message"} grok{match=\>"message","documentID:%{DATA:documentID}"]} } output{ elasticsearch{ host=\>"localhost" index\_type=\>"custom…

Visit Topic or reply to this email to respond.

To unsubscribe from these emails, click here.

---

<div class="post-metadata">

**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\
**Post date:** [March 6, 2018, 6:06am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/5 "2018-03-06T06:06:37Z")

</div>

> [@1243user](#):
>
> i want to set the value of elasticsearch \_id to the value 1 ,and set elasticsearch index to oodkfke , and set elasticsearch type to user

I thinks this is your requirement:

```
output {
 elasticsearch {
  hosts => ["loclahost:9200"]
  document_type => "user"
  document_id => "%{documentID}" // documentID you are getting from grok
}
}

```

---

<div class="post-metadata">

**Author:** ![1243user](https://avatars.discourse-cdn.com/v4/letter/1/e99b99/32.png) [@1243user](https://discuss.elastic.co/u/1243user)\
**Post date:** [March 6, 2018, 6:08am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/6 "2018-03-06T06:08:56Z")

</div>

i have tried , but not efficient , id is the elasticsearch default uuid .

---

<div class="post-metadata">

**Author:** ![1243user](https://avatars.discourse-cdn.com/v4/letter/1/e99b99/32.png) [@1243user](https://discuss.elastic.co/u/1243user)\
**Post date:** [March 6, 2018, 6:14am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/7 "2018-03-06T06:14:50Z")

</div>

1. List item  
my log data is 🙂

2018-03-01 18:03:13.504 INFO 30613 --- [nio-8040-exec-3] com.gh.controller.LogController : log/SearchIndex requestdata：|index:oodkfke|type:user|id=1|data={"data1":"data11","data2":"data22"}|

my gork is :

```
   match => { "message" => "%{DATESTAMP_CN:[@metadata][logdate]} .* | %{WORD:index}|%{WORD:type}|id:%{DATA:id}|%{WORD:data}|" }

```

your mean i use this ?  
output {  
elasticsearch {  
hosts =\> ["loclahost:9200"]  
document\_type =\> "user"  
document\_id =\> "%{id}" //  
}  
}

---

<div class="post-metadata">

**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\
**Post date:** [March 6, 2018, 6:31am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/8 "2018-03-06T06:31:05Z")

</div>

Yep. It wil take vale of field "id" which was created from grok pattern for every msg and will append to elasticsearch output. Ping error if u got any

⁣Sent from Blue ​

---

<div class="post-metadata">

**Author:** ![1243user](https://avatars.discourse-cdn.com/v4/letter/1/e99b99/32.png) [@1243user](https://discuss.elastic.co/u/1243user)\
**Post date:** [March 6, 2018, 6:40am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/9 "2018-03-06T06:40:31Z")

</div>

error is idid is nil  
and \_id = %{idid}  
obviously, the value not trans to idid ,  
i doubt my gork is right ? ,  
this website [http://grokdebug.herokuapp.com/?#](http://grokdebug.herokuapp.com/?#)  
i put my gork and input to this ,but no response ,so i cannot judge whether my gork is right or not

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 3, 2018, 6:41am UTC](https://discuss.elastic.co/t/logstash-dynamic-id/122627/10 "2018-04-03T06:41:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.