# Create a custom id

**URL:** <https://discuss.elastic.co/t/create-a-custom-id/119951>\
**Category:** Logstash\
**Created:** [February 15, 2018, 9:24am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951 "2018-02-15T09:24:08Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nayaz\_JH](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\
**Post date:** [February 15, 2018, 9:24am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/1 "2018-02-15T09:24:08Z")

</div>

"\_index": "access\_log",  
"\_type": "access\_logs",  
"\_id": "0udbmGEBevLW4jI4We7N",  
"\_score": 1,  
"\_source": {  
"@timestamp": "2018-02-15T07:26:18.252Z",  
"path": "V:/cc/7.7/mcs\_7\_indonesia\_dev/mpower/out/log/MessagingBroker\_access.log",

i want to change above id "\_id": "0udbmGEBevLW4jI4We7N", to 1 and auto increment it.

input{  
file{  
path =\> "V:/cc/7.7/mcs\_7\_indonesia\_dev/mpower/out/log/MessagingBroker\_access.log"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
if [message] =~ /^=/ {  
drop { }  
}  
}

output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "access\_log"  
document\_type =\> "access\_logs"  
}

stdout { }  
}  
Above is the config file and the above \_id is auto generating.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 15, 2018, 9:34am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/2 "2018-02-15T09:34:29Z")

</div>

Why do you need an autoincrementing id?

---

<div class="post-metadata">

**Author:** ![Nayaz\_JH](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\
**Post date:** [February 15, 2018, 9:43am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/3 "2018-02-15T09:43:01Z")

</div>

Just for the reference like as we do in mysql.

This id 0udbmGEBevLW4jI4We7N i am not able to understand.

is there any way change the above to 1 and keep on auto increment it

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 15, 2018, 9:47am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/4 "2018-02-15T09:47:54Z")

</div>

Then you will need to generate the ID externally in your application. Maintaining and generating strictly incrementing IDs in a distributed system can quickly become the bottleneck as it requires a lot of coordination across a cluster, which is why the current scheme is used.

---

<div class="post-metadata">

**Author:** ![Nayaz\_JH](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\
**Post date:** [February 15, 2018, 9:57am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/5 "2018-02-15T09:57:03Z")

</div>

Thank you

---

<div class="post-metadata">

**Author:** ![Nayaz\_JH](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\
**Post date:** [February 15, 2018, 9:58am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/6 "2018-02-15T09:58:18Z")

</div>

```
 "_index": "access_log",
    "_type": "access_logs",
    "_id": "0udbmGEBevLW4jI4We7N",
    "_score": 1,
    "_source": {
      "@timestamp": "2018-02-15T07:26:18.252Z",
      "path": "V:/cc/7.7/mcs_7_indonesia_dev/mpower/out/log/MessagingBroker_access.log",
      "message": "14/02/2018 13:14:56:800|MessagingRepository |INF|RGS-Nayaz-LT,MessagingBroker,PhilMoOptServlet,MO Request,4896,2018-02-14,null,639155146696,null,null,null,null,1,120,0,null,SUCCESS##MO Response : [200],null\r",
      "@version": "1",
      "host": "RGS-Nayaz-LT"
    }
  }

```

"message": "14/02/2018 13:14:56:800|MessagingRepository |INF|RGS-Nayaz-LT,MessagingBroker,PhilMoOptServlet,MO Request,4896,2018-02-14,null,639155146696,null,null,null,null,1,120,0,null,SUCCESS##MO Response : [200],null\r",

i want to storte above fields in specified columns.  
how i can do it

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 15, 2018, 10:03am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/7 "2018-02-15T10:03:37Z")

</div>

Use a combination of filters to extract the data into appropriate fields, e.g. [grok](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html), [dissect](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html) and/or [csv](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html) filters. It should be possible to find good examples by searching this forum.

---

<div class="post-metadata">

**Author:** ![Nayaz\_JH](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\
**Post date:** [February 15, 2018, 10:07am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/8 "2018-02-15T10:07:40Z")

</div>

i used csv i am favcing the errors below is my config file

input{  
file{  
path =\> "V:/cc/7.7/mcs\_7\_indonesia\_dev/mpower/out/log/MessagingBroker\_access.log"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}

filter{

if [message] =~ /^=/ {  
drop { }  
}  
csv{  
separator =\> ","  
columns =\> ["HOST","PROCESS","INTERFACE","METHOD","DURATION",  
"END\_TIME","URL","SUBSCRIBER\_ID","DEVICE","SESSION\_ID",  
"ENTITY\_ID","ENTITY\_NAME","CALL\_DIRECTION","TIMEZONE\_OFFSET",  
"STATUS","SERVICE\_PROVIDER","INPUT","NUMBER\_OF\_RESULTS",  
"DEVICE\_MODEL","STATUS\_DESC","ORIGIN"]  
}  
mutate {  
convert =\>{  
"DURATION" =\> "integer"  
"END\_TIME" =\>"timestamp"  
"CALL\_DIRECTION" =\> "integer"  
"TIMEZONE\_OFFSET" =\> "integer"  
"STATUS" =\> "integer"  
"NUMBER\_OF\_RESULTS" =\> "integer"  
}  
}  
}

output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "access\_log"  
document\_type =\> "access\_logs"  
}

stdout { }  
}

 ![Screenshot%20(6)](https://us1.discourse-cdn.com/elastic/original/3X/7/1/710788b50c364a334765730e5c35ebb57cd775d9.png)  
the attached screen shot is my log file contains the following data

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 15, 2018, 10:12am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/9 "2018-02-15T10:12:22Z")

</div>

> [@Nayaz\_JH](#):
>
> "END\_TIME" =\>"timestamp"

It would help if you explain what is not working and what error you are seeing. Note that `timestamp` is [not a valid conversion type](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-convert). For this field you will need to use the [date filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html).

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 15, 2018, 10:13am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/10 "2018-02-15T10:13:21Z")

</div>

Please do not post screenshots of text or data as it is hard to see and impossible to search.

---

<div class="post-metadata">

**Author:** ![Nayaz\_JH](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\
**Post date:** [February 15, 2018, 10:16am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/11 "2018-02-15T10:16:08Z")

</div>

[2018-02-15T15:45:00,256][ERROR][logstash.pipeline] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::FilterDelegator:0x39e26b91 @metric\_events\_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric\_events\_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric\_events\_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @id="116ee375390e8eca6f8c6da6b952c1a682d7acd0274595dd6ebacfad717f1a9e", @klass=LogStash::Filters::Mutate, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0x29efd3b3 @metric=#\<LogStash::Instrument::Metric:0x4a40761e @collector=#\<LogStash::Instrument::Collector:0x7bc75cb6 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x51530ebc @store=#\<Concurrent:🗺0x00000000000fb0 entries=3 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0xe2c8c9b, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb4 entries=70 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :filters, :"116ee375390e8eca6f8c6da6b952c1a682d7acd0274595dd6ebacfad717f1a9e", :events]\>, @filter=\<LogStash::Filters::Mutate convert=\>{"DURATION"=\>"integer", "END\_TIME"=\>"date", "CALL\_DIRECTION"=\>"integer", "TIMEZONE\_OFFSET"=\>"integer", "STATUS"=\>"integer", "NUMBER\_OF\_RESULTS"=\>"integer"}, id=\>"116ee375390e8eca6f8c6da6b952c1a682d7acd0274595dd6ebacfad717f1a9e", enable\_metric=\>true, periodic\_flush=\>false\>\>", :error=\>"translation missing: en.logstash.agent.configuration.invalid\_plugin\_register", :thread=\>"#\<Thread:0x70e90dd7 run\>"}  
[2018-02-15T15:45:00,301][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<LogStash::ConfigurationError: translation missing: en.logstash.agent.configuration.invalid\_plugin\_register\>, :backtrace=\>["C:/Users/Nayaz/Downloads/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.2.0/lib/logstash/filters/mutate.rb:190:in `block in register'", "org/jruby/RubyHash.java:1343:in`each'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/logstash-filter-mutate-3.2.0/lib/logstash/filters/mutate.rb:188:in `register'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/logstash-core/lib/logstash/pipeline.rb:341:in`register\_plugin'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/logstash-core/lib/logstash/pipeline.rb:352:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in`each'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/logstash-core/lib/logstash/pipeline.rb:352:in `register_plugins'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/logstash-core/lib/logstash/pipeline.rb:736:in`maybe\_setup\_out\_plugins'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/logstash-core/lib/logstash/pipeline.rb:362:in `start_workers'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/logstash-core/lib/logstash/pipeline.rb:289:in`run'", "C:/Users/Nayaz/Downloads/logstash-6.2.0/logstash-core/lib/logstash/pipeline.rb:249:in `block in start'"], :thread=\>"#\<Thread:0x70e90dd7 run\>"}  
[2018-02-15T15:45:00,349][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: LogStash::PipelineAction::Create/pipeline\_id:main, action\_result: false", :backtrace=\>nil}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 15, 2018, 10:19am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/12 "2018-02-15T10:19:05Z")

</div>

> [@Nayaz\_JH](#):
>
> \<LogStash::Filters::Mutate convert=\>{"DURATION"=\>"integer", "END\_TIME"=\>"date", "CALL\_DIRECTION"=\>"integer", "TIMEZONE\_OFFSET"=\>"integer", "STATUS"=\>"integer", "NUMBER\_OF\_RESULTS"=\>"integer"}

Please consult the documentation I linked to. You can not cast it to a date - you have to use a separate filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 15, 2018, 10:19am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951/13 "2018-03-15T10:19:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.