I need add field, and do value counter, how do it?

My config is:


input {
          path => "/home/elk/domains/rf_domains_test"
          start_position => "beginning"
          sincedb_path => "/dev/null"

filter {
        grok {
                match => { "message" => "%{HOSTNAME:Domain}%{SPACE}%{USERNAME:Registrator}%{SPACE}%{DATE_EU:Created}%{SPACE}%{DATE_EU:Paid-till}%{SPACE}%{DATE_EU:Free-date}" }
                add_field => { "ID" => "Hello world" }

        mutate {
                remove_field => ["message", "@timestamp", "path", "host", "@version"]

        date {
                match => ["Created","DD.MM.YYYY"]
                target => "Created"

        date {
                match => ["Paid-till","DD.MM.YYYY"]
                target => "Paid-till"

        date {
                match => ["Free-date","DD.MM.YYYY"]
                target => "Free-date"


output {
        elasticsearch {
            hosts    => "localhost:9200"
            index    =>"domains_rf_test"
        stdout {
           codec => rubydebug


Now every document i add field - "ID" => "Hello world"

Me need:

"ID" => "1"
"ID" => "2"
"ID" => "3"
"ID" => "4"
"ID" => "5"

How do it?

You need to hold state in Logstash, but it is not explicitly supported.

What is your purpose of using such incremental ids? Elasticsearch default _id is not enough?

  1. Thank you for you replay.
  2. May be you khow different solve this task?
  3. May be you know how not use standart -
    id for example_id:o_TT6X4B7HVNK6zt50Q8

i need


Not has this data (1,2,3...) in input ?

I need to get each document one by one, if incremental ids this is easy.

As it is difficult, you may need some alternatives. If you would fully explain the situation and the necessity, there could be some proposal.


  1. I has index, my index for example Contain 1000 docs
  2. I took 1 docs, took need me field, use it for my #bash script, get output my bash script, and POST new data in this doc.

I need do it for my 1000 docs.

I tnink my me i can use it -

Thank you, please answer for my question.

There is no need for incremental id.

If the size is only 1000, you can get the whole document with

  "size": 10000
  "query": {"match_all": {}}

If you need pagination, any sort should be ok. See NOTE in Search after.

Anyway, there are _id fields (something like {"_id" : "FaslK3QBySSL_rrj9zM5"}), where unique values are automatically assigned by Elasticsearch, to identify documents.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.