I need add field, and do value counter, how do it?

Elastic Stack Elasticsearch
1

My config is:

======================================

input {
        file{
          path => "/home/elk/domains/rf_domains_test"
          start_position => "beginning"
          sincedb_path => "/dev/null"
        }
}

filter {
        grok {
                match => { "message" => "%{HOSTNAME:Domain}%{SPACE}%{USERNAME:Registrator}%{SPACE}%{DATE_EU:Created}%{SPACE}%{DATE_EU:Paid-till}%{SPACE}%{DATE_EU:Free-date}" }
                add_field => { "ID" => "Hello world" }
        }

        mutate {
                remove_field => ["message", "@timestamp", "path", "host", "@version"]
                }

        date {
                match => ["Created","DD.MM.YYYY"]
                target => "Created"
             }

        date {
                match => ["Paid-till","DD.MM.YYYY"]
                target => "Paid-till"
                }

        date {
                match => ["Free-date","DD.MM.YYYY"]
                target => "Free-date"
                }


}

output {
        elasticsearch {
            hosts    => "localhost:9200"
            index    =>"domains_rf_test"
        }
        stdout {
           codec => rubydebug
        }
}

======================================

Now every document i add field - "ID" => "Hello world"

Me need:

"ID" => "1"
"ID" => "2"
"ID" => "3"
"ID" => "4"
"ID" => "5"
.......

How do it?

2

You need to hold state in Logstash, but it is not explicitly supported.

What is your purpose of using such incremental ids? Elasticsearch default _id is not enough?

3
  1. Thank you for you replay.
  2. May be you khow different solve this task?
  3. May be you know how not use standart -
    id for example_id:o_TT6X4B7HVNK6zt50Q8

i need

_id:1
_id:2

Not has this data (1,2,3...) in input ?

4

I need to get each document one by one, if incremental ids this is easy.

5

As it is difficult, you may need some alternatives. If you would fully explain the situation and the necessity, there could be some proposal.

6

Ok

  1. I has index, my index for example Contain 1000 docs
  2. I took 1 docs, took need me field, use it for my #bash script, get output my bash script, and POST new data in this doc.

I need do it for my 1000 docs.

I tnink my me i can use it -

7

Thank you, please answer for my question.

8

There is no need for incremental id.

If the size is only 1000, you can get the whole document with

{
  "size": 10000
  "query": {"match_all": {}}
}

If you need pagination, any sort should be ok. See NOTE in Search after.

Anyway, there are _id fields (something like {"_id" : "FaslK3QBySSL_rrj9zM5"}), where unique values are automatically assigned by Elasticsearch, to identify documents.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.