How to set "_id" value in elasticsearch document as my custom document id

[r.ganeshbabu (ganeshbabu)]

Hi All,

We are developing a customized beats for the monitoring tool and we are capturing metrics like cpu, memory, disk etc.. Data was sending it through output logstash which is configured in beat.yml file as like below,

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["localhost:5044"]

We are receiving fields from the monitoring tool and one of the field as test_serial_id and I want to set the test_serial_id as "_id" in the elasticsearch document in order to remove the duplicates.

How to achieve this in beats? Please share your thoughts and it would be very helpful.

Thanks,
Ganeshbabu R

[steffens (Steffen Siering)]

You can generate an ID and pubish it by setting adding the ID to your events meta data:

event := beat.Event{
  Timestamp: ...
  Fields: ...
  Meta: common.MapStr{"id": <unique event id>}
}

When publishing to elasticsearch, the id value will be used for _id. When publishing to logstash/redis/kafka, beats add a @metadata field to the event. This field will contain the id. You can configure the elasticsearch output in Logstash to use [@metadata][id].

When indexing, elasticsearch accepts an operation type. If you set it to 'create', then duplicate entries will be detected, but not overwritten. If you don't set it to create, then the old entry will be marked as deleted and the new event will be indexed.

[r.ganeshbabu (ganeshbabu)]

Thanks for sharing your inputs @steffens

@steffens

You can configure the elasticsearch output in Logstash to use [@metadata][id]

But we configured beats port in .yml file and sending data to port 5044 via output.logstash.

Can I send the document id through output.logstash?

Let me know your thoughts.

Regards,
Ganeshbabu R

[steffens (Steffen Siering)]

Can I send the document id through output.logstash?

Yes. Please read this paragraph again:

When publishing to logstash/redis/kafka, beats add a @metadata field to the event. This field will contain the id. You can configure the elasticsearch output in Logstash to use [@metadata][id].

You can test and inspect events in LS via:

input {
  beat { ... }
}

output {
  stdout {
    codec => rubydebug { metadata => true }
  }
}

This will print all events, include the @metadata section, which should contain the id field. If it's missing, then you need to update libbeat, or you didn't add it to your event in the beat.

[r.ganeshbabu (ganeshbabu)]

Hi @steffens,

@steffens

This will print all events, include the @metadata section, which should contain the id field. If it's missing, then you need to update libbeat, or you didn't add it to your event in the beat.

We tried printing all the events and we did find the metadata section in the result and below is the code snippet we have added in event beat,

event := beat.Event{
        Timestamp: time.Unix((i.Timestamp/1000), 0),
        Fields:common.MapStr{
                        "type" :"traversebeat",
                        "TestSerialNumber": TestSerialNumber,
                        "DeviceSerialNumber": DeviceSerialNumber,
                        "AccountSerialNumber": AccountSerialNumber,          
                },
                Meta: common.MapStr{"id": doc_id},
        }

and below is the printed event from the logstash, and I can see that @metadata "custom id" value is present,

{
      "type" => "testbeat",
     "DeviceSerialNumber" => 6020621,
    "AccountSerialNumber" => 5730000,
       "TestSerialNumber" => 6021700,
               "MinValue" => 446521,
                   "beat" => {
        "hostname" => "localhost",
            "name" => "localhost",
         "version" => "6.2.0"
    },    
              "@metadata" => {
           "beat" => "testbeat",
             "id" => "SCO-SRSWEB-PRD1-1534493100000-6021700",
           "type" => "doc",
        "version" => "6.2.0"
    },           
}

But when I sent the same data to beats port 5044 in logstash the _id value is not picking the same as above. Its creating a generated ID in the document.

Kindly share your thoughts.

Regards,
Ganeshbabu R

[steffens (Steffen Siering)]

Beats just forward some info via @metadata. Beats don't force you to make use of these information. You have to configure the Elasticsearch output in Logstash to use the id. Check out the Logstash elasticsearch outputs docs. Settings you might be interested in: action, document_id. Configure action => "create" and document_id => "[@metadata][id]".