A way to ensure additional field will be with correct type inside index

Elastic Stack Elasticsearch
,
1

Hello,

I was using the scripted field to get a date field from system.uptime.duration.ms field value. I used below painless script.

image
image468×510 30.5 KB 

String val = "system.uptime.duration.ms";

if (doc[val] != null){ 
    if (doc[val].size() != 0 ){
    
        long now = doc["@timestamp"].value.toInstant().toEpochMilli();
    
        long elapsedTime = now - doc[val].value;
    
        return elapsedTime;
    }
}

I wanted to migrate from this solution but not be forced to remember to add some additional fields whenever I do beats update.

I tried two painless scripts for the default pipeline.

1.) This one sets value from the object of ZonedDateTime but elasticsearch dynamically recognizes it as a keyword which messes up my dashboard.

if (ctx?.system?.uptime?.duration?.ms != null) {
        ZonedDateTime zdt = ZonedDateTime.parse(ctx['@timestamp']);
        long now = zdt.toInstant().toEpochMilli();
        long elapsedTime = now - ctx.system.uptime.duration.ms;
        Instant instant = Instant.ofEpochMilli(elapsedTime);
        ZonedDateTime zdt = ZonedDateTime.ofInstant(instant, ZoneId.of('Z'));
        ctx['system.uptime.since'] = zdt;
}

image
image565×637 45.6 KB

2.) This one sets it up as long but I cannot set up the date format inside the index pattern. So just big number is displayed inside of the date.

if (ctx?.system?.uptime?.duration?.ms != null) {
        ZonedDateTime zdt = ZonedDateTime.parse(ctx['@timestamp']);
        long now = zdt.toInstant().toEpochMilli();
        long elapsedTime = now - ctx.system.uptime.duration.ms;
        ctx['system.uptime.since'] = elapsedTime ;
}

image
image579×678 46.7 KB

Is there a way besides adding manually mapping to the index template that says that this field should be recognized as date not long or keyword?

ELK/Kibana 7.17-5

2

Honestly, your best bet is to do this in a template so that it's definitively matched.

Alternatively did you look at runtime fields?

3

I have just added this field as a runtime field to the index template of metricbeat indices.

image
image708×498 27 KB

I don't know how to force the index pattern to recognize this field. I guess I need to wait for a new index to generate from the updated index template or is there another way?

I think that this will work when the index pattern finally refreshes but please tell me what would be the best approach to make this field (automatically if possible) be added to every new metricbeat index template when I run metricbeat -e setup template command after updating beat to a new version?

4

Yes. But you can also add it directly to the mapping of the current index and it will start creating the field.

5

I just changed the index name for one of the beats. It works.

image

1.) What is the best way(as automatic as possible) to add this field to every new template that will be uploaded after a beat update and command metricbeat -e setup template.

2.) What is the best way to introduce this new field to the currently working weekly index? Some kind of reindex? The approach of reindexing the current index(i.e metricbeat-7.17.1-system-2022.30) to some other name(i.e metricbeat-7.17.1-system-2022.30_reindex) and just deleting(metricbeat-7.17.1-system-2022.30) it so the new would be generated would be appropriate? How to prevent any data loss?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.