Parsing metricbeat fields in logsatsh

Elastic Stack Beats
1

Hello,
I want to parse/filter the fields in Logstash, which are crated by metricbeat.
For example:
I want to store system.cpu.total.pct -> under custom field percentCpu

Reason for this change is:

I have 90% unix servers, where custom scripts are calculating 'percentCpu' and sending through filebeat.
On Windows machine we dont have such scripts so metricbeat need to be used.
Since visualization is using 'percentCpu' field for aggregation I have to do above change.

br,
Sunil.

2

Hi,

This functionality will be available in Beats 6.3 by the use of the new rename processor:

processors:
  - rename:
      fields:
        from: 'system.cpu.total.pct'
        to: 'percentCpu'

If you don't want to wait until 6.3 is out you can give it a try by using one of our daily snapshots:
https://beats-package-snapshots.s3.amazonaws.com/index.html?prefix=metricbeat/

Otherwise, you can use Logstash Mutate filter plugin or the Rename processor in Elasticsearch's Ingest Node.

3

It seems there are more features coming in Beats 6.3, Kibana 6.3 which I need desperately.
When are you guys releasing this version at least for non-production use?
What is the probable risk in using daily snapshot?
Is 'rename' processor available in today's snapshot?

br,
Sunil.

4

I'm sorry I can't give you a date when 6.3 will be out, as we don't announce release dates (delays happen and 6.3 involves many changes in the Elastic Stack).

I think your best option is to use one of our release candidates for 6.3. This way you can replace them easily with the final 6.3 when it comes out. The snapshots I shared before are from master (7.0.0-alpha1) so it can be tricky if later you want to use the official 6.3.

Here's metricbeat 64bit for Windows
Metricbeat 64bit for Linux (DEB)
Metricbeat 64bit for Linux (RPM)

Hope that covers everything.

5

Indeed. its a precise help!

thank you so much Tim!

6

Sorry to bother you Tim again.

How can I add more rename for example 4 fields to be renamed?

7

The correct way is to make fields an array, like this:

processors:
- rename:
    fields:
      - from: 'original_a'
        to: 'new_a'
      - from: 'original_b'
        to: 'new_b'

Every - starts a new element in the array.

See the documentation below, as you might be interested in using the ignore_missing or fail_on_error flags.

https://www.elastic.co/guide/en/beats/filebeat/master/rename-fields.html

8

Hello Tim,
rename is not working:

my MB configuration is as below.

  • module: system
    period: 10s
    metricsets:
    • cpu
      #- load
    • memory
      #- network
      #- process
      #- process_summary
      #- core
      #- diskio
      #- socket
      processes: ['.*']
      process.include_top_n:
      by_cpu: 5 # include top 5 processes by CPU
      by_memory: 5 # include top 5 processes by memory

#- module: system

metricsets:

- core

- diskio

- socket

  • module: system
    period: 1m
    metricsets:

    • filesystem
    • fsstat
      processors:
    • drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
    • rename:
      fields:
      from: 'beat.name'
      to: 'myHost'
      ignore_missing: false
      fail_on_error: true

  • module: system
    period: 15m
    metricsets:

    • uptime

#- module: system

period: 5m

metricsets:

- raid

raid.mount_point: '/'

9

I think your indentation is bad, but I can't tell for sure.

Can you enclose your configuration between triple-backtics? like this:
```
contents of configuration
```

10

hi,
It worked now.
However, I cant change the 'beat.name' to 'myHost'
How to change below fields?
beat.name
host.name
beat.hostname

using rename processor?

Br,
Sunil.

11

It might be that these fields, being added automatically, still don't exist by the time the processor is run. We'll look into that.

12

Hello Adrian,
Any update on this?
When can I have this in snapshot version?

br,
Sunil.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.