I had a difficult issue need to be fixed in terms of logstash drop fileds

Elastic Stack Logstash
1

Dear folks,

I had a difficult issue need to be fixed in terms of logstash drop fileds. the thing like this:

I predefined some fileds in metricbeat config file in order to send the difference index name to elasticsearch via logstash.
what we want is that we want to have different index name like "%{[dc]}-%{[env]}-%{[os]}-%{[service]}-%{+YYYY.MM.dd}" in terms of predefined fileds above.
But we don't want to insert those predefined fileds and its value into elasticsearch index..

Is there any way can to do this?

I'll appreciate it if anyone else can provide a good method to address this issue.

P.S.

===============metricbeat===============
fields:
    dc:       FR4
    env:      prod
    os:       linux
    service:  Apache
  fields_under_root: true
  
  
===============logstash===============  
output {
  elasticsearch {
     hosts => ["es2.abc.com:9200","es2.abc.com:9200"]
     template_overwrite => false
     manage_template => false
     index => "%{[dc]}-%{[env]}-%{[os]}-%{[service]}-%{+YYYY.MM.dd}"
     sniffing => false
   }
}

Thanks in advance

2

Fields stored under @metadata are available during processing but not part of the event being sent downstream, so you could use your fields to create an index prefix there and then drop them from the event. Having said that, it looks like this scheme could result in a large number of very small indices, which can be very inefficient and limit the amount of data a node can hold, so I would advise to to think about this carefully, especially if you plan to keep your data around in the cluster for a while.

3

Hi @Christian_Dahlqvist,
I want to index name consist of like fr4-prod-linux-apache-20170710,but I don't want to the data of following fields to go into the index I mentioned.
dc: FR4
env: prod
os: linux
service: Apache

Would you please give me an example to implement that?

4

any updates?

5

Do can do something like this in your filter section:

mutate {
  add_field => { "[@metadata][index_prefix]" => "%{dc}-%{env}-%{os}-%{service}" }
}

mutate {
  remove_field => ["dc","env","os","service"]
}

Then use this in the output:

elasticsearch {
   hosts => ["es2.abc.com:9200","es2.abc.com:9200"]
   template_overwrite => false
   manage_template => false
   index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"
   sniffing => false
}

I would however recommend carefully considering the number of indices and shards this index naming scheme may generate, as this has the potential to cause problems down the line. Use index templates in order to limit the number of primary shards each index has.

6

Hi @Christian_Dahlqvist,
Sorry for reply you late. today I tested the pipeline you gave me, seems the fields are deleted within index file properly, but the index prefix is not generated by the fields I expected.

P.S.

#metricbeat conf file

metricbeat.modules:

#system module
- module: system
  metricsets:
    - cpu
    - load
    - filesystem
    - fsstat
    - memory
    - network
    - process

  enabled: true
  period: 10s
  processes: ['.*']
  cpu_ticks: false
  tags:  "system"
  fields:
    env:      test 
    dc:       metricbeat-ld5  
    os:       linux
    service:  system
  fields_under_root: true
  raw: false

#pipline conf file
170713163220 root@uklogstash-test1 conf.d # cat beats.conf
#logstash for beats
input {
beats {
port => 5044
}
}

filter {
   mutate  {  add_field => { "[@metadata][index_prefix]" => "%{env}-%{dc}-%{os}-%{service}" } }   
   mutate  {  remove_field =>  ["dc","env","os","service"]  }

}


output {
  file {
    path => "/tmp/%{[@metadata][index_prefix]-%{+YYYY.MM.dd}"
  }
}

#result file

%{[@metadata][index_prefix]-%{+YYYY.MM.dd}

the index file i want is like this: test-metricbeat-ld5-linux-system-2017.07.13, not the index like above .

#contents of the result file

{"@timestamp":"2017-07-13T08:27:56.683Z","system":{"process":{"memory":{"rss":{"pct":8.0E-4,"bytes":3272704},"size":41332736,"share":2314240},"cmdline":"ntpd -u ntp:ntp -p /var/run/ntpd.pid -g","name":"ntpd","cpu":{"total":{"pct":0.0}},"pid":8694,"state":"sleeping","fd":{"limit":{"hard":4096,"soft":1024},"open":13}}},"beat":{"name":"uktelegraf-test1"},"@version":"1","metricset":{"module":"system","name":"process"},"type":"metricsets","tags":["system","beats_input_raw_event"]}
{"@timestamp":"2017-07-13T08:27:56.683Z","system":{"process":{"memory":{"rss":{"pct":0.0018,"bytes":7344128},"size":205381632,"share":2879488},"cmdline":"/usr/sbin/snmpd -LS0-6d -Lf /dev/null -p /var/run/snmpd.pid","name":"snmpd","cpu":{"total":{"pct":0.0}},"pid":8773,"state":"sleeping","fd":{"limit":{"hard":4096,"soft":1024},"open":10}}},"beat":{"name":"uktelegraf-test1"},"@version":"1","metricset":{"module":"system","name":"process"},"type":"metricsets","tags":["system","beats_input_raw_event"]}
{"@timestamp":"2017-07-13T08:27:56.683Z","system":{"process":{"memory":{"rss":{"pct":4.0E-4,"bytes":1548288},"size":305324032,"share":667648},"cmdline":"/usr/local/bacula/sbin/bacula-fd -c /usr/local/bacula/etc/bacula-fd.conf","name":"bacula-fd","cpu":{"total":{"pct":0.0}},"pid":8891,"state":"sleeping","fd":{"limit":{"hard":4096,"soft":1024},"open":4}}},"beat":{"name":"uktelegraf-test1"},"@version":"1","metricset":{"module":"system","name":"process"},"type":"metricsets","tags":["system","beats_input_raw_event"]}
{"@timestamp":"2017-07-13T08:27:56.683Z","system":{"process":{"cmdline":"/sbin/rsyslogd -i /var/run/syslogd.pid","memory":{"rss":{"pct":7.0E-4,"bytes":2768896},"size":253755392,"share":1179648},"name":"rsyslogd","cpu":{"total":{"pct":0.0}},"pid":8986,"state":"sleeping","fd":{"limit":{"hard":4096,"soft":1024},"open":9}}},"beat":{"name":"uktelegraf-test1"},"@version":"1","metricset":{"module":"system","name":"process"},"type":"metricsets","tags":["system","beats_input_raw_event"]}
{"@timestamp":"2017-07-13T08:27:56.683Z","system":{"process":{"cmdline":"/opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --logdest /var/log/puppetlabs/puppet/puppet.log","memory":{"rss":{"pct":0.0209,"bytes":84119552},"size":320401408,"share":3543040},"name":"puppet","cpu":{"total":{"pct":0.0}},"pid":9047,"state":"sleeping","fd":{"limit":{"hard":4096,"soft":1024},"open":8}}},"beat":{"name":"uktelegraf-test1"},"@version":"1","metricset":{"module":"system","name":"process"},"type":"metricsets","tags":["system","beats_input_raw_event"]}

Would you please give me more idea about this?

Thanks in adavance.

7

It looks like I missed a closing curly brace in the string.

8

@Christian_Dahlqvist
hi dude,
even if I appended the close braces, the index still not replace by variabes. it wired.

path => "/tmp/%{[@metadata][index_prefix]-%{+YYYY.MM.dd}}"
the index file gernated as following:

%{[@metadata][index_prefix]-%{+YYYY.MM.dd}}

9

I updated my example. You are still missing a closing curly brace after [index_prefix].

10

Hi man,
It works. much thanks to you. you help me a lot. :slight_smile:
By the way, where are you from? is there any way I can keep touch with you except here?

what you prefer to communicate with, twitter, facebook or something else?

Anyway, I follow your GitHub.

Thanks

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.