# Processing metric sets to reduce data amount **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [October 19, 2016, 10:38am UTC](https://discuss.elastic.co/t/processing-metric-sets-to-reduce-data-amount/63372 "2016-10-19T10:38:29Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![heilaaks](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@heilaaks](https://discuss.elastic.co/u/heilaaks)\ **Post date:** [October 19, 2016, 10:38am UTC](https://discuss.elastic.co/t/processing-metric-sets-to-reduce-data-amount/63372/1 "2016-10-19T10:38:29Z")
Has anyone succeeded to filter out metricbeat events with metricbeat-5.0.0-rc1-x86\_64.rpm? What I try to do is to get the memory and CPU load statistics only from few selected processes. How ever, I am not able to get the filters to work as expected. I was not able to do this with 'processes' field where I tried syntax variations like processes: ['kafka._|zookeeper._']. I was also not sure to what process fields this maps. Then I tried to create inverted regexp match with drop\_event but I was not able to get the basic filtering out of an event to work with example /1/. Has anyone created a filter with rc1 that enables getting CPU and memory only from selected processes? Is it possible to create inverted drop filter based regexp on on different keys like /2/? Some processes are running with specific user and some of them are running as root which requires using different fields to filter the data. The optimization that I am trying to do is that I have only a single node Elasticsearch to store measurements but multiple nodes that produce the data. The amount of events gets pretty large with default configuration if there are start to get more than 10 nodes that produce the data. Thanks. /1/ - module: system metricsets: /2/ processors: - drop\_event: when: regexp: system.process.username: '^nerve|^synapse' system.process.cmdline: '(^._synapse._)|(^._nerve._)' ---
**Author:** ![monica](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/monica/32/3696_2.png) [@monica](https://discuss.elastic.co/u/monica)\ **Post date:** [October 19, 2016, 12:42pm UTC](https://discuss.elastic.co/t/processing-metric-sets-to-reduce-data-amount/63372/2 "2016-10-19T12:42:56Z")
The `processes` configuration option accepts a list of regular expressions. I created a [golang playgroud](https://play.golang.org/p/6y8BKBvW9h) to help you construct the right regular expressions. You just need to add a function call `Match` in the main function to check that the string matches a certain regular expression. For example, I used the regular expression `kafka.|zookeeper.` to see if it matches the kafka process, and it fails: ```auto Match("kafka", "kafka.|zookeeper.") ``` Please let me know if you have further questions. ---
**Author:** ![heilaaks](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@heilaaks](https://discuss.elastic.co/u/heilaaks)\ **Post date:** [October 27, 2016, 11:44am UTC](https://discuss.elastic.co/t/processing-metric-sets-to-reduce-data-amount/63372/3 "2016-10-27T11:44:08Z")
I was able to get 'processes: ['java']' to match only the Java processes but I cannot get anything else to work. I tried also basic filtering with 'processes: ['kafka']' syntax but that did not result process level metrics. It may be that I have misunderstood how this is expected to work. I am assuming that this would for example match the user name running the process or the command line used to start the process /1/. I tried with 'processes: ['elastic']' and I think it matched to some zombie process or script that is named as 'elasticsearch\_h' which PID is constantly changing /2/. This did not seem to match to actual Elasticsearch service that we are running. This may be some dummy mistake from my end as well. If you can clarify to what Linux command output the 'processes: ' filtering matches and I could check how that is visible from our Linux (RHEL7.2) environment. Thanks /1/ Tasks: 313 total, 1 running, 309 sleeping, 0 stopped, 3 zombie %Cpu(s): 2.9 us, 0.8 sy, 0.0 ni, 95.9 id, 0.2 wa, 0.0 hi, 0.0 si, 0.1 st KiB Mem : 20888560 total, 8518116 free, 6685824 used, 5684620 buff/cache KiB Swap: 0 total, 0 free, 0 used. 13187196 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2728 kafka 20 0 12.378g 1.156g 15700 S 6.2 5.8 116:10.70 java ... $ ps -aef | grep 2728 kafka 2728 1 3 Oct25 ? 01:56:14 java -Xms kafka.Kafka /etc/kafka/server.properties /2/ { "\_index": "metricbeat-2016.10.27", "\_type": "metricsets", "\_id": "AVgF7YZtd5zAnjYIOEyt", "\_score": null, "\_source": { "@timestamp": "2016-10-27T11:36:41.438Z", "beat": { "hostname": "esps0.local", "name": "esps0.local" }, "metricset": { "module": "system", "name": "process", "rtt": 25662 }, "system": { "process": { "cpu": { "start\_time": "2016-10-27T11:36:37.000Z", "total": { "pct": 0 } }, "fd": { "limit": { "hard": 4096, "soft": 1024 }, "open": 0 }, "memory": { "rss": { "bytes": 0, "pct": 0 }, "share": 0, "size": 0 }, "name": "elasticsearch\_h", "pgid": 8800, "pid": 8800, "ppid": 1219, "state": "zombie", "username": "root" } }, "type": "metricsets" }, "fields": { "@timestamp": [ 1477568201438 ], "system.process.cpu.start\_time": [ 1477568197000 ] }, "highlight": { "[system.process.name](http://system.process.name)": [ "@kibana-highlighted-field@elasticsearch\_h@/kibana-highlighted-field@" ] }, "sort": [ 1477568201438 ] } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 9, 2016, 10:38am UTC](https://discuss.elastic.co/t/processing-metric-sets-to-reduce-data-amount/63372/4 "2016-11-09T10:38:45Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.