# Logstash grok to match the metricbeat type and add field

**URL:** <https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295>\
**Category:** Logstash\
**Created:** [March 2, 2018, 4:33pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295 "2018-03-02T16:33:43Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![arun\_prasath1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_prasath1/32/49539_2.png) [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\
**Post date:** [March 2, 2018, 4:33pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/1 "2018-03-02T16:33:43Z")

</div>

Hi Logstash Experts,

I have metricbeat to logstash to Kibana pipeline. In the logstash, I have the below grok filter to parse java process. But it is not adding the field for the matched pattern. secondly, it is additionally matching the filebeat logs. The expectation is to match only the metricbeat java process and add the field for the matching pattern.

filter {  
if [type] == "metricsets" {  
grok {  
match =\> { "system.process.cmdline" =\> ["(?java -jar)\s(?([-]?Env[=]?[prdeval]+[23]?))\s(?([-]?Int[=]?[a-zA-Z0-9-]+))\s%{GREEDYDATA:rest}"] }  
add\_field =\> { "service\_status" =\> "running" }  
}  
}  
}

From Kibana, json output  
{  
"\_index": "metric-2018.03.02",  
"\_type": "metricsets",  
"\_id": "AWHnPFRuOJtEtYrh-QW7",  
"\_score": null,  
"\_source": {  
"@timestamp": "2018-03-02T15:02:25.158Z",  
"system": {  
"process": {  
"memory": {  
"rss": {  
"pct": 0.0065,  
"bytes": 439881728  
},  
"size": 3891261440,  
"share": 14336000  
},  
"cmdline": "java -jar -Env=test2 -Int=3-0 -Dprotocol=https -Dssl\_enabled=true -Xms128m -Xmx256m /var/log",

please help here to fix this issue.

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [March 2, 2018, 6:13pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/2 "2018-03-02T18:13:03Z")

</div>

Hi,

Metricbeat should deliver as json already. At least it does if you let metricbeat write to file and ship this via filebeat.  
Then you can use json filter and the parsing is done automatically. Later you can add your own fields bases on the other fields.

You did not show the full json from kibana. Is there a tag like \_grokparsefailure? If grok is failing, i dont know if the fiel is added. I miss the **rest** field in your json. So I think thatnis thr issue you are facing.

---

<div class="post-metadata">

**Author:** ![arun\_prasath1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_prasath1/32/49539_2.png) [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\
**Post date:** [March 5, 2018, 2:51pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/3 "2018-03-05T14:51:03Z")

</div>

I am getting the filebeat data where i am supposed to get only metricbeat data.  
here is my code, what i am missing or what are the ways i can debug and find the issue ?

metricbeat.yml:

metricbeat.modules:

- module: system  
metricsets:
  - cpu
  - load
  - core
  - diskio
  - filesystem
  - fsstat
  - memory
  - network
  - process  
enabled: true  
period: 10s  
processes: ['.\*']  
output.logstash:  
hosts: ["logstash-server.domain:5044"]  
logging.level: warning

logstash-metric.conf:

```
input {
  beats {
    port => 5044
  }
}

filter {
    if [@metadata][beat] == "metricbeat" {
        grok {
            match => { "[system][process][cmdline]" => "[(?<projectdate>java -jar\s)(?<projectenv>([-]?Denv[=]?[prdeval]+[23]?\s))(?<projecttype>([-]?Dtype[=]?[a-zA-Z0-9-]+\s))(?<projectapp>([-]?Dapp[=]?[project23]+\s))(?<prof>([a-zA-Z0-9=-]+\s))(?<springprofile>([.a-zA-Z0-9=-]+\s))(?<servicename>([a-zA-Z0-9=-]+\s))(?<appserver>([.a-zA-Z0-9=-]+\s))(%{GREEDYDATA:rest})]" }
        add_field => { "service_status" => "running" }
        }
    }
}
output {
  elasticsearch {
    hosts => ["elasticsearch-server.domain:9200"]
    manage_template => false
    index => "metric-microservice-%{+YYYY.MM.dd}"
  }
}

```

I need to match only similar java processes and other processes only and not Filebeat data:

```
{
  "_index": "metric-microservice-2018.03.05",
  "_type": "metricsets",
  "_id": "AWH2k5q0OJtEtYrhjgpL",
  "_score": null,
  "_source": {
    "system": {
      "process": {
        "memory": {
          "rss": {
            "pct": 0.011,
            "bytes": 742027264
          },
          "size": 4006264832,
          "share": 11497472
        },
        "cmdline": "java -jar -Denv=ver -Dtype=3-0 -Dapp=project3 -Dprof=ver-3-0 -Dspring.profiles.active=env-3-0 -Dsvcname=project-component-3-0 -Dhostname=my-build-server.project.adml -Dsecname=my-build-server.fullname -Dapachsvr_port=8090 -Dprotocol=https -Dssl_enabled=true -Xms128m -Xmx256m -DLOG_PATH=/var/opt/logs -Dlog_path=/var/opt/logs /var/opt/pivotal/svcs/project-component.jar /tmp",
        "pgid": 18546,
        "name": "java",
        "cpu": {
          "start_time": "2018-02-20T16:53:40.000Z",
          "total": {
            "pct": 0
          }
        },
        "pid": 19615,
        "state": "sleeping",
        "fd": {
          "limit": {
            "hard": 4096,
            "soft": 4096
          },
          "open": 46
        },
        "ppid": 1,
        "username": "root"
      }
    },
    "@timestamp": "2018-03-05T14:32:03.089Z",
    "beat": {
      "hostname": "my-build-server",
      "name": "my-build-server",
      "version": "5.0.1"
    },
    "@version": "1",
    "host": "my-build-server",
    "service_status": "running",
    "metricset": {
      "rtt": 84209,
      "module": "system",
      "name": "process"
    },
    "type": "metricsets",
    "tags": [
      "beats_input_raw_event"
    ]
  },
  "fields": {
    "@timestamp": [
      1520260323089
    ]
  },
  "sort": [
    1520260323089
  ]
}
```

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [March 5, 2018, 3:32pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/4 "2018-03-05T15:32:07Z")

</div>

can you please format your json with the code button ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8b97b89bae26bef52cd0940148d134b3c879bf2c.png)?

> [@arun\_prasath1](#):
>
> match =\> { "system.process.cmdline" =\> ["(?java -jar)\s(?([-]?Env[=]?[prdeval]+[23]?))\s(?([-]?Int[=]?[a-zA-Z0-9-]+))\s%{GREEDYDATA:rest}"] }

Please try to change system.process.cmdline to [system][process][cmdline]. Thats the correct syntax to access nested fields.

Haven't checked your regex / grok.

---

<div class="post-metadata">

**Author:** ![arun\_prasath1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_prasath1/32/49539_2.png) [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\
**Post date:** [March 5, 2018, 3:36pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/5 "2018-03-05T15:36:52Z")

</div>

Yes, did changed to [\_source][system][process][cmdline] but still facing the same issue.

Below is the sample filebeat json data that is matched from this logstash configuration.

```
{
  "_index": "metric-microservice-2018.03.05",
  "_type": "apache_access",
  "_id": "AWH2qL8GOJtEtYrhkXtV",
  "_score": null,
  "_source": {
    "request": "/",
    "offset": 2508254,
    "auth": "-",
    "ident": "-",
    "input_type": "log",
    "verb": "GET",
    "source": "/usr/apache/logs/access_log.2018-03-05-00_00_00",
    "message": "172.26.35.2 - - [05/Mar/2018:15:55:07 +0100] \"GET / HTTP/1.0\" 200 25626 \"-\" \"HTTP-Monitor/1.1\" 1816",
    "type": "apache_access",
    "tags": [
      "dev",
      "beats_input_codec_plain_applied"
    ],
    "@timestamp": "2018-03-05T14:55:08.268Z",
    "response": "200",
    "bytes": "25626",
    "clientip": "192.56.35.2",
    "@version": "1",
    "beat": {
      "hostname": "my-webserver",
      "name": "my-webserver",
      "version": "5.0.1"
    },
    "host": "my-webserver",
    "httpversion": "1.0",
    "timestamp": "05/Mar/2018:15:55:07 +0100"
  },
  "fields": {
    "@timestamp": [
      1520261708268
    ]
  },
  "sort": [
    1520261708268
  ]
}
```

---

<div class="post-metadata">

**Author:** ![arun\_prasath1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_prasath1/32/49539_2.png) [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\
**Post date:** [March 6, 2018, 4:16pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/6 "2018-03-06T16:16:22Z")

</div>

@asp - how to format using code button? Is there any tool by which code button can be applied ?

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [March 7, 2018, 8:19am UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/7 "2018-03-07T08:19:00Z")

</div>

> [@arun\_prasath1](#):
>
> how to format using code button?

just set an empty line before and after your code. Then mark the text and hit the code button in the menu: ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c96014fcd49e5dee09549d2abf188dcca72acc82.png)

then you get sth like that

```
{
  "bla" : "blabla",
  "blub": {
    "test" : 1
    }
}

```

 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8ed80ac01f4d6a3c4ba21f9b71fc32e4f379782c.png)

otherwise spaces which are used for formating will be lost an saving.

---

<div class="post-metadata">

**Author:** ![arun\_prasath1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_prasath1/32/49539_2.png) [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\
**Post date:** [March 7, 2018, 11:35am UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/8 "2018-03-07T11:35:16Z")

</div>

thanks, formatted the code.

---

<div class="post-metadata">

**Author:** ![arun\_prasath1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arun_prasath1/32/49539_2.png) [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\
**Post date:** [March 7, 2018, 2:23pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/9 "2018-03-07T14:23:55Z")

</div>

@asp - thanks. done formatting as code.  
if [type] in ["metricsets", "metricbeat] solved the issue. Thanks a lot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 4, 2018, 2:24pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295/10 "2018-04-04T14:24:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.