# Grok filter by file path

**URL:** <https://discuss.elastic.co/t/grok-filter-by-file-path/199097>\
**Category:** Logstash\
**Created:** [September 11, 2019, 2:52pm UTC](https://discuss.elastic.co/t/grok-filter-by-file-path/199097 "2019-09-11T14:52:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenP](https://avatars.discourse-cdn.com/v4/letter/b/c37758/32.png) [@BenP](https://discuss.elastic.co/u/BenP)\
**Post date:** [September 11, 2019, 2:52pm UTC](https://discuss.elastic.co/t/grok-filter-by-file-path/199097/1 "2019-09-11T14:52:55Z")

</div>

Hello,

I'm currently banging my head against the wall trying to make my filters works.  
My grok patterns are OK but I can't make my conditions to work.  
The documents are added but there aren't transformed (I can confirm they aren't going through the filters on the pipeline monitoring page).

logstash.conf :

```
input {
  beats {
    port => 5044
  }
}

filter {
  if [log.file.path] =~ "WindowsUpdate" {
          grok {
                pattern_definitions => { "DATEP" => "%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}" }
                match => { "message" => "%{DATEP:date} %{TIME:time} %{NUMBER:id} %{BASE16NUM:hex} %{WORD:category} %{GREEDYDATA:message}" }
                overwrite => ["message"]
          }
    }
  else if [log.file.path] =~ "CacheSys" {
          grok {
                match => { "message" => "%{DATESTAMP:date} \(%{NUMBER:thread}\) %{NUMBER:number} %{GREEDYDATA:message}" }
                overwrite => ["message"]
          }
    }
  else if [log.file.path] =~ "Mpl" {
          grok {
                pattern_definitions => { "DATEP" => "%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}" }
                match => { "message" => "%{DATEP:date} %{TIME:time} \[%{DATA:thread}\] %{LOGLEVEL:loglevel} %{GREEDYDATA:composant} \[%{DATA:nested}\] \-\ %{GREEDYDATA:message}" }
                overwrite => ["message"]
                remove_field => ["nested"]
          }
    }
  else if [log.file.path] =~ "license" {
      xml {
                source => "LICENSE"
                target => "doc"
      }
  }
  else if [log.file.path] =~ "MONITOR" {
      xml {
                source => "REQUEST"
                target => "doc"
      }
  }
}

output {
  elasticsearch {
      hosts => "10.1.0.4:9200"
  }

```

Document example :

```
{
  "_index": "logstash-2019.08.30-000001",
  "_type": "_doc",
  "_id": "GEVbIG0B-90jmHDjwA1J",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "@timestamp": "2019-09-11T12:45:45.248Z",
    "message": "2019-09-11\t12:56:08:032\t 900\t6f0\tAU\tAU setting next sqm report timeout to 2019-09-12 10:56:08",
    "host": {
      "architecture": "x86_64",
      "os": {
        "kernel": "6.1.7601.24499 (win7sp1_ldr.190612-0600)",
        "family": "windows",
        "build": "7601.24494",
        "version": "6.1",
        "name": "Windows Server 2008 R2 Standard",
        "platform": "windows"
      },
      "name": "SERVERNAME",
      "id": "69440f3e-8a9c-4a2f-b4d5-e6474355a9f2",
      "hostname": "SERVERNAME"
    },
    "log": {
      "offset": 1108391,
      "file": {
        "path": "C:\\windows\\WindowsUpdate.log"
      }
    },
    "input": {
      "type": "log"
    },
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "ecs": {
      "version": "1.0.1"
    },
    "agent": {
      "hostname": "SERVERNAME",
      "id": "07f6741d-9f98-4e6f-98f5-4179a3fd5eca",
      "version": "7.3.1",
      "name": "SERVERNAME",
      "type": "filebeat",
      "ephemeral_id": "7d10470d-0f80-4b4d-b6a9-dfba343a41e2"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-09-11T12:45:45.248Z"
    ]
  },
  "sort": [
    1568205945248
  ]
}

```

What am I missing ?

2nd issue :  
Since I enabled security on my stack, I added a "logstash\_internal" user according to [this documentation](https://www.elastic.co/guide/en/logstash/7.3/ls-security.html) but Logstash is throwing a 403 error trying to add data to Elasticsearch :  
`[2019-09-11T14:50:45,451][INFO][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"security_exception", "reason"=>"action [indices:data/write/bulk[s]] is unauthorized for user [logstash_internal]"})`

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/e/d/ed03b27e15509f20be820f956e6747fdcc54540c.png)  
I'm temporarily using the elastic user to circumvent the issue. Any ideas ?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 11, 2019, 2:57pm UTC](https://discuss.elastic.co/t/grok-filter-by-file-path/199097/2 "2019-09-11T14:57:33Z")

</div>

> [@BenP](#):
>
> [log.file.path]

That refers to a field with two periods in its name. You have a [log] field that contains a [file] field that contains a [path] field. You can refer to it as [log][file][path]

---

<div class="post-metadata">

**Author:** ![BenP](https://avatars.discourse-cdn.com/v4/letter/b/c37758/32.png) [@BenP](https://discuss.elastic.co/u/BenP)\
**Post date:** [September 12, 2019, 9:43am UTC](https://discuss.elastic.co/t/grok-filter-by-file-path/199097/3 "2019-09-12T09:43:05Z")

</div>

Thanks that was it !

Now I have to refine some expressions and fix my logstash user issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 10, 2019, 9:43am UTC](https://discuss.elastic.co/t/grok-filter-by-file-path/199097/4 "2019-10-10T09:43:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.