# Parse filename before sending to ElasticSearch

**URL:** <https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 19, 2019, 8:28pm UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670 "2019-04-19T20:28:27Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahulnathan](https://avatars.discourse-cdn.com/v4/letter/r/71c47a/32.png) [@rahulnathan](https://discuss.elastic.co/u/rahulnathan)\
**Post date:** [April 19, 2019, 8:28pm UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/1 "2019-04-19T20:28:27Z")

</div>

I am trying to parse the filename to extract certain information. Is this possible today or need to send the data to LogStash to process further ?

Thanks,  
Rahul

---

<div class="post-metadata">

**Author:** ![Debashis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debashis/32/45290_2.png) [@Debashis](https://discuss.elastic.co/u/Debashis)\
**Post date:** [April 22, 2019, 9:05am UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/2 "2019-04-22T09:05:21Z")

</div>

@rahulnathan,  
It will be very helpful if you describe it with some example.

Filename can be fetched from json field "source" of publishable event and user can apply different processors on it with some regular expression and conditional statement.

If you elaborate with a sample example then it will be helpful to give proper solution.

Thanks

---

<div class="post-metadata">

**Author:** ![rahulnathan](https://avatars.discourse-cdn.com/v4/letter/r/71c47a/32.png) [@rahulnathan](https://discuss.elastic.co/u/rahulnathan)\
**Post date:** [April 22, 2019, 1:24pm UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/3 "2019-04-22T13:24:28Z")

</div>

My filename is of below format.  
"application-process-hostname-cluster-region.log.INFO.20190417-190942.1"  
Here I would like to publish process, hostname, cluster and region as additional fields to ElasticSearch.

Thanks,

---

<div class="post-metadata">

**Author:** ![Debashis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debashis/32/45290_2.png) [@Debashis](https://discuss.elastic.co/u/Debashis)\
**Post date:** [April 23, 2019, 5:27am UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/4 "2019-04-23T05:27:11Z")

</div>

Thanks @rahulnathan for giving details.

It is possible by filebeat but little bit tricky and version dependent. Kindly confirm the version of filebeat you are using. After that I will give you the exact solution.

---

<div class="post-metadata">

**Author:** ![rahulnathan](https://avatars.discourse-cdn.com/v4/letter/r/71c47a/32.png) [@rahulnathan](https://discuss.elastic.co/u/rahulnathan)\
**Post date:** [April 23, 2019, 12:22pm UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/5 "2019-04-23T12:22:36Z")

</div>

Filebeat version is 6.7.1

Thanks in advance for your help.

---

<div class="post-metadata">

**Author:** ![Debashis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debashis/32/45290_2.png) [@Debashis](https://discuss.elastic.co/u/Debashis)\
**Post date:** [April 24, 2019, 4:57am UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/6 "2019-04-24T04:57:37Z")

</div>

@rahulnathan, As you are using 6.7.1 so you can do with the help of "dissect" processor. To achieve your requirement you have to use "drop\_fields" processor with combination of "dissect".

```
  filebeat.inputs:
  - type: log
    enabled: true
    paths:
     - /var/log/application-process-hostname-cluster-region.log.INFO.20190417-190942.1
  processors:
     - dissect:
          tokenizer: "%{key1}-%{key2}-%{key3}-%{key4}-%{key5}.%{key6}"
          field: "source"
          target_prefix: ""  
     - drop_fields:
          when:
              has_fields: ['key1','key6']
          fields: ["key1","key6"]

```

The sample publishable event will be

```
  "@timestamp": "2019-04-24T04:52:21.749Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.4.1"
  },
  "message": "Sample",
  "input": {
    "type": "log"
  },
  "host": {
    "name": "localhost.localdomain"
  },
  "source": "/var/log/application-process-hostname-cluster-region.log.INFO.20190417-190942.1"
  "key2": "process",
  "key3": "hostname",
  "key4": "cluster",
  "key5": "region",
  "offset": 0,
  "prospector": {
    "type": "log"
  },
  "beat": {
    "hostname": "localhost.localdomain",
    "version": "6.4.1",
    "name": "localhost.localdomain"
  }, 
}

```

If you want to publish the value of process in your log filename as a field value of "process" then use the field name "process" in place of "key2". Make changes in similar way for the others (hostname, cluster and region).

---

<div class="post-metadata">

**Author:** ![Debashis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debashis/32/45290_2.png) [@Debashis](https://discuss.elastic.co/u/Debashis)\
**Post date:** [April 26, 2019, 11:17am UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/7 "2019-04-26T11:17:30Z")

</div>

@rahulnathan Is it solved? What is the current status regarding your problem?

---

<div class="post-metadata">

**Author:** ![rahulnathan](https://avatars.discourse-cdn.com/v4/letter/r/71c47a/32.png) [@rahulnathan](https://discuss.elastic.co/u/rahulnathan)\
**Post date:** [April 30, 2019, 5:12am UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/8 "2019-04-30T05:12:58Z")

</div>

@Debashis, Unfortunately, this did not work. Below fields are missing in the sent event

"source": "/var/log/application-process-hostname-cluster-region.log.INFO.20190417-190942.1"  
"key2": "process",  
"key3": "hostname",  
"key4": "cluster",  
"key5": "region",  
"offset": 0,  
I am not seeing any error in filebeat logs and message line is being sent to Elastic search without the additional fields.  
Is there a way we can enable more debugging in the logs to help identify the issue ?

Thanks,  
Rahil

---

<div class="post-metadata">

**Author:** ![Debashis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debashis/32/45290_2.png) [@Debashis](https://discuss.elastic.co/u/Debashis)\
**Post date:** [April 30, 2019, 5:20am UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/9 "2019-04-30T05:20:13Z")

</div>

@rahulnathan, I have executed this on my side and it is working fine and the events are getting published with those fields successfully. I can't understand what's problem is there on your side.

Kindly share your filebeat console log after enabling debug log and .yml file using \</\>. I will recheck it.

```
 logging.level: debug
 logging.selectors: ["*"]

```

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2019, 5:20am UTC](https://discuss.elastic.co/t/parse-filename-before-sending-to-elasticsearch/177670/10 "2019-05-28T05:20:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.