# Help with filter to create field from substring

**URL:** <https://discuss.elastic.co/t/help-with-filter-to-create-field-from-substring/364568>\
**Category:** Logstash\
**Created:** [August 8, 2024, 4:35am UTC](https://discuss.elastic.co/t/help-with-filter-to-create-field-from-substring/364568 "2024-08-08T04:35:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Big-Edd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/big-edd/32/131503_2.png) [@Big-Edd](https://discuss.elastic.co/u/Big-Edd)\
**Post date:** [August 8, 2024, 4:35am UTC](https://discuss.elastic.co/t/help-with-filter-to-create-field-from-substring/364568/1 "2024-08-08T04:35:23Z")

</div>

Hello Everyone,

We are inputting RabbitMQ messages that originates via SMTP, so the messages contain some SMTP information we need.

The logstash.conf file I have so far looks something like this.

```auto
input {
  rabbitmq {
    codec => "plain"
    tags => ["RabbitMQ","SMTP"]
    host => "rabbitmq"
    port => 5672
    queue => "smtpq"
    durable => true
    passive => true
    user => "consumer"
    password => "Bogus_Password"
    metadata_enabled => "extended"
  }
}

output {
  opensearch {
    hosts => ["opensearch:9200"]
    ssl => true
    ssl_certificate_verification => false
    user => "os_user"
    password => "Bogus_Password"
    healthcheck_path => ["/"]
    index => "data_from_smtp_%{+YYYY_MM}"
  }
  stdout { codec => rubydebug { metadata => true } }
}

```

The data in the resulting index looks something like this.

```auto
{
    "took": 319,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1784,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "index_for_some_data",
                "_id": "dkgirndy58gmdu69fys5",
                "_score": 1.0,
                "_source": {
                    "event": {
                        "original": "Return-Path: <some_one@some_where.com>\r\nX-Original-To: mailbox@needed.local\r\nReceived: from internal.host.local (internal.host.local .. [SNIP] .. \r\nSubject: Something of interset to us\r\n .. [SNIP] .. "
                    },
                    "message": "Return-Path: <some_one@some_where.com>\r\nX-Original-To: mailbox@needed.local\r\nReceived: from internal.host.local (internal.host.local .. [SNIP] .. \r\nSubject: Something of interset to us\r\n .. [SNIP] .. ",
                    "@version": "1",
                    "tags": [
                        "RabbitMQ",
                        "SMTP"
                    ],
                    "@timestamp": "2024-08-08T03:30:31.158337781Z"
                }
            },
.. [SNIP] ..  

```

I am hoping for some help on filter creation to `add_field` resulting in these from the above sample data.

```auto
"email_to": "mailbox@needed.local"
"email_subject": "Something of interset to us"

```

Cheers,  
Eddie.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 8, 2024, 5:07am UTC](https://discuss.elastic.co/t/help-with-filter-to-create-field-from-substring/364568/2 "2024-08-08T05:07:50Z")

</div>

Consider this....

```
input { generator { count => 1 lines => [ 'Return-Path: <some_one@some_where.com>
X-Original-To: mailbox@needed.local
Received: from internal.host.local (internal.host.local .. [SNIP] ..
Subject: Something of interset to us
  .. [SNIP] .. ' ] } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    mutate { remove_field => ["event", "host", "log"] }

    grok {
        break_on_match => false
        match => {
            "message" => [
                "^X-Original-To: %{EMAILADDRESS:email_to}",
                "^Subject: %{DATA:email_subject}
" ] }
    }
}

```

That gets me

```
"email_subject" => "Something of interset to us",
     "email_to" => "mailbox@needed.local",

```

You can use grok to pick single lines out of the message. Note the use of a literal newline embedded in the second grok pattern. If your source uses \r\n as newline and your logstash platform uses just \n then you may need to use mutate+gsub to remove the \r characters.

---

<div class="post-metadata">

**Author:** ![Big-Edd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/big-edd/32/131503_2.png) [@Big-Edd](https://discuss.elastic.co/u/Big-Edd)\
**Post date:** [August 8, 2024, 11:05am UTC](https://discuss.elastic.co/t/help-with-filter-to-create-field-from-substring/364568/3 "2024-08-08T11:05:42Z")

</div>

Works a treat.  
Thanks a bunch.