Help with filter to create field from substring

Elastic Stack Logstash
1

Hello Everyone,

We are inputting RabbitMQ messages that originates via SMTP, so the messages contain some SMTP information we need.

The logstash.conf file I have so far looks something like this.

input {
  rabbitmq {
    codec => "plain"
    tags => ["RabbitMQ","SMTP"]
    host => "rabbitmq"
    port => 5672
    queue => "smtpq"
    durable => true
    passive => true
    user => "consumer"
    password => "Bogus_Password"
    metadata_enabled => "extended"
  }
}

output {
  opensearch {
    hosts => ["opensearch:9200"]
    ssl => true
    ssl_certificate_verification => false
    user => "os_user"
    password => "Bogus_Password"
    healthcheck_path => ["/"]
    index => "data_from_smtp_%{+YYYY_MM}"
  }
  stdout { codec => rubydebug { metadata => true } }
}

The data in the resulting index looks something like this.

{
    "took": 319,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1784,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "index_for_some_data",
                "_id": "dkgirndy58gmdu69fys5",
                "_score": 1.0,
                "_source": {
                    "event": {
                        "original": "Return-Path: <some_one@some_where.com>\r\nX-Original-To: mailbox@needed.local\r\nReceived: from internal.host.local (internal.host.local  ..  [SNIP]  ..  \r\nSubject: Something of interset to us\r\n  ..  [SNIP]  ..  "
                    },
                    "message": "Return-Path: <some_one@some_where.com>\r\nX-Original-To: mailbox@needed.local\r\nReceived: from internal.host.local (internal.host.local  ..  [SNIP]  ..  \r\nSubject: Something of interset to us\r\n  ..  [SNIP]  ..  ",
                    "@version": "1",
                    "tags": [
                        "RabbitMQ",
                        "SMTP"
                    ],
                    "@timestamp": "2024-08-08T03:30:31.158337781Z"
                }
            },
..  [SNIP]  ..

I am hoping for some help on filter creation to add_field resulting in these from the above sample data.

"email_to": "mailbox@needed.local"
"email_subject": "Something of interset to us"

Cheers,
Eddie.

2

Consider this....

input { generator { count => 1 lines => [ 'Return-Path: <some_one@some_where.com>
X-Original-To: mailbox@needed.local
Received: from internal.host.local (internal.host.local  ..  [SNIP]  ..
Subject: Something of interset to us
  ..  [SNIP]  ..  ' ] } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    mutate { remove_field => [ "event", "host", "log" ] }

    grok {
        break_on_match => false
        match => {
            "message" => [
                "^X-Original-To: %{EMAILADDRESS:email_to}",
                "^Subject: %{DATA:email_subject}
" ] }
    }
}

That gets me

"email_subject" => "Something of interset to us",
     "email_to" => "mailbox@needed.local",

You can use grok to pick single lines out of the message. Note the use of a literal newline embedded in the second grok pattern. If your source uses \r\n as newline and your logstash platform uses just \n then you may need to use mutate+gsub to remove the \r characters.

1 Like
3

Works a treat.
Thanks a bunch.