Postfix parsing -

fribse · January 22, 2019, 8:52am

I found Tin Le's setup on Linked In, and I think it looked really nice, so I wanted to add those logfiles to our ELK as well.

I've created the patterns file, and modified my logstash.con, I'll be getting the logfiles with filebeat, so it's a bit different than Tin Le's.

I'm trying to add the template, but I get an error:

 curl -X PUT "elasticsearch:9200/_template/postfix" -H 'Content-Type: application/json' -d '{
    "template" : "maillog-*",
    "order" : 1,
    "settings" : {
        "number_of_shards" : 2,
        "index.refresh_interval" : "90s"
    },
    "mappings" : {
        "maillog" : {
            "properties" : {
                "reason" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "saslusername" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "postfix_action" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "relayip" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "messageid" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "pid" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
                "remote" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "type" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "qid" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "local" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "result" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "path" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "file" : { "index": "not_analyzed", "type" : "string" },
                "queuestatus" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "smtp_response" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "@version" : { "type" : "string" },
                "host" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "client" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "from" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "timestamp" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "nrcpt" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
                "responsecode" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
                "offset" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
                "relayhost" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "logsource" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "message" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "orig_to" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "command" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "tags" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "helo" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "saslmethod" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "component" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "@timestamp" : { "format" : "dateOptionalTime", "type" : "date" },
                "remotehost" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "size" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
                "anvilstatistic" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "proto" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "bouncequeueid" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "to" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
                "dsn" : { "index": "not_analyzed", "doc_values": true, "type" : "string" }
            }
        }
    }
}'

That gave me this error:

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [string] declared on field [reason]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [maillog]: No handler for type [string] declared on field [reason]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [string] declared on field [reason]"}},"status":400}[

Ignoring that, and sending the logfiles to logstash with filebeat, it is imported, but not interpreted.
A few logline examples from our mailservers:

Jan 22 10:53:42 alpha55002 postfix/smtp[23919]: 14117807B898: host gmail-smtp-in.l.google.com[108.177.127.27] said: 421-4.7.0 This message does not have authentication information or fails to pass 421-4.7.0 authentication checks. To best protect our users from spam, the 421-4.7.0 message has been blocked. Please visit 421-4.7.0  https://support.google.com/mail/answer/81126#authentication for more 421 4.7.0 information. s12-v6si1426609ejd.277 - gsmtp (in reply to end of DATA command)
Jan 22 10:53:45 alpha55002 postfix/smtp[23919]: Untrusted TLS connection established to alt1.gmail-smtp-in.l.google.com[74.125.24.27]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)

Magnus_Kessler · January 22, 2019, 11:49am

I'm going to assume that you're using a recent release of Elasticsearch. Since Elasticsearch 5.0 the way strings are handled have changed, and use either text or keyword instead of string/ {analyzed, not_analyzed). Please adapt your mapping accordingly.

In addition, you'll have to set up an appropriate filter in your Logstash configuration to break the message into individual fields. On its own, Filebeat will only send the original log entry and some metadata.

fribse · January 22, 2019, 12:06pm

Hi Magnus

Thankyou, oh wow, very new to the ELK world, so I don't have any 'history' with me about changed syntaxes

Regarding the filter, I found Tin Le's setup on Linked In (https://www.linkedin.com/pulse/monitoring-postfix-logs-elk-tin-le/?trk=pulse_spock-articles), which has some very nice grok filters (https://gist.github.com/TinLe/f9763ac68c122cd11b91) that I don't understand at all. I've also just started on Grok, and they are a bit more complicated than what I've done so far

fribse · January 22, 2019, 1:37pm

Not really sure I understand that fully.

I tried changing it according to Elasticsearch replaces string type with two new types text and keyword. | Elastic Blog, and then the top was changed to index_patterns, not sure if that has been changed in the syntax previously as well.

curl -X PUT "elasticsearch:9200/_template/postfix" -H 'Content-Type: application/json' -d '
{
    "index_patterns" : "maillog-*",
    "order" : 1,
    "settings" : {
        "number_of_shards" : 1,
        "index.refresh_interval" : "90s"
    },
    "mappings" : {
        "maillog" : {
            "properties" : {
                "reason" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "saslusername" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "postfix_action" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "relayip" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "messageid" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "pid" : { "index": "text", "doc_values": true, "type" : "long" },
                "remote" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "type" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "qid" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "local" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "result" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "path" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "file" : { "index": "text", "type" : "keyword" },
                "queuestatus" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "smtp_response" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "@version" : { "type" : "keyword" },
                "host" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "client" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "from" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "timestamp" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "nrcpt" : { "index": "text", "doc_values": true, "type" : "long" },
                "responsecode" : { "index": "text", "doc_values": true, "type" : "long" },
                "offset" : { "index": "text", "doc_values": true, "type" : "long" },
                "relayhost" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "logsource" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "message" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "orig_to" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "command" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "tags" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "helo" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "saslmethod" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "component" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "@timestamp" : { "format" : "dateOptionalTime", "type" : "date" },
                "remotehost" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "size" : { "index": "text", "doc_values": true, "type" : "long" },
                "anvilstatistic" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "proto" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "bouncequeueid" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "to" : { "index": "text", "doc_values": true, "type" : "keyword" },
                "dsn" : { "index": "text", "doc_values": true, "type" : "keyword" }
            }
        }
    }
}'

And that now gives me:

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Failed to parse mapping [maillog]: Could not convert [reason.index] to boolean"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [maillog]: Could not convert [reason.index] to boolean","caused_by":{"type":"illegal_argument_exception","reason":"Could not convert [reason.index] to boolean","caused_by":{"type":"illegal_argument_exception","reason":"Failed to parse value [text] as only [true] or [false] are allowed."}}},"status":400}[

Magnus_Kessler · January 22, 2019, 1:48pm

Try with e.g. "reason" : { "type" : "keyword" }, and make sure you read the documentation for mappings and datatypes.

fribse · January 25, 2019, 10:01am

Ok, so as I understand it, the index above was made for a previous version?

I tried changing it, but I get an error I don't understand, as far as I can see the format in the documentation is correct?
This is the new template definition I've made:

{
    "index_patters" : "maillog-*",
    "settings" : {
        "number_of_shards" : 1,
    },
    "mappings": {
        "maillog": {
            "properties": {
                "reason":
                  { "type": "keyword"
                  },
                "saslusername":
                  { "type": "keyword"
                  },
                "postfix_action":
                  { "type": "keyword"
                  },
                "relayip":
                  { "type": "keyword"
                  },
                "messageid":
                  { "type": "keyword"
                  },
                "pid":
                  { "type": "long"
                  },
                "remote":
                  { "type": "keyword"
                  },
                "type":
                  { "type": "keyword"
                  },
                "qid":
                  { "type": "keyword"
                  },
                "local":
                  { "type": "keyword"
                  },
                "result":
                  { "type" : "keyword"
                  },
                "path":
                  { "type" : "keyword"
                  },
                "file":
                  { "type" : "keyword"
                  },
                "queuestatus":
                  { "type" : "keyword"
                  },
                "smtp_response":
                  { "type" : "keyword"
                  },
                "@version":
                  { "type" : "keyword"
                  },
                "host":
                  { "type" : "keyword"
                  },
                "client":
                  { "type" : "keyword"
                  },
                "from":
                  { "type" : "keyword"
                  },
                "timestamp":
                  { "type" : "keyword"
                  },
                "nrcpt":
                  { "type" : "long"
                  },
                "responsecode":
                  { "type" : "long"
                  },
                "offset":
                  { "type" : "long"
                  },
                "relayhost":
                  { "type" : "keyword"
                  },
                "logsource":
                  { "type" : "keyword"
                  },
                "message":
                  { "type" : "keyword"
                  },
                "orig_to":
                  { "type" : "keyword"
                  },
                "command":
                  { "type" : "keyword"
                  },
                "tags":
                  { "type" : "keyword"
                  },
                "helo":
                  { "type" : "keyword"
                  },
                "saslmethod":
                  { "type" : "keyword"
                  },
                "component":
                  { "type" : "keyword"
                  },
                "@timestamp":
                  {  "type" : "date" , "format": "date_optional_time"
                  },
                "remotehost":
                  { "type" : "keyword"
                  },
                "size":
                  { "type" : "long"
                  },
                "anvilstatistic":
                  { "type" : "keyword"
                  },
                "proto":
                  { "type" : "keyword"
                  },
                "bouncequeueid":
                  { "type" : "keyword"
                  },
                "to":
                  { "type" : "keyword"
                  },
                "dsn":
                  { "type" : "keyword"
                  }
            }
        }
    }
}

But I get the message

{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to parse content to map"}],"type":"parse_exception","reason":"Failed to parse content to map","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@2eb92b4d; line: 2, column: 2]"}},"status":400}

As I understand the message I should give a field name, instead of a { somewhere?

Magnus_Kessler · January 25, 2019, 10:32am

There were a couple of issues with your template. index_patterns was mispelled, and also takes an array as its argument. A template must include the order field. The extra comma after the number_of_shards causes the reported error. JSON can be quite picky about its friends

{
  "index_patterns": [
    "maillog-*"
  ],
  "order": 0,
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "maillog": {
      "properties": {
        "reason": {
          "type": "keyword"
        },
        "saslusername": {
          "type": "keyword"
        },
        "postfix_action": {
          "type": "keyword"
        },
        "relayip": {
          "type": "keyword"
        },
        "messageid": {
          "type": "keyword"
        },
        "pid": {
          "type": "long"
        },
        "remote": {
          "type": "keyword"
        },
        "type": {
          "type": "keyword"
        },
        "qid": {
          "type": "keyword"
        },
        "local": {
          "type": "keyword"
        },
        "result": {
          "type": "keyword"
        },
        "path": {
          "type": "keyword"
        },
        "file": {
          "type": "keyword"
        },
        "queuestatus": {
          "type": "keyword"
        },
        "smtp_response": {
          "type": "keyword"
        },
        "@version": {
          "type": "keyword"
        },
        "host": {
          "type": "keyword"
        },
        "client": {
          "type": "keyword"
        },
        "from": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "keyword"
        },
        "nrcpt": {
          "type": "long"
        },
        "responsecode": {
          "type": "long"
        },
        "offset": {
          "type": "long"
        },
        "relayhost": {
          "type": "keyword"
        },
        "logsource": {
          "type": "keyword"
        },
        "message": {
          "type": "keyword"
        },
        "orig_to": {
          "type": "keyword"
        },
        "command": {
          "type": "keyword"
        },
        "tags": {
          "type": "keyword"
        },
        "helo": {
          "type": "keyword"
        },
        "saslmethod": {
          "type": "keyword"
        },
        "component": {
          "type": "keyword"
        },
        "@timestamp": {
          "type": "date",
          "format": "date_optional_time"
        },
        "remotehost": {
          "type": "keyword"
        },
        "size": {
          "type": "long"
        },
        "anvilstatistic": {
          "type": "keyword"
        },
        "proto": {
          "type": "keyword"
        },
        "bouncequeueid": {
          "type": "keyword"
        },
        "to": {
          "type": "keyword"
        },
        "dsn": {
          "type": "keyword"
        }
      }
    }
  }
}

fribse · January 25, 2019, 10:47am

Ok, had one more problem, an extra { had sneaked it's way into the PUT command.

Now, the order field is a bit of a surprise, as a lot of the examples in the doc doesn't show that?

Thankyou for the fix.

system · February 22, 2019, 10:48am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.