# Postfix parsing -
**URL:** https://discuss.elastic.co/t/postfix-parsing/165193
**Category:** Logstash
**Created:** [January 22, 2019, 8:52am UTC](https://discuss.elastic.co/t/postfix-parsing/165193 "2019-01-22T08:52:39Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@fribse](https://discuss.elastic.co/u/fribse)
#### Post date: [January 22, 2019, 8:52am UTC](https://discuss.elastic.co/t/postfix-parsing/165193/1 "2019-01-22T08:52:39Z")
I found Tin Le's setup on Linked In, and I think it looked really nice, so I wanted to add those logfiles to our ELK as well.
I've created the patterns file, and modified my logstash.con, I'll be getting the logfiles with filebeat, so it's a bit different than Tin Le's.
I'm trying to add the template, but I get an error:
```
curl -X PUT "elasticsearch:9200/_template/postfix" -H 'Content-Type: application/json' -d '{
"template" : "maillog-*",
"order" : 1,
"settings" : {
"number_of_shards" : 2,
"index.refresh_interval" : "90s"
},
"mappings" : {
"maillog" : {
"properties" : {
"reason" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"saslusername" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"postfix_action" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"relayip" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"messageid" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"pid" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
"remote" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"type" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"qid" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"local" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"result" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"path" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"file" : { "index": "not_analyzed", "type" : "string" },
"queuestatus" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"smtp_response" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"@version" : { "type" : "string" },
"host" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"client" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"from" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"timestamp" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"nrcpt" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
"responsecode" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
"offset" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
"relayhost" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"logsource" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"message" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"orig_to" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"command" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"tags" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"helo" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"saslmethod" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"component" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"@timestamp" : { "format" : "dateOptionalTime", "type" : "date" },
"remotehost" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"size" : { "index": "not_analyzed", "doc_values": true, "type" : "long" },
"anvilstatistic" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"proto" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"bouncequeueid" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"to" : { "index": "not_analyzed", "doc_values": true, "type" : "string" },
"dsn" : { "index": "not_analyzed", "doc_values": true, "type" : "string" }
}
}
}
}'
```
That gave me this error:
`{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [string] declared on field [reason]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [maillog]: No handler for type [string] declared on field [reason]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [string] declared on field [reason]"}},"status":400}[`
Ignoring that, and sending the logfiles to logstash with filebeat, it is imported, but not interpreted.
A few logline examples from our mailservers:
```
Jan 22 10:53:42 alpha55002 postfix/smtp[23919]: 14117807B898: host gmail-smtp-in.l.google.com[108.177.127.27] said: 421-4.7.0 This message does not have authentication information or fails to pass 421-4.7.0 authentication checks. To best protect our users from spam, the 421-4.7.0 message has been blocked. Please visit 421-4.7.0 https://support.google.com/mail/answer/81126#authentication for more 421 4.7.0 information. s12-v6si1426609ejd.277 - gsmtp (in reply to end of DATA command)
Jan 22 10:53:45 alpha55002 postfix/smtp[23919]: Untrusted TLS connection established to alt1.gmail-smtp-in.l.google.com[74.125.24.27]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
```
---
### Author:  [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)
#### Post date: [January 22, 2019, 11:49am UTC](https://discuss.elastic.co/t/postfix-parsing/165193/2 "2019-01-22T11:49:16Z")
I'm going to assume that you're using a recent release of Elasticsearch. Since Elasticsearch 5.0 the way strings are handled have changed, and use either `text` or `keyword` instead of `string`/ {`analyzed`, `not_analyzed`). Please adapt your mapping accordingly.
In addition, you'll have to set up an appropriate `filter` in your Logstash configuration to break the message into individual fields. On its own, Filebeat will only send the original log entry and some metadata.
---
### Author:  [@fribse](https://discuss.elastic.co/u/fribse)
#### Post date: [January 22, 2019, 12:06pm UTC](https://discuss.elastic.co/t/postfix-parsing/165193/3 "2019-01-22T12:06:30Z")
Hi Magnus
Thankyou, oh wow, very new to the ELK world, so I don't have any 'history' with me about changed syntaxes 🙂
Regarding the filter, I found Tin Le's setup on Linked In ([https://www.linkedin.com/pulse/monitoring-postfix-logs-elk-tin-le/?trk=pulse\_spock-articles](https://www.linkedin.com/pulse/monitoring-postfix-logs-elk-tin-le/?trk=pulse_spock-articles)), which has some very nice grok filters ([https://gist.github.com/TinLe/f9763ac68c122cd11b91](https://gist.github.com/TinLe/f9763ac68c122cd11b91)) that I don't understand at all. I've also just started on Grok, and they are a bit more complicated than what I've done so far 🙂
---
### Author:  [@fribse](https://discuss.elastic.co/u/fribse)
#### Post date: [January 22, 2019, 1:37pm UTC](https://discuss.elastic.co/t/postfix-parsing/165193/4 "2019-01-22T13:37:40Z")
> [@Magnus\_Kessler](#):
>
> use either `text` or `keyword` instead of `string` / { `analyzed` , `not_analyzed` ). Please adapt your mapping accordingly.
Not really sure I understand that fully.
I tried changing it according to [Elasticsearch replaces string type with two new types text and keyword. | Elastic Blog](https://www.elastic.co/blog/strings-are-dead-long-live-strings), and then the top was changed to index\_patterns, not sure if that has been changed in the syntax previously as well.
```
curl -X PUT "elasticsearch:9200/_template/postfix" -H 'Content-Type: application/json' -d '
{
"index_patterns" : "maillog-*",
"order" : 1,
"settings" : {
"number_of_shards" : 1,
"index.refresh_interval" : "90s"
},
"mappings" : {
"maillog" : {
"properties" : {
"reason" : { "index": "text", "doc_values": true, "type" : "keyword" },
"saslusername" : { "index": "text", "doc_values": true, "type" : "keyword" },
"postfix_action" : { "index": "text", "doc_values": true, "type" : "keyword" },
"relayip" : { "index": "text", "doc_values": true, "type" : "keyword" },
"messageid" : { "index": "text", "doc_values": true, "type" : "keyword" },
"pid" : { "index": "text", "doc_values": true, "type" : "long" },
"remote" : { "index": "text", "doc_values": true, "type" : "keyword" },
"type" : { "index": "text", "doc_values": true, "type" : "keyword" },
"qid" : { "index": "text", "doc_values": true, "type" : "keyword" },
"local" : { "index": "text", "doc_values": true, "type" : "keyword" },
"result" : { "index": "text", "doc_values": true, "type" : "keyword" },
"path" : { "index": "text", "doc_values": true, "type" : "keyword" },
"file" : { "index": "text", "type" : "keyword" },
"queuestatus" : { "index": "text", "doc_values": true, "type" : "keyword" },
"smtp_response" : { "index": "text", "doc_values": true, "type" : "keyword" },
"@version" : { "type" : "keyword" },
"host" : { "index": "text", "doc_values": true, "type" : "keyword" },
"client" : { "index": "text", "doc_values": true, "type" : "keyword" },
"from" : { "index": "text", "doc_values": true, "type" : "keyword" },
"timestamp" : { "index": "text", "doc_values": true, "type" : "keyword" },
"nrcpt" : { "index": "text", "doc_values": true, "type" : "long" },
"responsecode" : { "index": "text", "doc_values": true, "type" : "long" },
"offset" : { "index": "text", "doc_values": true, "type" : "long" },
"relayhost" : { "index": "text", "doc_values": true, "type" : "keyword" },
"logsource" : { "index": "text", "doc_values": true, "type" : "keyword" },
"message" : { "index": "text", "doc_values": true, "type" : "keyword" },
"orig_to" : { "index": "text", "doc_values": true, "type" : "keyword" },
"command" : { "index": "text", "doc_values": true, "type" : "keyword" },
"tags" : { "index": "text", "doc_values": true, "type" : "keyword" },
"helo" : { "index": "text", "doc_values": true, "type" : "keyword" },
"saslmethod" : { "index": "text", "doc_values": true, "type" : "keyword" },
"component" : { "index": "text", "doc_values": true, "type" : "keyword" },
"@timestamp" : { "format" : "dateOptionalTime", "type" : "date" },
"remotehost" : { "index": "text", "doc_values": true, "type" : "keyword" },
"size" : { "index": "text", "doc_values": true, "type" : "long" },
"anvilstatistic" : { "index": "text", "doc_values": true, "type" : "keyword" },
"proto" : { "index": "text", "doc_values": true, "type" : "keyword" },
"bouncequeueid" : { "index": "text", "doc_values": true, "type" : "keyword" },
"to" : { "index": "text", "doc_values": true, "type" : "keyword" },
"dsn" : { "index": "text", "doc_values": true, "type" : "keyword" }
}
}
}
}'
```
And that now gives me:
`{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Failed to parse mapping [maillog]: Could not convert [reason.index] to boolean"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [maillog]: Could not convert [reason.index] to boolean","caused_by":{"type":"illegal_argument_exception","reason":"Could not convert [reason.index] to boolean","caused_by":{"type":"illegal_argument_exception","reason":"Failed to parse value [text] as only [true] or [false] are allowed."}}},"status":400}[`
---
### Author:  [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)
#### Post date: [January 22, 2019, 1:48pm UTC](https://discuss.elastic.co/t/postfix-parsing/165193/5 "2019-01-22T13:48:01Z")
Try with e.g. `"reason" : { "type" : "keyword" }`, and make sure you read the [documentation for mappings and datatypes](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html).
---
### Author:  [@fribse](https://discuss.elastic.co/u/fribse)
#### Post date: [January 25, 2019, 10:01am UTC](https://discuss.elastic.co/t/postfix-parsing/165193/6 "2019-01-25T10:01:57Z")
Ok, so as I understand it, the index above was made for a previous version?
I tried changing it, but I get an error I don't understand, as far as I can see the format in the documentation is correct?
This is the new template definition I've made:
```
{
"index_patters" : "maillog-*",
"settings" : {
"number_of_shards" : 1,
},
"mappings": {
"maillog": {
"properties": {
"reason":
{ "type": "keyword"
},
"saslusername":
{ "type": "keyword"
},
"postfix_action":
{ "type": "keyword"
},
"relayip":
{ "type": "keyword"
},
"messageid":
{ "type": "keyword"
},
"pid":
{ "type": "long"
},
"remote":
{ "type": "keyword"
},
"type":
{ "type": "keyword"
},
"qid":
{ "type": "keyword"
},
"local":
{ "type": "keyword"
},
"result":
{ "type" : "keyword"
},
"path":
{ "type" : "keyword"
},
"file":
{ "type" : "keyword"
},
"queuestatus":
{ "type" : "keyword"
},
"smtp_response":
{ "type" : "keyword"
},
"@version":
{ "type" : "keyword"
},
"host":
{ "type" : "keyword"
},
"client":
{ "type" : "keyword"
},
"from":
{ "type" : "keyword"
},
"timestamp":
{ "type" : "keyword"
},
"nrcpt":
{ "type" : "long"
},
"responsecode":
{ "type" : "long"
},
"offset":
{ "type" : "long"
},
"relayhost":
{ "type" : "keyword"
},
"logsource":
{ "type" : "keyword"
},
"message":
{ "type" : "keyword"
},
"orig_to":
{ "type" : "keyword"
},
"command":
{ "type" : "keyword"
},
"tags":
{ "type" : "keyword"
},
"helo":
{ "type" : "keyword"
},
"saslmethod":
{ "type" : "keyword"
},
"component":
{ "type" : "keyword"
},
"@timestamp":
{ "type" : "date" , "format": "date_optional_time"
},
"remotehost":
{ "type" : "keyword"
},
"size":
{ "type" : "long"
},
"anvilstatistic":
{ "type" : "keyword"
},
"proto":
{ "type" : "keyword"
},
"bouncequeueid":
{ "type" : "keyword"
},
"to":
{ "type" : "keyword"
},
"dsn":
{ "type" : "keyword"
}
}
}
}
}
```
But I get the message
`{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to parse content to map"}],"type":"parse_exception","reason":"Failed to parse content to map","caused_by":{"type":"json_parse_exception","reason":"Unexpected character ('{' (code 123)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@2eb92b4d; line: 2, column: 2]"}},"status":400}`
As I understand the message I should give a field name, instead of a { somewhere?
---
### Author:  [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)
#### Post date: [January 25, 2019, 10:32am UTC](https://discuss.elastic.co/t/postfix-parsing/165193/7 "2019-01-25T10:32:11Z")
There were a couple of issues with your template. `index_patterns` was mispelled, and also takes an array as its argument. A template must include the `order` field. The extra comma after the `number_of_shards` causes the reported error. JSON can be quite picky about its friends 😄
```auto
{
"index_patterns": [
"maillog-*"
],
"order": 0,
"settings": {
"number_of_shards": 1
},
"mappings": {
"maillog": {
"properties": {
"reason": {
"type": "keyword"
},
"saslusername": {
"type": "keyword"
},
"postfix_action": {
"type": "keyword"
},
"relayip": {
"type": "keyword"
},
"messageid": {
"type": "keyword"
},
"pid": {
"type": "long"
},
"remote": {
"type": "keyword"
},
"type": {
"type": "keyword"
},
"qid": {
"type": "keyword"
},
"local": {
"type": "keyword"
},
"result": {
"type": "keyword"
},
"path": {
"type": "keyword"
},
"file": {
"type": "keyword"
},
"queuestatus": {
"type": "keyword"
},
"smtp_response": {
"type": "keyword"
},
"@version": {
"type": "keyword"
},
"host": {
"type": "keyword"
},
"client": {
"type": "keyword"
},
"from": {
"type": "keyword"
},
"timestamp": {
"type": "keyword"
},
"nrcpt": {
"type": "long"
},
"responsecode": {
"type": "long"
},
"offset": {
"type": "long"
},
"relayhost": {
"type": "keyword"
},
"logsource": {
"type": "keyword"
},
"message": {
"type": "keyword"
},
"orig_to": {
"type": "keyword"
},
"command": {
"type": "keyword"
},
"tags": {
"type": "keyword"
},
"helo": {
"type": "keyword"
},
"saslmethod": {
"type": "keyword"
},
"component": {
"type": "keyword"
},
"@timestamp": {
"type": "date",
"format": "date_optional_time"
},
"remotehost": {
"type": "keyword"
},
"size": {
"type": "long"
},
"anvilstatistic": {
"type": "keyword"
},
"proto": {
"type": "keyword"
},
"bouncequeueid": {
"type": "keyword"
},
"to": {
"type": "keyword"
},
"dsn": {
"type": "keyword"
}
}
}
}
}
```
---
### Author:  [@fribse](https://discuss.elastic.co/u/fribse)
#### Post date: [January 25, 2019, 10:47am UTC](https://discuss.elastic.co/t/postfix-parsing/165193/9 "2019-01-25T10:47:56Z")
Ok, had one more problem, an extra { had sneaked it's way into the PUT command.
Now, the order field is a bit of a surprise, as a lot of the examples in the doc doesn't show that?
Thankyou for the fix.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 22, 2019, 10:48am UTC](https://discuss.elastic.co/t/postfix-parsing/165193/10 "2019-02-22T10:48:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.