Fields in messages

[kayaHL (KayaSearch)]

Hi guys, I am new to Elasticsearch... I am using filebeat to ship the multiple line unstructured messages to logstash and use grok filter to structure the data. However, I noticed in the application logs, between messages, some messages have less field/value pair, some messages have more field/value pair, the grok
patterns doesn't work for all messages... so of the messages can't be structured. Could someone help me and give me a right direction for the solution?

Example:

message 1:

04:45:01.633 - R (1,2) msglen:89 New
Timestamp:109660 ItemCode:TEST1 ListingDate:20100102 Expiry:0 Term:

message 2:

04:45:01.633 - R (5,9) msglen:97 DELETE
Timestamp:20988 ItemCode:TEST2 ListingType:CIN ListingDate:20160102 Term: Processingdate: 2017-0108

Many thanks!

[dadoonet (David Pilato)]

I believe it's more a #logstash question so I moved your post.

BTW please format your code using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

Please edit your post.

[magnusbaeck (Magnus Bäck)]

Use a kv filter to parse lists of key/value pairs, not grok.

[kayaHL (KayaSearch)]

Thank you! I will give it a try.