# How to split log data into message fields when we have dynamic logs **URL:** \ **Category:** Logstash\ **Created:** [June 4, 2020, 10:04am UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725 "2020-06-04T10:04:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![hemant\_472](https://avatars.discourse-cdn.com/v4/letter/h/6bbea6/32.png) [@hemant\_472](https://discuss.elastic.co/u/hemant_472)\ **Post date:** [June 4, 2020, 10:04am UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725/1 "2020-06-04T10:04:08Z")
i have created some log patterns in logstash but when i get new log pattern it is parsed into the kibana as a message filed but is not split up into other fields. I would like to know do i need to add log patterns for every type of logs or is there any other way logstash or elastic search can divide all types of logs into our required fields This is a sample data ```auto Dec 27 06:25:05 desk.outlet19 MD: Sending desk is closed to UI Dec 27 06:25:03 desk.outlet17 MD: Sending desk is closed to UI Dec 27 06:25:03 desk.outlet17 MD: State change done! Dec 27 06:25:03 desk.outlet17 MD: Request state change enter standby to UI Dec 27 06:25:03 desk.outlet12 MD: Sending desk is closed to UI Dec 27 06:25:06 desk.outlet19 MD: Sending desk is closed to UI Dec 27 06:25:06 desk.outlet19 MD: State change done! ``` my logstash configuration for this is ```auto input { beats { port => "5044" } } filter { grok { match => {"message" => '(?[\w\s\d\:]+)\s(?[\w\d\.]+)\s(?[\w\:]+)\s(?[\w\s\!]+)' } }} output { elasticsearch { hosts => ["http://localhost:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } ``` but when i received new log data like this ```auto Dec 27 10:38:29 ff402-srv1 MC: DEBUG {2151} [PlanJob] ABSTRACT: SyncMode=Speed Speed=0.100 Delay=0.000 HasSG=0 StartRamp=0.030 StopRamp=0.030 MasterAxisId=0 Dec 27 10:38:29 ff402-srv1 MC: DEBUG {2151} [PlanJob] ABSTRACT.AxisId=312 AxisName=Center:v PlannedStart=4.161 Target=4.324 UseCurrPos=1 Dec 27 10:38:29 ff402-srv1 MC: DEBUG {2151} [PlanJob] EXPLICIT: DuraNoRamp=1.627 MaxMinDuraNoRamp=0.163 MaxSpeed=1.000 MinActP=0.000000 MaxActP=0.000000 ActP=0.000000 Dec 27 10:38:29 ff402-srv1 MC: DEBUG {2151} [PlanJob] EXPLICIT.AxisId=312 AxisName=Center:v PlannedStart=4.161 Start=4.161 ActIsPos=4.161 Target=4.324 Speed=0.100 StartRamp=0.030 StopRamp=0.030 LimRampScaling=1.000 Dec 27 10:38:29 ff402-srv1 MC: DEBUG {2151} [ApiUiJob] PlanJob(): return Speed=0.100000 Duration=4.658009 MinDuration=4.658009 MaxSpeed=1.000000 ``` how can i handle this data in my logstash , i mean is there any configuration which can only fetch the required data from the log files even when the logs are dynamic and it is difficult for a log pattern to recognize ---
**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\ **Post date:** [June 4, 2020, 12:24pm UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725/2 "2020-06-04T12:24:08Z")
you might want to share sample input and current logstash configuration to allow others to help you. ---
**Author:** ![hemant\_472](https://avatars.discourse-cdn.com/v4/letter/h/6bbea6/32.png) [@hemant\_472](https://discuss.elastic.co/u/hemant_472)\ **Post date:** [June 4, 2020, 12:45pm UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725/3 "2020-06-04T12:45:57Z")
Hi @ptamba , i have added the data to my post please check once and share your views ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 4, 2020, 1:48pm UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725/4 "2020-06-04T13:48:49Z")
If you add ``` kv {} ``` to your filter it will parse the key/value pairs, resulting in events like ``` { "AxisName" => "Center:v", "PlannedStart" => "4.161", "ActIsPos" => "4.161", ... "Start" => "4.161", "LimRampScaling" => "1.000", "@timestamp" => 2020-06-04T13:46:28.181Z, "outlet" => "27", "Target" => "4.324", "StartRamp" => "0.030", "StopRamp" => "0.030", "timestamp" => "Dec", "EXPLICIT.AxisId" => "312", "message" => "Dec 27 10:38:29 ff402-srv1 MC: DEBUG {2151} [PlanJob] EXPLICIT.AxisId=312 AxisName=Center:v PlannedStart=4.161 Start=4.161 ActIsPos=4.161 Target=4.324 Speed=0.100 StartRamp=0.030 StopRamp=0.030 LimRampScaling=1.000", "ID" => "10:38:29", "Status" => "ff402", "Speed" => "0.100" } ``` Personally I would use dissect rather than grok for this ``` dissect { mapping => { "message" => "%{timestamp} %{+timestamp} %{+timestamp} %{outlet} %{ID} %{Status} %{restOfLine}" } } kv { source => "restOfLine" } ``` ---
**Author:** ![hemant\_472](https://avatars.discourse-cdn.com/v4/letter/h/6bbea6/32.png) [@hemant\_472](https://discuss.elastic.co/u/hemant_472)\ **Post date:** [June 5, 2020, 8:40am UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725/5 "2020-06-05T08:40:26Z")
Hi @Badger thanks for sharing this pattern i will try with both patterns and will update if i get everything running ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 3, 2020, 8:51am UTC](https://discuss.elastic.co/t/how-to-split-log-data-into-message-fields-when-we-have-dynamic-logs/235725/6 "2020-07-03T08:51:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.