Create fields by splitting a log

[prasadlele]

Hi ,

I have configure logstash to read my log file which gives me below input. Now I want to split the message and create a field for each comma separated values against each. Tried to use mutate in my logstash.conf file however it did not work.

Log snippet (This is a single line)

werwerwer.com-1579808930169-13:1:1:1:634
03-Mar-2020 18:05:26,922 DEBUG [com.alu.cna.cloudmgmt.util.jms.JMSMessageHandler] (Message_Handler_Pool_Thread_1) message body: {"userName":null,"enterpriseName":null,"type":"DELETE","entityType":"alarm","entityVersion":null,"assoicatedEvent":true,"eventReceivedTime":1583238926914,"entities":[{"children":null,"parentType":"nsgateway","entityScope":"ENTERPRISE","embeddedMetadata":null,"lastUpdatedBy":"43f8868f-4bc1-472c-9d19-533dcfcb1ee0","lastUpdatedDate":1583238691000,"creationDate":1583238691000,"reason":"NSG:test NSG-C601-SIM(SystemID:216.87.90.41) - All configured NTP servers are unreachable.","description":"NSG:testNSG-C601-SIM(SystemID:216.87.90.41) - All configured NTP servers are unreachable.","acknowledged":false,"numberOfOccurances":1,"severity":"MAJOR","errorCondition":70002,"remedy":"Check the configured NTP servers.","owner":"43f8868f-4bc1-472c-9d19-533dcfcb1ee0","ID":"6b61d4ab-1eb0-4c4b-91d9-35b5437d87ac","parentID":"fe1b5d5d-3a7f-4e33-bbcb-81bfcf70f27a","externalID":null,"title":"No NTP Server Connection","timestamp":1583238691763,"targetObject":"nsgateway","alarmedObjectID":"fe1b5d5d-3a7f-4e33-bbcb-81bfcf70f27a","enterpriseID":"ff9dd969-5dd7-45e8-8afe-4f6215be462e"}],"diffMap":null,"associatedEntityType":null,"addedAssociations":null,"deletedAssociations":null,"ignoreDiffInMediationEvents":false,"updateMechanism":"DEFAULT","requestID":null,"sourceEnterpriseID":"ff9dd969-5dd7-45e8-8afe-4f6215be462e"}

cat /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/opt/jmsclient/log/jmsclient.log"
}
}
filter {
mutate {
split => { "userName" => "," }
split => { "enterpriseName" => "," }
split => { "type" => "," }
split => { "entityType" => "," }
split => { "entityVersion" => "," }
split => { "assoicatedEvent" => "," }
split => { "eventReceivedTime" => "," }
split => { "entities" => "," }
split => { "parentType" => "," }
split => { "entityScope" => "," }
split => { "lastUpdatedBy" => "," }
split => { "lastUpdatedDate" => "," }
split => { "creationDate" => "," }
split => { "name" => "," }
split => { "reason" => "," }
split => { "description" => "," }
output {
elasticsearch {
hosts => "10.3.1.86"
}
}

[prasadlele]

Can some one please help here?? I am in urgent need.

[dadoonet (David Pilato)]

Read this and specifically the "Also be patient" part.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

[dadoonet (David Pilato)]

Question moved to #logstash

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

[Badger]

I would use dissect to parse the beginning of the line, something like this. Then use a json filter to parse the rest of the line.

[prasadlele]

Thanks for your reply, Yes this was my first post over the discuss. I will try and be more sensible next time. Also will try out your suggestions.

[prasadlele]

Thanks @Badger I will try your suggestions too. However if you look at the message the second half is already in a JSON format. I tried to use JSON filter however it did not parse anything.

[Badger]

A json filter will not trim off random text before or after JSON. You have to do that.