Split row data to fields

Elastic Stack Logstash
1

Hi ,

When I load Json file to ElasticSearch , I sea the data in 1 column (message ).
For example:
{"Group": "abi", "ExTime": "", "LastSuccess": "", "PadDG%": "", "TestName": "PT_FOTA/abi/SUPPORT/test132", "WI": "", "TWrtRatio": "", "B-Error": "", "IMG-FS.DG%": "", "BupgRet": "0", "Policy": "", "Ratio": "", "B-DeltaSize": "0.161", "WrtRatio": "", "B-Fail-On": "", "Status": "Missing", "WrittenGap": "", "PadDelta": "", "Hash": "Miss", "Description": "", "Tags": "abi sr fsimg", "TargetImg": "", "lookUpStr": "", "TotalTime": "", "GenTime": "", "Base HTML": "http://qasrvlinux/Results/9.4/9.4.1.7/linux/PT_FOTA/abi/SUPPORT/test132/HTML/RUN/8A01021D-D8AB8450-9D170943.html", "FileDG%": "", "HTML": "http://qasrvlinux", "DeltaSize": "", "SourceImg": "", "B-ExTIme": "1", "B-TWritten": "", "B-GenTime": "5", "VersionParam": "", "RamGap%": "", "DG%": "", "TWrittenGap": "", "FileDelta": "", "SrcSize": "", "B-MinRam": "3.915", "TrgSize": "", "B-FileDelta": "", "B-LastSuccess": "PCUA 64b SimulateReset", "RamSize": "", "upgRet": "", "Range": "7", "IsOct": "No", "DeltaGap": "", "Error": "", "B-ErrType": "", "MinRam": "", "ExGap": "", "TWritten": "", "RamGap": "", "Fail-On": "", "NumBck": "", "Agent": "", "Bstatus": "Pass", "ramRatio": "", "ReqID": "", "SecSize": "", "GenGap": "", "ErrType": "", "B-Ratio": "", "B-PadDelta": "", "Subject": " abi"}

I upload the data to ElasticSearch with logstash , this is the conf file:
input {
file{
path => ["/root/scripts/logs/update/tmp1.json"]
type => "json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter{
grok {
match => [ 'message', '(?"TestName":.*"Agent":"[^"]+")' ]
add_field => [ "json_body", "{%{body}}" ]
}
json {
source => "json_body"
remove_field => ["message","body","json_body"]
}
mutate {
add_field => ["[geoip][location]","%{[SrcSize]}"]
add_field => ["[geoip][location]","%{[TrgSize]}"]
add_field => ["[geoip][location]","%{[B-DeltaSize]}"]
add_field => ["[geoip][location]","%{[DeltaSize]}"]
add_field => ["[geoip][location]","%{[B-Ratio]}"]
add_field => ["[geoip][location]","%{[Ratio]}"]
add_field => ["[geoip][location]","%{[B-MinRam]}"]
add_field => ["[geoip][location]","%{[MinRam]}"]
add_field => ["[geoip][location]","%{[ramRatio]}"]
add_field => ["[geoip][location]","%{[RamGap]}"]
add_field => ["[geoip][location]","%{[RamGap%]}"]
}
mutate {
convert => [ "[geoip][location]", "float"]
}
}

output {
stdout {
codec => rubydebug
}
elasticsearch {
host => "10.XX.XX.XXX"
protocol => "http"
index => "index_update"
}
}

If I want each one of the row will be in a separated column , how can I do it ?

BR,
Chen

2

add_field => [ "json_body", "{%{body}}" ]

Where is the body field created, and why are there curly braces at the beginning and end of this string?

What do you get to stdout when you run this?

3

Hi ,

I'm running logstash like this:
./bin/logstash agent -f logstash.conf
At the files there are almost 10,000 rows the same as the 1 I attached.
The result is the same if I put "[" character at the start , "]" at the end and "," between each one of the lines.
The first line at the json file is:
{"Group": "Group", "ExTime": "ExTime", "LastSuccess": "LastSuccess", "PadDG%": "PadDG%", "TestName": "TestName", "WI": "WI", "TWrtRatio": "TWrtRatio", "B-Error": "B-Error", "IMG-FS.DG%": "IMG-FS.DG%", "BupgRet": "BupgRet", "Policy": "Policy", "Ratio": "Ratio", "B-DeltaSize": "B-DeltaSize", "WrtRatio": "WrtRatio", "B-Fail-On": "B-Fail-On", "Status": "Status", "WrittenGap": "WrittenGap", "PadDelta": "PadDelta", "Hash": "Hash", "Description": "Description", "Tags": "Tags", "TargetImg": "TargetImg", "lookUpStr": "lookUpStr", "TotalTime": "TotalTime", "GenTime": "GenTime", "Base HTML": "Base HTML", "FileDG%": "FileDG%", "HTML": "HTML", "DeltaSize": "DeltaSize", "SourceImg": "SourceImg", "B-ExTIme": "B-ExTIme", "B-TWritten": "B-TWritten", "B-GenTime": "B-GenTime", "VersionParam": "VersionParam", "RamGap%": "RamGap%", "DG%": "DG%", "TWrittenGap": "TWrittenGap", "FileDelta": "FileDelta", "SrcSize": "SrcSize", "B-MinRam": "B-MinRam", "TrgSize": "TrgSize", "B-FileDelta": "B-FileDelta", "B-LastSuccess": "B-LastSuccess", "RamSize": "RamSize", "upgRet": "upgRet", "Range": "Range", "IsOct": "IsOct", "DeltaGap": "DeltaGap", "Error": "Error", "B-ErrType": "B-ErrType", "MinRam": "MinRam", "ExGap": "ExGap", "TWritten": "TWritten", "RamGap": "RamGap", "Fail-On": "Fail-On", "NumBck": "NumBck", "Agent": "Agent", "Bstatus": "Bstatus", "ramRatio": "ramRatio", "ReqID": "ReqID", "SecSize": "SecSize", "GenGap": "GenGap", "ErrType": "ErrType", "B-Ratio": "B-Ratio", "B-PadDelta": "B-PadDelta", "Subject": "Subject"}

I build my conf file according the examples I saw in the web.
how can I load my data according to the structure I need ?

BR,
Chen

4

You didn't answer my two questions. Unless you do that I can't help you.

5

This is a screenshot of the sdtout:

pic_es.PNG1510×711 27.2 KB

How can I know where the body field is created ?

6

Okay. As far as I can tell you have a bunch of unnecessary stuff in your configuration and judging by your most recent question you don't appear to know yourself what it's supposed to do. I reckon a simple json { source => "message" } filter should be enough to extract all the fields from the JSON string in the message field.

7

Thanks :slight_smile: