How to split one hit in kibana ( single field data) to diiferent hits for a json data?

[Muhammed_Danish]

Hi all, I have imported the json data in postgresql to elasticsearch using jdbc driver in logstash. But it is viewed as a single hit in kibana (as one filed data) and I can't create visualization using that. So, I need to split the json data to multiple hits in kibana. While using the split filter I am getting the error "NILCLASS IS NOT SPLITTABLE". The following data need to be splitted:

"ansible_local": {"tinc_facts": {"date": "Tue Oct 1 12:49:30 IST 2019"}, "cis_audit_status": {"Dns": "", "Fat": "1.1.1.8 - Mounting of vfat filesystems is not Disabled FAILED", "Ftp": "", "Gdm": "1.7.2 - GDM Login Banner is not configured FAILED", "Hfs": "1.1.1.4 - Mounting of hfs filesystems is not Disabled FAILED", "Mcs": "", "Nis": "", "Ntp": "2.2.1.2 - NTP is not configured FAILED", "Rds": "3.5.3 RDS is not Disabled FAILED", "Rsh": "", "Tmp": "1.1.2 - Separate partition not exists for /tmp FAILED", "Udf": "1.1.1.7 - Mounting of udf filesystems is not Disabled FAILED", "Var": "1.1.6 - Separate partition not exists for /var FAILED", "Aide": "1.3.1 - AIDE is not installed FAILED", "Aslr": "1.5.3 - Address Space Layout Randomization (ASLR) is not enabled FAILED", "Cron": "", "Cups": "", "Dccp": "3.5.1 DCCP is not Disabled FAILED", "Dhcp": "", "Echo": "", "Home": "", "Http": "", "Icmp": "3.2.2 - ICMP redirects are accpeted FAILED", "Imap": "", "Ipv6": "3.3.3 - Ensure IPv6 is not disabled FAILED", "Ldap": "", "Mail": "2.2.15 - Mail Transfer Agent is not configured for local-only mode FAILED", "Nisc": "2.3.1 - NIS Client is Installed FAILED",

Thanks in advance for the help.

[Badger]

What does your split filter configuration look like?

[Muhammed_Danish]

Sorry for the late reply and thanks for your reply. Now I can able to split the json to different fields in kibana by using ruby. But my intention is to split the data to multiple hits or events in kibana. That is, I need to view the data in separate rows of the elasticsearch in head plugin.
Following is my logstash conf file:
input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://x.x.x.x:5432/awx"
jdbc_user => "postgres"
jdbc_validate_connection => true
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql-jdbc.jar"
jdbc_driver_class => "org.postgresql.Driver"
clean_run => true
schedule => "*/1 * * * *"
statement => "SELECT DISTINCT ON (name) name,id,message->>'ansible_local'::TEXT AS message FROM main"
use_column_value => true
tracking_column => "id"
}

}

filter {
mutate {
copy => { "id" => "[@metadata][_id]"}
remove_field => ["id", "@version"]
}
}

filter {
mutate {
copy => { "message" => "[@metadata][_message]"}
remove_field => ["message", "@version"]
}
}

filter {
mutate {
copy => { "name" => "[@metadata][_name]"}
remove_field => ["name", "@version"]
}
}
filter {
json { source => "[@metadata][_message]" }
json {
skip_on_invalid_json => true
source => ["[@metadata][_message][0]"]
target => "[@metadata][_message]"
}
}
filter {
json {
source => "[@metadata][_message]"
}
ruby {
code => '
event.to_hash.each do |key, value|
if value.is_a?(Hash)
value.each do |field, child|
if child.is_a?(Array)
event.set("renameable_field", "[#{key}][#{field}]")
break
end
end
end
end
'
}
if [renameable_field] {
mutate {
rename => {"[%{renameable_field}]" => "field_that_needs_splitting"}
}
split {
field => "[field_that_needs_splitting]"
}
mutate {
rename => {"field_that_needs_splitting" => "[%{renameable_field}]"}
}
ruby {
code => '
renameable_field = event.remove("renameable_field")
inner = event.get(renameable_field)
if inner.is_a?(Hash)
inner.each do |field, child|
if child.is_a?(Array)
event.set("renameable_field", renameable_field + "[#{field}]")
break
end
end
end
'
}
if [renameable_field] {
mutate {
rename => {"[%{renameable_field}]" => "field_that_needs_splitting"}
}
split {
field => "[field_that_needs_splitting]"
}
mutate {
rename => {"field_that_needs_splitting" => "[%{renameable_field}]"}
}
}
}
}
output {
elasticsearch {
index => "%{[@metadata][_name]}-index"
document_id => "%{[@metadata][_id]}"
hosts => "x.x.x.x:9200"
}
}

Is there any way to split the fields to multiple hits in kibana from logstash? with field only I can't able to create visualization according to my need.