Hello, I'm new to the elastic ecosystem. I'm trying to use logstash to take a json file as input and then use a filter to split into different events but I'm drowned by all of the info there is online. This is a short version of my code { "proximity": [ { "network": "all_networks"…

If you have a working json codec then typically you do not need a json filter as well. However, unless you have complete JSON objects on each line a json codec will not work, you need a multiline codec. If you want to consume the entire file as a single event then configure the multiline codec as d…

Thanks for replying so fast. So I already have used the codec and have the whole json inside the "message" field and called it using the json filter. Unfortunately the split filter is not doing what i need (as you mentioned previously). Is there a good guide/link that explains how to get "events"…

[image] tcaetano1995: Unfortunately the split filter is not doing what i need (as you mentioned previously). What problem do you have with the results of the split filter?

I have applied the following filters json { source => "message" remove_field => [ "message" ] } mutate { remove_field => [ "host", "@version", "@timestamp", "path" ] } And my output looks like this: { "visitLength": [ (here I have the 2 elements) ], "loyaltyRecords": [ (another 2) ], "pro…

UPDATE I have found that in ruby I could create a new field with other fields data, ruby { code => 'event.set("my_field", [ event.get("proximity"), event.get("loyaltyRecords"), event.get("visitLength") ]) '} Problem is that I keep getting them nested, is there a way to get the all the elem…

I replaced the "doc" for one of my fields to get the objects out of the field into top level That seems to be the solution from what I can read of the code but I'm getting an exception Ruby exception occurred: no implicit conversion of Hash into String

[image] tcaetano1995: Ruby exception occurred: no implicit conversion of Hash into String That suggests that you are trying to do it to an array of hashes. You would need to split them first. What does an event look like if you send it to this?... output { stdout { codec => rubydebug } }

Without the split 2020-05-15T13:38:34,375][ERROR][logstash.filters.ruby ][main] Ruby exception occurred: no implicit conversion of Hash into String .7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "visitLength" => [ …

[image] tcaetano1995: ruby { code => 'event.set("my_field", [ event.get("proximity"), event.get("loyaltyRecords"), event.get("visitLength") ]) '} My filters are: json { source => "message" remove_field => [ "message" ] } mutate { remove_field => [ "host", "@version", "@timestamp", "path" ] …

Discuss the Elastic Stack

Json input and splitting

Elastic Stack Logstash

Badger May 15, 2020, 3:50pm 7

If you want to move nested fields to the top level you can do something like this.

Topic		Replies	Views
Logstash - split array into individual events Logstash	7	6614	June 16, 2019
Logstash and JSON array split Logstash	5	386	October 16, 2020
Splitting JSON arrays Logstash	3	309	March 26, 2019
Logstash and JSON array split [SOLVED] Logstash	11	31925	July 6, 2017
Have Logstash take a file containing an array of JSON objects Logstash	5	5260	October 26, 2018

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Json input and splitting

Related topics