Parsing Array Object in Logstash

Elastic Stack Logstash
1

Hi, i want to pars an array of object in Logstash. This is mi conf:

input {
stdin { }
}
filter {
grok {
match => {
"message" => "%{DATESTAMP:timestamp}%{SPACE}%{NOTSPACE:num_column}%{SPACE}%{NOTSPACE:sensor001}%{SPACE}%{NOTSPACE:sensor002}%{SPACE}%{NOTSPACE:sensor003}%{SPACE}%{NOTSPACE:sensor004}%{SPACE}%{NOTSPACE:sensor005}%{SPACE}%{NOTSPACE:sensor006}%{SPACE}%{NOTSPACE:sensor007}%{SPACE}%{NOTSPACE:sensor008}%{SPACE}%{NOTSPACE:sensor009}"
}

  add_field => { "data_sensor" => "[{'sensor':'001','value':'%{sensor001}'},{'sensor':'002','value':'%{sensor002}'},{'sensor':'003','value':'%{sensor003}'},{'sensor':'004','value':'%{sensor004}'},{'sensor':'005','value':'%{sensor005}'},{'sensor':'006','value':'%{sensor006}'},{'sensor':'007','value':'%{sensor007}'},{'sensor':'008','value':'%{sensor008}'},{'sensor':'009','value':'%{sensor009}'}]" }

  remove_field => ["num_column", "offset", "input_type", "message", "type", "beat", "garbage", "tags", "sensor001", "sensor002", "sensor003", "sensor004", "sensor005", "sensor006", "sensor007", "sensor008", "sensor009"]
}

}
output {
stdout { codec => json }
}

OUTPUT:
"data_sensor":"[{'sensor':'001','value':'1'},{'sensor':'002','value':'2'},{'sensor':'003','value':'3'},{'sensor':'004','value':'4'},{'sensor':'005','value':'5'},{'sensor':'006','value':'6'},{'sensor':'007','value':'7'},{'sensor':'008','value':'8'},{'sensor':'009','value':'9'}]"

Whit add_field i create structure, but it's a "message" and i can't access into fields of array object. Can i parse the message in JSON or there ara another solution ?

2

Indeed, your add_field line creates a single string field. To create a field that's an array of objects you'll have to use a ruby filter.

3

Hi Magnus, thanks for reply.
Ok now i try with ruby filter, but the syntax is the same of add_field ?

4

No, you'll have to write Ruby code. See the ruby filter documentation and https://www.elastic.co/guide/en/logstash/current/event-api.html.

5

Thanks a lots Magnus, i try it now!

6

ruby {
code => 'event.set("data_sensor", [{"sensor" => 001, "value" => event.get("sensor001")}, {"sensor" => 002, "value" => event.get("sensor002")}, {"sensor" => 003, "value" => event.get("sensor003")}, {"sensor" => 004, "value" => event.get("sensor004")}, {"sensor" => 005, "value" => event.get("sensor005")}, {"sensor" => 006, "value" => event.get("sensor006")}, {"sensor" => 007, "value" => event.get("sensor007")}, {"sensor" => 008, "value" => event.get("sensor008")}])'
}

It is work fine, but when i add {"sensor" => 008, "value" => event.get("sensor008")} get me this error:
SyntaxError: (ruby filter code):1: syntax error, unexpected tINTEGER
@codeblock = lambda { |event, &new_event_block| event.set("data_sensor", [{"sensor" => 001, "value" => event.get("sensor001")}, {"sensor" => 002, "value" => event.get("sensor002")}, {"sensor" => 003, "value" => event.get("sensor003")}, {"sensor" => 004, "value" => event.get("sensor004")}, {"sensor" => 005, "value" => event.get("sensor005")}, {"sensor" => 006, "value" => event.get("sensor006")}, {"sensor" => 007, "value" => event.get("sensor007")}, {"sensor" => 008, "value" => event.get("sensor008")}]) }
eval at org/jruby/RubyKernel.java:994
register at /Users/gargiu/Documents/ELK/logstash-6.0.0/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.0.4/lib/logstash/filters/ruby.rb:38
register at /Users/gargiu/Documents/ELK/logstash-6.0.0/vendor/jruby/lib/ruby/stdlib/forwardable.rb:189
register_plugin at /Users/gargiu/Documents/ELK/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:388
block in register_plugins at /Users/gargiu/Documents/ELK/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:399
each at org/jruby/RubyArray.java:1734
register_plugins at /Users/gargiu/Documents/ELK/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:399
maybe_setup_out_plugins at /Users/gargiu/Documents/ELK/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:801
start_workers at /Users/gargiu/Documents/ELK/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:409
run at /Users/gargiu/Documents/ELK/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:333
block in start at /Users/gargiu/Documents/ELK/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:293

7

Show us the exact ruby filter that doesn't work.

Also, you could of course write a loop that makes a list out of all sensorXXX fields and thereby avoid the repetition.

8

In fact i need to make a loop, in final i have 110 sensor. This is ruby filter:

ruby {
    code => 'event.set("data_sensor", [{"sensor" => "001", "value" => event.get("sensor001")}, {"sensor" => "002", "value" => event.get("sensor002")}, {"sensor" => "003", "value" => event.get("sensor003")}, {"sensor" => "004", "value" => event.get("sensor004")}, {"sensor" => "005", "value" => event.get("sensor005")}, {"sensor" => "006", "value" => event.get("sensor006")}, {"sensor" => "007", "value" => event.get("sensor007")}, {"sensor" => "008", "value" => event.get("sensor008")}])'
    }
  mutate {
    remove_field => ["sensor001", "sensor002", "sensor003", "sensor004", "sensor005", "sensor006", "sensor007", "sensor008", "sensor009"]
  }

When i write another object sand me the upper error

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.