Need help on creating Logstash_FILTER

Elastic Stack Logstash
1

log:{​​​​​​​​"time_stamp":"2021-06-11T04:58:32.22154414Z","labels":{​​​​​​​​"vm_index":0,"session_docker_id":"12d333","tenant_id":"21a64d4","user_id":"bob","session_uuid":"2a30"}​​​​​​​​,"event":{​​​​​​​​"kind":"Metric","category":"Session","type":"abcUser","abc":{​​​​​​​​"session_type":"High"}​​​​​​​​}​​​​​​​​}​​​​​​​​

I am Trying to create a logstash filter for the above code. Please let me know how to CODE the LOGSTASH_FILTER .

2

What do you want the filter to do?

3

Hi Badger, the above mentioned is the raw data.. I need the logstash filter to structure the data and indexed in ES For example the raw data should query

Labels.vm_index : 0
Session_id: 12345
Labels.tenant_id : foobar
Event: metric and so on

4

Use a json filter to parse it.

5

I am aware that we can use JSON filters, i wanted to know what will be the format of the filter, what logstash filters to be written

6

This is my logline "{"time_stamp":"2021-06-08T05:00:27.351201664Z","labels":{"vmm_index":0,"session_docker_id":"bf4c02a8ebfa0ce40d626f2b8280074431df2ad95c6f02360af38453b88b8135"},"event":{"kind":"Metric","category":"Session","type":"mysessioncontainer"}}

I need a Grok/JSON filter to be written, please assist

7

@YuWatanabe @jsvd @Badger

Hello Guys, Could you please help me with the above query

8

There is an example in the documentation.

9

@Badger Can you share me the draft of the filter from your end for the log i mentioned

10

@sumit_n

Sorry for late reply . I did not realize your mention until now.

Using json filter should parse your json even.

Assuming you input event is ,

{"time_stamp":"2021-06-08T05:00:27.351201664Z","labels":{"vmm_index":0,"session_docker_id":"bf4c02a8ebfa0ce40d626f2b8280074431df2ad95c6f02360af38453b88b8135"},"event":{"kind":"Metric","category":"Session","type":"mysessioncontainer"}}

And if you use pipeline like below ,

input {
  stdin {}
}

filter {
  json {
    source => "message"
  }
}

output {
  stdout {
    codec => rubydebug { metadata => true}
  }
}

You would get output something like,

[ywatanabe@laptop-archlinux logstash-7.13.4]$ echo '{"time_stamp":"2021-06-08T05:00:27.351201664Z","labels":{"vmm_index":0,"session_docker_id":"bf4c02a8ebfa0ce40d626f2b8280074431df2ad95c6f02360af38453b88b8135"},"event":{"kind":"Metric","category":"Session","type":"mysessioncontainer"}}' | bin/logstash -f config/json.conf 
Using bundled JDK: /home/ywatanabe/Downloads/logstash-7.13.4/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /home/ywatanabe/Downloads/logstash-7.13.4/logs which is now configured via log4j2.properties
[2021-08-10T21:26:10,792][INFO ][logstash.runner          ] Log4j configuration path used is: /home/ywatanabe/Downloads/logstash-7.13.4/config/log4j2.properties
[2021-08-10T21:26:10,798][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.4", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-08-10T21:26:11,011][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-10T21:26:11,507][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-08-10T21:26:11,793][INFO ][org.reflections.Reflections] Reflections took 33 ms to scan 1 urls, producing 24 keys and 48 values 
[2021-08-10T21:26:12,582][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/ywatanabe/Downloads/logstash-7.13.4/config/json.conf"], :thread=>"#<Thread:0x543111da run>"}
[2021-08-10T21:26:13,139][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.55}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/home/ywatanabe/Downloads/logstash-7.13.4/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2021-08-10T21:26:13,169][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-08-10T21:26:13,227][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "@timestamp" => 2021-08-10T12:26:13.209Z,
    "time_stamp" => "2021-06-08T05:00:27.351201664Z",
          "host" => "laptop-archlinux",
      "@version" => "1",
         "event" => {
            "kind" => "Metric",
            "type" => "mysessioncontainer",
        "category" => "Session"
    },
       "message" => "{\"time_stamp\":\"2021-06-08T05:00:27.351201664Z\",\"labels\":{\"vmm_index\":0,\"session_docker_id\":\"bf4c02a8ebfa0ce40d626f2b8280074431df2ad95c6f02360af38453b88b8135\"},\"event\":{\"kind\":\"Metric\",\"category\":\"Session\",\"type\":\"mysessioncontainer\"}}",
        "labels" => {
                "vmm_index" => 0,
        "session_docker_id" => "bf4c02a8ebfa0ce40d626f2b8280074431df2ad95c6f02360af38453b88b8135"
    }
}

Thanks.

11

Also using your first log , it would end up something like this.

[ywatanabe@laptop-archlinux logstash-7.13.4]$ echo '{"time_stamp":"2021-06-11T04:58:32.22154414Z","labels":{"vm_index":"0","session_docker_id":"12d333","tenant_id":"21a64d4","user_id":"bob","session_uuid":"2a30"},"event":{"kind":"Metric","category":"Session","type":"abcUser","abc":{"session_type":"High"}}}' | bin/logstash -f config/json.conf 
Using bundled JDK: /home/ywatanabe/Downloads/logstash-7.13.4/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /home/ywatanabe/Downloads/logstash-7.13.4/logs which is now configured via log4j2.properties
[2021-08-10T21:49:43,280][INFO ][logstash.runner          ] Log4j configuration path used is: /home/ywatanabe/Downloads/logstash-7.13.4/config/log4j2.properties
[2021-08-10T21:49:43,287][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.13.4", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[2021-08-10T21:49:43,503][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-10T21:49:43,989][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-08-10T21:49:44,248][INFO ][org.reflections.Reflections] Reflections took 26 ms to scan 1 urls, producing 24 keys and 48 values 
[2021-08-10T21:49:44,971][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/home/ywatanabe/Downloads/logstash-7.13.4/config/json.conf"], :thread=>"#<Thread:0x3dad353c run>"}
[2021-08-10T21:49:45,530][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.56}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/home/ywatanabe/Downloads/logstash-7.13.4/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2021-08-10T21:49:45,566][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-08-10T21:49:45,620][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "@timestamp" => 2021-08-10T12:49:45.609Z,
    "time_stamp" => "2021-06-11T04:58:32.22154414Z",
      "@version" => "1",
          "host" => "laptop-archlinux",
       "message" => "{\"time_stamp\":\"2021-06-11T04:58:32.22154414Z\",\"labels\":{\"vm_index\":\"0\",\"session_docker_id\":\"12d333\",\"tenant_id\":\"21a64d4\",\"user_id\":\"bob\",\"session_uuid\":\"2a30\"},\"event\":{\"kind\":\"Metric\",\"category\":\"Session\",\"type\":\"abcUser\",\"abc\":{\"session_type\":\"High\"}}}",
         "event" => {
             "abc" => {
            "session_type" => "High"
        },
            "kind" => "Metric",
            "type" => "abcUser",
        "category" => "Session"
    },
        "labels" => {
                "tenant_id" => "21a64d4",
                 "vm_index" => "0",
                  "user_id" => "bob",
        "session_docker_id" => "12d333",
             "session_uuid" => "2a30"
    }
}
[2021-08-10T21:49:45,886][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2021-08-10T21:49:46,203][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2021-08-10T21:49:46,280][INFO ][logstash.runner          ] Logstash shut down.
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.