Modify json field

manasapp · February 19, 2019, 8:11am

Hi Everyone,

I need to add one timestamp value to JSON. i share u code n output

input
    {
      stdin{}
    }
    filter{
      if ([message] =~ "{\"index") {
          drop {}
      }

      json {
        source => "message"
        target => "parsedJson"
        }
      }
     if ([parsedJson][frame]) {

    mutate {
          add_field => {
            
    		"{[parsedJson][frame][timestamp]}" => "%{[parsedJson][timestamp]}"
          }
        }
        mutate {
          add_field => {
            "frame" => '%{[parsedJson][frame]}'
          }
        }
      }
    }
    output{
    stdout{}
    }

I'm trying to add timestamp as value to frame JSON i will share sample json output which i got. this is the frame json

{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data"}

but i want output as

{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data","timestamp":"2019-02-19T08:07:28.847Z"}

pls help me solve this issue.

A_B · February 19, 2019, 8:45am

Hi @manasapp,

what is the exact problem you are having? Your Logstash config doesn't really match up with your sample JSON.

E.g. [parsedJson][layers][frame][timestamp] is not found in your sample JSON.

I think some of the syntax in your Logstash config is a bit off as well. I tried to fix it below.

input {
  stdin {
    codec => "json"
  }
}
filter {
  if ([message] =~ "{\"index") {
    drop {}
  }

  json {
    source => "message"
    target => "parsedJson"
  }

  if [parsedJson][layers][frame] {
    mutate {
      add_field => {
        "[parsedJson][layers][frame][timestamp]" => "%{[parsedJson][timestamp]}"
          }
        }
    mutate {
      add_field => {
        "frame" => "%{[parsedJson][layers][frame]}"
      }
    }
  }
}
output{
  stdout { codec => rubydebug }
}

manasapp · February 19, 2019, 9:37am

yeah Mr. A_B I'm trying to add that timestamp as value into the frame JSON. still i didn't get any timestamp after adding.help me to change frame json

A_B · February 19, 2019, 10:06am

I took your sample JSON and used this Logstash config

input {
  stdin {
    codec => "json"
  }
}
filter {

  date {
    match => [ "frame_frame_time_epoch", "UNIX" ]
  }

}
output{
  stdout { codec => rubydebug }
}

and the result is

{
           "frame_frame_time_delta_displayed" => "0.000509962",
                                 "@timestamp" => 2019-02-19T07:41:15.603Z,
                     "frame_frame_encap_type" => "1",
                   "frame_frame_offset_shift" => "0.000000000",
                        "frame_frame_cap_len" => "1514",
                         "frame_frame_number" => "228",
                     "frame_frame_time_epoch" => "1550562075.603254333",
                            "frame_frame_len" => "1514",
                                   "@version" => "1",
                                       "host" => "mg1500.log0.mad1.bwcom.net",
                  "frame_frame_time_relative" => "12.076067648",
                     "frame_frame_time_delta" => "0.000509962",
                        "frame_frame_ignored" => "0",
                      "frame_frame_protocols" => "eth:ethertype:ip:udp:data",
                         "frame_frame_marked" => "0",
    "frame_interface_id_frame_interface_name" => "eth0",
                   "frame_frame_interface_id" => "0"
}

In the result frame_frame_time_epoch is parsed as date and populated the filed @timestamp. You can change the target field to what you want.

manasapp · February 19, 2019, 10:30am

how u got output Mr. A_B. i did but no change in the outside there is no timestamp value

A_B · February 19, 2019, 10:41am

What I did was

download Logstash, if you don't have it
Put my above config in a file e.g. test.conf
Start Logstash with /path/to/logstash -f /path/to/test.conf
Wait for Logstash to start
Paste in sample JSON

manasapp · February 19, 2019, 11:12am

no i followed the procedure to solve this issue but i didn't get output as u shown

A_B · February 19, 2019, 11:28am

Which version of Logstash do you use?

Copy/paste error?

The date filter should be included by default and I do not think Logstash would start without it if it is referenced in the config but you can check installed plugins with

# logstash-6.3.1/bin/logstash-plugin list filter-*
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-jdbc_static
logstash-filter-jdbc_streaming
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-xml

A_B · February 19, 2019, 11:32am

And here is the complete console output from my test machine (running Debian 9)

~/tmp # cat ls-elastic.conf
input {
  stdin {
    codec => "json"
  }
}
filter {

  date {
    match => [ "frame_frame_time_epoch", "UNIX" ]
  }

}
output{
  stdout { codec => rubydebug }
}
~/tmp # logstash-6.3.1/bin/logstash -f ls-elastic.conf
Sending Logstash's logs to /root/tmp/logstash-6.3.1/logs which is now configured via log4j2.properties
[2019-02-19T11:30:14,336][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-19T11:30:14,455][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.1"}
[2019-02-19T11:30:14,874][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-19T11:30:14,902][INFO ][logstash.inputs.stdin    ] Automatically switching from json to json_lines codec {:plugin=>"stdin"}
[2019-02-19T11:30:14,918][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x42aa582f@/root/tmp/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:245 sleep>"}
The stdin plugin is now waiting for input:
[2019-02-19T11:30:14,931][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-19T11:30:14,972][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data"}
{
                   "frame_frame_interface_id" => "0",
                        "frame_frame_ignored" => "0",
                     "frame_frame_encap_type" => "1",
    "frame_interface_id_frame_interface_name" => "eth0",
                        "frame_frame_cap_len" => "1514",
                      "frame_frame_protocols" => "eth:ethertype:ip:udp:data",
                                       "host" => "mg1500.log0.mad1.bwcom.net",
                     "frame_frame_time_epoch" => "1550562075.603254333",
                  "frame_frame_time_relative" => "12.076067648",
                                   "@version" => "1",
           "frame_frame_time_delta_displayed" => "0.000509962",
                            "frame_frame_len" => "1514",
                     "frame_frame_time_delta" => "0.000509962",
                         "frame_frame_marked" => "0",
                   "frame_frame_offset_shift" => "0.000000000",
                                 "@timestamp" => 2019-02-19T07:41:15.603Z,
                         "frame_frame_number" => "228"
}

After Logstash has started I paste in the sample JSON and hit Enter.

manasapp · February 19, 2019, 11:59am

no no actually not like this i want to like this output as file
file {
codec => line { format => "%{frame}" }
path => "E:/tsharkfiletest.txt"
}
do frame variable contains this @timestamp no rgt?

manasapp · February 19, 2019, 12:00pm

logstash version- 6.2.2

A_B · February 19, 2019, 12:56pm

I take it you mean to get %{frame} from??

    mutate {
      add_field => {
        "frame" => "%{[parsedJson][layers][frame]}"
      }
    }

You have lost me... You have nothing like [parsedJson][layers][frame] in the sample JSON, which means frame will not be populated which means the Logstash output config doesn't know what to set the line codec format to...

system · March 19, 2019, 12:56pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Add field from JSON / logstash filter Logstash	4	8992	January 15, 2017
How can i add a field to the original message (json) that was sent to logstash? Logstash	15	4006	May 31, 2018
How logstash uses add_field to add a JSON array Logstash	3	856	September 6, 2021
Filter mutate add_field : select a value in an inner field Logstash	2	534	April 16, 2018
Add field from JSON -logstash filter Logstash	2	4675	June 2, 2017

Modify json field

Related topics