Modify json field

manasapp · February 19, 2019, 8:11am

Hi Everyone,

I need to add one timestamp value to JSON. i share u code n output

input
    {
      stdin{}
    }
    filter{
      if ([message] =~ "{\"index") {
          drop {}
      }

      json {
        source => "message"
        target => "parsedJson"
        }
      }
     if ([parsedJson][frame]) {

    mutate {
          add_field => {
            
    		"{[parsedJson][frame][timestamp]}" => "%{[parsedJson][timestamp]}"
          }
        }
        mutate {
          add_field => {
            "frame" => '%{[parsedJson][frame]}'
          }
        }
      }
    }
    output{
    stdout{}
    }

I'm trying to add timestamp as value to frame JSON i will share sample json output which i got. this is the frame json

{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data"}

but i want output as

{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data","timestamp":"2019-02-19T08:07:28.847Z"}

pls help me solve this issue.

A_B · February 19, 2019, 8:45am

Hi @manasapp,

what is the exact problem you are having? Your Logstash config doesn't really match up with your sample JSON.

E.g. [parsedJson][layers][frame][timestamp] is not found in your sample JSON.

I think some of the syntax in your Logstash config is a bit off as well. I tried to fix it below.

input {
  stdin {
    codec => "json"
  }
}
filter {
  if ([message] =~ "{\"index") {
    drop {}
  }

  json {
    source => "message"
    target => "parsedJson"
  }

  if [parsedJson][layers][frame] {
    mutate {
      add_field => {
        "[parsedJson][layers][frame][timestamp]" => "%{[parsedJson][timestamp]}"
          }
        }
    mutate {
      add_field => {
        "frame" => "%{[parsedJson][layers][frame]}"
      }
    }
  }
}
output{
  stdout { codec => rubydebug }
}

manasapp · February 19, 2019, 9:37am

yeah Mr. A_B I'm trying to add that timestamp as value into the frame JSON. still i didn't get any timestamp after adding.help me to change frame json

A_B · February 19, 2019, 10:06am

I took your sample JSON and used this Logstash config

input {
  stdin {
    codec => "json"
  }
}
filter {

  date {
    match => [ "frame_frame_time_epoch", "UNIX" ]
  }

}
output{
  stdout { codec => rubydebug }
}

and the result is

{
           "frame_frame_time_delta_displayed" => "0.000509962",
                                 "@timestamp" => 2019-02-19T07:41:15.603Z,
                     "frame_frame_encap_type" => "1",
                   "frame_frame_offset_shift" => "0.000000000",
                        "frame_frame_cap_len" => "1514",
                         "frame_frame_number" => "228",
                     "frame_frame_time_epoch" => "1550562075.603254333",
                            "frame_frame_len" => "1514",
                                   "@version" => "1",
                                       "host" => "mg1500.log0.mad1.bwcom.net",
                  "frame_frame_time_relative" => "12.076067648",
                     "frame_frame_time_delta" => "0.000509962",
                        "frame_frame_ignored" => "0",
                      "frame_frame_protocols" => "eth:ethertype:ip:udp:data",
                         "frame_frame_marked" => "0",
    "frame_interface_id_frame_interface_name" => "eth0",
                   "frame_frame_interface_id" => "0"
}

In the result frame_frame_time_epoch is parsed as date and populated the filed @timestamp. You can change the target field to what you want.

manasapp · February 19, 2019, 10:30am

how u got output Mr. A_B. i did but no change in the outside there is no timestamp value

A_B · February 19, 2019, 10:41am

What I did was

download Logstash, if you don't have it
Put my above config in a file e.g. test.conf
Start Logstash with /path/to/logstash -f /path/to/test.conf
Wait for Logstash to start
Paste in sample JSON

manasapp · February 19, 2019, 11:12am

no i followed the procedure to solve this issue but i didn't get output as u shown

A_B · February 19, 2019, 11:28am

Which version of Logstash do you use?

Copy/paste error?

The date filter should be included by default and I do not think Logstash would start without it if it is referenced in the config but you can check installed plugins with

# logstash-6.3.1/bin/logstash-plugin list filter-*
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-jdbc_static
logstash-filter-jdbc_streaming
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-xml

A_B · February 19, 2019, 11:32am

And here is the complete console output from my test machine (running Debian 9)

~/tmp # cat ls-elastic.conf
input {
  stdin {
    codec => "json"
  }
}
filter {

  date {
    match => [ "frame_frame_time_epoch", "UNIX" ]
  }

}
output{
  stdout { codec => rubydebug }
}
~/tmp # logstash-6.3.1/bin/logstash -f ls-elastic.conf
Sending Logstash's logs to /root/tmp/logstash-6.3.1/logs which is now configured via log4j2.properties
[2019-02-19T11:30:14,336][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-19T11:30:14,455][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.1"}
[2019-02-19T11:30:14,874][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-19T11:30:14,902][INFO ][logstash.inputs.stdin    ] Automatically switching from json to json_lines codec {:plugin=>"stdin"}
[2019-02-19T11:30:14,918][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x42aa582f@/root/tmp/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:245 sleep>"}
The stdin plugin is now waiting for input:
[2019-02-19T11:30:14,931][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-19T11:30:14,972][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data"}
{
                   "frame_frame_interface_id" => "0",
                        "frame_frame_ignored" => "0",
                     "frame_frame_encap_type" => "1",
    "frame_interface_id_frame_interface_name" => "eth0",
                        "frame_frame_cap_len" => "1514",
                      "frame_frame_protocols" => "eth:ethertype:ip:udp:data",
                                       "host" => "mg1500.log0.mad1.bwcom.net",
                     "frame_frame_time_epoch" => "1550562075.603254333",
                  "frame_frame_time_relative" => "12.076067648",
                                   "@version" => "1",
           "frame_frame_time_delta_displayed" => "0.000509962",
                            "frame_frame_len" => "1514",
                     "frame_frame_time_delta" => "0.000509962",
                         "frame_frame_marked" => "0",
                   "frame_frame_offset_shift" => "0.000000000",
                                 "@timestamp" => 2019-02-19T07:41:15.603Z,
                         "frame_frame_number" => "228"
}

After Logstash has started I paste in the sample JSON and hit Enter.

manasapp · February 19, 2019, 11:59am

no no actually not like this i want to like this output as file
file {
codec => line { format => "%{frame}" }
path => "E:/tsharkfiletest.txt"
}
do frame variable contains this @timestamp no rgt?

manasapp · February 19, 2019, 12:00pm

logstash version- 6.2.2

A_B · February 19, 2019, 12:56pm

I take it you mean to get %{frame} from??

    mutate {
      add_field => {
        "frame" => "%{[parsedJson][layers][frame]}"
      }
    }

You have lost me... You have nothing like [parsedJson][layers][frame] in the sample JSON, which means frame will not be populated which means the Logstash output config doesn't know what to set the line codec format to...

Topic		Replies	Views
How can i add a field to the original message (json) that was sent to logstash? Logstash	15	4529	May 31, 2018
Logstash Does not extract timestamp from JSON Logstash	6	5058	July 6, 2017
Json logs with a field named "time", won´t be shown in kibana Logstash	1	381	December 27, 2017
How to have provide value for @timestamp when using JSON? Logstash	4	548	March 8, 2018
Epoch Time Via Logstash Logstash	4	76	December 16, 2024

Modify json field

Related topics