Modify json field

Hi Everyone,

I need to add one timestamp value to JSON. i share u code n output

input
    {
      stdin{}
    }
    filter{
      if ([message] =~ "{\"index") {
          drop {}
      }

      json {
        source => "message"
        target => "parsedJson"
        }
      }
     if ([parsedJson][frame]) {

    mutate {
          add_field => {
            
    		"{[parsedJson][frame][timestamp]}" => "%{[parsedJson][timestamp]}"
          }
        }
        mutate {
          add_field => {
            "frame" => '%{[parsedJson][frame]}'
          }
        }
      }
    }
    output{
    stdout{}
    }

I'm trying to add timestamp as value to frame JSON i will share sample json output which i got. this is the frame json

{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data"}

but i want output as

{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data","timestamp":"2019-02-19T08:07:28.847Z"}

pls help me solve this issue.:confused:

Hi @manasapp,

what is the exact problem you are having? Your Logstash config doesn't really match up with your sample JSON.

E.g. [parsedJson][layers][frame][timestamp] is not found in your sample JSON.

I think some of the syntax in your Logstash config is a bit off as well. I tried to fix it below.

input {
  stdin {
    codec => "json"
  }
}
filter {
  if ([message] =~ "{\"index") {
    drop {}
  }

  json {
    source => "message"
    target => "parsedJson"
  }

  if [parsedJson][layers][frame] {
    mutate {
      add_field => {
        "[parsedJson][layers][frame][timestamp]" => "%{[parsedJson][timestamp]}"
          }
        }
    mutate {
      add_field => {
        "frame" => "%{[parsedJson][layers][frame]}"
      }
    }
  }
}
output{
  stdout { codec => rubydebug }
}

yeah Mr. A_B I'm trying to add that timestamp as value into the frame JSON. still i didn't get any timestamp after adding.help me to change frame json

I took your sample JSON and used this Logstash config

input {
  stdin {
    codec => "json"
  }
}
filter {

  date {
    match => [ "frame_frame_time_epoch", "UNIX" ]
  }

}
output{
  stdout { codec => rubydebug }
}

and the result is

{
           "frame_frame_time_delta_displayed" => "0.000509962",
                                 "@timestamp" => 2019-02-19T07:41:15.603Z,
                     "frame_frame_encap_type" => "1",
                   "frame_frame_offset_shift" => "0.000000000",
                        "frame_frame_cap_len" => "1514",
                         "frame_frame_number" => "228",
                     "frame_frame_time_epoch" => "1550562075.603254333",
                            "frame_frame_len" => "1514",
                                   "@version" => "1",
                                       "host" => "mg1500.log0.mad1.bwcom.net",
                  "frame_frame_time_relative" => "12.076067648",
                     "frame_frame_time_delta" => "0.000509962",
                        "frame_frame_ignored" => "0",
                      "frame_frame_protocols" => "eth:ethertype:ip:udp:data",
                         "frame_frame_marked" => "0",
    "frame_interface_id_frame_interface_name" => "eth0",
                   "frame_frame_interface_id" => "0"
}

In the result frame_frame_time_epoch is parsed as date and populated the filed @timestamp. You can change the target field to what you want.

how u got output Mr. A_B. i did but no change in the outside there is no timestamp value

What I did was

  1. download Logstash, if you don't have it
  2. Put my above config in a file e.g. test.conf
  3. Start Logstash with /path/to/logstash -f /path/to/test.conf
  4. Wait for Logstash to start
  5. Paste in sample JSON

no i followed the procedure to solve this issue but i didn't get output as u shown

Which version of Logstash do you use?

Copy/paste error?

The date filter should be included by default and I do not think Logstash would start without it if it is referenced in the config but you can check installed plugins with

# logstash-6.3.1/bin/logstash-plugin list filter-*
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-jdbc_static
logstash-filter-jdbc_streaming
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-xml

And here is the complete console output from my test machine (running Debian 9)

~/tmp # cat ls-elastic.conf
input {
  stdin {
    codec => "json"
  }
}
filter {

  date {
    match => [ "frame_frame_time_epoch", "UNIX" ]
  }

}
output{
  stdout { codec => rubydebug }
}
~/tmp # logstash-6.3.1/bin/logstash -f ls-elastic.conf
Sending Logstash's logs to /root/tmp/logstash-6.3.1/logs which is now configured via log4j2.properties
[2019-02-19T11:30:14,336][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-02-19T11:30:14,455][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.1"}
[2019-02-19T11:30:14,874][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-19T11:30:14,902][INFO ][logstash.inputs.stdin    ] Automatically switching from json to json_lines codec {:plugin=>"stdin"}
[2019-02-19T11:30:14,918][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x42aa582f@/root/tmp/logstash-6.3.1/logstash-core/lib/logstash/pipeline.rb:245 sleep>"}
The stdin plugin is now waiting for input:
[2019-02-19T11:30:14,931][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-19T11:30:14,972][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}
{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"eth0","frame_frame_encap_type":"1","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1550562075.603254333","frame_frame_time_delta":"0.000509962","frame_frame_time_delta_displayed":"0.000509962","frame_frame_time_relative":"12.076067648","frame_frame_number":"228","frame_frame_len":"1514","frame_frame_cap_len":"1514","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:udp:data"}
{
                   "frame_frame_interface_id" => "0",
                        "frame_frame_ignored" => "0",
                     "frame_frame_encap_type" => "1",
    "frame_interface_id_frame_interface_name" => "eth0",
                        "frame_frame_cap_len" => "1514",
                      "frame_frame_protocols" => "eth:ethertype:ip:udp:data",
                                       "host" => "mg1500.log0.mad1.bwcom.net",
                     "frame_frame_time_epoch" => "1550562075.603254333",
                  "frame_frame_time_relative" => "12.076067648",
                                   "@version" => "1",
           "frame_frame_time_delta_displayed" => "0.000509962",
                            "frame_frame_len" => "1514",
                     "frame_frame_time_delta" => "0.000509962",
                         "frame_frame_marked" => "0",
                   "frame_frame_offset_shift" => "0.000000000",
                                 "@timestamp" => 2019-02-19T07:41:15.603Z,
                         "frame_frame_number" => "228"
}

After Logstash has started I paste in the sample JSON and hit Enter.

no no actually not like this i want to like this output as file
file {
codec => line { format => "%{frame}" }
path => "E:/tsharkfiletest.txt"
}
do frame variable contains this @timestamp no rgt?

logstash version- 6.2.2

I take it you mean to get %{frame} from??

    mutate {
      add_field => {
        "frame" => "%{[parsedJson][layers][frame]}"
      }
    }

You have lost me... You have nothing like [parsedJson][layers][frame] in the sample JSON, which means frame will not be populated which means the Logstash output config doesn't know what to set the line codec format to...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.