How to get part of a field of parsed json message in logstash

Elastic Stack Logstash
1

I am trying to get just the date part from @report_timestamp field of a parsed json message in logstash configuration so that I can name the output log file name based on date, say request_2017_08_01.log
my message looks like this:
"{"udf": 1, "report_timestamp": "2017-11-29T17:58:28.967388", "service_engine": "test", "vcpu_id": 1, "log_id": 911550, "client_ip": "93.20.1.24", "response_code": 200}"
my logstash conf looks like this:
filter {
json {
source => "message"
target => "parsedjson"
}
mutate {
add_field => {"event_generated_on" => "%{[parsedjson][report_timestamp]}"}

}

Expecting date part of the report_timestamp field value assigned to "event_generated_on".

Thanks

2

Show an example event produced by Logstash. Use a stdout { codec => rubydebug } to get a raw dump of it.

3

Hi Magnus,
Output is something like this; changed message due to security reasons.
Thanks
{

        "@timestamp" => 2017-11-29T17:58:29.495Z,
"event_generated_on" => "2017-11-29T17:58:28.967388",
        "parsedjson" => {
               "server_response_code" => 200,
     "server_response_time_last_byte" => 38,
                            "vs_name" => "test_443",
                            "message" => "{\"udf\": 1, \"report_timestamp\": \"2017-11-29T17:58:28.967388\", \"service_engine\": \"test\", \"vcpu_id\": 1, \"log_id\": 911550, \"client_ip\": \"93.20.1.24\", \"response_code\": 200}",
          "server_ssl_session_reused" => 1,
                   "report_timestamp" => "2017-11-29T17:58:28.967388",
           "server_connection_reused" => 1,
                      "request_state" => "HTTP_REQUEST_STATE_SEND_TO_CLIENT",
                         "@timestamp" => "2017-11-29T17:58:29.451Z",
       "headers_received_from_server" => "Date: Wed, 29 Nov 2017 17:58:49 GMT  X-Powered-By: Servlet/3.0  Set-Cookie: JSESSIONID_IC_BMIX_T=0000fwSnDd2:ICCommon01; Path=/; Domain=.ibm.com; HttpOnly  Expires: Thu, 01 Dec 1994 16:00:00 GMT  Cache-Control: no-cache=set-cookie, set-cookie2  Vary: User-Agent,Accept-Encoding  Transfer-Encoding: chunked  Content-Type: text/html;charset=ISO-8859-1  Content-Language: en-US  "
},
          "@version" => "1",
              "host" => "9.207.131.217",
           "message" => "same as above, changed due to security reasons"

}

4

You can use a grok filter to extract the date, but if your end goal is to name your output file e.g. request_2017_08_01.log just use the %{+YYYY_MM_dd} notation to get the timestamp from @timestamp formatted in yyyy_mm_dd format.

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf

5

Not sure if @timestamp will have values same as that of report_timestamp (What if I put a past date ) which is a part of json message field. please suggest.

6

Well, I don't know your data so I can't tell if report_timestamp should be copied to @timestamp but it probably should be if report_timestamp is the "main" timestamp of the event.

If not you should, as I said, use a grok filter to extract the date part from report_timestamp.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.