Simple question about timestamp in logs

[Mark_S (Baldrick)]

When I create an index and use Logstash to send logfiles, I use the option start_position => "beginning".

However when viewing "discover" mode in kibana, I only see the logs after the index was created. Shouldn't there be also older logs ?

When creating the index pattern I chose the @timestamp - I thought that was the way to use the log timestamp...

[stephenb (Stephen Brown)]

Hi @Mark_S

Excellent question.

If you do not parse the log... Then Logstash will set the @timestamp to when it read the log.

In order to use the timestamp that that is contained within the message of the log you will need to do a little parsing if the log.

Provide a couple sample log lines and your logstash pipeline .conf file and perhaps we can help.

[Mark_S (Baldrick)]

All right! I do some parsing, mainly adding some tags and concatenating java exceptions.
Here is the conf file

input {
       file {
          path => ["/opt/applications/logs/*.log"]
          start_position => "beginning"
          codec => multiline {
          # Grok pattern names are valid! :)
                   pattern => "^%{TIMESTAMP_ISO8601} "
                   negate => true
                   what => "previous"
                   }
           }
}

filter {

if "app-testserver1" in [path] {
      mutate { add_tag => "app-testserver1" }
   }
if "app-testserver2" in [path] {
      mutate { add_tag => "app-testserver2" }
   }



}


output {
  elasticsearch {
    hosts => ["https://elastic-testserv:9200"]
    index => "applogs"
    user => "elastic"
    password => "XXXXXXXXXXXXXX"
    cacert => "/etc/logstash/es-ca.pem"
    ssl => true
    }

}

The log format is as follows:

2023-03-12 10:48:14.003  WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter   : processing_end, processing_end some more-characters-here

[stephenb (Stephen Brown)]

@Mark_S You will need to do some additional parsing and be sure to add the correct timezone ... The timezone will default to your local timezone. If it's something different you need to set it.

here is a sample

input {
	file {
		path => "/Users/sbrown/workspace/sample-data/discuss/test-logstash-timestamp.log"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => multiline {
			# Grok pattern names are valid! :)
			pattern => "^%{TIMESTAMP_ISO8601} "
			negate => true
			what => "previous"
		}
	}
}


filter{
 grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp}  %{DATA:message_detail}" }
      }

  date {
    match => [ "log_timestamp", "ISO8601" ]
    # Set Time Zone since log does not have it, otherwise it will use local time zone
    timezone => "PST8PDT"
  }
}

output {
  stdout { codec => rubydebug }
}

Sample

{
               "log" => {
        "file" => {
            "path" => "/Users/sbrown/workspace/sample-data/discuss/test-logstash-timestamp.log"
        }
    },
        "@timestamp" => 2023-03-12T17:48:14.003Z,
     "log_timestamp" => "2023-03-12 10:48:14.003",
              "host" => {
        "name" => "hyperion"
    },
             "event" => {
        "original" => "2023-03-12 10:48:14.003  WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter   : processing_end, processing_end some more-characters-here"
    },
    "message_detail" => "WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter   : processing_end, processing_end some more-characters-here",
           "message" => "2023-03-12 10:48:14.003  WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter   : processing_end, processing_end some more-characters-here",
          "@version" => "1"
}