# Simple question about timestamp in logs

**URL:** <https://discuss.elastic.co/t/simple-question-about-timestamp-in-logs/327485>\
**Category:** Kibana\
**Created:** [March 11, 2023, 3:07pm UTC](https://discuss.elastic.co/t/simple-question-about-timestamp-in-logs/327485 "2023-03-11T15:07:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 11, 2023, 3:07pm UTC](https://discuss.elastic.co/t/simple-question-about-timestamp-in-logs/327485/1 "2023-03-11T15:07:10Z")

</div>

When I create an index and use Logstash to send logfiles, I use the option start\_position =\> "beginning".

However when viewing "discover" mode in kibana, I only see the logs after the index was created. Shouldn't there be also older logs ?

When creating the index pattern I chose the @timestamp - I thought that was the way to use the log timestamp...

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 11, 2023, 3:23pm UTC](https://discuss.elastic.co/t/simple-question-about-timestamp-in-logs/327485/2 "2023-03-11T15:23:04Z")

</div>

Hi @Mark_S

Excellent question.

If you do not parse the log... Then Logstash will set the `@timestamp` to when it read the log.

In order to use the timestamp that that is contained within the message of the log you will need to do a little parsing if the log.

Provide a couple sample log lines and your logstash pipeline .conf file and perhaps we can help.

---

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 12, 2023, 9:00am UTC](https://discuss.elastic.co/t/simple-question-about-timestamp-in-logs/327485/3 "2023-03-12T09:00:12Z")

</div>

All right! I do some parsing, mainly adding some tags and concatenating java exceptions.  
Here is the conf file

```auto
input {
       file {
          path => ["/opt/applications/logs/*.log"]
          start_position => "beginning"
          codec => multiline {
          # Grok pattern names are valid! :)
                   pattern => "^%{TIMESTAMP_ISO8601} "
                   negate => true
                   what => "previous"
                   }
           }
}

filter {

if "app-testserver1" in [path] {
      mutate { add_tag => "app-testserver1" }
   }
if "app-testserver2" in [path] {
      mutate { add_tag => "app-testserver2" }
   }

}

output {
  elasticsearch {
    hosts => ["https://elastic-testserv:9200"]
    index => "applogs"
    user => "elastic"
    password => "XXXXXXXXXXXXXX"
    cacert => "/etc/logstash/es-ca.pem"
    ssl => true
    }

}

```

The log format is as follows:

```auto
2023-03-12 10:48:14.003 WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end some more-characters-here

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 12, 2023, 4:31pm UTC](https://discuss.elastic.co/t/simple-question-about-timestamp-in-logs/327485/4 "2023-03-12T16:31:42Z")

</div>

@Mark_S You will need to do some additional parsing and be sure to add the correct timezone ... The timezone will default to your local timezone. If it's something different you need to set it.

here is a sample

```auto
input {
	file {
		path => "/Users/sbrown/workspace/sample-data/discuss/test-logstash-timestamp.log"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => multiline {
			# Grok pattern names are valid! :)
			pattern => "^%{TIMESTAMP_ISO8601} "
			negate => true
			what => "previous"
		}
	}
}

filter{
 grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{DATA:message_detail}" }
      }

  date {
    match => ["log_timestamp", "ISO8601"]
    # Set Time Zone since log does not have it, otherwise it will use local time zone
    timezone => "PST8PDT"
  }
}

output {
  stdout { codec => rubydebug }
}

```

Sample

```auto
{
               "log" => {
        "file" => {
            "path" => "/Users/sbrown/workspace/sample-data/discuss/test-logstash-timestamp.log"
        }
    },
        "@timestamp" => 2023-03-12T17:48:14.003Z,
     "log_timestamp" => "2023-03-12 10:48:14.003",
              "host" => {
        "name" => "hyperion"
    },
             "event" => {
        "original" => "2023-03-12 10:48:14.003 WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end some more-characters-here"
    },
    "message_detail" => "WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end some more-characters-here",
           "message" => "2023-03-12 10:48:14.003 WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end some more-characters-here",
          "@version" => "1"
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2023, 4:32pm UTC](https://discuss.elastic.co/t/simple-question-about-timestamp-in-logs/327485/5 "2023-04-09T16:32:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.