# Drop milliseconds from timestamp

**URL:** <https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264>\
**Category:** Logstash\
**Created:** [May 12, 2020, 4:04pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264 "2020-05-12T16:04:21Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![alberto1](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@alberto1](https://discuss.elastic.co/u/alberto1)\
**Post date:** [May 12, 2020, 4:04pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/1 "2020-05-12T16:04:21Z")

</div>

Hi, I am trying to drop the 4 miliseconds from the timestamp but i cannot make it work and it's given me an error of \_dateparsefailure

the bit where I have the error is:

```
grok {
			match => ["message", "%{TIMESTAMP_ISO8601:fechalog} %{LOGLEVEL:Severity} %{GREEDYDATA:Message}"]
		}
		mutate {
			gsub => ["fechalog", "\.\d{4}$"]
		}
		date {
			match => ["fechalog", "YYYY-MM-dd HH:mm:ss"]
			timezone => "UTC"
		}
		mutate {
			add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"} 
		}
		mutate {
			remove_tag => ["beats_input_codec_plain_applied"] 
    		}

```

an example log...

```
2020-05-09 22:34:05.0880 ERROR DatosGruas.BOL.Trama.Leer 

```

I can see that the `fechalog` field its been filled with the right timestamp. I guess my mutate is not right and that is making fail my date filter???

Thanks in advance

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 12, 2020, 11:09pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/2 "2020-05-12T23:09:23Z")

</div>

> [@alberto1](#):
>
> mutate { gsub =\> ["fechalog", ".\d{4}$"] }

That should produce an error message (logstash.agent.configuration.invalid\_plugin\_register). Try

```
mutate { gsub => ["fechalog", "\.\d{4}$", ""] }

```

---

<div class="post-metadata">

**Author:** ![alberto1](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@alberto1](https://discuss.elastic.co/u/alberto1)\
**Post date:** [May 13, 2020, 4:12pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/3 "2020-05-13T16:12:36Z")

</div>

That works, thanks.

Anyway I have another little issue...  
So my date filter get the logtime and replace it at @timestamp but when I check the logs in Kibana they are 2 hours late.

I know that elasticserarch works with UTC. In kibana you have a option in the settings to change the timezone to Europe/Madrid and also I have change all the timezones to Europe/Madrid in the date filter but still two hours for behind....

Any idea why? What I am doing wrong?

```
filter {
if "Datos_Gruas" in [tags] {
		grok {
			match => ["message", "%{TIMESTAMP_ISO8601:fechalog} %{LOGLEVEL:Severity} %{GREEDYDATA:Message}"]
		}
		mutate {
			gsub => ["fechalog", "\.\d{4}$", ""]
		}
		date {
			match => ["fechalog", "YYYY-MM-dd HH:mm:ss"]
			timezone => "Europe/Madrid"
		}
		mutate {
			add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"} 
		}
		mutate {
			remove_tag => ["beats_input_codec_plain_applied"] 
    		}
} else if "MonitorizacionReefers" in [tags] {
	grok {
		match => ["message", "%{TIME:Hora} - %{DATE_EU:Fecha} %{LOGLEVEL:Severity} %{GREEDYDATA:Message}"]
	}
	mutate {
		add_field => { "logfecha" => "%{Fecha} %{Hora}"} 
	}
	date {
		match => ["logfecha", "dd-MM-yyyy HH:mm:ss"]
		timezone => "Europe/Madrid"
	}
	mutate {
		add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"} 
	}
	mutate {
		remove_tag => ["beats_input_codec_plain_applied"] 
    	}
} else if "Procesos2_SVC" in [tags] {
		grok {
			match => ["message", "%{BASE10NUM:Fecha} %{TIME:Hora} - %{GREEDYDATA:Message}"]
		}
		mutate {
			add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"}
			add_field => { "logtime" => "%{Fecha} %{Hora}"} 
		}
		date {
			match => ["logfecha", "yyyyMMdd HH:mm:ss"]
			timezone => "Europe/Madrid"
		}
		mutate {
			remove_tag => ["beats_input_codec_plain_applied"] 
    		}

} else if "Procesos_SVC" in [tags] {
		grok {
			match => ["message", "%{BASE10NUM:Fecha} %{TIME:Hora} - %{GREEDYDATA:Message}"]
		}
		mutate {
			add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"}
			add_field => { "logdate" => "%{Fecha} %{Hora}"}
		}
		date {
			match => ["logdate", "yyyyMMdd HH:mm:ss"]
			timezone => "Europe/Madrid"
		}
		mutate {
			remove_tag => ["beats_input_codec_plain_applied"] 
    		}
} else {
	mutate {
		add_field => { "Ubicacion" => "%{[host][hostname]}-%{[log][file][path]}"} 
		}
		mutate {
		remove_tag => ["beats_input_codec_plain_applied"] 
    		}
} }

```

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 13, 2020, 10:14pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/4 "2020-05-13T22:14:26Z")

</div>

Are you sure the timestamps in the log file are Europe/Madrid and not UTC? If you query elasticsearch directly, without letting kibana do the presentation, does the timestamp look right?

---

<div class="post-metadata">

**Author:** ![alberto1](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@alberto1](https://discuss.elastic.co/u/alberto1)\
**Post date:** [May 13, 2020, 10:38pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/5 "2020-05-13T22:38:50Z")

</div>

Hi Badger,  
Yeah i,m sure the timestamp un the logs are Europe/Madrid, and in logstash the @timestsmp chance with the actual log time but un kibana move two hours earlier.  
How can i query elasticsearch? I can ser the Index with get /indexname but don't know how to see a log from elasticsearch API  
Thanks for your help

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 13, 2020, 11:06pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/6 "2020-05-13T23:06:41Z")

</div>

Use a [match all](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-all-query.html#query-dsl-match-all-query) query (it will return a limited subset of the documents). You could use the console in the kibana dev tools to do the PUT to elasticsearch, or connect directly to elasticsearch with curl.

---

<div class="post-metadata">

**Author:** ![alberto1](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@alberto1](https://discuss.elastic.co/u/alberto1)\
**Post date:** [May 14, 2020, 7:06am UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/7 "2020-05-14T07:06:52Z")

</div>

Hi Badger,

I did try to do the search in elasticsearch and it looks like the timestamp is alright.

```
{
  "took" : 3468,
  "timed_out" : false,
  "_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
  },
  "hits" : {
"total" : {
  "value" : 10000,
  "relation" : "gte"
},
"max_score" : 1.0,
"hits" : [
  {
    "_index" : "srv-refcon-procesos-svc",
    "_type" : "_doc",
    "_id" : "zK_GDXIBcHQrn9J24IsS",
    "_score" : 1.0,
    "_source" : {
      "ecs" : {
        "version" : "1.4.0"
      },
      "message" : "20170603 00:25:33 - Ejecuto VERMAS_MANAGER",
      "host" : {
        "architecture" : "x86_64",
        "hostname" : "REFCON",
        "id" : "9be686f6-e89c-4e29-8e5d-685038fea9e8",
        "os" : {
          "version" : "6.3",
          "build" : "9600.19653",
          "family" : "windows",
          "kernel" : "6.3.9600.19",
          "platform" : "windows",
          "name" : "Windows Server"
        },
        "name" : "REFCON"
      },
      "Message" : "Ejecuto VERMAS_MANAGER",
      "Hora" : "00:25:33",
      "logdate" : "20170603 00:25:33",
      "@version" : "1",
      "tags" : [
        "Procesos_SVC",
        "SRV_REFCON"
      ],
      "Fecha" : "20170603",
      "Ubicacion" : """\Procesos_SVC\20170603.log""",
      "agent" : {
        "version" : "7.6.2",
        "hostname" : "REFCON",
        "ephemeral_id" : "ded8bc91-c177-4242-b233-14ff0738c881",
        "type" : "filebeat",
        "id" : "7f9fb557-1c4b-4be8-bee6-695aabdb628a"
      },
      "log" : {
        "offset" : 33555,
        "file" : {
          "path" : """C:\Procesos_SVC\20170603.log"""
        }
      },
      "@timestamp" : "2017-06-03T00:25:33.000Z",
      "input" : {
        "type" : "log"
      }
    }
  }

```

As you can see the @timestamp has the real logtime but then when I see the logs in kibana they are two hours earlier  
Don't know why?

---

<div class="post-metadata">

**Author:** ![alberto1](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@alberto1](https://discuss.elastic.co/u/alberto1)\
**Post date:** [May 14, 2020, 2:35pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/8 "2020-05-14T14:35:23Z")

</div>

Hi Badger,

I did manage to sort it out, yeah you were right and one of my logs was using UTC already that's why it was giving me that 2 hours miss match.  
All sort it now and working!! Thank you for all your help in this topic and other before 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2020, 2:35pm UTC](https://discuss.elastic.co/t/drop-milliseconds-from-timestamp/232264/9 "2020-06-11T14:35:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.