# Logs time 6 hours behind local time **URL:** \ **Category:** Metrics\ **Created:** [February 23, 2019, 7:05pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679 "2019-02-23T19:05:51Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![cachedout](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cachedout/32/38707_2.png) [@cachedout](https://discuss.elastic.co/u/cachedout)\ **Post date:** [March 5, 2019, 6:33pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/21 "2019-03-05T18:33:59Z")
Could we see a few lines from both the Elasticsearch log file that's being delivered as well as one from the Logstash log file, please? ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [March 5, 2019, 6:57pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/22 "2019-03-05T18:57:20Z")
Elasticsearch: > [2019-03-05T08:44:21,781][INFO][o.e.d.DiscoveryModule] [met-elk-exp6a] using discovery type [zen] and host providers [settings] > [2019-03-05T08:44:22,606][INFO][o.e.n.Node] [met-elk-exp6a] initialized > [2019-03-05T08:44:22,607][INFO][o.e.n.Node] [met-elk-exp6a] starting ... > [2019-03-05T08:44:22,839][INFO][o.e.t.TransportService] [met-elk-exp6a] publish\_address {128.206.0.162:9300}, bound\_addresses {[::]:9300} > [2019-03-05T08:44:22,941][INFO][o.e.b.BootstrapChecks] [met-elk-exp6a] bound or publishing to a non-loopback address, enforcing bootstrap checks > [2019-03-05T08:44:26,003][INFO][o.e.c.s.MasterService] [met-elk-exp6a] zen-disco-elected-as-master ([0] nodes joined), reason: new\_master {met-elk-exp6a}{8lmPMbyUS6-0v6v87BNmOA}{NSWlpR3SQCafgv\_LPxWWZQ}{128.206.0.162}{128.206.0.162:9300}{ml.machine\_memory=12392050688, rack=hyperv, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true} > [2019-03-05T08:44:26,012][INFO][o.e.c.s.ClusterApplierService] [met-elk-exp6a] new\_master {met-elk-exp6a}{8lmPMbyUS6-0v6v87BNmOA}{NSWlpR3SQCafgv\_LPxWWZQ}{128.206.0.162}{128.206.0.162:9300}{ml.machine\_memory=12392050688, rack=hyperv, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {met-elk-exp6a}{8lmPMbyUS6-0v6v87BNmOA}{NSWlpR3SQCafgv\_LPxWWZQ}{128.206.0.162}{128.206.0.162:9300}{ml.machine\_memory=12392050688, rack=hyperv, xpack.installed=true, ml.max\_open\_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]]) > [2019-03-05T08:44:26,066][INFO][o.e.h.n.Netty4HttpServerTransport] [met-elk-exp6a] publish\_address {128.206.0.162:9200}, bound\_addresses {[::]:9200} > [2019-03-05T08:44:26,066][INFO][o.e.n.Node] [met-elk-exp6a] started > [2019-03-05T08:44:26,845][INFO][o.e.c.s.ClusterSettings] [met-elk-exp6a] updating [xpack.monitoring.collection.enabled] from [false] to [true] > [2019-03-05T08:44:27,354][WARN][r.suppressed] [met-elk-exp6a] path: /.kibana/doc/kql-telemetry%3Akql-telemetry, params: {index=.kibana, id=kql-telemetry:kql-telemetry, type=doc} > org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE\_UNAVAILABLE/1/state not recovered / initialized]; > [2019-03-05T12:00:03,766][INFO][o.e.c.m.MetaDataMappingService] [met-elk-exp6a] [winlogbeat-6.6.0-dc-mst-2019.03.05/aiNgaDhJQveJ3RDL-GkfPA] update\_mapping [doc] logstash: > [2019-03-05T08:57:49,802][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} > [2019-03-05T08:57:49,806][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} > [2019-03-05T08:57:49,935][WARN][logstash.outputs.elasticsearch] You are using a deprecated config setting "document\_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=\>"document\_type", :plugin=\>\"/\_xpack/monitoring/\_bulk?system\_id=logstash&system\_api\_version=2&interval=1s", hosts=\>[[http://localhost:9200](http://localhost:9200)], sniffing=\>false, manage\_template=\>false, id=\>"2196aa69258f6adaaf9506d8988cc76ab153e658434074dcf2e424e0aca0d381", document\_type=\>"%{[@metadata][document\_type]}", enable\_metric=\>true, codec=\>\"plain\_e3ed3e0d-5293-47ad-b780-bbc0dd1002c4", enable\_metric=\>true, charset=\>"UTF-8"\>, workers=\>1, template\_name=\>"logstash", template\_overwrite=\>false, doc\_as\_upsert=\>false, script\_type=\>"inline", script\_lang=\>"painless", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_initial\_interval=\>2, retry\_max\_interval=\>64, retry\_on\_conflict=\>1, ilm\_enabled=\>false, ilm\_rollover\_alias=\>"logstash", ilm\_pattern=\>"{now/d}-000001", ilm\_policy=\>"logstash-policy", action=\>"index", ssl\_certificate\_verification=\>true, sniffing\_delay=\>5, timeout=\>60, pool\_max=\>1000, pool\_max\_per\_route=\>100, resurrect\_delay=\>5, validate\_after\_inactivity=\>10000, http\_compression=\>false\>} > [2019-03-05T08:57:49,951][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} > [2019-03-05T08:57:49,959][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} > [2019-03-05T09:59:18,545][WARN][logstash.runner] SIGTERM received. Shutting down. > [2019-03-05T09:59:23,635][WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight\_count"=\>0, "stalling\_threads\_info"=\>{"other"=\>[{"thread\_id"=\>50, "name"=\>"[filebeat]\"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.8-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>47, "name"=>"[filebeat]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read\_batch'"}, {"thread\_id"=\>48, "name"=\>"[filebeat]\>worker1", "current\_call"=\>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>49, "name"=>"[filebeat]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read\_batch'"}]}} > [2019-03-05T09:59:23,639][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information. > [2019-03-05T09:59:30,572][WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight\_count"=\>0, "stalling\_threads\_info"=\>{"other"=\>[{"thread\_id"=\>37, "name"=\>"[exchange]\"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.1.8-java/lib/logstash/inputs/beats.rb:212:in `run'"}, {"thread_id"=>34, "name"=>"[exchange]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read\_batch'"}, {"thread\_id"=\>35, "name"=\>"[exchange]\>worker1", "current\_call"=\>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read_batch'"}, {"thread_id"=>36, "name"=>"[exchange]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/pipeline.rb:333:in `read\_batch'"}]}} > [2019-03-05T09:59:30,572][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information. > [2019-03-05T09:59:58,999][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"} ---
**Author:** ![cachedout](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cachedout/32/38707_2.png) [@cachedout](https://discuss.elastic.co/u/cachedout)\ **Post date:** [March 5, 2019, 8:44pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/23 "2019-03-05T20:44:58Z")
Hmmm. If you modify the pipeline so that timezone is explicitly set to `UTC` in the "date" block does it still throw the errors? Replace: `{< if .convert_timezone >}"timezone": "{{ event.timezone }}",{< end >}` With: `"timezone": "UTC"` ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [March 5, 2019, 10:37pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/24 "2019-03-05T22:37:03Z")
So that is in the filebeat dynamic creation of the pipeline? I couldn't get that to reload the pipelines, would they be overwritten or would I have to delete them? I edited the pipelines in Dev Tools. I got elasticsearch working. I used CST6CDT for the timezone since the logs have local time, not UTC. I've noticed that the pipelines refer to event.timezone (for elasticsearch) and event.timezone (for logstash), but neither exist. "beat.timezone" exists in the elasticsearch events, but I can't get anything to ingest from logstash now, not sure why, no errors, no nothing. Or add\_locale: ~ isn't working... ---
**Author:** ![Mike\_Place](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_place/32/39555_2.png) [@Mike\_Place](https://discuss.elastic.co/u/Mike_Place)\ **Post date:** [March 5, 2019, 10:59pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/25 "2019-03-05T22:59:25Z")
A lot of the code that seems to be related to what we're seeing here landed in v6.6.1 via this PR: [https://github.com/elastic/beats/pull/9797](https://github.com/elastic/beats/pull/9797) I'm going to link this issue in that PR to see if the original author can have a look. ---
**Author:** ![Mike\_Place](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_place/32/39555_2.png) [@Mike\_Place](https://discuss.elastic.co/u/Mike_Place)\ **Post date:** [March 5, 2019, 11:06pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/26 "2019-03-05T23:06:15Z")
Additionally, I think I can confirm what you're seeing. I think that what might possibly have happened was that the PR referenced a field that was changed in 7.0 but the fix went into 6.6.1. I will need to confirm this with the original developer but it would align with what we're seeing here. ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [March 5, 2019, 11:21pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/27 "2019-03-05T23:21:40Z")
Again, thanks for looking, this is just something I'm poking at while waiting for "the project" to start 🙂 ---
**Author:** ![Mike\_Place](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_place/32/39555_2.png) [@Mike\_Place](https://discuss.elastic.co/u/Mike_Place)\ **Post date:** [March 6, 2019, 12:05am UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/28 "2019-03-06T00:05:17Z")
You're quite welcome. I think there could well be a legitimate bug here so I'm glad you brought it to our attention. Much appreciated! ---
**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Post date:** [March 6, 2019, 4:50am UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/29 "2019-03-06T04:50:38Z")
Another question. Since I just started filebeat with these modules enabled, then discovered convert\_timestamp, the pipelines were already loaded without the date conversion. I wonder if the pipelines need to be deleted before the changed code will be stored? ---
**Author:** ![Mike\_Place](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_place/32/39555_2.png) [@Mike\_Place](https://discuss.elastic.co/u/Mike_Place)\ **Post date:** [March 6, 2019, 3:56pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/30 "2019-03-06T15:56:15Z")
You can get the pipelines to update by running `filebeat setup --pipelines`. Otherwise, they are not updated if they exist in Elasticsearch already. ---
**Author:** ![Mike\_Place](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_place/32/39555_2.png) [@Mike\_Place](https://discuss.elastic.co/u/Mike_Place)\ **Post date:** [March 8, 2019, 5:40pm UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/31 "2019-03-08T17:40:08Z")
By way of an update here, we have confirmed what you're seeing as a legitimate bug and have written a patch for it that is currently under review. Please find it here: > Thanks again for bringing this to our attention. Much appreciated and apologies for the trouble. ---
**Author:** ![Ofelia44](https://avatars.discourse-cdn.com/v4/letter/o/ebca7d/32.png) [@Ofelia44](https://discuss.elastic.co/u/Ofelia44)\ **Post date:** [March 30, 2019, 4:40am UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/32 "2019-03-30T04:40:44Z")
Yes u can get the pipelines to update by running the code filebeat setup --pipelines.And check their updation,bcoz chances are high that they exist in Elasticsearch already[.](https://mykfcexperience.website/) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 27, 2019, 4:42am UTC](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679/33 "2019-04-27T04:42:12Z")
Closed on April 27, 2019, 4:42am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/logs-time-6-hours-behind-local-time/169679.md?page=1)