# Multiple problems with logstash

**URL:** <https://discuss.elastic.co/t/multiple-problems-with-logstash/90554>\
**Category:** Logstash\
**Created:** [June 22, 2017, 10:51pm UTC](https://discuss.elastic.co/t/multiple-problems-with-logstash/90554 "2017-06-22T22:51:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sfunk](https://avatars.discourse-cdn.com/v4/letter/s/7993a0/32.png) [@sfunk](https://discuss.elastic.co/u/sfunk)\
**Post date:** [June 22, 2017, 10:51pm UTC](https://discuss.elastic.co/t/multiple-problems-with-logstash/90554/1 "2017-06-22T22:51:13Z")

</div>

I'm encountering a couple issues with logstash, it's probably me not understanding the documentation.

First, I cannot get logstash to start where it left off when filling ES from an Oracle datastore. When it gets about 2 million records in, logstash appears to stop, then restart (with debug logging enabled, it produces no error) and then runs the exact same query over again instead of using a recorded :sql\_last\_value. I have tried a myriad of configurations in order to try to use specific fields for tracking, clean run and the other associated fields (clean run false with an empty file simply doesn't work, because it's trying to use "Null", apparently, for :sql\_last\_value) and I'm out of ideas at this point. I can record an arbitrary timestmap in the last\_metadata file, but instead of overwritingthe value I manually place in there, it just reads from it repeatedly.

Here's a log snippet from logstash starting up, and for whatever, starting at 0 date which it shouldn't be starting at, based on the configuration:

> <https://gist.github.com/anonymous/fe44527673536be47ea0a1f6bbee3208>

And here's a log snippet where logstash automagically restarts with no human input, and of course, no logged reason _why_ it would be restarting. And of course, it executes the same query from the beginning of time, as opposed to the timestamp that it _should_ be recording or keeping track of somehow, according to the configuration and documentation:

> <https://gist.github.com/anonymous/0efdd3b76bbd6c7e246cf1435529da2d>

Clearly I'm doing something completely wrong here but it's not incredibly clear from the jdbc documentation page, and there are no provided examples for this scenario so maybe I added too many options, or not including enough...

Second, my date filter appears to create a \_dateparseerror in the current configuration. Apparently, no debug information from the filter is ever generated and I don't see in the documentation where I can put the date filter in a debug or trace logging state - only logstash as a whole (and sql debug to see the query, I'm guessing - which is enabled). When I isolate the date match statement in it's own configuration file, I can echo the timestamp into logstash and logstash will match it. I wrapped the date filter with a conditional to ensure that it can see the "start\_ts" field, and it indeed does see it, but apparently the timestamp it's getting is not what is being reported in the log (??) - as the date filter is not matching the timestamp and I'm clearly not matching against the correct format. I would expect the date match to fail when I isolate the date filter, but it doesn't - and takes whatever timestamp string I give it and applies it to the @timestamp field. No such luck when feeding from jdbc, though.

Here's a clip of console from the logs when I isolate the filter:

> <https://gist.github.com/anonymous/815a23b4119a436655f7bad91089c01c>

Here's the logstash yaml config:

> <https://gist.github.com/anonymous/3d081655814831453108be4ac5dc804c>

Any help is appreciated here. I"m a bit burned out on this as it really should be pretty straight forward, but has been anything but.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 26, 2017, 5:53am UTC](https://discuss.elastic.co/t/multiple-problems-with-logstash/90554/2 "2017-06-26T05:53:34Z")

</div>

Does Logstash have sufficient permissions to update /usr/share/logstash/.jdbc\_last\_run? I'd try setting it to e.g. /tmp/.jdbc\_last\_run just to be sure.

Regarding your date problem see this issue:

> <https://github.com/logstash-plugins/logstash-filter-date/issues/95>

I think you can use the mutate filter's convert option to convert the timestamp field into a string which then can be parsed by the date filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 24, 2017, 5:53am UTC](https://discuss.elastic.co/t/multiple-problems-with-logstash/90554/3 "2017-07-24T05:53:48Z")

</div>

Closed on July 24, 2017, 5:53am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.