# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=725 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 726 --- ## [Getting error INVALID\_FETCH\_SESSION\_EPOCH in logstash](https://discuss.elastic.co/t/getting-error-invalid-fetch-session-epoch-in-logstash/139340)
**Author:** [@Krishna\_Oza](https://discuss.elastic.co/u/Krishna_Oza)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 12:14pm UTC](https://discuss.elastic.co/t/getting-error-invalid-fetch-session-epoch-in-logstash/139340 "2018-07-10T12:14:51Z")
I am getting the error INVALID\_FETCH\_SESSION\_EPOCH while reading messages from kafka. The error I get is as below. \[2018-07-10T17:42:20,547\]\[INFO \]\[org.apache.kafka.clients.FetchSessionHandler\] \[Consumer clientId=logst… --- ## [Logstash stuck after all data inserted without any error](https://discuss.elastic.co/t/logstash-stuck-after-all-data-inserted-without-any-error/139339)
**Author:** [@FrantisekHrdina](https://discuss.elastic.co/u/FrantisekHrdina)\ **Replies:** 2\ **Last updated:** [July 10, 2018, 12:11pm UTC](https://discuss.elastic.co/t/logstash-stuck-after-all-data-inserted-without-any-error/139339 "2018-07-10T12:11:23Z")
Hi, I am trying to parse some mactimes and upload them to elastic using logstash. All data will get there without problem, but the logstash process won't stop, I have to stop it manualy. Also nothing is logged in /var/l… --- ## [Unable to parse specific log with logstash 6.3](https://discuss.elastic.co/t/unable-to-parse-specific-log-with-logstash-6-3/139301)
**Author:** [@Loudone](https://discuss.elastic.co/u/Loudone)\ **Replies:** 4\ **Last updated:** [July 10, 2018, 11:30am UTC](https://discuss.elastic.co/t/unable-to-parse-specific-log-with-logstash-6-3/139301 "2018-07-10T11:30:08Z")
Dear all, I'm new on logstash, and I'm trying to parse this log with logstash: 2018-07-10T10:46:25+0200 coll\_bddf \[3\] \[0\] \[6\] \[24\] \[2\] \[5\] so, I've create the logstash configuration file below: input { file { path … --- ## [No pattern in log entries/bulk entries with Logstash](https://discuss.elastic.co/t/no-pattern-in-log-entries-bulk-entries-with-logstash/139311)
**Author:** [@venkatesh\_m](https://discuss.elastic.co/u/venkatesh_m)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 9:29am UTC](https://discuss.elastic.co/t/no-pattern-in-log-entries-bulk-entries-with-logstash/139311 "2018-07-10T09:29:38Z")
I need to combine no specific(No Pattern) log entries in to logstash or ignore the multiple entries in one minute and send it as one event. Can anyone please suggest me. --- ## [Duplication of data using the aggregate plugin](https://discuss.elastic.co/t/duplication-of-data-using-the-aggregate-plugin/139284)
**Author:** [@wm1994](https://discuss.elastic.co/u/wm1994)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 7:19am UTC](https://discuss.elastic.co/t/duplication-of-data-using-the-aggregate-plugin/139284 "2018-07-10T07:19:24Z")
The first time we used logstash to process data today, there was a problem of data duplication or loss.This is logstash's bug or I've got a mistake? And my logstash.conf is: input { stdin {} jdbc { jdbc\_connection\_… --- ## [Visa topic](https://discuss.elastic.co/t/visa-topic/139279)
**Author:** [@Chen\_Yingmin](https://discuss.elastic.co/u/Chen_Yingmin)\ **Replies:** 1\ **Last updated:** [July 10, 2018, 7:09am UTC](https://discuss.elastic.co/t/visa-topic/139279 "2018-07-10T07:09:07Z")
visa topic --- ## [Dynamic dictionary?](https://discuss.elastic.co/t/dynamic-dictionary/138798)
**Author:** [@alex3](https://discuss.elastic.co/u/alex3)\ **Replies:** 7\ **Last updated:** [July 10, 2018, 5:32am UTC](https://discuss.elastic.co/t/dynamic-dictionary/138798 "2018-07-10T05:32:03Z")
For some reason I can't reference a field in my dictionary\_path, is this even possible? I want to use a different dictionary depending on a field. --- ## [Override analyzer for Logstash extracted field](https://discuss.elastic.co/t/override-analyzer-for-logstash-extracted-field/139245)
**Author:** [@daniel.becroft](https://discuss.elastic.co/u/daniel.becroft)\ **Replies:** 0\ **Last updated:** [July 9, 2018, 11:42pm UTC](https://discuss.elastic.co/t/override-analyzer-for-logstash-extracted-field/139245 "2018-07-09T23:42:18Z")
We are working on setting up logstash to manage our log files. One of the issues we have, is that some of the mesages are written with "\*\*" in the string, and we need to retain that when we try and search (so that we can… --- ## [Import Lat/Lon from SQL Server using JDBC](https://discuss.elastic.co/t/import-lat-lon-from-sql-server-using-jdbc/139244)
**Author:** [@Scott\_Newby](https://discuss.elastic.co/u/Scott_Newby)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 11:48pm UTC](https://discuss.elastic.co/t/import-lat-lon-from-sql-server-using-jdbc/139244 "2018-07-09T23:48:29Z")
I'm trying to import a few fields into an existing index, 3 doubles and a geo\_point. Spent the better part of the day with this one without luck. Here is my index mapping setup: "mappings": { "\_doc": { "properties": … --- ## [Netflow codec issues with VMware DVS](https://discuss.elastic.co/t/netflow-codec-issues-with-vmware-dvs/139240)
**Author:** [@1fe60245fe160bc0f8ab](https://discuss.elastic.co/u/1fe60245fe160bc0f8ab)\ **Replies:** 0\ **Last updated:** [July 9, 2018, 10:20pm UTC](https://discuss.elastic.co/t/netflow-codec-issues-with-vmware-dvs/139240 "2018-07-09T22:20:34Z")
Hi All, We've been trying to use our ELK stack (version 6.3.1) to gather IPFIX from VMware VDS (version 5.5 update 3, currently upgrading to 6.5 but it's going to take a while). When we configure our DVS to export Netf… --- ## [Dataparse failures](https://discuss.elastic.co/t/dataparse-failures/139045)
**Author:** [@heskez](https://discuss.elastic.co/u/heskez)\ **Replies:** 3\ **Last updated:** [July 9, 2018, 9:18pm UTC](https://discuss.elastic.co/t/dataparse-failures/139045 "2018-07-09T21:18:31Z")
Hi I'm getting dataparse failure errors because of invalid timestamp format: fortigate\_subtype:%{subtype} host:192.168.1.1 logTimestamp:%{date} %{time} @version:1 syslog5424\_pri:188 fortigate\_type:syslog message:%COPY-W… --- ## [Missing /config file and yml files after new installation](https://discuss.elastic.co/t/missing-config-file-and-yml-files-after-new-installation/138954)
**Author:** [@bamptini](https://discuss.elastic.co/u/bamptini)\ **Replies:** 3\ **Last updated:** [July 9, 2018, 8:22pm UTC](https://discuss.elastic.co/t/missing-config-file-and-yml-files-after-new-installation/138954 "2018-07-09T20:22:24Z")
Hi, I know very little about all this stuff. Im trying to learn form scratch and after a long drawn out day of installing the products logstash, elasticsearch and kibana. I am now stuck. Maybe this is not an issue but I… --- ## [Logstash plugin error](https://discuss.elastic.co/t/logstash-plugin-error/139231)
**Author:** [@xpaths](https://discuss.elastic.co/u/xpaths)\ **Replies:** 0\ **Last updated:** [July 9, 2018, 7:38pm UTC](https://discuss.elastic.co/t/logstash-plugin-error/139231 "2018-07-09T19:38:54Z")
This is the error I was getting: \[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will restart this plugin. Plugin: \ **Author:** [@eviatenne](https://discuss.elastic.co/u/eviatenne)\ **Replies:** 2\ **Last updated:** [July 9, 2018, 7:27pm UTC](https://discuss.elastic.co/t/how-to-create-array-data-type-from-csv/139179 "2018-07-09T19:27:16Z") Hey, let's assume I have the following CSV and I would like to populate it into world\_cup\_2018 index csv: groups.csv Group A|Uruguay,Russia,Saudi Arabia,Egypt Group B|Spain,Portugal,Iran,Morocco .... I would like to… --- ## [Issue in using Logstash as service with jdbc-input plugin](https://discuss.elastic.co/t/issue-in-using-logstash-as-service-with-jdbc-input-plugin/138992)
**Author:** [@Shailesh\_Vaidya](https://discuss.elastic.co/u/Shailesh_Vaidya)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 6:21pm UTC](https://discuss.elastic.co/t/issue-in-using-logstash-as-service-with-jdbc-input-plugin/138992 "2018-07-09T18:21:04Z")
Hello, We are using Logstash to push data from Oracle database to Elastic. We are using jdbc-input plugin in logstash. We are using tracking column names 'insertts' to control records to be pushed to elastic. I observ… --- ## [Flatten JSON array (Simple, but I don't know ruby) Please help!](https://discuss.elastic.co/t/flatten-json-array-simple-but-i-dont-know-ruby-please-help/139212)
**Author:** [@Ryan.Ivis](https://discuss.elastic.co/u/Ryan.Ivis)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 6:15pm UTC](https://discuss.elastic.co/t/flatten-json-array-simple-but-i-dont-know-ruby-please-help/139212 "2018-07-09T18:15:40Z")
Here is what my Objects look like. { "severity": "", "organization\_name": "", "source": "", "organization\_id": "", "server\_or\_region": "", "created\_at": "", "title": "", "id": "" }, { "severity": "", "organi… --- ## [Use time same with the host local timezone](https://discuss.elastic.co/t/use-time-same-with-the-host-local-timezone/139197)
**Author:** [@poper](https://discuss.elastic.co/u/poper)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 6:14pm UTC](https://discuss.elastic.co/t/use-time-same-with-the-host-local-timezone/139197 "2018-07-09T18:14:27Z")
Hi There, I have read more topics but I don't understand about timezone. So I just need to create the file with the hour in the day. output e.g. 21.log, 22.log, 23.log, 00.log, 01.log My logstash.conf file { path =\>"… --- ## [@timestamp is not getting updated through date filter. getting \_dateparsefailure](https://discuss.elastic.co/t/timestamp-is-not-getting-updated-through-date-filter-getting-dateparsefailure/137784)
**Author:** [@Honda\_fred\_elk](https://discuss.elastic.co/u/Honda_fred_elk)\ **Replies:** 3\ **Last updated:** [July 9, 2018, 3:14pm UTC](https://discuss.elastic.co/t/timestamp-is-not-getting-updated-through-date-filter-getting-dateparsefailure/137784 "2018-07-09T15:14:09Z")
I have following logstash configuration for Websphere App server SystemErr logs. As per the grok pattern, I do see the logs are getting parsed, but Overriding @timestamp is not working. I'm getting \_dateparsefailure. h… --- ## [Path.data: /var/lib/logstash](https://discuss.elastic.co/t/path-data-var-lib-logstash/139111)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 11:59am UTC](https://discuss.elastic.co/t/path-data-var-lib-logstash/139111 "2018-07-09T11:59:51Z")
Hi, \[root@frghcslnetv10 elasticsearch\]# df -kh Filesystem Size Used Avail Use% Mounted on /dev/mapper/cl\_frghcslnetv10-root 22G 16G 6.1G 72% / devtmpfs 3.9G 0… --- ## [How to access existing field in KV filter plugin?](https://discuss.elastic.co/t/how-to-access-existing-field-in-kv-filter-plugin/139134)
**Author:** [@kelvink](https://discuss.elastic.co/u/kelvink)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 11:56am UTC](https://discuss.elastic.co/t/how-to-access-existing-field-in-kv-filter-plugin/139134 "2018-07-09T11:56:59Z")
How can we access a field that was created in a previous grok match step (eg. \[app\_name\]) and dynamically use that value to control the value of "target" in the kv filter plugin? eg: grok { match =\> { "\[inputdata\]" =\> … --- ## [Logstash arcsight integration module (problem)](https://discuss.elastic.co/t/logstash-arcsight-integration-module-problem/139161)
**Author:** [@tsm-elastic](https://discuss.elastic.co/u/tsm-elastic)\ **Replies:** 0\ **Last updated:** [July 9, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-arcsight-integration-module-problem/139161 "2018-07-09T11:56:32Z")
Hi, I am currently setting up the ArcSight integration module on Logstash so that I could feed security logs to the elastic stack via smartconnectors. Here's the config I am using as instructed in the guide: bin/logstas… --- ## [Mutate filter](https://discuss.elastic.co/t/mutate-filter/139156)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 2\ **Last updated:** [July 9, 2018, 11:44am UTC](https://discuss.elastic.co/t/mutate-filter/139156 "2018-07-09T11:44:52Z")
Can I exchange the field for the existed field? Like this.. mutate { update =\> {"\[matchingCid\]\[0\]\[CID\]" =\> \[matchingCid\]\[0\]\[OTC2\_CID\]} } \[matchingCid\]\[0\]\[CID\] and \[matchingCid\]\[0\]\[OTC2\_CID\] are existed field.. --- ## [How to identify and track logstash running or not at scheduled timeframe](https://discuss.elastic.co/t/how-to-identify-and-track-logstash-running-or-not-at-scheduled-timeframe/138710)
**Author:** [@sumanbehara](https://discuss.elastic.co/u/sumanbehara)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 11:42am UTC](https://discuss.elastic.co/t/how-to-identify-and-track-logstash-running-or-not-at-scheduled-timeframe/138710 "2018-07-09T11:42:44Z")
Hi All, I have a requirement to identify logstash scheduler running or not at scheduled time. My application should know which scheduler are running , which are not running. current my schedulers are running fine with… --- ## [How to call a method of ksh file(shell script) from a conf file(logstash file)](https://discuss.elastic.co/t/how-to-call-a-method-of-ksh-file-shell-script-from-a-conf-file-logstash-file/138910)
**Author:** [@prashansa1](https://discuss.elastic.co/u/prashansa1)\ **Replies:** 5\ **Last updated:** [July 9, 2018, 11:33am UTC](https://discuss.elastic.co/t/how-to-call-a-method-of-ksh-file-shell-script-from-a-conf-file-logstash-file/138910 "2018-07-09T11:33:38Z")
How to call a method of external ksh file(shell script) from a conf file(logstash file). Shell script (ksh file) a.ksh Test() { x ="$1" if (x \> 10 ) return x; } Logstash Script (conf file) acon.conf I want to… --- ## [Logstash, Filebeat and Winlogbeat](https://discuss.elastic.co/t/logstash-filebeat-and-winlogbeat/138746)
**Author:** [@Marcos\_Felix](https://discuss.elastic.co/u/Marcos_Felix)\ **Replies:** 12\ **Last updated:** [July 9, 2018, 11:28am UTC](https://discuss.elastic.co/t/logstash-filebeat-and-winlogbeat/138746 "2018-07-09T11:28:07Z")
Is there any good tutorials out there with good configuration to install these tools so that I can ship event logs to my Kibana interface? Thanks in advance --- ## [How to pass timestamp value into other log file](https://discuss.elastic.co/t/how-to-pass-timestamp-value-into-other-log-file/139122)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 5\ **Last updated:** [July 9, 2018, 10:40am UTC](https://discuss.elastic.co/t/how-to-pass-timestamp-value-into-other-log-file/139122 "2018-07-09T10:40:31Z")
hi am new to elk,how to pass timestamp value into other log file any body suggestions?? --- ## [Is there any method to filter enterprise library logs?](https://discuss.elastic.co/t/is-there-any-method-to-filter-enterprise-library-logs/139138)
**Author:** [@Avinash\_Kumar](https://discuss.elastic.co/u/Avinash_Kumar)\ **Replies:** 0\ **Last updated:** [July 9, 2018, 10:24am UTC](https://discuss.elastic.co/t/is-there-any-method-to-filter-enterprise-library-logs/139138 "2018-07-09T10:24:28Z")
sample is here: =-=-=-=-=-=-=-=- Timestamp: Wednesday, May 23, 2018 9:45:51 AM Message: HandlingInstanceID: 2d398f81-fddd-4adb-914e-acb179551a4e An exception of type 'System.Exception' occurred and was caught. 05/23/… --- ## [Deleting files after logstash processes them](https://discuss.elastic.co/t/deleting-files-after-logstash-processes-them/139133)
**Author:** [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 10:16am UTC](https://discuss.elastic.co/t/deleting-files-after-logstash-processes-them/139133 "2018-07-09T10:16:06Z")
Hi, I would like to know if there is a way of deleting log files once logstash has finished processing. We have files sent to an input folder, logstash then processes them. I would like to clear them down once it is do… --- ## [Best practices for reading elastic data to logstash](https://discuss.elastic.co/t/best-practices-for-reading-elastic-data-to-logstash/138570)
**Author:** [@dmamalis](https://discuss.elastic.co/u/dmamalis)\ **Replies:** 8\ **Last updated:** [July 9, 2018, 9:23am UTC](https://discuss.elastic.co/t/best-practices-for-reading-elastic-data-to-logstash/138570 "2018-07-09T09:23:35Z")
Greetings to everybody, having only scratched the surface of the ELK stack I found myself in the following situation: one of my linux servers runs a few services which have built-in functionality to send logs directly t… --- ## [Is it possible to overwrite the select statement each time when db is called](https://discuss.elastic.co/t/is-it-possible-to-overwrite-the-select-statement-each-time-when-db-is-called/139086)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 9\ **Last updated:** [July 9, 2018, 7:47am UTC](https://discuss.elastic.co/t/is-it-possible-to-overwrite-the-select-statement-each-time-when-db-is-called/139086 "2018-07-09T07:47:30Z")
i have two dates, eg :(from)2018/01/01 to now ..i have to fetch the records between these two dates and after fetching from will be replaced with now timing and now with new timestamp..how can v do these?? any body??a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=724) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=726)