# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=955 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 956 --- ## [Resend the data from the beginning of a file with sincedb setting](https://discuss.elastic.co/t/resend-the-data-from-the-beginning-of-a-file-with-sincedb-setting/91211)
**Author:** [@SongJieCN](https://discuss.elastic.co/u/SongJieCN)\ **Replies:** 3\ **Last updated:** [June 29, 2017, 7:13am UTC](https://discuss.elastic.co/t/resend-the-data-from-the-beginning-of-a-file-with-sincedb-setting/91211 "2017-06-29T07:13:54Z")
Hi Guys, Version: 5.0.1 Operating System: CentOS6.8 Config File (if you have sensitive info, please remove it): input { file { path =\> \[ "/opt/Logs/\*", \] start\_position =\> "beginning" sincedb\_path =\> "/opt/tmp/.sincedb… --- ## [Logstash kafka input plugin doesn't pull data](https://discuss.elastic.co/t/logstash-kafka-input-plugin-doesnt-pull-data/88589)
**Author:** [@guy\_es](https://discuss.elastic.co/u/guy_es)\ **Replies:** 1\ **Last updated:** [June 29, 2017, 6:46am UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-doesnt-pull-data/88589 "2017-06-29T06:46:41Z")
Hi all, I have an ubuntu machine with the following containers and settings, but logstash doesn't pull events from kafka at all, using the same configuration on a machine witch the services install directly on (not doc… --- ## [File input plugin lost first part of log line](https://discuss.elastic.co/t/file-input-plugin-lost-first-part-of-log-line/61989)
**Author:** [@Juan\_Rella](https://discuss.elastic.co/u/Juan_Rella)\ **Replies:** 1\ **Last updated:** [June 29, 2017, 6:18am UTC](https://discuss.elastic.co/t/file-input-plugin-lost-first-part-of-log-line/61989 "2017-06-29T06:18:24Z")
We're working with the version 2.3.x of logstash and we've defined a input file plugin to read apache logs on this way file { add\_field =\> \["timestamp", ""\] path =\> "/var/log/httpd/access\_log.\*" start\_position =\> en… --- ## [\[BUG\] Logstash's temp file auto remove fail](https://discuss.elastic.co/t/bug-logstashs-temp-file-auto-remove-fail/90800)
**Author:** [@maddison](https://discuss.elastic.co/u/maddison)\ **Replies:** 0\ **Last updated:** [June 26, 2017, 9:18am UTC](https://discuss.elastic.co/t/bug-logstashs-temp-file-auto-remove-fail/90800 "2017-06-26T09:18:13Z")
Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here. For all general issue… --- ## [Logstash showing error](https://discuss.elastic.co/t/logstash-showing-error/91130)
**Author:** [@rp311286](https://discuss.elastic.co/u/rp311286)\ **Replies:** 2\ **Last updated:** [June 29, 2017, 5:49am UTC](https://discuss.elastic.co/t/logstash-showing-error/91130 "2017-06-29T05:49:24Z")
here is my logstash\_car.conf file file{ path =\> "C:\\Users\\ravi\\Downloads\\data\\car.csv" start\_position =\>"beginning" sincedb\_path =\> "/dev/null" } filter{ csv{ separator=\>"," columns =\>\["maker","model","mileage","… --- ## [Grok filter not working to capture filename](https://discuss.elastic.co/t/grok-filter-not-working-to-capture-filename/91197)
**Author:** [@Deepthi\_V](https://discuss.elastic.co/u/Deepthi_V)\ **Replies:** 1\ **Last updated:** [June 29, 2017, 5:17am UTC](https://discuss.elastic.co/t/grok-filter-not-working-to-capture-filename/91197 "2017-06-29T05:17:46Z")
Hello, I have a grok filter that looks like below, which I wrote to capture only the filename from the "path" field. grok { match =\> { "path" =\> "%{GREEDYDATA}/%{GREEDYDATA:filename}.log" } … --- ## [Logstash template not used if Elasticsearch not ready?](https://discuss.elastic.co/t/logstash-template-not-used-if-elasticsearch-not-ready/90556)
**Author:** [@ankh](https://discuss.elastic.co/u/ankh)\ **Replies:** 2\ **Last updated:** [June 28, 2017, 11:21pm UTC](https://discuss.elastic.co/t/logstash-template-not-used-if-elasticsearch-not-ready/90556 "2017-06-28T23:21:09Z")
I am supplying a template file with my Logstash job using: template =\> "/conf/myindex-mapping-template.json" template\_name =\> "myindex" template\_overwrite =\> true This works well, and I see: 12:27:19.383 \[\[main\]-pip… --- ## [Aggregate filter does not always work](https://discuss.elastic.co/t/aggregate-filter-does-not-always-work/91206)
**Author:** [@Shuxin\_Lin](https://discuss.elastic.co/u/Shuxin_Lin)\ **Replies:** 1\ **Last updated:** [June 28, 2017, 11:12pm UTC](https://discuss.elastic.co/t/aggregate-filter-does-not-always-work/91206 "2017-06-28T23:12:25Z")
I use aggregate filter. It is super powerful. But it does not work consistently. What I mean is the aggregate filter sometimes failed to aggregate certain logs. I would say in 90% of cases it works. The failure is not re… --- ## [Logstash is not passing the data to elastisearch](https://discuss.elastic.co/t/logstash-is-not-passing-the-data-to-elastisearch/91072)
**Author:** [@vijjun123](https://discuss.elastic.co/u/vijjun123)\ **Replies:** 3\ **Last updated:** [June 28, 2017, 9:57pm UTC](https://discuss.elastic.co/t/logstash-is-not-passing-the-data-to-elastisearch/91072 "2017-06-28T21:57:24Z")
when i pass the excel data in logstash and after running the below command and its strucking not able to pass the values. \[root@ip-10-3-1-152 logstash\]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf S… --- ## [Creating multiple output files in Logstash](https://discuss.elastic.co/t/creating-multiple-output-files-in-logstash/90517)
**Author:** [@Deepthi\_V](https://discuss.elastic.co/u/Deepthi_V)\ **Replies:** 2\ **Last updated:** [June 28, 2017, 9:57pm UTC](https://discuss.elastic.co/t/creating-multiple-output-files-in-logstash/90517 "2017-06-28T21:57:12Z")
Hello, I have my input section where I have a given path which has multiple directories and log files in it. I have written the following: input { file { path =\> "/home/deepthi/Desktop/logs/\*\*… --- ## [Is it possible apply grok in oldest logs?](https://discuss.elastic.co/t/is-it-possible-apply-grok-in-oldest-logs/91104)
**Author:** [@joao](https://discuss.elastic.co/u/joao)\ **Replies:** 7\ **Last updated:** [June 28, 2017, 7:59pm UTC](https://discuss.elastic.co/t/is-it-possible-apply-grok-in-oldest-logs/91104 "2017-06-28T19:59:15Z")
Hello, I need to apply the grok filter in oldest logs, is it possible? Because I need to creat new fields. Thank you very much. --- ## [Logstash Timestamp date field filter not parsing properly](https://discuss.elastic.co/t/logstash-timestamp-date-field-filter-not-parsing-properly/91154)
**Author:** [@ViJaY\_S](https://discuss.elastic.co/u/ViJaY_S)\ **Replies:** 10\ **Last updated:** [June 28, 2017, 7:22pm UTC](https://discuss.elastic.co/t/logstash-timestamp-date-field-filter-not-parsing-properly/91154 "2017-06-28T19:22:04Z")
Hello, I'm unable to parse the Timestamp field from logstash configuration, I didn't see any errors while running configuration but the date field is not matching my index field same as in IIS logs. All my logs are ap… --- ## [Logstash with Slack plugin cannot be monitored in Kibana](https://discuss.elastic.co/t/logstash-with-slack-plugin-cannot-be-monitored-in-kibana/90348)
**Author:** [@Bob04](https://discuss.elastic.co/u/Bob04)\ **Replies:** 5\ **Last updated:** [June 28, 2017, 5:19pm UTC](https://discuss.elastic.co/t/logstash-with-slack-plugin-cannot-be-monitored-in-kibana/90348 "2017-06-28T17:19:50Z")
Hi All, After installing Slack plugin, Logstash cannot be monitored from Kibana, though it sends data to Elasticsearch. This is a Dockerfile: FROM logstash:5.3.0 RUN logstash-plugin install logstash-output-slack … --- ## [Logstash servers consuming all CPUs](https://discuss.elastic.co/t/logstash-servers-consuming-all-cpus/90477)
**Author:** [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)\ **Replies:** 4\ **Last updated:** [June 28, 2017, 4:43pm UTC](https://discuss.elastic.co/t/logstash-servers-consuming-all-cpus/90477 "2017-06-28T16:43:46Z")
Can some advise why logstash using all CPUs I am using filebeat 5.4 and logstash5.4 logstash config input { beats { client\_inactivity\_timeout =\> 86400 port =\> 5044 codec =\> multiline { pattern … --- ## [Load logstash environment in tests](https://discuss.elastic.co/t/load-logstash-environment-in-tests/91150)
**Author:** [@Conall\_Prendergast](https://discuss.elastic.co/u/Conall_Prendergast)\ **Replies:** 0\ **Last updated:** [June 28, 2017, 4:16pm UTC](https://discuss.elastic.co/t/load-logstash-environment-in-tests/91150 "2017-06-28T16:16:31Z")
Hi, I am trying to run tests using a logstash environment from a downloaded logstash. In my spec file, I "require" the "environment.rb" file of a logstash installation, and then run LogStash::Bundler.setup!({:with… --- ## [How to specify number of logstash workers in configuration file?](https://discuss.elastic.co/t/how-to-specify-number-of-logstash-workers-in-configuration-file/32972)
**Author:** [@harshafrnd4u](https://discuss.elastic.co/u/harshafrnd4u)\ **Replies:** 9\ **Last updated:** [June 28, 2017, 3:24pm UTC](https://discuss.elastic.co/t/how-to-specify-number-of-logstash-workers-in-configuration-file/32972 "2017-06-28T15:24:45Z")
is there any way to specify number of filter workers from configuration file. ? Right now I am using "service logstash start" command to start logstash, can we pass somehow pass number of worker threads to this command… --- ## [Parsing logs with Regexs](https://discuss.elastic.co/t/parsing-logs-with-regexs/91031)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 9\ **Last updated:** [June 28, 2017, 3:07pm UTC](https://discuss.elastic.co/t/parsing-logs-with-regexs/91031 "2017-06-28T15:07:12Z")
I am parsing logs that have follow along the lines of: timestamp... NEW EVENT ...timestamp I am having an issue in storing the NEW EVENT part of the log. When I try %{DATESTAMP:datestamp}... (%{WORD:acti… --- ## [How to compare for values in previous records](https://discuss.elastic.co/t/how-to-compare-for-values-in-previous-records/91131)
**Author:** [@Tejas\_Ghutukade](https://discuss.elastic.co/u/Tejas_Ghutukade)\ **Replies:** 0\ **Last updated:** [June 28, 2017, 2:52pm UTC](https://discuss.elastic.co/t/how-to-compare-for-values-in-previous-records/91131 "2017-06-28T14:52:41Z")
Hi, i'am trying to read a csv file with log data that is arranged according to timestamp. i want to check the current record value with all the previous records values and if the condition is true set a field with valu… --- ## [Logstash is it possible to drop only field?](https://discuss.elastic.co/t/logstash-is-it-possible-to-drop-only-field/91116)
**Author:** [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\ **Replies:** 3\ **Last updated:** [June 28, 2017, 2:43pm UTC](https://discuss.elastic.co/t/logstash-is-it-possible-to-drop-only-field/91116 "2017-06-28T14:43:07Z")
Hi, My usecase i want to drop a field if it doesnt satisfy the value like if \[ServerName\] ! "data-0","data-1","data-2" drop ServerName (i.e. I want to drop the ServerName field in an event if it doesn't have eith… --- ## [Multiple Grok Filters](https://discuss.elastic.co/t/multiple-grok-filters/90501)
**Author:** [@Kevin\_Wiegand](https://discuss.elastic.co/u/Kevin_Wiegand)\ **Replies:** 4\ **Last updated:** [June 28, 2017, 10:02am UTC](https://discuss.elastic.co/t/multiple-grok-filters/90501 "2017-06-28T10:02:14Z")
Hey there i got a question about my grok filter... I'm dealing with Logfiles which are looking like this: 2017-05-23 14:41:09 DEBUG \[09-exec-38\] PreparedStatement (27 ) - {pstm-100637} Executing Statement:… --- ## [Duplicating data when i am using http plollar input](https://discuss.elastic.co/t/duplicating-data-when-i-am-using-http-plollar-input/91061)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 6\ **Last updated:** [June 28, 2017, 9:53am UTC](https://discuss.elastic.co/t/duplicating-data-when-i-am-using-http-plollar-input/91061 "2017-06-28T09:53:33Z")
I am using http \_pollar input and output as elastic search to get data. From input i am getting data as xml format so i used xml filter in filter section.Everything coming properly into elastic search but when logstash r… --- ## [Logstash not working for multiple instances](https://discuss.elastic.co/t/logstash-not-working-for-multiple-instances/90876)
**Author:** [@Prateek\_Kshtriya](https://discuss.elastic.co/u/Prateek_Kshtriya)\ **Replies:** 13\ **Last updated:** [June 28, 2017, 9:15am UTC](https://discuss.elastic.co/t/logstash-not-working-for-multiple-instances/90876 "2017-06-28T09:15:03Z")
Hi All, I am getting below while trying to run second instance of logstash- Error- \[FATAL\]\[logstash.runner \] Logstash could not b e started because there is already another instance using the configured data… --- ## [Logstash kafka input plugin](https://discuss.elastic.co/t/logstash-kafka-input-plugin/91066)
**Author:** [@elk\_user1](https://discuss.elastic.co/u/elk_user1)\ **Replies:** 0\ **Last updated:** [June 28, 2017, 8:51am UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin/91066 "2017-06-28T08:51:18Z")
Hi All, I am new to logstash and was trying to use it to ingest data from an existing kafka topic. I am using the following conf : input { kafka { topics =\> "test" bootstrap\_servers =\> "X.X.X.X:90… --- ## [JDBC SQLite integration with logstash](https://discuss.elastic.co/t/jdbc-sqlite-integration-with-logstash/90801)
**Author:** [@floxtrott](https://discuss.elastic.co/u/floxtrott)\ **Replies:** 17\ **Last updated:** [June 28, 2017, 8:30am UTC](https://discuss.elastic.co/t/jdbc-sqlite-integration-with-logstash/90801 "2017-06-28T08:30:24Z")
Hi there, I have a bunch of SQlite files with multiple tables each that I want to integrate into ES. All of ES is installed on Debian using the deb packages. Integration is done using JDBC with the following input conf… --- ## [Can Logstash process archived data in the .tar.gz format?](https://discuss.elastic.co/t/can-logstash-process-archived-data-in-the-tar-gz-format/91029)
**Author:** [@eranlog](https://discuss.elastic.co/u/eranlog)\ **Replies:** 2\ **Last updated:** [June 28, 2017, 7:14am UTC](https://discuss.elastic.co/t/can-logstash-process-archived-data-in-the-tar-gz-format/91029 "2017-06-28T07:14:19Z")
Hi, I am trying to figure out a solution. - Feed Logstash with .tar.gz archive - Extract from .tar.gz archive files of interest - output files of interest to Elasticsearch Does LS have support for the .tar.gz … --- ## [Logstash bulk request](https://discuss.elastic.co/t/logstash-bulk-request/91049)
**Author:** [@sharongur](https://discuss.elastic.co/u/sharongur)\ **Replies:** 1\ **Last updated:** [June 28, 2017, 7:07am UTC](https://discuss.elastic.co/t/logstash-bulk-request/91049 "2017-06-28T07:07:52Z")
Hi, I understand that Logstash sends to ES bulk requests and manages it. My question is, is there a way to send bulks TO Logstash? At the moment im sending only single requests, which can turn to be thousands of requ… --- ## [Filebeat to logstash seems OK. Logstash to Elasticsearch seems to be having authentication issues](https://discuss.elastic.co/t/filebeat-to-logstash-seems-ok-logstash-to-elasticsearch-seems-to-be-having-authentication-issues/91022)
**Author:** [@twelsh37](https://discuss.elastic.co/u/twelsh37)\ **Replies:** 3\ **Last updated:** [June 28, 2017, 6:08am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-seems-ok-logstash-to-elasticsearch-seems-to-be-having-authentication-issues/91022 "2017-06-28T06:08:11Z")
Hi All, Once again, please be kind. im 2 days into this and still getting round the learning curve. ES 5.4.2 installed from tar.gz filebeat-5.4.2-1.x86\_64 Installed from RPM logstash-5.4.3-1.noarch Installed from RPM … --- ## [How to maintain the order in which records are read](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015)
**Author:** [@Tejas\_Ghutukade](https://discuss.elastic.co/u/Tejas_Ghutukade)\ **Replies:** 5\ **Last updated:** [June 28, 2017, 6:06am UTC](https://discuss.elastic.co/t/how-to-maintain-the-order-in-which-records-are-read/91015 "2017-06-28T06:06:45Z")
M trying to read a csv file which is sorted by time-stamp. m trying to add a field with incremental value as per the ascending order of time-stamp But the incremental value is getting assigned in random order to the … --- ## [As per field condition should change fields](https://discuss.elastic.co/t/as-per-field-condition-should-change-fields/90687)
**Author:** [@Amit2015](https://discuss.elastic.co/u/Amit2015)\ **Replies:** 3\ **Last updated:** [June 28, 2017, 6:04am UTC](https://discuss.elastic.co/t/as-per-field-condition-should-change-fields/90687 "2017-06-28T06:04:23Z")
Hi, I have csv file, as per field value next fields should be change. i.e Field name : ABC ABC field have different values: x, y, z if ABC=x value next fields are A1,A2,A3,A4,A5... which should be combine in new … --- ## [If conditional statement field source](https://discuss.elastic.co/t/if-conditional-statement-field-source/91043)
**Author:** [@elaair](https://discuss.elastic.co/u/elaair)\ **Replies:** 1\ **Last updated:** [June 28, 2017, 5:44am UTC](https://discuss.elastic.co/t/if-conditional-statement-field-source/91043 "2017-06-28T05:44:32Z")
I have defined a simple parser and would like to setup a if statement in log stash.conf to have a grok match on one of the fields. I am working towards having many log sources but am starting with only one. I have a fiel… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=954) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=956)