# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=327 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 328 --- ## [Managing an arbitrary number of pipelines](https://discuss.elastic.co/t/managing-an-arbitrary-number-of-pipelines/241745)
**Author:** [@Sean.Lim](https://discuss.elastic.co/u/Sean.Lim)\ **Replies:** 1\ **Last updated:** [July 18, 2020, 5:58pm UTC](https://discuss.elastic.co/t/managing-an-arbitrary-number-of-pipelines/241745 "2020-07-18T17:58:29Z")
I am working on a project with these requirements: Each event contains a field identifying the syslog server to send the output to. Transport failure to any server should not block output to other servers. From requir… --- ## [We are not receiving the log from kafka to logstash](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472)
**Author:** [@saravananveera](https://discuss.elastic.co/u/saravananveera)\ **Replies:** 3\ **Last updated:** [July 18, 2020, 5:08am UTC](https://discuss.elastic.co/t/we-are-not-receiving-the-log-from-kafka-to-logstash/241472 "2020-07-18T05:08:28Z")
Hi I have received the logs from beat to Kafaka, but i can't able to receive logs from kafka to logstash . Kafka log: \[root@ip kafka\]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sscplblog… --- ## [\[WEIRD\] Only some Pipelines in Pipelines.yml stopped working, without making any changes to the conf](https://discuss.elastic.co/t/weird-only-some-pipelines-in-pipelines-yml-stopped-working-without-making-any-changes-to-the-conf/241719)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [July 18, 2020, 1:35am UTC](https://discuss.elastic.co/t/weird-only-some-pipelines-in-pipelines-yml-stopped-working-without-making-any-changes-to-the-conf/241719 "2020-07-18T01:35:22Z")
Hi, In monday about half of pipelines in my pipelines.yml, stopped working. I didn make any changes to the single pipelines configurations and to pipelines.yml. When I run them separately they work, when I run them vi… --- ## [Logstash on docker: configure sincedb file to use the last indexed position whenever container restarts/new](https://discuss.elastic.co/t/logstash-on-docker-configure-sincedb-file-to-use-the-last-indexed-position-whenever-container-restarts-new/241537)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 11\ **Last updated:** [July 17, 2020, 9:22pm UTC](https://discuss.elastic.co/t/logstash-on-docker-configure-sincedb-file-to-use-the-last-indexed-position-whenever-container-restarts-new/241537 "2020-07-17T21:22:57Z")
Hi Team I mounted the logstash sincedb position to a persistent volume assuming logstash will remember the position of last indexed log. But every time I create/restart a container it's pushing all the logs again to ela… --- ## [Wildcards in logstash remove\_field](https://discuss.elastic.co/t/wildcards-in-logstash-remove-field/241644)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 11\ **Last updated:** [July 17, 2020, 8:40pm UTC](https://discuss.elastic.co/t/wildcards-in-logstash-remove-field/241644 "2020-07-17T20:40:27Z")
Hi All, I have a data source with almost 692 fields, out of which only 200 fields are valid, i want to remove those fields , i tried using below one, but no luck mutate { remove\_field =\> \[ ".value" \] } all the unwant… --- ## [Logstash filter basted on log file age](https://discuss.elastic.co/t/logstash-filter-basted-on-log-file-age/241350)
**Author:** [@Bader](https://discuss.elastic.co/u/Bader)\ **Replies:** 6\ **Last updated:** [July 17, 2020, 4:16pm UTC](https://discuss.elastic.co/t/logstash-filter-basted-on-log-file-age/241350 "2020-07-17T16:16:48Z")
Hello. I am running ELK 6.6 on a CentOS 7 box. I have filebeat configured on a Windows machine to forward specific logs to Logstash. The problem I have is that my Logstash filter has a throttling mechanism setup as to … --- ## [Write log into 2 elasticsearch in realtime](https://discuss.elastic.co/t/write-log-into-2-elasticsearch-in-realtime/241111)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 4\ **Last updated:** [July 17, 2020, 3:44pm UTC](https://discuss.elastic.co/t/write-log-into-2-elasticsearch-in-realtime/241111 "2020-07-17T15:44:53Z")
I would like to write with logstash into elasticsearch pseudonymized data in a way that it can no be linked to a single subject without the use of additional data. Also this additional data should be kept separate … --- ## [Migrating 3 millions of records from RDBMS to Elastic Search using logstash](https://discuss.elastic.co/t/migrating-3-millions-of-records-from-rdbms-to-elastic-search-using-logstash/240427)
**Author:** [@rao7806](https://discuss.elastic.co/u/rao7806)\ **Replies:** 7\ **Last updated:** [July 17, 2020, 3:27pm UTC](https://discuss.elastic.co/t/migrating-3-millions-of-records-from-rdbms-to-elastic-search-using-logstash/240427 "2020-07-17T15:27:33Z")
Hi All, We are trying to migrate around 3 million records from oracle to Elastic Search using Logstash. We are applying a couple of jdbc\_streaming filters as a part of our logstash script, one to load connecting nested… --- ## [Further details of how a pipeline configuration works](https://discuss.elastic.co/t/further-details-of-how-a-pipeline-configuration-works/241687)
**Author:** [@damasu](https://discuss.elastic.co/u/damasu)\ **Replies:** 0\ **Last updated:** [July 17, 2020, 3:24pm UTC](https://discuss.elastic.co/t/further-details-of-how-a-pipeline-configuration-works/241687 "2020-07-17T15:24:48Z")
I wondering about my pipeline setup: - pipeline.id: bal path.config: "/usr/share/logstash/pipeline/1\*.conf" pipeline.workers: 24 pipeline.batch.size: 2000 - pipeline.id: soul path.config: "/usr/share/logstash/pi… --- ## [Logstash memcached not returning value](https://discuss.elastic.co/t/logstash-memcached-not-returning-value/241649)
**Author:** [@WhensTheWeekend](https://discuss.elastic.co/u/WhensTheWeekend)\ **Replies:** 0\ **Last updated:** [July 17, 2020, 1:31pm UTC](https://discuss.elastic.co/t/logstash-memcached-not-returning-value/241649 "2020-07-17T13:31:41Z")
Hi, New to memcached and am using is as a filter in logstash as follows; memcached { id=\>"getconditionterminology" hosts =\> \["${memcacheserver}"\] namespace =\> "code\_concept" get =\> { "%{rsnstcode}" =\> "\[mm\_rsnstcod… --- ## [Cannot receive logs from Filebeat in Logstash, SSL Problem](https://discuss.elastic.co/t/cannot-receive-logs-from-filebeat-in-logstash-ssl-problem/241496)
**Author:** [@ngerpiel](https://discuss.elastic.co/u/ngerpiel)\ **Replies:** 2\ **Last updated:** [July 17, 2020, 12:35pm UTC](https://discuss.elastic.co/t/cannot-receive-logs-from-filebeat-in-logstash-ssl-problem/241496 "2020-07-17T12:35:31Z")
I have a problem with sending logs from filebeat to logstash node using ssl certificates. The error I get is this: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL\_internal:SSLV3\_ALERT\_BAD\_CERTI… --- ## [Have logstash automatically parse any .json files](https://discuss.elastic.co/t/have-logstash-automatically-parse-any-json-files/241610)
**Author:** [@elk6](https://discuss.elastic.co/u/elk6)\ **Replies:** 6\ **Last updated:** [July 17, 2020, 11:55am UTC](https://discuss.elastic.co/t/have-logstash-automatically-parse-any-json-files/241610 "2020-07-17T11:55:54Z")
I'm planning on shipping lots of logs into logstash. Is it possible to configure logstash in such a way so that only if a file ends with .json it would go through: json { source =\> "message" } Thanks ahea… --- ## [How to skip second row in log file while processing it through CSV filter](https://discuss.elastic.co/t/how-to-skip-second-row-in-log-file-while-processing-it-through-csv-filter/241314)
**Author:** [@Ashish\_kapoor](https://discuss.elastic.co/u/Ashish_kapoor)\ **Replies:** 4\ **Last updated:** [July 17, 2020, 8:49am UTC](https://discuss.elastic.co/t/how-to-skip-second-row-in-log-file-while-processing-it-through-csv-filter/241314 "2020-07-17T08:49:29Z")
Hi I want to skip second line in my logs while processing these logs with CSV filter. second line of my logs looks like this: #Remark Values: ComponentType="OAM" ReleaseVersion="11.1.1.9.0" and logstash conf for pro… --- ## [Errors after updating plugins in 6.8](https://discuss.elastic.co/t/errors-after-updating-plugins-in-6-8/241331)
**Author:** [@jong99](https://discuss.elastic.co/u/jong99)\ **Replies:** 1\ **Last updated:** [July 17, 2020, 8:40am UTC](https://discuss.elastic.co/t/errors-after-updating-plugins-in-6-8/241331 "2020-07-17T08:40:24Z")
I'm using Logstash 6.8 and am using some updated versions of plugins which have been updated with the command: logstash-plugin update logstash-filter-kv logstash-input-s3 logstash-filter-json This has worked fine for a… --- ## [Logstash date filter can not convert certain range of unix time](https://discuss.elastic.co/t/logstash-date-filter-can-not-convert-certain-range-of-unix-time/241425)
**Author:** [@Rex\_lin](https://discuss.elastic.co/u/Rex_lin)\ **Replies:** 2\ **Last updated:** [July 17, 2020, 6:09am UTC](https://discuss.elastic.co/t/logstash-date-filter-can-not-convert-certain-range-of-unix-time/241425 "2020-07-17T06:09:46Z")
I have met some date filter problem, when tried to convert my unix time to seconds. As my picture, when I input "2903299199" , I except the result is "2061-12-31T23:59:59+00:00 in ISO 8601, however it returned failure… --- ## [Aggregate filter gets different output even with same input](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394)
**Author:** [@Aberry](https://discuss.elastic.co/u/Aberry)\ **Replies:** 4\ **Last updated:** [July 17, 2020, 2:33am UTC](https://discuss.elastic.co/t/aggregate-filter-gets-different-output-even-with-same-input/241394 "2020-07-17T02:33:42Z")
I encountered similar problem again while using a mass of test data. This time to set pipeline.workers=1 and pipeline.java\_execution = false in logstash.yml cannot help anymore. I have 2 input files like below - inboun… --- ## [Convert winbeats back to Raw windows event](https://discuss.elastic.co/t/convert-winbeats-back-to-raw-windows-event/241567)
**Author:** [@bcl](https://discuss.elastic.co/u/bcl)\ **Replies:** 0\ **Last updated:** [July 17, 2020, 1:56am UTC](https://discuss.elastic.co/t/convert-winbeats-back-to-raw-windows-event/241567 "2020-07-17T01:56:16Z")
Hi All, We currently use ELK in the following way winbeats,filebeats,syslog -\> logstash ingest nodes -\> kafka cluster -\> logstash grok nodes -\> Elastic cluster We have a requirement to send winbeats logs to our securi… --- ## [Datestamp to timestamp](https://discuss.elastic.co/t/datestamp-to-timestamp/241550)
**Author:** [@merlock](https://discuss.elastic.co/u/merlock)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 11:12pm UTC](https://discuss.elastic.co/t/datestamp-to-timestamp/241550 "2020-07-16T23:12:35Z")
hello all, i have the next format of log 06/17/2020 00:00:00 - 1.6.250.107 o00044696e20022001200940005 my filter match =\> { "message" =\> ".%{DATESTAMP:LOGTIME}.-.\*%{IP}.\*w(?.{10})(?\.{5})(?.{3})(?\ **Author:** [@elk6](https://discuss.elastic.co/u/elk6)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 9:49pm UTC](https://discuss.elastic.co/t/make-logstash-filter-by-name/241526 "2020-07-16T21:49:59Z") I have a log file called "/var/log/commands.log" that I'm trying to separate into fields with logstash & grok. I've got it working. Now, I'm trying to make logstash only do this to the file "/var/log/commands.log" and no… --- ## [Logstash Netflow Codec Issues](https://discuss.elastic.co/t/logstash-netflow-codec-issues/241500)
**Author:** [@csgeek](https://discuss.elastic.co/u/csgeek)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 8:26pm UTC](https://discuss.elastic.co/t/logstash-netflow-codec-issues/241500 "2020-07-16T20:26:26Z")
I'm having issues handling netflow data through the logstash codec. I was wondering if anyone had any insight that could help. Logstash version: 7.6.2 (I'm happy to upgrade if need be not tied to a specific version) f… --- ## [Tag \_grokparsefailure when parsing logs via logstash](https://discuss.elastic.co/t/tag-grokparsefailure-when-parsing-logs-via-logstash/241489)
**Author:** [@Rahul\_Aggarwal](https://discuss.elastic.co/u/Rahul_Aggarwal)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 8:18pm UTC](https://discuss.elastic.co/t/tag-grokparsefailure-when-parsing-logs-via-logstash/241489 "2020-07-16T20:18:58Z")
Below are 2 logs of same cloudfront distribution, patterns of logs same but when processing with logstash same filter showing tag \_grokparsefailure for first line but no any failure for second line, Please help to remove… --- ## [Logstash Patterns - Firewalls - Best Place to Find it](https://discuss.elastic.co/t/logstash-patterns-firewalls-best-place-to-find-it/239107)
**Author:** [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Replies:** 4\ **Last updated:** [July 16, 2020, 8:15pm UTC](https://discuss.elastic.co/t/logstash-patterns-firewalls-best-place-to-find-it/239107 "2020-07-16T20:15:43Z")
My use case is Cisco ASA firewall logs but I think these questions apply more broadly. I am trying do the parsing of Cisco ASA logs in logstash, not using Filebeat. I'd like to not reinvent the wheel so where can I f… --- ## [Logstash sFlow Codec Input Syntax :: Implement "optional\_removed\_field"?](https://discuss.elastic.co/t/logstash-sflow-codec-input-syntax-implement-optional-removed-field/241527)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 7:25pm UTC](https://discuss.elastic.co/t/logstash-sflow-codec-input-syntax-implement-optional-removed-field/241527 "2020-07-16T19:25:41Z")
Hi Logstash Megaminds, I'm using my Logstash instance as an sFlow collector, and it works great. When I read the documentation on the sFlow Codec, I see this: By default \[some sFlow\] fields are removed from \[every\] e… --- ## [Logstash date filter match timestamp issue](https://discuss.elastic.co/t/logstash-date-filter-match-timestamp-issue/241506)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 5\ **Last updated:** [July 16, 2020, 7:24pm UTC](https://discuss.elastic.co/t/logstash-date-filter-match-timestamp-issue/241506 "2020-07-16T19:24:08Z")
Hi Team I want to match the timestamp with date from logs. But in kibana I see different time for timestamp and logdate. why is this happening and how to resolve this ? Kibana data Logstash.conf Any suggestions w… --- ## [Logstash pipeline workers and imap input plugin](https://discuss.elastic.co/t/logstash-pipeline-workers-and-imap-input-plugin/241534)
**Author:** [@Diptee](https://discuss.elastic.co/u/Diptee)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 7:20pm UTC](https://discuss.elastic.co/t/logstash-pipeline-workers-and-imap-input-plugin/241534 "2020-07-16T19:20:34Z")
Hi, We have configured logstash(using default pipeline configuration). We have just one pipeline running. When I start logstash I see it has 4 pipeline workers and batch size of 125. I am using IMAP input plugin to pro… --- ## [Few worker threads keep running even after logstash is shutdown](https://discuss.elastic.co/t/few-worker-threads-keep-running-even-after-logstash-is-shutdown/241535)
**Author:** [@Diptee](https://discuss.elastic.co/u/Diptee)\ **Replies:** 0\ **Last updated:** [July 16, 2020, 7:14pm UTC](https://discuss.elastic.co/t/few-worker-threads-keep-running-even-after-logstash-is-shutdown/241535 "2020-07-16T19:14:34Z")
Hi, We have configured logstash(using default pipeline configuration). We have just one pipeline running. When I start logstash I see it has 4 pipeline workers and batch size of 125. I am using IMAP input plugin to pro… --- ## [Logstash multiple inputs parallel](https://discuss.elastic.co/t/logstash-multiple-inputs-parallel/241523)
**Author:** [@dbargo](https://discuss.elastic.co/u/dbargo)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 6:10pm UTC](https://discuss.elastic.co/t/logstash-multiple-inputs-parallel/241523 "2020-07-16T18:10:51Z")
If I have multiple inputs in my logstash and one output does it collects the logs from the inputs in serial way ( one by one ) or in parallel way ( all of them in the same time) ? Couldn't find an answer to it --- ## [GeoIP lookup failed](https://discuss.elastic.co/t/geoip-lookup-failed/241414)
**Author:** [@Alex\_Liu](https://discuss.elastic.co/u/Alex_Liu)\ **Replies:** 3\ **Last updated:** [July 16, 2020, 5:51pm UTC](https://discuss.elastic.co/t/geoip-lookup-failed/241414 "2020-07-16T17:51:01Z")
Hi, how to debug GeoIP lookup failure? For below attached screenshot, why the first clientIP: 123.58.60.5 lookup failed, but the second one 112.85.231.147 succeed? the configuration is simple, and the GeoLIte2-city.m… --- ## [Logstash on k8s - high memory usage](https://discuss.elastic.co/t/logstash-on-k8s-high-memory-usage/241299)
**Author:** [@zachshani](https://discuss.elastic.co/u/zachshani)\ **Replies:** 4\ **Last updated:** [July 16, 2020, 5:45pm UTC](https://discuss.elastic.co/t/logstash-on-k8s-high-memory-usage/241299 "2020-07-16T17:45:24Z")
Hi, I'm implementing logstash as a deployment (pod) on k8s. I've noticed that it consumes high memory - even when no data is being send to it. I've tried setting jvm.options (Xms4g & Xmx4g), but I got same results - h… --- ## [How logstash read multiline data line by line](https://discuss.elastic.co/t/how-logstash-read-multiline-data-line-by-line/241498)
**Author:** [@ashish2](https://discuss.elastic.co/u/ashish2)\ **Replies:** 0\ **Last updated:** [July 16, 2020, 3:58pm UTC](https://discuss.elastic.co/t/how-logstash-read-multiline-data-line-by-line/241498 "2020-07-16T15:58:53Z")
i am new to elk .am phasing some problem about logstash ,tell me that i can solve this problem with logstash or not? i have a csv file like this. a: rajat b: 123,321a,543, 123a....... a: ashish b: 878, 543, 321, 123… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=326) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=328)