# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=361 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 362 --- ## [Help with parsing similar bit in a message](https://discuss.elastic.co/t/help-with-parsing-similar-bit-in-a-message/231737)
**Author:** [@SigmazGFX](https://discuss.elastic.co/u/SigmazGFX)\ **Replies:** 1\ **Last updated:** [May 8, 2020, 3:46pm UTC](https://discuss.elastic.co/t/help-with-parsing-similar-bit-in-a-message/231737 "2020-05-08T15:46:16Z")
Hey gang. I'm trying to set up GEOIP and I need to pick out he client IP from the message string. problem is that I need the client IP and while I think my patch would work, Im afraid it'll freak out when it sees the sec… --- ## [Implement Dynamic Hash Table?](https://discuss.elastic.co/t/implement-dynamic-hash-table/231625)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 2\ **Last updated:** [May 8, 2020, 3:27pm UTC](https://discuss.elastic.co/t/implement-dynamic-hash-table/231625 "2020-05-08T15:27:15Z")
Hi Logstash Gurus, I think I need to use the Memcached Plugin for a design problem… but I’m not sure. Here’s my issue: I have a Docker container Logstash instance (v7.4.0, yes I need to upgrade) currently processing s… --- ## [Multi Pipelines don't all start](https://discuss.elastic.co/t/multi-pipelines-dont-all-start/231379)
**Author:** [@DWbank](https://discuss.elastic.co/u/DWbank)\ **Replies:** 4\ **Last updated:** [May 8, 2020, 2:38pm UTC](https://discuss.elastic.co/t/multi-pipelines-dont-all-start/231379 "2020-05-08T14:38:26Z")
I for the longest time only had one pipeline but now I need to add the second. The first pipeline starts and works as it always has, the second does not open the ports in a netstat -an also looked at iptables and that i… --- ## [Kafka-Logstash-ES-Kibana](https://discuss.elastic.co/t/kafka-logstash-es-kibana/231627)
**Author:** [@A\_K1](https://discuss.elastic.co/u/A_K1)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 9:46pm UTC](https://discuss.elastic.co/t/kafka-logstash-es-kibana/231627 "2020-05-07T21:46:26Z")
Recently I integrated the ELK pipeline including Kafka, Logstash, Elastic Search and Kibana. I had a working instance of Kafka cluster, Elastic search(7.6) both in AWS and this blog just describes the steps involved with… --- ## [Output to both Kafka and Elasticsearch](https://discuss.elastic.co/t/output-to-both-kafka-and-elasticsearch/231691)
**Author:** [@omersiar](https://discuss.elastic.co/u/omersiar)\ **Replies:** 1\ **Last updated:** [May 8, 2020, 12:40pm UTC](https://discuss.elastic.co/t/output-to-both-kafka-and-elasticsearch/231691 "2020-05-08T12:40:40Z")
Hello All, I would like to know that if Logstash is capable of outputting the filtered data to both Kafka and Elasticsearch, see example below: input { } filter { # dissect scope dissect { … --- ## [Auto-run Logstash on Insert on SQL Server](https://discuss.elastic.co/t/auto-run-logstash-on-insert-on-sql-server/231706)
**Author:** [@ManishNarula](https://discuss.elastic.co/u/ManishNarula)\ **Replies:** 0\ **Last updated:** [May 8, 2020, 11:23am UTC](https://discuss.elastic.co/t/auto-run-logstash-on-insert-on-sql-server/231706 "2020-05-08T11:23:58Z")
Hi, I am working with Logstash and Elasticsearch but I have a problem. In my case I want Elasticsearch to be updated with the new data added into a SQL Database, all done via Logstash. But here is the catch i don't wan… --- ## [How to accept logs from FileBeat in Logstash?](https://discuss.elastic.co/t/how-to-accept-logs-from-filebeat-in-logstash/231508)
**Author:** [@Vlad\_Piratov](https://discuss.elastic.co/u/Vlad_Piratov)\ **Replies:** 2\ **Last updated:** [May 8, 2020, 11:22am UTC](https://discuss.elastic.co/t/how-to-accept-logs-from-filebeat-in-logstash/231508 "2020-05-08T11:22:25Z")
Hi! How do you accept logs from Filebeam in Logstash? I have configured configurations in the yml file for output in Logstash.. In the settings, Logstash specified that without a filter, logs go directly to elasticsearch… --- ## [Logstash - SNMP Input - Trouble Converting / Using MIB Files](https://discuss.elastic.co/t/logstash-snmp-input-trouble-converting-using-mib-files/230374)
**Author:** [@Sinnott](https://discuss.elastic.co/u/Sinnott)\ **Replies:** 4\ **Last updated:** [May 8, 2020, 10:09am UTC](https://discuss.elastic.co/t/logstash-snmp-input-trouble-converting-using-mib-files/230374 "2020-05-08T10:09:46Z")
Hi Elastic-Forum, I am trying to use the Logstash SNMP Input plugin to use OID's from a MIB file but am having trouble. I have tried to follow the guide below and can make standard OID calls, but not device specific on… --- ## [Logstash plc4x Input Plugin output to Elastic not Working](https://discuss.elastic.co/t/logstash-plc4x-input-plugin-output-to-elastic-not-working/231677)
**Author:** [@baltweck](https://discuss.elastic.co/u/baltweck)\ **Replies:** 0\ **Last updated:** [May 8, 2020, 8:33am UTC](https://discuss.elastic.co/t/logstash-plc4x-input-plugin-output-to-elastic-not-working/231677 "2020-05-08T08:33:11Z")
Iam Using the apache plc4x logstash plugin, as far as i see in logstash debug output my job will be polled and the opc ua namespaces seems to be right for my plc. I dont get a Index into Elasticsearch, OPC UA Server is … --- ## [Merge two fields into one](https://discuss.elastic.co/t/merge-two-fields-into-one/231592)
**Author:** [@alberto1](https://discuss.elastic.co/u/alberto1)\ **Replies:** 2\ **Last updated:** [May 8, 2020, 6:17am UTC](https://discuss.elastic.co/t/merge-two-fields-into-one/231592 "2020-05-08T06:17:49Z")
Hi, First of all sorry for ask this, I know that is lots of info about this but i cannot make it work. So what I would like to do is combine the fields hostname and path. "@timestamp" =\> 2020-05-07T16:45:36.042Z } { … --- ## [Logstash XML parsing failure due to no namespace defined](https://discuss.elastic.co/t/logstash-xml-parsing-failure-due-to-no-namespace-defined/230890)
**Author:** [@Saravana\_Maadavan](https://discuss.elastic.co/u/Saravana_Maadavan)\ **Replies:** 3\ **Last updated:** [May 7, 2020, 8:51pm UTC](https://discuss.elastic.co/t/logstash-xml-parsing-failure-due-to-no-namespace-defined/230890 "2020-05-07T20:51:31Z")
Hello, I would like to store XMLs in elastic, so when I give "store\_xml =\> true", I get a parsing exception because my whole XML is wrapped by a tag "\" and the prefix "C" is not having any definition unfortun… --- ## [Last IP from possible comma separated list of IPs](https://discuss.elastic.co/t/last-ip-from-possible-comma-separated-list-of-ips/231583)
**Author:** [@clarkritchie](https://discuss.elastic.co/u/clarkritchie)\ **Replies:** 2\ **Last updated:** [May 7, 2020, 7:43pm UTC](https://discuss.elastic.co/t/last-ip-from-possible-comma-separated-list-of-ips/231583 "2020-05-07T19:43:17Z")
I'm struggling to match the last IP in pattern which may contain a comma separated list of IPs. e.g. both of these appear in my logs: foo\\t102.35.174.94\\tbar 102.35.174.94 is my desired match foo\\t102.35.174.94,66.… --- ## [Using Mutate filter not able to convert string to integer](https://discuss.elastic.co/t/using-mutate-filter-not-able-to-convert-string-to-integer/231613)
**Author:** [@souravatta](https://discuss.elastic.co/u/souravatta)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 7:19pm UTC](https://discuss.elastic.co/t/using-mutate-filter-not-able-to-convert-string-to-integer/231613 "2020-05-07T19:19:40Z")
--- ## [Lumberjack plugin](https://discuss.elastic.co/t/lumberjack-plugin/230809)
**Author:** [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)\ **Replies:** 2\ **Last updated:** [May 7, 2020, 7:18pm UTC](https://discuss.elastic.co/t/lumberjack-plugin/230809 "2020-05-07T19:18:32Z")
I am trying to send Metricbeat ----\> Logstash-1 -------\> Logstash-2 -------\> Elasticsearch. I tried both tcp and Lumberjack protocols. Successfully start both logstshes but not found data in elasticsearch. \`\`\`\`\`\`\`\`\`\`lO… --- ## [Logstash SQS output failing makes pipeline stop working entirely](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425)
**Author:** [@robertknutzen](https://discuss.elastic.co/u/robertknutzen)\ **Replies:** 4\ **Last updated:** [May 7, 2020, 6:04pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425 "2020-05-07T18:04:07Z")
I have a logstash pipeline that uses an SQS output. I'm seeing errors where logstash will just stop working if it cannot connect to SQS. The logs seem to indicate that logstash attempts to restart, fails and then just do… --- ## [Difference in time between timestamp and @timestamp](https://discuss.elastic.co/t/difference-in-time-between-timestamp-and-timestamp/231456)
**Author:** [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Replies:** 4\ **Last updated:** [May 7, 2020, 4:04pm UTC](https://discuss.elastic.co/t/difference-in-time-between-timestamp-and-timestamp/231456 "2020-05-07T16:04:33Z")
I have written the date filter as follows date { match =\> \["timestamp","yyyy-MM-dd HH:mm:ss.SSSSSSSSS"\] target =\> "@timestamp" timezone =\> "Asia/Kolkata" } timestamp : 2020-04-17 12:10:15.… --- ## [How can I use %{+yyyy.mm.dd} in input?](https://discuss.elastic.co/t/how-can-i-use-yyyy-mm-dd-in-input/231535)
**Author:** [@jinjinbari](https://discuss.elastic.co/u/jinjinbari)\ **Replies:** 1\ **Last updated:** [May 7, 2020, 3:55pm UTC](https://discuss.elastic.co/t/how-can-i-use-yyyy-mm-dd-in-input/231535 "2020-05-07T15:55:20Z")
Hello. I want to use date variable in input, like below input { elasticsearch{ hosts =\> \["localhost"\] index =\> "test-%{+yyyy-MM-dd}" codec =\> "json" } } but it occur error like this. Error: \[404\] {"err… --- ## [Problem with Ruby Filter](https://discuss.elastic.co/t/problem-with-ruby-filter/231408)
**Author:** [@rildo](https://discuss.elastic.co/u/rildo)\ **Replies:** 6\ **Last updated:** [May 7, 2020, 3:23pm UTC](https://discuss.elastic.co/t/problem-with-ruby-filter/231408 "2020-05-07T15:23:40Z")
Hello People, I would like a help with my problem. Currently I have a database with postgresql and I have some queries in logstash to extract some information and send it to elasticsearch.One query return a list of da… --- ## [Logstash log level change requires restart?](https://discuss.elastic.co/t/logstash-log-level-change-requires-restart/231568)
**Author:** [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 3:01pm UTC](https://discuss.elastic.co/t/logstash-log-level-change-requires-restart/231568 "2020-05-07T15:01:53Z")
If I change log level in logstash.yaml, it seems restart is required. Will it take effect without restart? --- ## [Error while running logstash: org.logstash.instrument.metrics.gauge.LazyDelegatingGauge](https://discuss.elastic.co/t/error-while-running-logstash-org-logstash-instrument-metrics-gauge-lazydelegatinggauge/231552)
**Author:** [@elen](https://discuss.elastic.co/u/elen)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 1:43pm UTC](https://discuss.elastic.co/t/error-while-running-logstash-org-logstash-instrument-metrics-gauge-lazydelegatinggauge/231552 "2020-05-07T13:43:31Z")
Hello, Trying to send data Logstash -\> Elasticsearch ( both are running in docker, configs are tested and are ok) and I'm experiencing issue that actual data is not sent (this is the only warning I get while running log… --- ## [How can filter diiferent logs with different index generated by logstash in kibana](https://discuss.elastic.co/t/how-can-filter-diiferent-logs-with-different-index-generated-by-logstash-in-kibana/231066)
**Author:** [@Ekta](https://discuss.elastic.co/u/Ekta)\ **Replies:** 9\ **Last updated:** [May 7, 2020, 1:40pm UTC](https://discuss.elastic.co/t/how-can-filter-diiferent-logs-with-different-index-generated-by-logstash-in-kibana/231066 "2020-05-07T13:40:19Z")
Hi I take log file from different paths and make different config files according to it. Then for output all indexes are created as given in config file but while in kibana it filter with different index it show all lo… --- ## [Delete Duplicate Documents with Elasticsearch and Ruby](https://discuss.elastic.co/t/delete-duplicate-documents-with-elasticsearch-and-ruby/231479)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 4\ **Last updated:** [May 7, 2020, 12:34pm UTC](https://discuss.elastic.co/t/delete-duplicate-documents-with-elasticsearch-and-ruby/231479 "2020-05-07T12:34:46Z")
I use a script ruby for duplicate Document input { beats { port =\> "5044" } } filter { csv { separator =\> "," columns =\> \["chaine", "job", "date\_plan", "statut", "date\_debut"… --- ## [Filtering csv headers with different lengths](https://discuss.elastic.co/t/filtering-csv-headers-with-different-lengths/231526)
**Author:** [@eeijlar](https://discuss.elastic.co/u/eeijlar)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 11:47am UTC](https://discuss.elastic.co/t/filtering-csv-headers-with-different-lengths/231526 "2020-05-07T11:47:21Z")
I have two slightly different csv headers i would like to catch under a filter: ACTION,DISCRIMINATOR,CHECKPOINT\_ID,INSTALLATION\_ID,NUMERIC\_VALUE,STRING\_VALUE,COMPONENT\_NAME,TRANSACTION\_ID,STATISTIC\_DURATION,STATISTIC\_NA… --- ## [Proofpoint sendmail/ syslog](https://discuss.elastic.co/t/proofpoint-sendmail-syslog/231522)
**Author:** [@jamesy281](https://discuss.elastic.co/u/jamesy281)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 11:38am UTC](https://discuss.elastic.co/t/proofpoint-sendmail-syslog/231522 "2020-05-07T11:38:29Z")
Hey there, I am currently ingesting Proofpoint mail/ filter/ security, etc logs into our Rapid7 SIEM. I found though that I am unable to do a simple log search for say messages from & to a user as proofpoint/ sendmail b… --- ## [Using logstash kafka plugin 10.2.0 with logstash 6.8.8](https://discuss.elastic.co/t/using-logstash-kafka-plugin-10-2-0-with-logstash-6-8-8/231521)
**Author:** [@ItamarBenjamin](https://discuss.elastic.co/u/ItamarBenjamin)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 11:31am UTC](https://discuss.elastic.co/t/using-logstash-kafka-plugin-10-2-0-with-logstash-6-8-8/231521 "2020-05-07T11:31:24Z")
Is it possible to use versions 10.2.0 of the kafka plugins (especially kafka input plugin) with logstash 6.8.8? when i run the upgrade command from the 6.8.8 docker this is what i get: bash-4.2$ logstash-plugin update … --- ## [Output to indexes based on \[agent.type\] (or any conditionals) not working](https://discuss.elastic.co/t/output-to-indexes-based-on-agent-type-or-any-conditionals-not-working/231363)
**Author:** [@SteveParker](https://discuss.elastic.co/u/SteveParker)\ **Replies:** 5\ **Last updated:** [May 7, 2020, 10:57am UTC](https://discuss.elastic.co/t/output-to-indexes-based-on-agent-type-or-any-conditionals-not-working/231363 "2020-05-07T10:57:26Z")
Sorry to ask this, I have searched long for an answer but not found anything. My winlogbeat, packetbeat and metricbeat are all sent to logstash on 5044. I want logstash to output them to dedicated indexes. I am using th… --- ## [Logstash cannot support postgres geometry type](https://discuss.elastic.co/t/logstash-cannot-support-postgres-geometry-type/231492)
**Author:** [@ohoroyoi](https://discuss.elastic.co/u/ohoroyoi)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 9:01am UTC](https://discuss.elastic.co/t/logstash-cannot-support-postgres-geometry-type/231492 "2020-05-07T09:01:23Z")
There is a column of geometry type in the postgres table. I checked that the rest of the columns worked fine postgres-\> logstash-\> elasticsearch. However, I am not sure how to cast the geometry type. Please help me. Can… --- ## [How to use custom grok pattern for syslog input](https://discuss.elastic.co/t/how-to-use-custom-grok-pattern-for-syslog-input/231494)
**Author:** [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Replies:** 0\ **Last updated:** [May 7, 2020, 9:06am UTC](https://discuss.elastic.co/t/how-to-use-custom-grok-pattern-for-syslog-input/231494 "2020-05-07T09:06:05Z")
Hi, I listening to the port 31230 to get syslogs from a router. The logs are coming in but it doesnt use my grok filter. Kibana uses some default index pattern for syslogs. input { syslog { type =\> "syslog"… --- ## [Logstash Aggregate map working only for few rows while merging two data sources](https://discuss.elastic.co/t/logstash-aggregate-map-working-only-for-few-rows-while-merging-two-data-sources/231293)
**Author:** [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Replies:** 1\ **Last updated:** [May 7, 2020, 9:02am UTC](https://discuss.elastic.co/t/logstash-aggregate-map-working-only-for-few-rows-while-merging-two-data-sources/231293 "2020-05-07T09:02:55Z")
I am using the latest version of logstash(7.6.2). I am trying to merge rows from two different files using a common ID. Find below the samples of the two data sources. Also, find below my desired output. Sample Data 1 w… --- ## [Permission denied error while collecting logs from external ip](https://discuss.elastic.co/t/permission-denied-error-while-collecting-logs-from-external-ip/231112)
**Author:** [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Replies:** 9\ **Last updated:** [May 7, 2020, 7:35am UTC](https://discuss.elastic.co/t/permission-denied-error-while-collecting-logs-from-external-ip/231112 "2020-05-07T07:35:24Z")
Hi, I am trying to collect syslogs from a juniper router. I have configured the router to send logs to the IP where Logstash is running. I have written the input config for Logstash as: input { syslog { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=360) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=362)