# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=487 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 488 --- ## [S3 input with multiple log types](https://discuss.elastic.co/t/s3-input-with-multiple-log-types/196810)
**Author:** [@uma\_rengasamy](https://discuss.elastic.co/u/uma_rengasamy)\ **Replies:** 0\ **Last updated:** [August 26, 2019, 4:27pm UTC](https://discuss.elastic.co/t/s3-input-with-multiple-log-types/196810 "2019-08-26T16:27:55Z")
Does input supports multiple codec , since we have multiple log type in single s3 bucket and s3 prefix doesn't support regex , How can we add type for each logs ? --- ## [Logstash Configuration error - logstash shut down](https://discuss.elastic.co/t/logstash-configuration-error-logstash-shut-down/196801)
**Author:** [@deemanu](https://discuss.elastic.co/u/deemanu)\ **Replies:** 1\ **Last updated:** [August 26, 2019, 3:37pm UTC](https://discuss.elastic.co/t/logstash-configuration-error-logstash-shut-down/196801 "2019-08-26T15:37:15Z")
Hi , I am trying to parse a log line (log file customized by me ) but there is some error msg Sending Logstash logs to C:/Users/mau/Desktop/ELK/logstash-7.3.0/logs which is now configured via log4j2.properties \[2019-08… --- ## [Grokking works in debugger but differs in Logstash](https://discuss.elastic.co/t/grokking-works-in-debugger-but-differs-in-logstash/196793)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 2\ **Last updated:** [August 26, 2019, 2:48pm UTC](https://discuss.elastic.co/t/grokking-works-in-debugger-but-differs-in-logstash/196793 "2019-08-26T14:48:11Z")
Hello, I'm using a regex which would remove the first word and parse the rest. grok { match =\> { "resource" =\> "/\[^/\]+(/|)(?\\[^/\]+)?(/%{GREEDYDATA:resource\_path})?" } } Test Messages: /list/Lighter-t… --- ## [Need help to parse xml file with logstash](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464)
**Author:** [@ROok](https://discuss.elastic.co/u/ROok)\ **Replies:** 11\ **Last updated:** [August 26, 2019, 2:30pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464 "2019-08-26T14:30:10Z")
Hi All, I need help for parsing my xml file My xml file look like this: \ \ \3312348555\ \308014204145695\ … --- ## [Can we use another database with same table after sync is completed in Elasticsearch using Logstash](https://discuss.elastic.co/t/can-we-use-another-database-with-same-table-after-sync-is-completed-in-elasticsearch-using-logstash/196790)
**Author:** [@subhashpurohit](https://discuss.elastic.co/u/subhashpurohit)\ **Replies:** 0\ **Last updated:** [August 26, 2019, 2:12pm UTC](https://discuss.elastic.co/t/can-we-use-another-database-with-same-table-after-sync-is-completed-in-elasticsearch-using-logstash/196790 "2019-08-26T14:12:21Z")
I have imported all the data from a table in postgres master database to Elasticsearch using Logstash JDBC plugin. Now I want to use one of my slave database which have only read permission. I am not sure if I will chan… --- ## [Assistance with grokparsefailure](https://discuss.elastic.co/t/assistance-with-grokparsefailure/196720)
**Author:** [@unknown\_user](https://discuss.elastic.co/u/unknown_user)\ **Replies:** 2\ **Last updated:** [August 26, 2019, 2:06pm UTC](https://discuss.elastic.co/t/assistance-with-grokparsefailure/196720 "2019-08-26T14:06:54Z")
Hi All, Very new to using logstash, so would appreciate any help. Would would be the best way to parse this file? I'm still getting some grokparsefailures but it is picking up everything except besides the "cpu" which … --- ## [Convert array with 2 equal values to single value](https://discuss.elastic.co/t/convert-array-with-2-equal-values-to-single-value/196763)
**Author:** [@programagor](https://discuss.elastic.co/u/programagor)\ **Replies:** 3\ **Last updated:** [August 26, 2019, 2:02pm UTC](https://discuss.elastic.co/t/convert-array-with-2-equal-values-to-single-value/196763 "2019-08-26T14:02:05Z")
Greetings My events sometimes have a field "foo" equal to a single value, sometimes an array of multiple distinct values, and sometimes an array of exactly two equal values. I would like to convert the last case to the… --- ## [Logstash 7.3. ruby undefined method split](https://discuss.elastic.co/t/logstash-7-3-ruby-undefined-method-split/196778)
**Author:** [@jimmyvalmer19](https://discuss.elastic.co/u/jimmyvalmer19)\ **Replies:** 4\ **Last updated:** [August 26, 2019, 2:00pm UTC](https://discuss.elastic.co/t/logstash-7-3-ruby-undefined-method-split/196778 "2019-08-26T14:00:08Z")
Hi, I don't know how to use the filebeat fields listed here: https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html In logstash. I need to extract the application name from "log.file.path" fi… --- ## [Conditional on elasticsearch output not working as expected](https://discuss.elastic.co/t/conditional-on-elasticsearch-output-not-working-as-expected/196772)
**Author:** [@glenacota](https://discuss.elastic.co/u/glenacota)\ **Replies:** 3\ **Last updated:** [August 26, 2019, 1:54pm UTC](https://discuss.elastic.co/t/conditional-on-elasticsearch-output-not-working-as-expected/196772 "2019-08-26T13:54:32Z")
Hi folks, I'm using the elasticsearch output plugin, and I want to define two alternatives configurations for it, depending on a certain condition (see code below). ... output { if \[condition\] { elasticsearch { … --- ## [Data modeling qustion](https://discuss.elastic.co/t/data-modeling-qustion/196641)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [August 26, 2019, 12:58pm UTC](https://discuss.elastic.co/t/data-modeling-qustion/196641 "2019-08-26T12:58:58Z")
I have a user record database which has employee, userinfo1, userinfo2, managername foo1, info1, info2, foo2 foo2, info1, info2, foo3 foo3, info1, info2, foo4 as now I have manager level foo4 -\> foo3 -\> foo2 -\> foo… --- ## [Having a hard time parsing artifactory logs using 7.3.0](https://discuss.elastic.co/t/having-a-hard-time-parsing-artifactory-logs-using-7-3-0/195170)
**Author:** [@starrynight](https://discuss.elastic.co/u/starrynight)\ **Replies:** 10\ **Last updated:** [August 26, 2019, 12:58pm UTC](https://discuss.elastic.co/t/having-a-hard-time-parsing-artifactory-logs-using-7-3-0/195170 "2019-08-26T12:58:21Z")
Having a hard time parsing artifactory logs -- I've utilized the grok debugger in kibana as well as referenced jfrog's page regarding parsing logs -- but still it seems to not register. I've inserted greedydata in a lot … --- ## [ES cluster restart, any impact on Logstash output?](https://discuss.elastic.co/t/es-cluster-restart-any-impact-on-logstash-output/196767)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 0\ **Last updated:** [August 26, 2019, 12:27pm UTC](https://discuss.elastic.co/t/es-cluster-restart-any-impact-on-logstash-output/196767 "2019-08-26T12:27:58Z")
Hello, We need to restart ES cluster (node by node) due to some changes. Our concenr is to know whether it will have any impact on LS. We have mentioned ES-output to logstash using hosts uri. https://www.elastic.co/g… --- ## [Grok pattern for SYSLOGPROG](https://discuss.elastic.co/t/grok-pattern-for-syslogprog/196515)
**Author:** [@Kragh](https://discuss.elastic.co/u/Kragh)\ **Replies:** 3\ **Last updated:** [August 26, 2019, 12:27pm UTC](https://discuss.elastic.co/t/grok-pattern-for-syslogprog/196515 "2019-08-26T12:27:50Z")
Hi there, I'm currently looking into Logstash which recieves our logs from the syslog server, which recieves logs from our windows hosts (logged with NXLog, parsed as JSON). Currently I'm having having a issue with the… --- ## [Error while diverting output to multiple secured kafka topic](https://discuss.elastic.co/t/error-while-diverting-output-to-multiple-secured-kafka-topic/196746)
**Author:** [@amrutadale](https://discuss.elastic.co/u/amrutadale)\ **Replies:** 0\ **Last updated:** [August 26, 2019, 10:12am UTC](https://discuss.elastic.co/t/error-while-diverting-output-to-multiple-secured-kafka-topic/196746 "2019-08-26T10:12:53Z")
I am trying to divert output to different secured kafka topics based on the topic name received from HTTP end point. for single topic my configuration is working fine. If I add if else based on topic name condition inte… --- ## [SSL Authentication between beat agent and Logstash](https://discuss.elastic.co/t/ssl-authentication-between-beat-agent-and-logstash/196742)
**Author:** [@radw](https://discuss.elastic.co/u/radw)\ **Replies:** 0\ **Last updated:** [August 26, 2019, 9:55am UTC](https://discuss.elastic.co/t/ssl-authentication-between-beat-agent-and-logstash/196742 "2019-08-26T09:55:07Z")
Hi all. I'm trying to setup the SSL authentication between beats agents(Metricbeat, Filebeat etc) but I'm stuck! I'm using the free/community edition of ELK stack, v6.8.2. Logstash configuration file(logstash.yml): p… --- ## [Multiple outputs with different formats](https://discuss.elastic.co/t/multiple-outputs-with-different-formats/196285)
**Author:** [@programagor](https://discuss.elastic.co/u/programagor)\ **Replies:** 3\ **Last updated:** [August 26, 2019, 8:42am UTC](https://discuss.elastic.co/t/multiple-outputs-with-different-formats/196285 "2019-08-26T08:42:38Z")
Greetings I am using Logstash to parse and index event logs into Elasticsearch. My pipeline has the following form: input{ # input plugins and their config } filter{ # lots of parsing # in some cases, event is t… --- ## [SystemOut log grokking](https://discuss.elastic.co/t/systemout-log-grokking/196694)
**Author:** [@ravi\_varma](https://discuss.elastic.co/u/ravi_varma)\ **Replies:** 0\ **Last updated:** [August 26, 2019, 3:36am UTC](https://discuss.elastic.co/t/systemout-log-grokking/196694 "2019-08-26T03:36:56Z")
How to i use logstash to format for these kind of different log file every sec. \[8/23/19 10:30:29:643 MYT\] 0000147c JavaScriptInt I com.worklight.integration.js.JavaScriptIntegrationLibraryImplementation info @@@@20190… --- ## [Input S3 does NOT work properly with prefix option](https://discuss.elastic.co/t/input-s3-does-not-work-properly-with-prefix-option/196426)
**Author:** [@jasony](https://discuss.elastic.co/u/jasony)\ **Replies:** 2\ **Last updated:** [August 26, 2019, 2:00am UTC](https://discuss.elastic.co/t/input-s3-does-not-work-properly-with-prefix-option/196426 "2019-08-26T02:00:56Z")
Hello, I am trying to set up a logstash pipeline with input source AWS S3. I set up as below and tested, and saw all log files were successfully retrieved by logstash. but only problem was from using "prefix" option. i… --- ## [Logstash ILM](https://discuss.elastic.co/t/logstash-ilm/196217)
**Author:** [@farrukh](https://discuss.elastic.co/u/farrukh)\ **Replies:** 4\ **Last updated:** [August 26, 2019, 12:41am UTC](https://discuss.elastic.co/t/logstash-ilm/196217 "2019-08-26T00:41:17Z")
Hi, I'm setting up a test environment where i want Winlogbeat and packetbeat to send Windows event logs to Elasticsearch through Logstash. I need to know where do i suppose to enable ILM? on Beats side or logstash? I'v… --- ## [Aggregation filter timeout settings](https://discuss.elastic.co/t/aggregation-filter-timeout-settings/196208)
**Author:** [@jratliff](https://discuss.elastic.co/u/jratliff)\ **Replies:** 4\ **Last updated:** [August 26, 2019, 12:33am UTC](https://discuss.elastic.co/t/aggregation-filter-timeout-settings/196208 "2019-08-26T00:33:00Z")
I'm having some trouble with the logstash aggregate filter. We are trying to combine sflow events together. These events are not sent to logstash in real time. The sensor data is collected and sent to us in a batch on a… --- ## [Path not loading in logstash 7.3](https://discuss.elastic.co/t/path-not-loading-in-logstash-7-3/196353)
**Author:** [@SachinRaj](https://discuss.elastic.co/u/SachinRaj)\ **Replies:** 9\ **Last updated:** [August 24, 2019, 5:27pm UTC](https://discuss.elastic.co/t/path-not-loading-in-logstash-7-3/196353 "2019-08-24T17:27:03Z")
hello, Good Day !! Please help me on this installed ELK 7.3 in Windows 10 Laptop below mentioned data is the conf file i am trying to execute in Logslatsh and no error. but the file (path) is not loading except that … --- ## [How to convert KVpair value from string to an integer in logstash pipeline](https://discuss.elastic.co/t/how-to-convert-kvpair-value-from-string-to-an-integer-in-logstash-pipeline/196421)
**Author:** [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\ **Replies:** 5\ **Last updated:** [August 24, 2019, 4:27pm UTC](https://discuss.elastic.co/t/how-to-convert-kvpair-value-from-string-to-an-integer-in-logstash-pipeline/196421 "2019-08-24T16:27:01Z")
I have a setup as below source for kvpair: Duration: {2297} | Kernel: {171} | User: {1218} KVblock is as below kv { trim\_value =\> " {}\\r\\n" trim\_key =\> " \\r\\n" value\_split =\> ":" … --- ## [Start time much longer after upgrade from 6.x to 7.x](https://discuss.elastic.co/t/start-time-much-longer-after-upgrade-from-6-x-to-7-x/196613)
**Author:** [@mmolho](https://discuss.elastic.co/u/mmolho)\ **Replies:** 0\ **Last updated:** [August 24, 2019, 8:28am UTC](https://discuss.elastic.co/t/start-time-much-longer-after-upgrade-from-6-x-to-7-x/196613 "2019-08-24T08:28:44Z")
Hi, I've upgraded my Logstash instances from 6.8 to 7.3. After the upgrade, I've noticed a startup time ~3x higher, going from ~3min to ~9min. I'm surprised because I red in the release notes that the startup time is ex… --- ## [Mikrotik logs](https://discuss.elastic.co/t/mikrotik-logs/196610)
**Author:** [@Anand\_Malhotra](https://discuss.elastic.co/u/Anand_Malhotra)\ **Replies:** 0\ **Last updated:** [August 24, 2019, 7:06am UTC](https://discuss.elastic.co/t/mikrotik-logs/196610 "2019-08-24T07:06:24Z")
I want to parse the logs from mikrotik router..... My logs are of type..... "tags" =\> \[ \[0\] "\_geoip\_lookup\_failure" \], "type" =\> "syslog", "@version" =\> "1", "@timestamp" =\> 2019-08-24T05:09:51.875Z, "message" =\> … --- ## [Identifying when logstash jdbc SQL statement has completed execution](https://discuss.elastic.co/t/identifying-when-logstash-jdbc-sql-statement-has-completed-execution/196574)
**Author:** [@stark525](https://discuss.elastic.co/u/stark525)\ **Replies:** 0\ **Last updated:** [August 23, 2019, 6:38pm UTC](https://discuss.elastic.co/t/identifying-when-logstash-jdbc-sql-statement-has-completed-execution/196574 "2019-08-23T18:38:51Z")
I've set up a logstash pipeline watching a directory for logstash.conf s with a jdbc input and elasticsearch output. I enabled .logstash\_jdbc\_test\_last\_run , which logs the time when query is executed: --- 2019-08-23 1… --- ## [Logstash host field set as docker bridge network gateway](https://discuss.elastic.co/t/logstash-host-field-set-as-docker-bridge-network-gateway/196395)
**Author:** [@dcmott86](https://discuss.elastic.co/u/dcmott86)\ **Replies:** 5\ **Last updated:** [August 23, 2019, 6:31pm UTC](https://discuss.elastic.co/t/logstash-host-field-set-as-docker-bridge-network-gateway/196395 "2019-08-23T18:31:07Z")
Running logstash 7.3.0 as a docker container. I have a UDP input receiving messages from from some Cisco ASA's and the host field is populating with the docker bridge network gateway address vs the firewall's IP. Ruby … --- ## [Ruby request and logstash](https://discuss.elastic.co/t/ruby-request-and-logstash/196570)
**Author:** [@He\_Pamela](https://discuss.elastic.co/u/He_Pamela)\ **Replies:** 1\ **Last updated:** [August 23, 2019, 6:17pm UTC](https://discuss.elastic.co/t/ruby-request-and-logstash/196570 "2019-08-23T18:17:58Z")
Hi, I'm trying to make a ruby request to Elasticsearch using Logstash and ruby. this is my filter part in the config file. filter { json { source =\> "message" } } ruby {… --- ## [Can you apply another grok filter in case of \_grokparsefailure?](https://discuss.elastic.co/t/can-you-apply-another-grok-filter-in-case-of-grokparsefailure/196557)
**Author:** [@Siri\_sour](https://discuss.elastic.co/u/Siri_sour)\ **Replies:** 1\ **Last updated:** [August 23, 2019, 6:12pm UTC](https://discuss.elastic.co/t/can-you-apply-another-grok-filter-in-case-of-grokparsefailure/196557 "2019-08-23T18:12:51Z")
Hey, I'm working with different log structure and l Tried to adapt my grok filter for most of logs, but there still some that cause a grokparsefailure. I wonder what is the best way to handle a grokparsefailure? In m… --- ## [Configuring Pipeline To Handle Duplicates In Rollover Indices](https://discuss.elastic.co/t/configuring-pipeline-to-handle-duplicates-in-rollover-indices/196563)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [August 23, 2019, 5:00pm UTC](https://discuss.elastic.co/t/configuring-pipeline-to-handle-duplicates-in-rollover-indices/196563 "2019-08-23T17:00:38Z")
For awhile, I've been using the fingerprint filter and setting the resulting value to the document id when outputting to Elasticsearch to handle duplicate prevention. I've recently discovered (the hard way) that this me… --- ## [Centralized Pipelines referencing logstash-keystore for passwords](https://discuss.elastic.co/t/centralized-pipelines-referencing-logstash-keystore-for-passwords/196033)
**Author:** [@pattercw](https://discuss.elastic.co/u/pattercw)\ **Replies:** 3\ **Last updated:** [August 23, 2019, 4:07pm UTC](https://discuss.elastic.co/t/centralized-pipelines-referencing-logstash-keystore-for-passwords/196033 "2019-08-23T16:07:36Z")
Hello: So we're implementing SAML on our cluster which has led to the implementation of SSL communications instead of http. That journey is completed and functioning properly including logstash monitoring and manageme… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=486) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=488)