# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=485 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 486 --- ## [Two logstash servers configuration](https://discuss.elastic.co/t/two-logstash-servers-configuration/197352)
**Author:** [@Yashwant\_Shettigar](https://discuss.elastic.co/u/Yashwant_Shettigar)\ **Replies:** 0\ **Last updated:** [August 29, 2019, 1:30pm UTC](https://discuss.elastic.co/t/two-logstash-servers-configuration/197352 "2019-08-29T13:30:34Z")
I'm trying to configure cluster with two log-stash nodes. Can someone please let me know, where and what will be the configuration changes to be made ? Thanks --- ## [Use a match field from the message field in a conditional of the logstash output](https://discuss.elastic.co/t/use-a-match-field-from-the-message-field-in-a-conditional-of-the-logstash-output/197274)
**Author:** [@dferez](https://discuss.elastic.co/u/dferez)\ **Replies:** 1\ **Last updated:** [August 29, 2019, 12:52pm UTC](https://discuss.elastic.co/t/use-a-match-field-from-the-message-field-in-a-conditional-of-the-logstash-output/197274 "2019-08-29T12:52:12Z")
Hello, it is my first question, if there is something wrong, please forgive me. I present my problem to you: My Logstash I get logs from Sophos XG. The sophos logs reach me in the "message" field, so with the filter, gr… --- ## [Changing the workers number](https://discuss.elastic.co/t/changing-the-workers-number/197098)
**Author:** [@Alisa\_Faingold](https://discuss.elastic.co/u/Alisa_Faingold)\ **Replies:** 6\ **Last updated:** [August 29, 2019, 12:44pm UTC](https://discuss.elastic.co/t/changing-the-workers-number/197098 "2019-08-29T12:44:41Z")
Hi, I'm trying to define transactions with elapsed filter, and I got the "elapsed\_end\_without\_start" tag. I found that the same issue happens to other people and the solution that has been suggested is changing the wor… --- ## [Logstash tried to load a plugin's code (Google BigQuery)](https://discuss.elastic.co/t/logstash-tried-to-load-a-plugins-code-google-bigquery/197322)
**Author:** [@Sindre\_E\_de\_Lange](https://discuss.elastic.co/u/Sindre_E_de_Lange)\ **Replies:** 0\ **Last updated:** [August 29, 2019, 11:10am UTC](https://discuss.elastic.co/t/logstash-tried-to-load-a-plugins-code-google-bigquery/197322 "2019-08-29T11:10:24Z")
I am trying to send data to BigQuery using LogStash. The BigQuery output plugin is installed, as shown by using bin/logstash-plugin list, but when running a .conf file consisting of: output { google\_bigquery { projec… --- ## [Check if a field is a given type in logstash](https://discuss.elastic.co/t/check-if-a-field-is-a-given-type-in-logstash/197304)
**Author:** [@alchy](https://discuss.elastic.co/u/alchy)\ **Replies:** 0\ **Last updated:** [August 29, 2019, 9:35am UTC](https://discuss.elastic.co/t/check-if-a-field-is-a-given-type-in-logstash/197304 "2019-08-29T09:35:02Z")
dear, from a source (winlogbeat 7.11) I'm getting the messages which should contain specific type, but sometime, the type is not the type expected and elastic won't index the data. i would like to have a condition in l… --- ## [Grokparsefailure message in the case of using grok filter](https://discuss.elastic.co/t/grokparsefailure-message-in-the-case-of-using-grok-filter/197275)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [August 29, 2019, 8:19am UTC](https://discuss.elastic.co/t/grokparsefailure-message-in-the-case-of-using-grok-filter/197275 "2019-08-29T08:19:22Z")
Hi all, i have some log as following template: 1925/05/01;2019-07-23;00:00:55.193;\[\[ (MD) 19915478 CR=25895;AC=TLK;Id=2458 I am using following filter in logstash script: filter { csv { columns =\> \["ODate"… --- ## [PFSense logs in ELK Stack](https://discuss.elastic.co/t/pfsense-logs-in-elk-stack/197267)
**Author:** [@mohamad.faisel](https://discuss.elastic.co/u/mohamad.faisel)\ **Replies:** 0\ **Last updated:** [August 29, 2019, 7:10am UTC](https://discuss.elastic.co/t/pfsense-logs-in-elk-stack/197267 "2019-08-29T07:10:40Z")
Hi, I have installed the ELK Stack on a server followed steps published in Digital Ocean tutorial (https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubunt… --- ## [Kv filter seems to be working](https://discuss.elastic.co/t/kv-filter-seems-to-be-working/197259)
**Author:** [@Abed\_Bishara](https://discuss.elastic.co/u/Abed_Bishara)\ **Replies:** 1\ **Last updated:** [August 29, 2019, 6:19am UTC](https://discuss.elastic.co/t/kv-filter-seems-to-be-working/197259 "2019-08-29T06:19:16Z")
Hi I configured the logstash with kv filter and it is not working When I simply add the argument: kv { source =\> "syslog\_message" field\_split =\> " " } The logstash stops working and I see no logs in kibana … --- ## [Logstash error Failed to execute action](https://discuss.elastic.co/t/logstash-error-failed-to-execute-action/197234)
**Author:** [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\ **Replies:** 2\ **Last updated:** [August 29, 2019, 2:15am UTC](https://discuss.elastic.co/t/logstash-error-failed-to-execute-action/197234 "2019-08-29T02:15:57Z")
Hello: I get the following error message when starting logstash. My setup is as follows. Logstash isn't communicating to the Elasticsearch servers because it can't create a pipeline or something. Not sure what I setu… --- ## [Multiple different outputs](https://discuss.elastic.co/t/multiple-different-outputs/197033)
**Author:** [@JohnM](https://discuss.elastic.co/u/JohnM)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 11:58pm UTC](https://discuss.elastic.co/t/multiple-different-outputs/197033 "2019-08-28T23:58:50Z")
We are running logstash to collect syslog and forwarding to ES cluster now which is working fine. /etc/logstash/conf.d/5-listen.conf input { udp { port =\> 514 type =\> syslog } tcp { … --- ## [Logstash xpack.monitoring.elasticsearch.password](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604)
**Author:** [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\ **Replies:** 18\ **Last updated:** [August 28, 2019, 11:38pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604 "2019-08-28T23:38:12Z")
Hi there, Is it possible to use a keystore to store this password. I don't want to have any plain text passwords in config file I'm specifically talking about this setting xpack.monitoring.elasticsearch.username: "log… --- ## [Convert timestamp format using date filter](https://discuss.elastic.co/t/convert-timestamp-format-using-date-filter/197073)
**Author:** [@Suhee\_Kim](https://discuss.elastic.co/u/Suhee_Kim)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 11:06pm UTC](https://discuss.elastic.co/t/convert-timestamp-format-using-date-filter/197073 "2019-08-28T23:06:25Z")
Hi, I have the following timestamp format Aug 28, 2019 @ 17:51:45.501 and this is the format i want to convert 2019-08-28 08:52:07 how can I convert it? Thank you --- ## [File format/type for S3 output plugin](https://discuss.elastic.co/t/file-format-type-for-s3-output-plugin/197217)
**Author:** [@Cloud\_enthu](https://discuss.elastic.co/u/Cloud_enthu)\ **Replies:** 0\ **Last updated:** [August 28, 2019, 7:52pm UTC](https://discuss.elastic.co/t/file-format-type-for-s3-output-plugin/197217 "2019-08-28T19:52:34Z")
Can we change the file format/type to JSON or Avro when using the Logstash Amazon S3 output plugin? --- ## [Entire File as "one line"](https://discuss.elastic.co/t/entire-file-as-one-line/197192)
**Author:** [@paiz1556](https://discuss.elastic.co/u/paiz1556)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 7:47pm UTC](https://discuss.elastic.co/t/entire-file-as-one-line/197192 "2019-08-28T19:47:58Z")
Is there a good strategy to read an entire file as the "message" value you would send to ElasticSearch? I see "read" mode as part of file input plugin but that wants me to write the input to a different file. I want… --- ## [Cannot install logstash-input-salesforce](https://discuss.elastic.co/t/cannot-install-logstash-input-salesforce/197210)
**Author:** [@trent.dierking](https://discuss.elastic.co/u/trent.dierking)\ **Replies:** 1\ **Last updated:** [August 28, 2019, 7:47pm UTC](https://discuss.elastic.co/t/cannot-install-logstash-input-salesforce/197210 "2019-08-28T19:47:06Z")
After a fresh install of logstash # /usr/share/logstash/bin/logstash-plugin install --version 3.0.5 --no-verify logstash-input-salesforce Installing logstash-input-salesforce Plugin version conflict, aborting ERROR: Ins… --- ## [How to load a log file and apply grok filters without using Filebeat](https://discuss.elastic.co/t/how-to-load-a-log-file-and-apply-grok-filters-without-using-filebeat/197185)
**Author:** [@kmiklas](https://discuss.elastic.co/u/kmiklas)\ **Replies:** 1\ **Last updated:** [August 28, 2019, 7:24pm UTC](https://discuss.elastic.co/t/how-to-load-a-log-file-and-apply-grok-filters-without-using-filebeat/197185 "2019-08-28T19:24:26Z")
How can a log file be loaded, and grok filters in our .conf file applied without using filebeat? In other words, and I manually load a log into logstash and apply our grok filters (as we would in Filebeat)? Thx, Keith :… --- ## [Is it possible to create two index for the same day using logstash templete](https://discuss.elastic.co/t/is-it-possible-to-create-two-index-for-the-same-day-using-logstash-templete/197022)
**Author:** [@RITZ\_VERMA](https://discuss.elastic.co/u/RITZ_VERMA)\ **Replies:** 7\ **Last updated:** [August 28, 2019, 6:14pm UTC](https://discuss.elastic.co/t/is-it-possible-to-create-two-index-for-the-same-day-using-logstash-templete/197022 "2019-08-28T18:14:49Z")
I am using logstash to send data elasticsearch on daily basis but the size of the indexes created is too large .that is why i want ot create two index for the same day.is it possible using logstash --- ## [Usage of "or" operator in logstash grok filters](https://discuss.elastic.co/t/usage-of-or-operator-in-logstash-grok-filters/197132)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 10\ **Last updated:** [August 28, 2019, 5:31pm UTC](https://discuss.elastic.co/t/usage-of-or-operator-in-logstash-grok-filters/197132 "2019-08-28T17:31:18Z")
Hello, I have a condition to check if the 7th-word matches either /list/ or /list or /simple/ or /simple using below condition \[@metadata\]\[copyOfMessage\]\[6\] =~ /^\\/list\\// or /^\\/list/ or /^\\/simple\\// or /^\\/simple/ T… --- ## [How use gsub in a Ruby code](https://discuss.elastic.co/t/how-use-gsub-in-a-ruby-code/197166)
**Author:** [@Thiago\_Paiva](https://discuss.elastic.co/u/Thiago_Paiva)\ **Replies:** 1\ **Last updated:** [August 28, 2019, 5:06pm UTC](https://discuss.elastic.co/t/how-use-gsub-in-a-ruby-code/197166 "2019-08-28T17:06:48Z")
Hi friend, I'm trying use gsub in ruby code. But my syntax didn't work. matches\[ x \] = "GenericLog\\tGenericDetail\\t4\\t2019-08-23 01:23:05" Code: matches.each\_index{ |x| x.gsub(/(GenericLog).\*\\\\t4\\\\t/… --- ## [Unable to Change Timezone](https://discuss.elastic.co/t/unable-to-change-timezone/197158)
**Author:** [@MR551](https://discuss.elastic.co/u/MR551)\ **Replies:** 4\ **Last updated:** [August 28, 2019, 4:45pm UTC](https://discuss.elastic.co/t/unable-to-change-timezone/197158 "2019-08-28T16:45:38Z")
Hello, Trying to configure logstash to ingest error logs into elastic from a txt file, so far its working and data gets delivered into elastic, but the date plugin won't get convert it to Eastern time. Sample error fro… --- ## [Facing an issue to send the log file Elastic](https://discuss.elastic.co/t/facing-an-issue-to-send-the-log-file-elastic/197117)
**Author:** [@Hari21](https://discuss.elastic.co/u/Hari21)\ **Replies:** 3\ **Last updated:** [August 28, 2019, 4:39pm UTC](https://discuss.elastic.co/t/facing-an-issue-to-send-the-log-file-elastic/197117 "2019-08-28T16:39:24Z")
Hi All, Am unable to send the log message to elastic using logstash , it would be great help if someone can help me on this. here is my test code. (in fact i tried several scripts but unsuccessful :frowning: Example on… --- ## [Grok jenkins console logs](https://discuss.elastic.co/t/grok-jenkins-console-logs/194577)
**Author:** [@chaymaa](https://discuss.elastic.co/u/chaymaa)\ **Replies:** 1\ **Last updated:** [August 28, 2019, 4:15pm UTC](https://discuss.elastic.co/t/grok-jenkins-console-logs/194577 "2019-08-28T16:15:31Z")
Hi All, I have a problem on groking jenkins console log, i have tested the filter instructions using input type file and it's OK but when i implement this configuration using a tcp input type the results is not the same… --- ## [Ruby Filter to delete array element with regexp](https://discuss.elastic.co/t/ruby-filter-to-delete-array-element-with-regexp/197131)
**Author:** [@Thiago\_Paiva](https://discuss.elastic.co/u/Thiago_Paiva)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 2:55pm UTC](https://discuss.elastic.co/t/ruby-filter-to-delete-array-element-with-regexp/197131 "2019-08-28T14:55:27Z")
Hi, I try to delete an array element using a regexp, But code isn't run. Imput json: \`{"test":"The http module allows to expose\\nThe http module allows to expose\\ncamp:using asynchronous communication\\nusing asynchrono… --- ## [Prune not parsing date component of custom elasticsearch index](https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129)
**Author:** [@iiz](https://discuss.elastic.co/u/iiz)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 2:26pm UTC](https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129 "2019-08-28T14:26:03Z")
I have json logs being shipped to logstash by filebeat on multiple server instances. I would like to whitelist a specific set of fields to include. The whitelist feature works quite well, except for the fact that it does… --- ## [Problem in sending data from HTTP plugin to kafka plugin in logstash](https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110)
**Author:** [@DZ1](https://discuss.elastic.co/u/DZ1)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 1:44pm UTC](https://discuss.elastic.co/t/problem-in-sending-data-from-http-plugin-to-kafka-plugin-in-logstash/197110 "2019-08-28T13:44:30Z")
I am trying to send JSON data to HTTP plugin and that output to Kafka using Kafka plugin. In this HTTP input plugin shows output in the first attempt and sends only a small amount of data to Kafka. When I hit it next tim… --- ## [Metrics for the Logstash persistent queue](https://discuss.elastic.co/t/metrics-for-the-logstash-persistent-queue/197064)
**Author:** [@Luigi\_Clemente](https://discuss.elastic.co/u/Luigi_Clemente)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 1:34pm UTC](https://discuss.elastic.co/t/metrics-for-the-logstash-persistent-queue/197064 "2019-08-28T13:34:34Z")
Hi all I am trying to implement a cluster of Logstash nodes in AWS using Autoscaling group. I cannot find a clear, reliable metric in the Logstash API that can tell me if the Logstash persistent queue is completely empt… --- ## [Filter on TCP input plugin](https://discuss.elastic.co/t/filter-on-tcp-input-plugin/197083)
**Author:** [@DZ1](https://discuss.elastic.co/u/DZ1)\ **Replies:** 2\ **Last updated:** [August 28, 2019, 12:03pm UTC](https://discuss.elastic.co/t/filter-on-tcp-input-plugin/197083 "2019-08-28T12:03:24Z")
I am fetching input using the TCP input plugin. I am hitting TCP endpoint with an API so while fetching data from the endpoint it fetches data like header information (Content-length, User-Agent etc) . So is there any wa… --- ## [Logstash doesn't work with XML file](https://discuss.elastic.co/t/logstash-doesnt-work-with-xml-file/197032)
**Author:** [@Nurbek](https://discuss.elastic.co/u/Nurbek)\ **Replies:** 1\ **Last updated:** [August 28, 2019, 11:47am UTC](https://discuss.elastic.co/t/logstash-doesnt-work-with-xml-file/197032 "2019-08-28T11:47:22Z")
XML file \ -\ -\ \Зар… --- ## [Licence error after starting Logstash](https://discuss.elastic.co/t/licence-error-after-starting-logstash/197095)
**Author:** [@\_bawaji](https://discuss.elastic.co/u/_bawaji)\ **Replies:** 0\ **Last updated:** [August 28, 2019, 11:27am UTC](https://discuss.elastic.co/t/licence-error-after-starting-logstash/197095 "2019-08-28T11:27:35Z")
I installed ES first, it worked fine. Then installed Kibana, both Kibana and ES worked fine after enabling Authentication as well. Now installed Logstash, faced issues while starting, now managed to start it but getting… --- ## [Another @timestamp question... non-standard timestamp transformation](https://discuss.elastic.co/t/another-timestamp-question-non-standard-timestamp-transformation/197017)
**Author:** [@Kalydrae](https://discuss.elastic.co/u/Kalydrae)\ **Replies:** 3\ **Last updated:** [August 28, 2019, 11:08am UTC](https://discuss.elastic.co/t/another-timestamp-question-non-standard-timestamp-transformation/197017 "2019-08-28T11:08:13Z")
Hi there, I'm new to all of this. I'm using Logstash with Elastic Search and finding the @timestamp transformation frustrating for my non-standard log format. Any help you can provide me would be appreciated. Here is … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=484) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=486)