# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=622 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 623 --- ## [Most basic Logstash to secure cluster.conf](https://discuss.elastic.co/t/most-basic-logstash-to-secure-cluster-conf/163376)
**Author:** [@bigdamhero](https://discuss.elastic.co/u/bigdamhero)\ **Replies:** 2\ **Last updated:** [January 8, 2019, 3:42pm UTC](https://discuss.elastic.co/t/most-basic-logstash-to-secure-cluster-conf/163376 "2019-01-08T15:42:14Z")
Continued troubles, frustration rising. Version 6.5.4 Logstash running on Ubuntu 16 - OpenJDK I'm looking for recommendations on the easiest .conf file to test a connection. Should I pull syslog? no filter? What wo… --- ## [Getting month from weekyear date in logstash output](https://discuss.elastic.co/t/getting-month-from-weekyear-date-in-logstash-output/162579)
**Author:** [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\ **Replies:** 1\ **Last updated:** [January 8, 2019, 3:42pm UTC](https://discuss.elastic.co/t/getting-month-from-weekyear-date-in-logstash-output/162579 "2019-01-08T15:42:28Z")
Hello. I use this kind of date format for index naming index =\> "logstash-%{+xxxx.MM-ww}". And it had been working for last year till today. When index name should have "2019.01-01" in date (year.month-week) it has "20… --- ## [Help with the logstash ruby filter.Ruby exception occurred: undefined method \`time' for 1546441403:Fixnum](https://discuss.elastic.co/t/help-with-the-logstash-ruby-filter-ruby-exception-occurred-undefined-method-time-for-1546441403-fixnum/162676)
**Author:** [@simmonHuong](https://discuss.elastic.co/u/simmonHuong)\ **Replies:** 1\ **Last updated:** [January 2, 2019, 5:18pm UTC](https://discuss.elastic.co/t/help-with-the-logstash-ruby-filter-ruby-exception-occurred-undefined-method-time-for-1546441403-fixnum/162676 "2019-01-02T17:18:20Z")
filter { if \[is\_try\] == 1 { drop { } } ruby { code =\> " fmt = '%Y-%m-%dT%H:%M:%S.%L%z' tz = '+08:00' create\_time = event.get('\[create\_time\]').time … --- ## [What does Logstash responds with when it back pressures Filebeat?](https://discuss.elastic.co/t/what-does-logstash-responds-with-when-it-back-pressures-filebeat/163381)
**Author:** [@xojima](https://discuss.elastic.co/u/xojima)\ **Replies:** 1\ **Last updated:** [January 8, 2019, 3:18pm UTC](https://discuss.elastic.co/t/what-does-logstash-responds-with-when-it-back-pressures-filebeat/163381 "2019-01-08T15:18:46Z")
We recently dropped Redis from our ELK pipeline and started sending logs directly using the beats input from Filebeat (6.1.3) to two Logstash instances(5.6.2) with persistent queues and loadbalance set to true. However w… --- ## [How to differentiate between field references and list memberships with "in"/"not in"?](https://discuss.elastic.co/t/how-to-differentiate-between-field-references-and-list-memberships-with-in-not-in/163257)
**Author:** [@skierx](https://discuss.elastic.co/u/skierx)\ **Replies:** 5\ **Last updated:** [January 8, 2019, 3:09pm UTC](https://discuss.elastic.co/t/how-to-differentiate-between-field-references-and-list-memberships-with-in-not-in/163257 "2019-01-08T15:09:59Z")
If I have a sample event: { "client" =\> "someone" } I'm curious as to what is going on in the following logstash configurations. Configuration A: if \[client\] not in \["someone", "foo"\] { // do something... } Config… --- ## [How to add a new field and value with field name and value taken from another fields value?](https://discuss.elastic.co/t/how-to-add-a-new-field-and-value-with-field-name-and-value-taken-from-another-fields-value/163368)
**Author:** [@hpniemitube](https://discuss.elastic.co/u/hpniemitube)\ **Replies:** 1\ **Last updated:** [January 8, 2019, 2:17pm UTC](https://discuss.elastic.co/t/how-to-add-a-new-field-and-value-with-field-name-and-value-taken-from-another-fields-value/163368 "2019-01-08T14:17:23Z")
I would like to know how I could do this kind of new field: Field "sesssion.variable\_name" exists and has a value for example "session.username" Field "session.variable\_value" exists and has a value for example "Mike" … --- ## [Logstash not recognizing the JSON objects from my logs and displays entire log set in message field](https://discuss.elastic.co/t/logstash-not-recognizing-the-json-objects-from-my-logs-and-displays-entire-log-set-in-message-field/162842)
**Author:** [@utki](https://discuss.elastic.co/u/utki)\ **Replies:** 4\ **Last updated:** [January 8, 2019, 1:30pm UTC](https://discuss.elastic.co/t/logstash-not-recognizing-the-json-objects-from-my-logs-and-displays-entire-log-set-in-message-field/162842 "2019-01-08T13:30:19Z")
I need to send below logs using logstash and want to create a dashboard using request\_header\_x\_forwarded\_for and request\_username fields which are JSON objects Dec 26 00:32:50 Web1 mars-auth-capture: \[INFO \]\[mars\]\[auth… --- ## [Logstash to redirect output to icingaweb2 and elasticsearch simultaneously](https://discuss.elastic.co/t/logstash-to-redirect-output-to-icingaweb2-and-elasticsearch-simultaneously/163348)
**Author:** [@baidurjya](https://discuss.elastic.co/u/baidurjya)\ **Replies:** 5\ **Last updated:** [January 8, 2019, 12:55pm UTC](https://discuss.elastic.co/t/logstash-to-redirect-output-to-icingaweb2-and-elasticsearch-simultaneously/163348 "2019-01-08T12:55:34Z")
Hello Team, I want to create single logstash.conf file that must redirect nginx log to elasticsearch and in case of any http error (e.g. 500) , it must redirect nginx error to icingaweb2 as well as elasticsearch simulta… --- ## [Jdbc\_static performance issues](https://discuss.elastic.co/t/jdbc-static-performance-issues/163364)
**Author:** [@darefilz](https://discuss.elastic.co/u/darefilz)\ **Replies:** 0\ **Last updated:** [January 8, 2019, 12:41pm UTC](https://discuss.elastic.co/t/jdbc-static-performance-issues/163364 "2019-01-08T12:41:29Z")
Hello there, I am comparing Jdbc\_static performance to Jdbc\_streaming performance. I expected to see that Jdbc\_static is slower for the first few event but scales better the more events use that filter. Oddly Jdbc\_strea… --- ## [Netflow flow direction on v5 vs v9](https://discuss.elastic.co/t/netflow-flow-direction-on-v5-vs-v9/163025)
**Author:** [@samplonius](https://discuss.elastic.co/u/samplonius)\ **Replies:** 1\ **Last updated:** [January 8, 2019, 11:28am UTC](https://discuss.elastic.co/t/netflow-flow-direction-on-v5-vs-v9/163025 "2019-01-08T11:28:20Z")
The Netflow module forces v5 flows to be of direction ingress. But for v9 messages, the module uses the "direction" field from the message, but if "direction" is not included, the \_\_netflow\_direction\_not\_recognized tag … --- ## [Covert syslog to jason format and store it as an index in elastic search](https://discuss.elastic.co/t/covert-syslog-to-jason-format-and-store-it-as-an-index-in-elastic-search/163347)
**Author:** [@varunsaun](https://discuss.elastic.co/u/varunsaun)\ **Replies:** 0\ **Last updated:** [January 8, 2019, 10:50am UTC](https://discuss.elastic.co/t/covert-syslog-to-jason-format-and-store-it-as-an-index-in-elastic-search/163347 "2019-01-08T10:50:16Z")
Hello Team, I have a syslog server which accepts logs from a group of servers. I have installed logstash in the same syslog server. I want logstash to convert syslog into jason format and store the converted logs as … --- ## [JDBC properties duplication](https://discuss.elastic.co/t/jdbc-properties-duplication/163327)
**Author:** [@dharana](https://discuss.elastic.co/u/dharana)\ **Replies:** 0\ **Last updated:** [January 8, 2019, 9:26am UTC](https://discuss.elastic.co/t/jdbc-properties-duplication/163327 "2019-01-08T09:26:58Z")
I use multiple data import using jdbc inputs (from single database). so I use same code many times, can I keep jdbc properties in different folder or one place and can I reuse it --- ## [How do I load data from hive to elasticsearch 6.5 using logstash?](https://discuss.elastic.co/t/how-do-i-load-data-from-hive-to-elasticsearch-6-5-using-logstash/163303)
**Author:** [@patil-akshay](https://discuss.elastic.co/u/patil-akshay)\ **Replies:** 1\ **Last updated:** [January 8, 2019, 9:26am UTC](https://discuss.elastic.co/t/how-do-i-load-data-from-hive-to-elasticsearch-6-5-using-logstash/163303 "2019-01-08T09:26:41Z")
How can I load data from hive tables to elasticsearch? please help me with the logsatsh config file. --- ## [Logstash grok filter is having issue](https://discuss.elastic.co/t/logstash-grok-filter-is-having-issue/163212)
**Author:** [@nitin.jangid](https://discuss.elastic.co/u/nitin.jangid)\ **Replies:** 3\ **Last updated:** [January 8, 2019, 8:15am UTC](https://discuss.elastic.co/t/logstash-grok-filter-is-having-issue/163212 "2019-01-08T08:15:08Z")
Hi there, I am trying ingest data from a log file and using grok to filter it and writing in a log file. Here is input log file content "122.164.121.231 - - \[23/Nov/2015:07:19:54 -0500\] "GET /configs/config/thecellose… --- ## [Jdbc-input-plugin -\> how to supress sql statement in log?](https://discuss.elastic.co/t/jdbc-input-plugin-how-to-supress-sql-statement-in-log/163234)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 2\ **Last updated:** [January 8, 2019, 7:26am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-how-to-supress-sql-statement-in-log/163234 "2019-01-08T07:26:26Z")
Hi, I am using jdbc input plugin. I noticed that each time it runs, the complete sql statement is written to log. Is there a way to suppress this if needed? Thanks, Andreas --- ## [Splitting inputs](https://discuss.elastic.co/t/splitting-inputs/163297)
**Author:** [@Jukocross](https://discuss.elastic.co/u/Jukocross)\ **Replies:** 1\ **Last updated:** [January 8, 2019, 7:17am UTC](https://discuss.elastic.co/t/splitting-inputs/163297 "2019-01-08T07:17:06Z")
I will like to split input from UDP into two different output but have no idea how For example: Input A & B =\> Logstash UDP Logstash UDP A =\> Output A Logstash UDP B =\> Output B --- ## [\](https://discuss.elastic.co/t/csv-illegal-quoting-in-line-1/162822)
**Author:** [@pbroek](https://discuss.elastic.co/u/pbroek)\ **Replies:** 2\ **Last updated:** [January 8, 2019, 6:27am UTC](https://discuss.elastic.co/t/csv-illegal-quoting-in-line-1/162822 "2019-01-08T06:27:59Z")
Hello, I am trying to load csv files using the logstash pipeline. The pipeline works but the indexed results is not what I expected. When I use the same csv file and import it using the data visualizer in machine learn… --- ## [Not able to insert JSON from PostgreSQL to elasticsearch. Getting error - "Exception when executing JDBC query"](https://discuss.elastic.co/t/not-able-to-insert-json-from-postgresql-to-elasticsearch-getting-error-exception-when-executing-jdbc-query/163173)
**Author:** [@poonia](https://discuss.elastic.co/u/poonia)\ **Replies:** 0\ **Last updated:** [January 7, 2019, 9:24am UTC](https://discuss.elastic.co/t/not-able-to-insert-json-from-postgresql-to-elasticsearch-getting-error-exception-when-executing-jdbc-query/163173 "2019-01-07T09:24:02Z")
Hi All, I am trying to migrate data from postgresql server to elasticsearch. The postgres data is in JSONB format. When I am starting the river, I am getting the below error. \[INFO \]\[logstash.agent \] Successf… --- ## [Removing Metadata from logs in logstash](https://discuss.elastic.co/t/removing-metadata-from-logs-in-logstash/162994)
**Author:** [@mayanks8](https://discuss.elastic.co/u/mayanks8)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 10:25pm UTC](https://discuss.elastic.co/t/removing-metadata-from-logs-in-logstash/162994 "2019-01-07T22:25:15Z")
Hi All, i have a scenirio where my Jenkins is pushing builld logs to logstash line by line. the issue here is the data reccived by the ELK has the meta data which is common to all the logs reccived by logstash for a sin… --- ## [ElasticSearch Output Disconnects and OOM](https://discuss.elastic.co/t/elasticsearch-output-disconnects-and-oom/163277)
**Author:** [@sbaranoff](https://discuss.elastic.co/u/sbaranoff)\ **Replies:** 0\ **Last updated:** [January 7, 2019, 10:23pm UTC](https://discuss.elastic.co/t/elasticsearch-output-disconnects-and-oom/163277 "2019-01-07T22:23:32Z")
Problem: LogStash is going OOM and restarting itself causing data loss. Our average document size is in the 1-2KB range. We have a few large\* 175MB JSON documents which chew up almost 5GB of RAM going through LogStash f… --- ## [Logstash deal with 24:00 time representation](https://discuss.elastic.co/t/logstash-deal-with-24-00-time-representation/163007)
**Author:** [@Joapfel](https://discuss.elastic.co/u/Joapfel)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 9:44pm UTC](https://discuss.elastic.co/t/logstash-deal-with-24-00-time-representation/163007 "2019-01-07T21:44:19Z")
Hello, I have a dataset containing a datetime representation like this MM/01/yyyy 24:00 instead of MM/02/yyyy 00:00 or MM/01/yyyy 23:59 So the end of the day is represented as 24:00 and therefore not recognized by Elas… --- ## [Logstash to S3 error](https://discuss.elastic.co/t/logstash-to-s3-error/163264)
**Author:** [@earl\_potter](https://discuss.elastic.co/u/earl_potter)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 9:43pm UTC](https://discuss.elastic.co/t/logstash-to-s3-error/163264 "2019-01-07T21:43:26Z")
I'm trying to get logstash to write to an S3 bucket using the information here: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#plugins-outputs-s3-additional\_settings My s3 output config in logs… --- ## [Convert a column in csv to hour time](https://discuss.elastic.co/t/convert-a-column-in-csv-to-hour-time/163098)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 9:38pm UTC](https://discuss.elastic.co/t/convert-a-column-in-csv-to-hour-time/163098 "2019-01-07T21:38:17Z")
I have these 2 cells the first is Day which was converted fine using the date filter but the HOUR column is appearing as a random date time stamp with no hours my logstash date config: date { match =\> \["DAY", "M/d/y… --- ## [Removing field reference in field as string value](https://discuss.elastic.co/t/removing-field-reference-in-field-as-string-value/163099)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 9:17pm UTC](https://discuss.elastic.co/t/removing-field-reference-in-field-as-string-value/163099 "2019-01-07T21:17:19Z")
I am adding a field that references another field, add\_field =\> { "newfield" =\> "%{oldfield}"}. Sometimes the referenced field is empty, resulting in newfield containing a string with the value %{oldfield}. How can I r… --- ## [Key value pair](https://discuss.elastic.co/t/key-value-pair/163260)
**Author:** [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 8:31pm UTC](https://discuss.elastic.co/t/key-value-pair/163260 "2019-01-07T20:31:09Z")
Good afternoon. I am trying to use KVP to separate the key value pair in my "Message" field. "{"EventTime":"2019-01-07 18:20:42.826361Z","Hostname":"186590cbd0f7","Message":"EPS: 99","EventReceivedTime":"2019-01-07 18:… --- ## [Multiple Logstash pipelines outputting into same index](https://discuss.elastic.co/t/multiple-logstash-pipelines-outputting-into-same-index/163269)
**Author:** [@jackdw](https://discuss.elastic.co/u/jackdw)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 8:30pm UTC](https://discuss.elastic.co/t/multiple-logstash-pipelines-outputting-into-same-index/163269 "2019-01-07T20:30:14Z")
I have two Filebeat pipes inputting into Logstash. One parses out log errors that I actually care about from one service while the other takes each line in order to keep track of the health of another service that often … --- ## [Can anybody help me to parse apache web and api logs](https://discuss.elastic.co/t/can-anybody-help-me-to-parse-apache-web-and-api-logs/163231)
**Author:** [@Vinit\_Kumar](https://discuss.elastic.co/u/Vinit_Kumar)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 7:45pm UTC](https://discuss.elastic.co/t/can-anybody-help-me-to-parse-apache-web-and-api-logs/163231 "2019-01-07T19:45:44Z")
Do we have any predefined pattern to parse my apache web and apache api logs. Below is my logs Apache Web Logs \[06/Jan/2019:03:28:19 +0000\] 66.249.79.85 "GET /pages/argylle-derego/5c3048c6e4b081094857431f HTTP/1.1" "M… --- ## [Output plugin IBM Qradar](https://discuss.elastic.co/t/output-plugin-ibm-qradar/162695)
**Author:** [@raffutz85](https://discuss.elastic.co/u/raffutz85)\ **Replies:** 3\ **Last updated:** [January 7, 2019, 5:43pm UTC](https://discuss.elastic.co/t/output-plugin-ibm-qradar/162695 "2019-01-07T17:43:48Z")
Good morning, I'm planning to use logstash to aggregate, filter and enrich logs from different log sources like syslog (proxy, firewall and AV) but also Microsoft Windows (WEC with agent for sending logs to the LogStash)… --- ## [Preprocessing logs with logstash and kafka](https://discuss.elastic.co/t/preprocessing-logs-with-logstash-and-kafka/163240)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 0\ **Last updated:** [January 7, 2019, 4:13pm UTC](https://discuss.elastic.co/t/preprocessing-logs-with-logstash-and-kafka/163240 "2019-01-07T16:13:42Z")
I want to know if there are blacklisted ip address that contact my web services in real-time. Every day I download a blacklist of ip address and I want to add this information to my logs. My idea is to pre-process logs… --- ## [Logstash performance crippled between 6.2.4 and 6.3.0(+)](https://discuss.elastic.co/t/logstash-performance-crippled-between-6-2-4-and-6-3-0/163207)
**Author:** [@bjornryden](https://discuss.elastic.co/u/bjornryden)\ **Replies:** 0\ **Last updated:** [January 7, 2019, 12:45pm UTC](https://discuss.elastic.co/t/logstash-performance-crippled-between-6-2-4-and-6-3-0/163207 "2019-01-07T12:45:44Z")
Hi, After quite a bit of debugging, I've found that our Logstash performance cripples moving from 6.2.4 to 6.3.0 (and have tested up to 6.5.4, all being identically poor) We're using RabbitMQ as input and elasticsearch… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=621) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=623)