# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=713 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 714 --- ## [Date\_formatter plug-in issue](https://discuss.elastic.co/t/date-formatter-plug-in-issue/140958)
**Author:** [@HRG](https://discuss.elastic.co/u/HRG)\ **Replies:** 4\ **Last updated:** [July 27, 2018, 5:45pm UTC](https://discuss.elastic.co/t/date-formatter-plug-in-issue/140958 "2018-07-27T17:45:50Z")
Can you please help on what is missing. I have got date\_formatter.rb code and able to install the plug-in ./logstash-plugin install logstash-filter-date\_formatter Validating logstash-filter-date\_formatter Installing l… --- ## [Logstash input by log4j](https://discuss.elastic.co/t/logstash-input-by-log4j/140085)
**Author:** [@springCkWyy](https://discuss.elastic.co/u/springCkWyy)\ **Replies:** 1\ **Last updated:** [July 27, 2018, 4:21pm UTC](https://discuss.elastic.co/t/logstash-input-by-log4j/140085 "2018-07-27T16:21:35Z")
Now I use logstash plugin "log4j" as input. And I have a question, log4j as a server to receive the log from applications, and when logstash is down and dead, the link between logstash and app is close. and Can log4j can… --- ## [Winlogbeat fails to add geotags](https://discuss.elastic.co/t/winlogbeat-fails-to-add-geotags/139948)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 1\ **Last updated:** [July 27, 2018, 4:11pm UTC](https://discuss.elastic.co/t/winlogbeat-fails-to-add-geotags/139948 "2018-07-27T16:11:13Z")
Hi, We are shipping some windows logs using winlogbeat to our logstash nodes and then to our elastic cluster. We have successfully installed the geoip plugin on all of our elasticsearch nodes. Here is our logstash confi… --- ## [Unable to create elastic index through logstash output plugin with es index template in elastic DB](https://discuss.elastic.co/t/unable-to-create-elastic-index-through-logstash-output-plugin-with-es-index-template-in-elastic-db/139926)
**Author:** [@Thirumoorthi\_J](https://discuss.elastic.co/u/Thirumoorthi_J)\ **Replies:** 1\ **Last updated:** [July 27, 2018, 4:07pm UTC](https://discuss.elastic.co/t/unable-to-create-elastic-index-through-logstash-output-plugin-with-es-index-template-in-elastic-db/139926 "2018-07-27T16:07:38Z")
Scenario 1: The elastic index creation template is PUT into elastic DB first. The elastic index creation is attempted next with the mapping json as payload using PUT request. The elastic index is created successfully … --- ## [The grok part from logstash didn't take effect in elasticsearch](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867)
**Author:** [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Replies:** 15\ **Last updated:** [July 27, 2018, 4:04pm UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867 "2018-07-27T16:04:09Z")
Below is my logstash config: And I find only the document\_type is effect after I start the logstash. I go to elasticsearch: hostname:9200/indexName and find as below: So I find the grok part didn't take effect. Is … --- ## [Using multiple outputs and reliability](https://discuss.elastic.co/t/using-multiple-outputs-and-reliability/141610)
**Author:** [@Mark\_van\_Rossum](https://discuss.elastic.co/u/Mark_van_Rossum)\ **Replies:** 5\ **Last updated:** [July 27, 2018, 4:01pm UTC](https://discuss.elastic.co/t/using-multiple-outputs-and-reliability/141610 "2018-07-27T16:01:04Z")
We currently have a web cluster in Azure which streams HTTP logs from HAProxy over syslog to an Azure hosted logstash instance. We then have two outputs, the 1st writes the logs to an Azure storage bucket - which is rel… --- ## [Confused about error](https://discuss.elastic.co/t/confused-about-error/141966)
**Author:** [@wes](https://discuss.elastic.co/u/wes)\ **Replies:** 0\ **Last updated:** [July 27, 2018, 3:39pm UTC](https://discuss.elastic.co/t/confused-about-error/141966 "2018-07-27T15:39:08Z")
Can someone help me understand this error I am seeing? I switched my index from index =\> "%{\[@metadata\]\[beat\]}-%{\[@metadata\]\[version\]}-%{+YYYY.MM.dd}" to index =\> "%{\[fields\]\[group\]}-%{\[@metadata\]\[beat\]}" And now I g… --- ## [Logstash output: If field exists, use other index](https://discuss.elastic.co/t/logstash-output-if-field-exists-use-other-index/141958)
**Author:** [@johann](https://discuss.elastic.co/u/johann)\ **Replies:** 2\ **Last updated:** [July 27, 2018, 3:03pm UTC](https://discuss.elastic.co/t/logstash-output-if-field-exists-use-other-index/141958 "2018-07-27T15:03:25Z")
Hi all, I want to use an other index, if a specific field exists... My output-config looks like this: output { if \[kubernetes.namespace\] { elasticsearch { hosts =\> "my-elastic.local:9200" … --- ## [Aggregate data to month level from csv file](https://discuss.elastic.co/t/aggregate-data-to-month-level-from-csv-file/141956)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [July 27, 2018, 2:54pm UTC](https://discuss.elastic.co/t/aggregate-data-to-month-level-from-csv-file/141956 "2018-07-27T14:54:14Z")
Hi All I have a dataset in below form in the csv format Order Date Amount Order001 10-Jan-2018 10 Order001 11-Jan-2018 20 Order001 12-Jan-2018 30 Order001 13-Jan-2018 40 Order002 14-Jan-2018 50 Order… --- ## [Date type in logstash](https://discuss.elastic.co/t/date-type-in-logstash/141929)
**Author:** [@waitangi](https://discuss.elastic.co/u/waitangi)\ **Replies:** 1\ **Last updated:** [July 27, 2018, 2:35pm UTC](https://discuss.elastic.co/t/date-type-in-logstash/141929 "2018-07-27T14:35:44Z")
I really don't know it's the best place to ask my question. I use logstash to save logs data in elasticsearch. Grok: match =\> \["message", "%{TIMESTAMP\_ISO8601:request\_timestamp} %{IP:server\_ip} %{WORD:request\_method} %… --- ## ["object mapping for \[host\] tried to parse field \[host\] as object, but found a concrete value"](https://discuss.elastic.co/t/object-mapping-for-host-tried-to-parse-field-host-as-object-but-found-a-concrete-value/141761)
**Author:** [@Marcos\_Felix](https://discuss.elastic.co/u/Marcos_Felix)\ **Replies:** 6\ **Last updated:** [July 27, 2018, 2:32pm UTC](https://discuss.elastic.co/t/object-mapping-for-host-tried-to-parse-field-host-as-object-but-found-a-concrete-value/141761 "2018-07-27T14:32:24Z")
I am getting this error on my logstash log: "object mapping for \[host\] tried to parse field \[host\] as object, but found a concrete value" this started happening when I tried to log the syslog coming from fortigate, her… --- ## [GROK Parsing Problem - IP within brackets AND parentheses](https://discuss.elastic.co/t/grok-parsing-problem-ip-within-brackets-and-parentheses/141845)
**Author:** [@NeQter](https://discuss.elastic.co/u/NeQter)\ **Replies:** 9\ **Last updated:** [July 27, 2018, 1:12pm UTC](https://discuss.elastic.co/t/grok-parsing-problem-ip-within-brackets-and-parentheses/141845 "2018-07-27T13:12:14Z")
Hello. I am having some trouble parsing the following log: \<14\>Jul 26 13:37:17 NL-Syn1-RI Connection: User \[SYNNAS\\WIN7$\] from \[192.168.10.111(192.168.10.111)\] via \[CIFS(SMB2)\] accessed shared folder \[sysvol\]. This is … --- ## [Logstash is keep on running](https://discuss.elastic.co/t/logstash-is-keep-on-running/141880)
**Author:** [@Deepak\_Dandotiya](https://discuss.elastic.co/u/Deepak_Dandotiya)\ **Replies:** 1\ **Last updated:** [July 27, 2018, 11:52am UTC](https://discuss.elastic.co/t/logstash-is-keep-on-running/141880 "2018-07-27T11:52:53Z")
Hi, I am reading data from CSV filetest and dumping to Elasticsearch. Data is getting done properly and index is also getting created but Logstash process never ends. It has to be killed manually. please do help as the … --- ## [Handling 409 Exception -version conflict](https://discuss.elastic.co/t/handling-409-exception-version-conflict/141919)
**Author:** [@TANISHQ\_BATRA](https://discuss.elastic.co/u/TANISHQ_BATRA)\ **Replies:** 0\ **Last updated:** [July 27, 2018, 9:50am UTC](https://discuss.elastic.co/t/handling-409-exception-version-conflict/141919 "2018-07-27T09:50:55Z")
Hi, In our use case , We have a document , which has a collection of nested documents named "notes". Each document can have nested notes upto 1000 under field"notes". So when multiple nested objects i.e notes are adde… --- ## [Same message being sent twice](https://discuss.elastic.co/t/same-message-being-sent-twice/141893)
**Author:** [@xShadow](https://discuss.elastic.co/u/xShadow)\ **Replies:** 4\ **Last updated:** [July 27, 2018, 8:13am UTC](https://discuss.elastic.co/t/same-message-being-sent-twice/141893 "2018-07-27T08:13:47Z")
Hi, I am not sure if I should place this topic under logstash or elasticsearch but I was testing the function of sending logs from logstash to elasticsearch and I noticed that the same message appears twice in the elast… --- ## [Syslog pri not working as expected](https://discuss.elastic.co/t/syslog-pri-not-working-as-expected/141902)
**Author:** [@sreejiths](https://discuss.elastic.co/u/sreejiths)\ **Replies:** 0\ **Last updated:** [July 27, 2018, 8:06am UTC](https://discuss.elastic.co/t/syslog-pri-not-working-as-expected/141902 "2018-07-27T08:06:02Z")
We have ELK stack running on V5.6.8 for syslog analytics from Network devices ..We used syslog pri filter (give below . Section 1)on the grok to determine the severity levels of the syslogs .. I use syslog\_severity field… --- ## [How can I improve Logstash performance?](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882)
**Author:** [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\ **Replies:** 24\ **Last updated:** [July 27, 2018, 7:50am UTC](https://discuss.elastic.co/t/how-can-i-improve-logstash-performance/140882 "2018-07-27T07:50:02Z")
Hello, I have logs in file and my logstash.conf looks in the way shown below. When I run sudo ./logstash -f /path/logstash.conf -b 100000 -w 1 and monitor elasticsearch node in Kibana I see on the graph in document count… --- ## [Logstash 6.3 and redis conditional on id](https://discuss.elastic.co/t/logstash-6-3-and-redis-conditional-on-id/140735)
**Author:** [@wanexa](https://discuss.elastic.co/u/wanexa)\ **Replies:** 4\ **Last updated:** [July 27, 2018, 7:15am UTC](https://discuss.elastic.co/t/logstash-6-3-and-redis-conditional-on-id/140735 "2018-07-27T07:15:18Z")
Hello I have multiple redis input , I want to do some conditionnal based on key type \[id\] for the output but it doesn't work example: input { redis { key =\> "winlogbeat" host =\> "x.x.x.x" port =\> "6379" password … --- ## [Running Logstash only 1 row affected to elasticsearch](https://discuss.elastic.co/t/running-logstash-only-1-row-affected-to-elasticsearch/141868)
**Author:** [@dedi\_mi](https://discuss.elastic.co/u/dedi_mi)\ **Replies:** 2\ **Last updated:** [July 27, 2018, 6:33am UTC](https://discuss.elastic.co/t/running-logstash-only-1-row-affected-to-elasticsearch/141868 "2018-07-27T06:33:07Z")
Hi, I run the logstash via command prompt : logstash -f logstash.conf input { jdbc { jdbc\_driver\_library =\> "C:\\ELASTIC\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServ… --- ## [How to add a tag if device lookup using JDBC fails?](https://discuss.elastic.co/t/how-to-add-a-tag-if-device-lookup-using-jdbc-fails/140887)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 4\ **Last updated:** [July 27, 2018, 5:55am UTC](https://discuss.elastic.co/t/how-to-add-a-tag-if-device-lookup-using-jdbc-fails/140887 "2018-07-27T05:55:27Z")
Hi We have a MySQL database with all of our devices stored as a table with an ID, FQDN, IP and a service name. Now we want to have some assurance that if the device that is sending logs to our ELK stack isn't in our dat… --- ## [Http call in logstash filter](https://discuss.elastic.co/t/http-call-in-logstash-filter/141779)
**Author:** [@chandra0651](https://discuss.elastic.co/u/chandra0651)\ **Replies:** 2\ **Last updated:** [July 27, 2018, 5:21am UTC](https://discuss.elastic.co/t/http-call-in-logstash-filter/141779 "2018-07-27T05:21:15Z")
I have a document coming in from logstash input using HTTP poller , in filter section i need to make a http call to other end point with one specific field of document and get the output from end point in filter, at this… --- ## [Logstash Multiple Grok Add Field](https://discuss.elastic.co/t/logstash-multiple-grok-add-field/141683)
**Author:** [@insomniac](https://discuss.elastic.co/u/insomniac)\ **Replies:** 2\ **Last updated:** [July 27, 2018, 2:19am UTC](https://discuss.elastic.co/t/logstash-multiple-grok-add-field/141683 "2018-07-27T02:19:41Z")
I'm trying to have a single message parse through a hash array of match, and then check the message type after to add a field. grok { match =\> { "message" =\> \[ "%{NOTSPACE:syslogversion}%{SYSLOGTIMESTAMP:timest… --- ## [Invalid Frame Type, received: 82 and 79](https://discuss.elastic.co/t/invalid-frame-type-received-82-and-79/140113)
**Author:** [@nutchalum.k](https://discuss.elastic.co/u/nutchalum.k)\ **Replies:** 5\ **Last updated:** [July 26, 2018, 8:43pm UTC](https://discuss.elastic.co/t/invalid-frame-type-received-82-and-79/140113 "2018-07-26T20:43:27Z")
I've deploy Logstash (6.3.1) behind kubernetes ingress nginx that expose via TCP method and ELB (kubernetes service type is LoadBalancer). kind: ConfigMap apiVersion: v1 metadata: name: configmap-ingress-tcp-conf n… --- ## [Logstash not handling Byte Order Mark (BOM) correctly](https://discuss.elastic.co/t/logstash-not-handling-byte-order-mark-bom-correctly/141804)
**Author:** [@smbullet](https://discuss.elastic.co/u/smbullet)\ **Replies:** 2\ **Last updated:** [July 26, 2018, 7:46pm UTC](https://discuss.elastic.co/t/logstash-not-handling-byte-order-mark-bom-correctly/141804 "2018-07-26T19:46:50Z")
See reply for update. I have an rsyslog config that looks like this: $ModLoad imtcp $ModLoad imudp $template myFormat,"\<%pri%\> %timestamp% \<%syslogfacility%.%syslogpriority%\> %hostname% %syslogtag%: %msg%\\n" $ActionFi… --- ## [Remove fields object event](https://discuss.elastic.co/t/remove-fields-object-event/141826)
**Author:** [@robsonkades](https://discuss.elastic.co/u/robsonkades)\ **Replies:** 2\ **Last updated:** [July 26, 2018, 5:49pm UTC](https://discuss.elastic.co/t/remove-fields-object-event/141826 "2018-07-26T17:49:39Z")
Hello how do I remove all event objects dynamically https://www.rubydoc.info/gems/logstash-core-event/2.3.4/LogStash/Event I need this because my message comes through the metada, fields that are in it is also in the … --- ## [Possible to use external file as part of filter?](https://discuss.elastic.co/t/possible-to-use-external-file-as-part-of-filter/141398)
**Author:** [@S\_Chase](https://discuss.elastic.co/u/S_Chase)\ **Replies:** 3\ **Last updated:** [July 26, 2018, 4:20pm UTC](https://discuss.elastic.co/t/possible-to-use-external-file-as-part-of-filter/141398 "2018-07-26T16:20:33Z")
6.3.0 I've got certain tags being applied if the message body has certain IP Addresses in it. is it possible to maintain that separate IP address (or username) list in a separate file that the filter can look at along … --- ## [Order of treatment](https://discuss.elastic.co/t/order-of-treatment/141783)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 1\ **Last updated:** [July 26, 2018, 3:15pm UTC](https://discuss.elastic.co/t/order-of-treatment/141783 "2018-07-26T15:15:21Z")
Hello ! Here is my input config : input { file{ path =\> "C:\\Users\\GAUTSCPI\\Documents\\Elasticsearch\\Logs\\CDR-SBC\\prod/\*" start\_position =\> "end" tags =\> \["cdr"\] codec =\> plain { charset =\> "ISO-8859-1" } } } … --- ## [Xml filter: create filter definition based on xsd](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581)
**Author:** [@JPelastic](https://discuss.elastic.co/u/JPelastic)\ **Replies:** 8\ **Last updated:** [July 26, 2018, 3:08pm UTC](https://discuss.elastic.co/t/xml-filter-create-filter-definition-based-on-xsd/141581 "2018-07-26T15:08:21Z")
Hi Context: I need to put (several millions) XML invoices into ES and query and aggregate over their content. Proposed solution: use Logstash xml filter plugin to map the XML to JSON First question: For my context, wo… --- ## [Logstash (6.3) not starting anymore with DLQ](https://discuss.elastic.co/t/logstash-6-3-not-starting-anymore-with-dlq/140917)
**Author:** [@af\_andi](https://discuss.elastic.co/u/af_andi)\ **Replies:** 4\ **Last updated:** [July 26, 2018, 2:33pm UTC](https://discuss.elastic.co/t/logstash-6-3-not-starting-anymore-with-dlq/140917 "2018-07-26T14:33:41Z")
Hi Guys, since i'd upgraded to 6.3 i have a problem with the dead\_letter\_queue on one Logstash host (at the second everything works fine). I just enable the feature in the logstash.yml and leave the other dlq options t… --- ## [Logstash Elasticsearch Output with specific fields to index](https://discuss.elastic.co/t/logstash-elasticsearch-output-with-specific-fields-to-index/140914)
**Author:** [@chandra0651](https://discuss.elastic.co/u/chandra0651)\ **Replies:** 3\ **Last updated:** [July 26, 2018, 1:37pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-with-specific-fields-to-index/140914 "2018-07-26T13:37:50Z")
Is there possibility for logstash to ingest whole document to one index and specific fields to other index in the elastic search output. Example: If i have 10 fields coming in from input , can i write all the fields to … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=712) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=714)