# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=698 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 699 --- ## [Log Filter for different cluster](https://discuss.elastic.co/t/log-filter-for-different-cluster/144766)
**Author:** [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\ **Replies:** 2\ **Last updated:** [August 21, 2018, 11:06pm UTC](https://discuss.elastic.co/t/log-filter-for-different-cluster/144766 "2018-08-21T23:06:30Z")
Hi, I'm collecting logs using Filebeat and Logstash from multiple Kafka clusters. I want to filter this logs based on log-level in Logstash, so my current configuration is like: if \[log\_level\] == "TRACE" or \[log\_leve… --- ## [\[logstash.outputs.elasticsearch\] An unknown error occurred sending a bulk request to Elasticsearch](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-an-unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/145435)
**Author:** [@perryparktung](https://discuss.elastic.co/u/perryparktung)\ **Replies:** 1\ **Last updated:** [August 21, 2018, 9:25pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-an-unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/145435 "2018-08-21T21:25:20Z")
Hi, I am getting below error when starting Logstash. I tried restarting Elasticsearch then Logstash, but still have such issue. Please advise cause of it. Thanks! \[2018-08-21T11:54:17,570\]\[ERROR\]\[logstash.outputs.elas… --- ## [Logstash.conf error](https://discuss.elastic.co/t/logstash-conf-error/145440)
**Author:** [@nipunthathsara](https://discuss.elastic.co/u/nipunthathsara)\ **Replies:** 1\ **Last updated:** [August 21, 2018, 6:55pm UTC](https://discuss.elastic.co/t/logstash-conf-error/145440 "2018-08-21T18:55:21Z")
I'm following this tutorial to setup logstash for ballerina. https://ballerina.io/learn/how-to-observe-ballerina-code/#elastic-stack Created the logstash.conf with below content. input { beats { port =\> 5044 } } … --- ## [Extract information from field](https://discuss.elastic.co/t/extract-information-from-field/145376)
**Author:** [@saisimo02](https://discuss.elastic.co/u/saisimo02)\ **Replies:** 13\ **Last updated:** [August 21, 2018, 6:52pm UTC](https://discuss.elastic.co/t/extract-information-from-field/145376 "2018-08-21T18:52:01Z")
Hello, I am using Logstash to parse an XML file, I have something like that: \ I am looking for a way to get a new field with SPM whic… --- ## [Logstash filter optimisation](https://discuss.elastic.co/t/logstash-filter-optimisation/145403)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 3\ **Last updated:** [August 21, 2018, 6:48pm UTC](https://discuss.elastic.co/t/logstash-filter-optimisation/145403 "2018-08-21T18:48:15Z")
Hi, We are having logstash filter with around 30 different grok filters for diff types. What we suspect is whenever we onboard new type of logs, data takes long time to index in elasticsearch. I want to know how to br… --- ## [JDBC I/P Plugin -Connecting to DB2](https://discuss.elastic.co/t/jdbc-i-p-plugin-connecting-to-db2/145433)
**Author:** [@parvipul](https://discuss.elastic.co/u/parvipul)\ **Replies:** 0\ **Last updated:** [August 21, 2018, 4:49pm UTC](https://discuss.elastic.co/t/jdbc-i-p-plugin-connecting-to-db2/145433 "2018-08-21T16:49:32Z")
I trying to connect to DB2 database on Mainframe from my local m/c. I am using JDBC input plugin. It seems the jdbc driver license file is not getting loaded. Can someone please tell me how can I load the db2jcc\_license… --- ## [Kafka-logstash-database](https://discuss.elastic.co/t/kafka-logstash-database/145427)
**Author:** [@arp220](https://discuss.elastic.co/u/arp220)\ **Replies:** 0\ **Last updated:** [August 21, 2018, 4:03pm UTC](https://discuss.elastic.co/t/kafka-logstash-database/145427 "2018-08-21T16:03:11Z")
I have a server with two network cards. (192.168.1.20 , 192.168.1.50) Logstash is installed on this server. I want to logstash the first interface that reads data from Kafka. (192.168.1.20) And output the logstash wit… --- ## [Data sent to elasticsearch only after SIGTERM](https://discuss.elastic.co/t/data-sent-to-elasticsearch-only-after-sigterm/145384)
**Author:** [@nicolasl](https://discuss.elastic.co/u/nicolasl)\ **Replies:** 2\ **Last updated:** [August 21, 2018, 3:01pm UTC](https://discuss.elastic.co/t/data-sent-to-elasticsearch-only-after-sigterm/145384 "2018-08-21T15:01:58Z")
Hello, I'm using logstash to parse an XML file, split it, do some basic extraction and send results to elasticsearch. I also use kibana. All in kubernetes. Logstash's version is 6.2.4. Elasticsearch's version is 6.2.4… --- ## [How to disable parsing of nested filelds](https://discuss.elastic.co/t/how-to-disable-parsing-of-nested-filelds/145227)
**Author:** [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Replies:** 6\ **Last updated:** [August 21, 2018, 1:47pm UTC](https://discuss.elastic.co/t/how-to-disable-parsing-of-nested-filelds/145227 "2018-08-21T13:47:45Z")
Below is my logstash configuration input { beats { port =\> 5001 } kafka { bootstrap\_servers =\> "kafka-xxxxxxxxxxxxxxxx:9092" topics =\> \[ "java-prod", "java-pro… --- ## [Using wildcard on mutate for fieldname](https://discuss.elastic.co/t/using-wildcard-on-mutate-for-fieldname/145115)
**Author:** [@oms1226](https://discuss.elastic.co/u/oms1226)\ **Replies:** 3\ **Last updated:** [August 21, 2018, 12:57pm UTC](https://discuss.elastic.co/t/using-wildcard-on-mutate-for-fieldname/145115 "2018-08-21T12:57:22Z")
I used below in logstash.config mutate { convert =\> { "goog\*" =\> "float" } } But didn't work. I want to convert all the fieldname getting prefix 'goog' TO float type. What will I fix ? I am waiting your help! --- ## [Logstash- http\_poller: Filter need to execute after getting first and second URL request’s response](https://discuss.elastic.co/t/logstash-http-poller-filter-need-to-execute-after-getting-first-and-second-url-request-s-response/145373)
**Author:** [@Muhammed\_Aboobacker](https://discuss.elastic.co/u/Muhammed_Aboobacker)\ **Replies:** 0\ **Last updated:** [August 21, 2018, 11:11am UTC](https://discuss.elastic.co/t/logstash-http-poller-filter-need-to-execute-after-getting-first-and-second-url-request-s-response/145373 "2018-08-21T11:11:12Z")
I have two URLs and i need execute filters functionality after getting the responses of each . Because i need merge the response of these url response and parse responses and put it into the elastic search.But when i t… --- ## [How to create Arrays and Loop them such as "FOR" in logstash config file](https://discuss.elastic.co/t/how-to-create-arrays-and-loop-them-such-as-for-in-logstash-config-file/145352)
**Author:** [@snehabh](https://discuss.elastic.co/u/snehabh)\ **Replies:** 1\ **Last updated:** [August 21, 2018, 10:30am UTC](https://discuss.elastic.co/t/how-to-create-arrays-and-loop-them-such-as-for-in-logstash-config-file/145352 "2018-08-21T10:30:41Z")
I am new to ELK and need help to understand and execute the code. I need to create two arrays Array1--\>Errcode {E00,E001,E002} Array2--\>ErrorShortDesc {a1,a2,a3,a4,a5,a6,a7} now based to the above array values i need… --- ## [DNS Filter and Winlogbeat problem](https://discuss.elastic.co/t/dns-filter-and-winlogbeat-problem/145345)
**Author:** [@ronnykwon](https://discuss.elastic.co/u/ronnykwon)\ **Replies:** 0\ **Last updated:** [August 21, 2018, 9:37am UTC](https://discuss.elastic.co/t/dns-filter-and-winlogbeat-problem/145345 "2018-08-21T09:37:53Z")
I'm new to logstash and I've been using winlogbeat to get log event and I wanted to do a DNS lookup on the field Workstation While trying the dns filter I encounter a strange bug my pipeline configuration is as followe… --- ## [Howto convert UNIX timestamp json field into ISO8601 format](https://discuss.elastic.co/t/howto-convert-unix-timestamp-json-field-into-iso8601-format/145109)
**Author:** [@Roberto\_Giordani](https://discuss.elastic.co/u/Roberto_Giordani)\ **Replies:** 5\ **Last updated:** [August 21, 2018, 8:47am UTC](https://discuss.elastic.co/t/howto-convert-unix-timestamp-json-field-into-iso8601-format/145109 "2018-08-21T08:47:21Z")
Hi, I've allowed filebeat to parse a json log, but the Unix timestamp fields like this "EdgeStartTimestamp": 1534732311104000000, on Kibana is converted in this way EdgeStartTimestamp 1,534,732,311,104,000,0… --- ## [Email alert for message in tag](https://discuss.elastic.co/t/email-alert-for-message-in-tag/145162)
**Author:** [@WarrenG](https://discuss.elastic.co/u/WarrenG)\ **Replies:** 13\ **Last updated:** [August 21, 2018, 7:30am UTC](https://discuss.elastic.co/t/email-alert-for-message-in-tag/145162 "2018-08-21T07:30:37Z")
Hi All First try with emailing logs, I found this bit of information but im having issues working out how this would work for me. if "FOUND" in \[message\] { email { from =\> "logstash\_alert@compa… --- ## [Logstash - After a certain amount of time, can not connect to the database](https://discuss.elastic.co/t/logstash-after-a-certain-amount-of-time-can-not-connect-to-the-database/145314)
**Author:** [@hyukjun\_choi](https://discuss.elastic.co/u/hyukjun_choi)\ **Replies:** 0\ **Last updated:** [August 21, 2018, 7:26am UTC](https://discuss.elastic.co/t/logstash-after-a-certain-amount-of-time-can-not-connect-to-the-database/145314 "2018-08-21T07:26:06Z")
I am using version 6.2.1 of logstash. Use jdbc input to read MSSQL data every hour and then send data to remote logstash. 3-4 days work well. But after that I get the following error, but I have not found a reason yet. … --- ## [How do I know which filters apply to many types of logs?](https://discuss.elastic.co/t/how-do-i-know-which-filters-apply-to-many-types-of-logs/144922)
**Author:** [@jeongsyhd](https://discuss.elastic.co/u/jeongsyhd)\ **Replies:** 6\ **Last updated:** [August 21, 2018, 5:22am UTC](https://discuss.elastic.co/t/how-do-i-know-which-filters-apply-to-many-types-of-logs/144922 "2018-08-21T05:22:07Z")
I am currently receiving firewalls and IPSec VPN, WAF, IPS logs. There are four types of logs: Firewall, IPSec VPN, WAF, and IPS. I create an index pattern for each device's log and set it on the grok filter. So how d… --- ## [Logstash uses protobuf, running error: NoMethodError", :message=\>"undefined method \`msgclass' for nil:NilClass"](https://discuss.elastic.co/t/logstash-uses-protobuf-running-error-nomethoderror-message-undefined-method-msgclass-for-nil-nilclass/144806)
**Author:** [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\ **Replies:** 0\ **Last updated:** [August 17, 2018, 2:44am UTC](https://discuss.elastic.co/t/logstash-uses-protobuf-running-error-nomethoderror-message-undefined-method-msgclass-for-nil-nilclass/144806 "2018-08-17T02:44:11Z")
I use logstash to accept message data in the format protobuf， the protobuf.conf is setting : input { beats { port =\> 5044 ssl =\> false codec =\> protobuf { class\_name =\> \["Elk.ElkData"\] include\_path =\> \["/hom… --- ## [Logstash future](https://discuss.elastic.co/t/logstash-future/145099)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 3\ **Last updated:** [August 20, 2018, 9:30pm UTC](https://discuss.elastic.co/t/logstash-future/145099 "2018-08-20T21:30:55Z")
Is Elastic planning to deprecate the Logstash development? --- ## [SOLUTION FOR ERROR \*\*\* Could not find or load main class Files\*\*\*](https://discuss.elastic.co/t/solution-for-error-could-not-find-or-load-main-class-files/145241)
**Author:** [@ARTUSO](https://discuss.elastic.co/u/ARTUSO)\ **Replies:** 0\ **Last updated:** [August 20, 2018, 8:51pm UTC](https://discuss.elastic.co/t/solution-for-error-could-not-find-or-load-main-class-files/145241 "2018-08-20T20:51:21Z")
Hi folks, For fix the error Could not find or load main class Files when to run logstash.bat, you must add quote in the environment variable %CLASSPATH%. See it below more detais: Before: %JAVA% %JAVA\_OPTS% -cp %C… --- ## [Elasticsearch documents getting updated with same document id](https://discuss.elastic.co/t/elasticsearch-documents-getting-updated-with-same-document-id/144920)
**Author:** [@chandra0651](https://discuss.elastic.co/u/chandra0651)\ **Replies:** 1\ **Last updated:** [August 20, 2018, 6:57pm UTC](https://discuss.elastic.co/t/elasticsearch-documents-getting-updated-with-same-document-id/144920 "2018-08-20T18:57:19Z")
Hello i am using one of the field to be made as document id when ingesting in ES and so that if any document comes with same document id, it shouldn't get ingested or updated. But i see the documents are getting updated.… --- ## [Logstash filter issue remove unwanted fields](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048)
**Author:** [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Replies:** 7\ **Last updated:** [August 20, 2018, 6:31pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048 "2018-08-20T18:31:06Z")
hi All I have syslog setup and running ok, now try to use logstash and remove some unecessary logs before i send to ES below output show there is 2 date and stamps : like to remove 1 time stamp to stream line the messa… --- ## [Logstash.codec.sflow doesn't understand some record formats (1003, 1008, 1009 etc)](https://discuss.elastic.co/t/logstash-codec-sflow-doesnt-understand-some-record-formats-1003-1008-1009-etc/145221)
**Author:** [@TheZeke](https://discuss.elastic.co/u/TheZeke)\ **Replies:** 0\ **Last updated:** [August 20, 2018, 6:29pm UTC](https://discuss.elastic.co/t/logstash-codec-sflow-doesnt-understand-some-record-formats-1003-1008-1009-etc/145221 "2018-08-20T18:29:37Z")
I'm using ElastiFlow but seeing hundreds of warnings every minute for sFlow records saying logstash.codec.sflow does not understand records with certain formats. The logstash.codecs.flow GitHub hasn't had any activity o… --- ## [XML with multiple fields to Elastic Search](https://discuss.elastic.co/t/xml-with-multiple-fields-to-elastic-search/145218)
**Author:** [@rogueleader](https://discuss.elastic.co/u/rogueleader)\ **Replies:** 0\ **Last updated:** [August 20, 2018, 6:12pm UTC](https://discuss.elastic.co/t/xml-with-multiple-fields-to-elastic-search/145218 "2018-08-20T18:12:43Z")
Hi, I am a newbie to ELK and have been browsing these forums and other tutorials in search of some comprehensive guide for handling XML files.Now most of the examples i saw are handling simple XMLs ( and pretty small one… --- ## [Logstash is modifying my SQL and causing it crash](https://discuss.elastic.co/t/logstash-is-modifying-my-sql-and-causing-it-crash/145215)
**Author:** [@nomoneynoproblems](https://discuss.elastic.co/u/nomoneynoproblems)\ **Replies:** 0\ **Last updated:** [August 20, 2018, 5:39pm UTC](https://discuss.elastic.co/t/logstash-is-modifying-my-sql-and-causing-it-crash/145215 "2018-08-20T17:39:36Z")
This query runs fine in MS SQL management studio: SELECT documents.\*, FCCorgsstandardized.ein, FCCorgsstandardized.orgid, FCCorgsstandardized.sourcestring, FCCorgsstandardized.cmteid, FCCorgsstandardized.unique… --- ## [Grok comma separated text](https://discuss.elastic.co/t/grok-comma-separated-text/145186)
**Author:** [@MarcusCaepio](https://discuss.elastic.co/u/MarcusCaepio)\ **Replies:** 2\ **Last updated:** [August 20, 2018, 3:26pm UTC](https://discuss.elastic.co/t/grok-comma-separated-text/145186 "2018-08-20T15:26:33Z")
Hi all, I want to grok a message like blablabla: entry\_1, entry\_2, entry\_3 the message has at least 1 entry after the : and CAN HAVE up to x entries, which are separated with ",". I need a grok which matches the firs… --- ## [Automatic data enrichment](https://discuss.elastic.co/t/automatic-data-enrichment/145198)
**Author:** [@Andrew\_Vdovin](https://discuss.elastic.co/u/Andrew_Vdovin)\ **Replies:** 0\ **Last updated:** [August 20, 2018, 3:07pm UTC](https://discuss.elastic.co/t/automatic-data-enrichment/145198 "2018-08-20T15:07:35Z")
This will be very helpful for many threathunters https://twitter.com/andriinb/status/1031554843083780097 --- ## [Running Logstash config in background](https://discuss.elastic.co/t/running-logstash-config-in-background/144915)
**Author:** [@Divit\_Sharma](https://discuss.elastic.co/u/Divit_Sharma)\ **Replies:** 10\ **Last updated:** [August 20, 2018, 2:03pm UTC](https://discuss.elastic.co/t/running-logstash-config-in-background/144915 "2018-08-20T14:03:12Z")
I have created several config files and all are running with single run and stops when I close the terminal. How to make those config run forever in background and also do I have to run them again when server goes down/r… --- ## [Error in config when logstash runs](https://discuss.elastic.co/t/error-in-config-when-logstash-runs/145171)
**Author:** [@afs2r](https://discuss.elastic.co/u/afs2r)\ **Replies:** 6\ **Last updated:** [August 20, 2018, 1:59pm UTC](https://discuss.elastic.co/t/error-in-config-when-logstash-runs/145171 "2018-08-20T13:59:30Z")
Hello, First, logstash does not listen on tcp port 5544 as I mentioned in my config file... And second, I cannot figure it out : I have this error repeatedly (the end of config file): \[2018-08-20T14:51:51,439\]\[ERROR\]… --- ## [How to parse irregular xml logs](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646)
**Author:** [@dinesh1](https://discuss.elastic.co/u/dinesh1)\ **Replies:** 10\ **Last updated:** [August 20, 2018, 12:45pm UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646 "2018-08-20T12:45:03Z")
Hi How to parse irregular xml logs sample log is shown below. DEBUG 2018/08/16 13:01:17.307842 Response XML:- \
GR\_IFA\_IN\_B2C\_127\_0\_0\_1\_01\