# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=899 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 900 --- ## [Remove\_field for "type" is not working](https://discuss.elastic.co/t/remove-field-for-type-is-not-working/102618)
**Author:** [@surajbs](https://discuss.elastic.co/u/surajbs)\ **Replies:** 0\ **Last updated:** [October 3, 2017, 9:19pm UTC](https://discuss.elastic.co/t/remove-field-for-type-is-not-working/102618 "2017-10-03T21:19:24Z")
Hi, I am using multiple files as input, where type =\> csv is for loading csv file into elatsicsearch and type =\> syslog is for tracking the exception. My config looks like below: input { file { path =\> "path-to-csv… --- ## [Logstash filter conditional if tags or](https://discuss.elastic.co/t/logstash-filter-conditional-if-tags-or/102479)
**Author:** [@ktella](https://discuss.elastic.co/u/ktella)\ **Replies:** 2\ **Last updated:** [October 3, 2017, 7:16pm UTC](https://discuss.elastic.co/t/logstash-filter-conditional-if-tags-or/102479 "2017-10-03T19:16:47Z")
Am i doing something wrong here ? Logs with tag dcsaghosts are the ones only processed by the filter and ignoring the logs with tag pcwsaghosts filter { if "dcsaghosts" in \[tags\] or "pcwsaghosts" in \[tags\] { if "dc-i… --- ## [Logstash monitoring pipeline and SSL verification](https://discuss.elastic.co/t/logstash-monitoring-pipeline-and-ssl-verification/102600)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 0\ **Last updated:** [October 3, 2017, 6:16pm UTC](https://discuss.elastic.co/t/logstash-monitoring-pipeline-and-ssl-verification/102600 "2017-10-03T18:16:58Z")
Hi All, I'm wondering if it is possible to disable SSL certificate verification by LS v.5.6.2 completely? with the following settings: xpack.monitoring.elasticsearch.url: \["https://es-node"\] xpack.monitoring.elasticse… --- ## [Issue with Aggregate task\_id undefined variable](https://discuss.elastic.co/t/issue-with-aggregate-task-id-undefined-variable/102314)
**Author:** [@ffernandez](https://discuss.elastic.co/u/ffernandez)\ **Replies:** 1\ **Last updated:** [October 3, 2017, 6:14pm UTC](https://discuss.elastic.co/t/issue-with-aggregate-task-id-undefined-variable/102314 "2017-10-03T18:14:15Z")
Hello! I'm trying to use the aggregate filter plugin to merge a couple of events using the field "thread" obtained via the grok filter plugin. However, every time I try to run logstash, whether is from the cli or system… --- ## [Installing plugins behind proxy when installed from deb package](https://discuss.elastic.co/t/installing-plugins-behind-proxy-when-installed-from-deb-package/102405)
**Author:** [@wmathews](https://discuss.elastic.co/u/wmathews)\ **Replies:** 2\ **Last updated:** [October 3, 2017, 5:16pm UTC](https://discuss.elastic.co/t/installing-plugins-behind-proxy-when-installed-from-deb-package/102405 "2017-10-03T17:16:52Z")
I'm running 5.6, and I've had a hell of a time just trying to get the logstash-plugin install command to work. I've read it needs to be run as logstash user with http\_proxy and the JRUBY\_OPTS set, but none of these thin… --- ## [Logstash not accepting Filebeat data - possible java issue?](https://discuss.elastic.co/t/logstash-not-accepting-filebeat-data-possible-java-issue/102594)
**Author:** [@h3llocomputer](https://discuss.elastic.co/u/h3llocomputer)\ **Replies:** 1\ **Last updated:** [October 3, 2017, 4:50pm UTC](https://discuss.elastic.co/t/logstash-not-accepting-filebeat-data-possible-java-issue/102594 "2017-10-03T16:50:24Z")
Hey guys. I am a newbie and I'm testing out ELK. I am running Filebeat 5.6 on a server and attempting to forward some json files to Logstash (5.6). On the Filebeat server, I see the following errors: 2017/10/03 16:0… --- ## [Logstash to logstash results in illegible data](https://discuss.elastic.co/t/logstash-to-logstash-results-in-illegible-data/102593)
**Author:** [@Justin.DynamicD](https://discuss.elastic.co/u/Justin.DynamicD)\ **Replies:** 0\ **Last updated:** [October 3, 2017, 4:41pm UTC](https://discuss.elastic.co/t/logstash-to-logstash-results-in-illegible-data/102593 "2017-10-03T16:41:53Z")
We have a scenario where we are attempting to configure a logstash server within a location/vpc with multiple inputs in order to gather the appropriate information (lets call it LS-A), but then wish to use a singular out… --- ## [Grok - extract file extension from file name](https://discuss.elastic.co/t/grok-extract-file-extension-from-file-name/102528)
**Author:** [@evild3ad](https://discuss.elastic.co/u/evild3ad)\ **Replies:** 3\ **Last updated:** [October 3, 2017, 2:15pm UTC](https://discuss.elastic.co/t/grok-extract-file-extension-from-file-name/102528 "2017-10-03T14:15:46Z")
Hi, I have problems with creating a grok filter to extract the file extension from the file name field. Example data: 9151\_to\_9152.log 80bddb9998f9f66bc44b59d9899bbbb853b8958ecd9188e14fd0828e00246050.txt ($FILE\_NAME) … --- ## [Logstash re-register on POSIX SIG](https://discuss.elastic.co/t/logstash-re-register-on-posix-sig/102558)
**Author:** [@Conall\_Prendergast](https://discuss.elastic.co/u/Conall_Prendergast)\ **Replies:** 0\ **Last updated:** [October 3, 2017, 1:03pm UTC](https://discuss.elastic.co/t/logstash-re-register-on-posix-sig/102558 "2017-10-03T13:03:21Z")
Logstash will listen for a SIGHUP and reload the configuration file when it receives one. It listens for SIGSTP and will safely shutdown the pipeline. Is there a way to get logstash to re-register a plugin by giving it… --- ## [Grok custom pattern is not working](https://discuss.elastic.co/t/grok-custom-pattern-is-not-working/102545)
**Author:** [@Aparna\_Thomas](https://discuss.elastic.co/u/Aparna_Thomas)\ **Replies:** 1\ **Last updated:** [October 3, 2017, 12:38pm UTC](https://discuss.elastic.co/t/grok-custom-pattern-is-not-working/102545 "2017-10-03T12:38:21Z")
In logstash, we were trying to parse a sample log using custom grok pattern. The sample log looks like: 2017.09.26 00:59:47:158 UTC | Info | Sip | UserProfileNonCall \[Thread #231\] | +7777777777 | callhalf-1000000… --- ## [Secure Communication Between Filebeat and Logstash](https://discuss.elastic.co/t/secure-communication-between-filebeat-and-logstash/102552)
**Author:** [@aditya\_soni](https://discuss.elastic.co/u/aditya_soni)\ **Replies:** 0\ **Last updated:** [October 3, 2017, 11:47am UTC](https://discuss.elastic.co/t/secure-communication-between-filebeat-and-logstash/102552 "2017-10-03T11:47:24Z")
Hi All, I am trying to configuring the secure communication between Logstash and Filebeat by the reference of the link provided https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html#testi… --- ## [How to identify JDBC pipeline](https://discuss.elastic.co/t/how-to-identify-jdbc-pipeline/102281)
**Author:** [@feijiangnan](https://discuss.elastic.co/u/feijiangnan)\ **Replies:** 4\ **Last updated:** [October 3, 2017, 9:39am UTC](https://discuss.elastic.co/t/how-to-identify-jdbc-pipeline/102281 "2017-10-03T09:39:36Z")
Hello, I have pipelines of Filebeat which are identified by \[fields\]\[document\_type\] in logstash, and the indices are named with \[fields\]\[document\_type\] . Now, I am adding JDBC pipeline, how can I set \[fields\]\[document\_t… --- ## [Io timeout using filebeat and logstash](https://discuss.elastic.co/t/io-timeout-using-filebeat-and-logstash/102535)
**Author:** [@lishunan246](https://discuss.elastic.co/u/lishunan246)\ **Replies:** 0\ **Last updated:** [October 3, 2017, 9:09am UTC](https://discuss.elastic.co/t/io-timeout-using-filebeat-and-logstash/102535 "2017-10-03T09:09:37Z")
I am using filebeat 5.5.1 and logstash 5.5.1 on Debian 8 to collect logs and send them to InfluxDB. Logstash and filebeat are installed as a whole on 20 servers forwarding logs to a single InfluxDB server. Logstash-inp… --- ## [Modifying / using beats field in logstash filter](https://discuss.elastic.co/t/modifying-using-beats-field-in-logstash-filter/102473)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 1\ **Last updated:** [October 3, 2017, 5:50am UTC](https://discuss.elastic.co/t/modifying-using-beats-field-in-logstash-filter/102473 "2017-10-03T05:50:51Z")
Hi there, I'm sending some data via filebeat into logstash and have the connections working properly. I'm trying to use the data in beat.hostname to cut out ".local" from a string and store that into a new field called … --- ## [Logstash failed](https://discuss.elastic.co/t/logstash-failed/102385)
**Author:** [@Jose\_Campos](https://discuss.elastic.co/u/Jose_Campos)\ **Replies:** 2\ **Last updated:** [October 2, 2017, 9:59pm UTC](https://discuss.elastic.co/t/logstash-failed/102385 "2017-10-02T21:59:28Z")
Hello I was carrying the load of my information with public and internal IP's and the internal ones I load them, but the public gives me this error object field starting or ending with a \[.\] makes object resolution amb… --- ## [Kafka output plugin loses all data when Kafka is down](https://discuss.elastic.co/t/kafka-output-plugin-loses-all-data-when-kafka-is-down/102468)
**Author:** [@quadd5856](https://discuss.elastic.co/u/quadd5856)\ **Replies:** 1\ **Last updated:** [October 2, 2017, 8:22pm UTC](https://discuss.elastic.co/t/kafka-output-plugin-loses-all-data-when-kafka-is-down/102468 "2017-10-02T20:22:48Z")
I am investigating why we lost data from certain servers during a network outage. We are able to reproduce the conditions and can confirm that when the Kafka server is unavailable, anything sent to logstash is lost. The… --- ## [An error occurs when loading loads with " logstash "](https://discuss.elastic.co/t/an-error-occurs-when-loading-loads-with-logstash/102367)
**Author:** [@taetaetae](https://discuss.elastic.co/u/taetaetae)\ **Replies:** 7\ **Last updated:** [October 2, 2017, 5:51pm UTC](https://discuss.elastic.co/t/an-error-occurs-when-loading-loads-with-logstash/102367 "2017-10-02T17:51:41Z")
hi An error occurs when loading loads with " logstash ". The setup file is shown below. input { http { host =\> "0.0.0.0" port =\> "28080" codec =\> es\_bulk {} } } filter { mutate { remove\_field =\> \["host","@tim… --- ## [Kv filter](https://discuss.elastic.co/t/kv-filter/102455)
**Author:** [@Pola](https://discuss.elastic.co/u/Pola)\ **Replies:** 1\ **Last updated:** [October 2, 2017, 4:53pm UTC](https://discuss.elastic.co/t/kv-filter/102455 "2017-10-02T16:53:54Z")
Hi, I am trying to filter key values from 'message' field "message": "\<188\>Oct 2 16:17:39.000 68.85.2.39 alarm\[1318\]: Alarm set: License color=YELLOW, class=SECOND, reason=Protocol usage requires a license\\n" I wante… --- ## [Back up of Elastic Serach Data](https://discuss.elastic.co/t/back-up-of-elastic-serach-data/102438)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 1\ **Last updated:** [October 2, 2017, 3:58pm UTC](https://discuss.elastic.co/t/back-up-of-elastic-serach-data/102438 "2017-10-02T15:58:16Z")
Hi - My ES is hosted on a Virtual Machine, receiving data from multiple nodes, Disk is reaching to the maximum with the data (indices). I'd like to take the backup of last 30 days of data and purge it and move the back… --- ## [Private ips and dns name into the worldmap](https://discuss.elastic.co/t/private-ips-and-dns-name-into-the-worldmap/102325)
**Author:** [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Replies:** 5\ **Last updated:** [October 2, 2017, 2:34pm UTC](https://discuss.elastic.co/t/private-ips-and-dns-name-into-the-worldmap/102325 "2017-10-02T14:34:08Z")
Hi folks! Just think about any chance how to perform visualization over the wordlmap in case I've manipulate with domain name or internal ip? Any mechanism to do that? --- ## [Please distribute libjffi-1.2.so for s390x](https://discuss.elastic.co/t/please-distribute-libjffi-1-2-so-for-s390x/102304)
**Author:** [@benjsmi](https://discuss.elastic.co/u/benjsmi)\ **Replies:** 2\ **Last updated:** [October 2, 2017, 1:51pm UTC](https://discuss.elastic.co/t/please-distribute-libjffi-1-2-so-for-s390x/102304 "2017-10-02T13:51:53Z")
When I download the latest stable Logstash from here, which at the time of writing is v5.6.2, I see a structure of folders that has me perplexed. There is a compiled shared object binary called libjffi that is required … --- ## [JDBC Input and Scheduled Filter - events being missed](https://discuss.elastic.co/t/jdbc-input-and-scheduled-filter-events-being-missed/99603)
**Author:** [@stevedearl](https://discuss.elastic.co/u/stevedearl)\ **Replies:** 2\ **Last updated:** [October 2, 2017, 10:51am UTC](https://discuss.elastic.co/t/jdbc-input-and-scheduled-filter-events-being-missed/99603 "2017-10-02T10:51:09Z")
Hi All, Not sure there’s much that can be done about this but I thought I’d ask the question. I’m running the JDBC Input plugin to obtain new events from our database. The input is set to run every 2 minutes. I’m also… --- ## [Logstash faield. "Page file size 0 different to configured page"](https://discuss.elastic.co/t/logstash-faield-page-file-size-0-different-to-configured-page/102335)
**Author:** [@vl.zemtsov](https://discuss.elastic.co/u/vl.zemtsov)\ **Replies:** 2\ **Last updated:** [October 2, 2017, 5:15am UTC](https://discuss.elastic.co/t/logstash-faield-page-file-size-0-different-to-configured-page/102335 "2017-10-02T05:15:07Z")
Hello. Logstash version - 5.5.1 Logstash cant start. In log: \` Logstash failed to create queue {"exception"=\>"Page file size 0 different to configured page capacity 262144000 for /usr/share/logstash/data/queue/main/… --- ## [Logstash JDBC MySQL \> Elastic... not all Records are imported](https://discuss.elastic.co/t/logstash-jdbc-mysql-elastic-not-all-records-are-imported/101175)
**Author:** [@open7](https://discuss.elastic.co/u/open7)\ **Replies:** 3\ **Last updated:** [October 2, 2017, 10:30am UTC](https://discuss.elastic.co/t/logstash-jdbc-mysql-elastic-not-all-records-are-imported/101175 "2017-10-02T10:30:50Z")
We have a MySQL Table with this kind of Datasets (10 Tables, a few hundred thousands Entries): field\_129: {"MANDT":"200","BUKRS":"3000","ANLN1":"000000002130","ANLN2":"0194","ANLKL":"00002000","GEGST":"","ANLAR":"","E… --- ## [Problem with unicode](https://discuss.elastic.co/t/problem-with-unicode/102422)
**Author:** [@cwt](https://discuss.elastic.co/u/cwt)\ **Replies:** 0\ **Last updated:** [October 2, 2017, 10:03am UTC](https://discuss.elastic.co/t/problem-with-unicode/102422 "2017-10-02T10:03:00Z")
Hi all, How can I avoid such messages: \[2017-10-02T12:57:26,652\]\[WARN \]\[logstash.codecs.plain \] Received an event that has a different character encoding than you configured. {:text=\>"\<118\>Oct 2 09:54:27 fw01-ips … --- ## [Separate configs, different indexes yet both logs go to both indexes](https://discuss.elastic.co/t/separate-configs-different-indexes-yet-both-logs-go-to-both-indexes/102411)
**Author:** [@X\_Calibur](https://discuss.elastic.co/u/X_Calibur)\ **Replies:** 1\ **Last updated:** [October 2, 2017, 8:18am UTC](https://discuss.elastic.co/t/separate-configs-different-indexes-yet-both-logs-go-to-both-indexes/102411 "2017-10-02T08:18:06Z")
I have the following 2 config files: ############IMAP Email reader############ input { imap { host =\> "imap.gmail.com" user =\> "account@gmail.com" password =\> "pass" secure =\> true port =\> 993 check\_interval =\> … --- ## [Jdbc\_streaming filter - unable to map netflow fields?](https://discuss.elastic.co/t/jdbc-streaming-filter-unable-to-map-netflow-fields/102369)
**Author:** [@djluko](https://discuss.elastic.co/u/djluko)\ **Replies:** 3\ **Last updated:** [October 2, 2017, 7:33am UTC](https://discuss.elastic.co/t/jdbc-streaming-filter-unable-to-map-netflow-fields/102369 "2017-10-02T07:33:41Z")
Hi all, having trouble mapping fields from netflow records, onto an SQL statement exected by the jdbc\_streaming driver against a MySQL database. Scenario: logstash is configured to read netflow data from UDP port 2055. … --- ## [Merge values in JSON (jdbc events logstash)](https://discuss.elastic.co/t/merge-values-in-json-jdbc-events-logstash/99367)
**Author:** [@malhotras](https://discuss.elastic.co/u/malhotras)\ **Replies:** 13\ **Last updated:** [October 2, 2017, 6:49am UTC](https://discuss.elastic.co/t/merge-values-in-json-jdbc-events-logstash/99367 "2017-10-02T06:49:48Z")
I have a JSON structure like this: { "details": \[ { "place": "abc", "group": 3, "year": 2006, "id": 1304, "street": "xyz 14", "lf\_number": "0118", "code": 4433, "name": "abc coorperation", "group2": 3817… --- ## [Does anyone have working bind9 logstash config?](https://discuss.elastic.co/t/does-anyone-have-working-bind9-logstash-config/102407)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 0\ **Last updated:** [October 2, 2017, 6:29am UTC](https://discuss.elastic.co/t/does-anyone-have-working-bind9-logstash-config/102407 "2017-10-02T06:29:44Z")
Hi Guys, Does anyone have working bind9 logstash config file to parse the DNS logs? Please let me know Thanks in advance! --- ## [Logstatsh plugins issue](https://discuss.elastic.co/t/logstatsh-plugins-issue/102404)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 2\ **Last updated:** [October 2, 2017, 6:05am UTC](https://discuss.elastic.co/t/logstatsh-plugins-issue/102404 "2017-10-02T06:05:54Z")
Hi Guys, How do I run plugins with logstash? I mean I tried entering below setting in my /etc/logstash/logstash.yml but after restarting its not picking up the desired setting and my netflow ports not listing. However … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=898) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=900)