# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=640 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 641 --- ## [Is the logstash mutate filter enough to get field indexed in ES with the correct type](https://discuss.elastic.co/t/is-the-logstash-mutate-filter-enough-to-get-field-indexed-in-es-with-the-correct-type/158973)
**Author:** [@lightonseo](https://discuss.elastic.co/u/lightonseo)\ **Replies:** 4\ **Last updated:** [December 1, 2018, 11:37am UTC](https://discuss.elastic.co/t/is-the-logstash-mutate-filter-enough-to-get-field-indexed-in-es-with-the-correct-type/158973 "2018-12-01T11:37:41Z")
Hi I m not sure to have well understood what the mutate filter do and what it doesn't do. After a lot of pain I've finaly imported CSV datas into elastic. Logstash was treating every single csv field as a string, obvio… --- ## [An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error\_message=\>"\\"\\\\xC4\\" from ASCII-8BIT to UTF-8", :error\_class=\>"LogStash::Json::GeneratorError"](https://discuss.elastic.co/t/an-unknown-error-occurred-sending-a-bulk-request-to-elasticsearch-we-will-retry-indefinitely-error-message-xc4-from-ascii-8bit-to-utf-8-error-class-logstash-generatorerror/155826)
**Author:** [@pank](https://discuss.elastic.co/u/pank)\ **Replies:** 0\ **Last updated:** [November 8, 2018, 6:51am UTC](https://discuss.elastic.co/t/an-unknown-error-occurred-sending-a-bulk-request-to-elasticsearch-we-will-retry-indefinitely-error-message-xc4-from-ascii-8bit-to-utf-8-error-class-logstash-generatorerror/155826 "2018-11-08T06:51:54Z")
I am using logstash 6.4.0 and elasticsearch 6.4.0. I have these config files in test.config and test1.config. I have created pipeline id in files pipelines.yml . I have simple two config file test input { jdbc … --- ## [I m failing in getting csv data and index those on elastic using logstash on Windows](https://discuss.elastic.co/t/i-m-failing-in-getting-csv-data-and-index-those-on-elastic-using-logstash-on-windows/158756)
**Author:** [@lightonseo](https://discuss.elastic.co/u/lightonseo)\ **Replies:** 2\ **Last updated:** [November 30, 2018, 8:48pm UTC](https://discuss.elastic.co/t/i-m-failing-in-getting-csv-data-and-index-those-on-elastic-using-logstash-on-windows/158756 "2018-11-30T20:48:59Z")
hi everybody, please apolgize me for my english I’m new to ELK, running on Windows 10 for the moment. I'm using wamp server (not sure if that's a relevant informations). It’s two days now that I’m facing an issue that… --- ## [TCP Input - behaviour under load](https://discuss.elastic.co/t/tcp-input-behaviour-under-load/158913)
**Author:** [@paulkeogh](https://discuss.elastic.co/u/paulkeogh)\ **Replies:** 1\ **Last updated:** [November 30, 2018, 5:57pm UTC](https://discuss.elastic.co/t/tcp-input-behaviour-under-load/158913 "2018-11-30T17:57:16Z")
I was looking at the method in the TCP input plugin that enqueues incoming documents onto the output queue. def enqueue\_decorated(event, client\_ip\_address, client\_address, client\_port, socket) event.set(HOST\_FIELD, clie… --- ## [Logstash Config File Error: type 'string'](https://discuss.elastic.co/t/logstash-config-file-error-type-string/158915)
**Author:** [@victorhearnyeates](https://discuss.elastic.co/u/victorhearnyeates)\ **Replies:** 1\ **Last updated:** [November 30, 2018, 5:50pm UTC](https://discuss.elastic.co/t/logstash-config-file-error-type-string/158915 "2018-11-30T17:50:49Z")
Hi, I'm currently trying to ingest a CSV file containing blogposts, and I'm getting the error Exiting: error loading config file: type 'string' is not supported on top level of config, only dictionary or list (source:'… --- ## [What is the best way to monitor SNMP?](https://discuss.elastic.co/t/what-is-the-best-way-to-monitor-snmp/158939)
**Author:** [@rueldu](https://discuss.elastic.co/u/rueldu)\ **Replies:** 0\ **Last updated:** [November 30, 2018, 4:21pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-monitor-snmp/158939 "2018-11-30T16:21:17Z")
Hello, I am monitoring SNMP through logstash and its plugins, but it is a pain of head since they are 20 switch core and I have to build the OID for each of the switches since they are of different brands. My question is… --- ## [Test on beat version doesn't work](https://discuss.elastic.co/t/test-on-beat-version-doesnt-work/158152)
**Author:** [@Yilmaz\_Cam](https://discuss.elastic.co/u/Yilmaz_Cam)\ **Replies:** 8\ **Last updated:** [November 30, 2018, 4:20pm UTC](https://discuss.elastic.co/t/test-on-beat-version-doesnt-work/158152 "2018-11-30T16:20:29Z")
Hello Currently i have 2 versions of beat on my infra 5.5 and 6.4, my logstash version is 5.6 so compatible with beat client 5.5 and 6.4. I try to detect the beat version of the document and redirect it to correct outp… --- ## [Snmp input plugin doesn't show any activity when configuring snmpv3](https://discuss.elastic.co/t/snmp-input-plugin-doesnt-show-any-activity-when-configuring-snmpv3/158886)
**Author:** [@olivier974](https://discuss.elastic.co/u/olivier974)\ **Replies:** 9\ **Last updated:** [November 30, 2018, 3:25pm UTC](https://discuss.elastic.co/t/snmp-input-plugin-doesnt-show-any-activity-when-configuring-snmpv3/158886 "2018-11-30T15:25:58Z")
Hi all, I'm trying to get the snmp input plugin work using SNMPv3 and, if the plugin doesn't show any error on start up, It doesn't retreive any information either. I'm using logstash 6.5 with the following configurati… --- ## [Elasticsearch input plugin](https://discuss.elastic.co/t/elasticsearch-input-plugin/158891)
**Author:** [@imaad](https://discuss.elastic.co/u/imaad)\ **Replies:** 10\ **Last updated:** [November 30, 2018, 3:25pm UTC](https://discuss.elastic.co/t/elasticsearch-input-plugin/158891 "2018-11-30T15:25:33Z")
Hello, I am running a simple logstash configuration : input { elasticsearch { hosts =\> "http://:9200/" index =\> "A" query =\> '{ "query": {"bool": {"must": \[ {"term": { "name": { "value": "X" } }} \]}} }' s… --- ## [Kv filter killing logstash performance (II) \[Solved\]](https://discuss.elastic.co/t/kv-filter-killing-logstash-performance-ii-solved/158435)
**Author:** [@nacesprin](https://discuss.elastic.co/u/nacesprin)\ **Replies:** 6\ **Last updated:** [November 30, 2018, 10:48am UTC](https://discuss.elastic.co/t/kv-filter-killing-logstash-performance-ii-solved/158435 "2018-11-30T10:48:19Z")
Hello. I have exactly the same problem like KV filter killing logstash performance but I continue on this post due to I cannot reply on it. I mean, by using KV filter, the CPU go after some hours at 100%. If I disable K… --- ## [Logstash - Manipulate field with special characters](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203)
**Author:** [@g.le](https://discuss.elastic.co/u/g.le)\ **Replies:** 15\ **Last updated:** [November 30, 2018, 10:26am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203 "2018-11-30T10:26:08Z")
Hello, I'm trying to manipulate a bunch of fields that have names with special characters (e.g. @ or dots). This is a usual case if someone uses Logstash-logback-encoder for shipping logs from an application to logstas… --- ## [Is it possible to have a reusable template for the line codec format?](https://discuss.elastic.co/t/is-it-possible-to-have-a-reusable-template-for-the-line-codec-format/158880)
**Author:** [@naga\_kunchala](https://discuss.elastic.co/u/naga_kunchala)\ **Replies:** 3\ **Last updated:** [November 30, 2018, 10:12am UTC](https://discuss.elastic.co/t/is-it-possible-to-have-a-reusable-template-for-the-line-codec-format/158880 "2018-11-30T10:12:28Z")
\*\*Is it possible to have a reusable template for the line codec format ? The same is in multiple places so it would be nice to have it once so all messages are the same regardless of which file { … } it was generated fro… --- ## [Mapper \[xxxx\] of different type, current\_type \[date\], merged\_type \[text\]](https://discuss.elastic.co/t/mapper-xxxx-of-different-type-current-type-date-merged-type-text/158626)
**Author:** [@Saad\_Rabhi](https://discuss.elastic.co/u/Saad_Rabhi)\ **Replies:** 2\ **Last updated:** [November 30, 2018, 9:39am UTC](https://discuss.elastic.co/t/mapper-xxxx-of-different-type-current-type-date-merged-type-text/158626 "2018-11-30T09:39:19Z")
Hi everybody, I'm a newbie on ELK and for some unknown reasons, as it was working without any changes on my side in any configuration, I'm facing to following issue : mapper \[hits.data.billing\_address.postal\_code\] of d… --- ## [The mutate's gsub result is incorrect](https://discuss.elastic.co/t/the-mutates-gsub-result-is-incorrect/158838)
**Author:** [@jmkim](https://discuss.elastic.co/u/jmkim)\ **Replies:** 10\ **Last updated:** [November 30, 2018, 9:09am UTC](https://discuss.elastic.co/t/the-mutates-gsub-result-is-incorrect/158838 "2018-11-30T09:09:19Z")
Hello everyone! Here is the code: m\_mkt="CUK45:cuk45\_moonv2\_ab4533fd85f7d55ce6a1a695fdfb9300;" result: CUK45=cuk45\_moonv2\_ab4533fd85f7d55ce6a1a695fdfb9300 However, the result is this. CUK45 = CUK45:cuk45\_moonv2\_ab4… --- ## [Multiple inputs using the same port](https://discuss.elastic.co/t/multiple-inputs-using-the-same-port/158784)
**Author:** [@przemolb](https://discuss.elastic.co/u/przemolb)\ **Replies:** 6\ **Last updated:** [November 30, 2018, 9:08am UTC](https://discuss.elastic.co/t/multiple-inputs-using-the-same-port/158784 "2018-11-30T09:08:10Z")
Hello, we use currently filebeat for shipping logs to logstash. It is in internal network so no SSL. But I have a request to send logs from another place (from fluent bit: from fluent bit to logstash) over Internet so … --- ## [Logstash Giving failed to parse \[Records.apiVersion\]](https://discuss.elastic.co/t/logstash-giving-failed-to-parse-records-apiversion/156781)
**Author:** [@Suresh\_Pal](https://discuss.elastic.co/u/Suresh_Pal)\ **Replies:** 6\ **Last updated:** [November 30, 2018, 6:19am UTC](https://discuss.elastic.co/t/logstash-giving-failed-to-parse-records-apiversion/156781 "2018-11-30T06:19:26Z")
Hi Team, Great morning, Hope you are doing awesome. I am trying to ship cloudtrail logs in my ELK. But I'm getting following warning \[WARN \] 2018-11-14 23:45:37.775 \[Ruby-0-Thread-6@\[main\]\>worker0: :1\] elasticsearch -… --- ## [Need help in parsing log in grok](https://discuss.elastic.co/t/need-help-in-parsing-log-in-grok/158843)
**Author:** [@luongthenghi](https://discuss.elastic.co/u/luongthenghi)\ **Replies:** 1\ **Last updated:** [November 30, 2018, 4:23am UTC](https://discuss.elastic.co/t/need-help-in-parsing-log-in-grok/158843 "2018-11-30T04:23:24Z")
Hi support team I have raw logs below: GlobalProtect gateway user authentication succeeded. Login from: 14.15.10.62, Source region: VN, Login from: 14.187.170.62, User name: dungh, Auth type: profile, Client OS version… --- ## [Some problem of logstash elapsed filter](https://discuss.elastic.co/t/some-problem-of-logstash-elapsed-filter/158111)
**Author:** [@zxc654951](https://discuss.elastic.co/u/zxc654951)\ **Replies:** 12\ **Last updated:** [November 30, 2018, 2:19am UTC](https://discuss.elastic.co/t/some-problem-of-logstash-elapsed-filter/158111 "2018-11-30T02:19:58Z")
I use elapsed filter to calculate events duration, but it has something wrong. The elapsed\_time got negative number in my case. config below: Can someone help me? Thanks. --- ## [Logstash - Metricbeat indice doesn't populate](https://discuss.elastic.co/t/logstash-metricbeat-indice-doesnt-populate/158648)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 8\ **Last updated:** [November 29, 2018, 7:33pm UTC](https://discuss.elastic.co/t/logstash-metricbeat-indice-doesnt-populate/158648 "2018-11-29T19:33:06Z")
I'm trying to get Metricbeat to populate data in Elastic but can't seem to figure out the missing pieces are. I currently have a Windows 2016 Server sandbox setup with Elastic, Kibana and Logstash on one node, abc, and … --- ## [Syslog input filter parse failure](https://discuss.elastic.co/t/syslog-input-filter-parse-failure/158803)
**Author:** [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Replies:** 0\ **Last updated:** [November 29, 2018, 5:48pm UTC](https://discuss.elastic.co/t/syslog-input-filter-parse-failure/158803 "2018-11-29T17:48:46Z")
I'm getting a parse failure on all logs from Cisco ironport appliances. I see a syslog PRI field of \<38\>. RFC 3164 tables seem to say 3 is daemons but 8 is invalid. Another method described here https://gist.github.c… --- ## [Logstash Pipeline Configuration & inputs](https://discuss.elastic.co/t/logstash-pipeline-configuration-inputs/158288)
**Author:** [@Carlos\_Magalhaes](https://discuss.elastic.co/u/Carlos_Magalhaes)\ **Replies:** 14\ **Last updated:** [November 29, 2018, 3:15pm UTC](https://discuss.elastic.co/t/logstash-pipeline-configuration-inputs/158288 "2018-11-29T15:15:19Z")
Hi all, Wanted to check an understanding on multiple pipelines: My understanding of having multiple pipelines is it allows you to have different inputs and outputs for a specific filter and provides better performance … --- ## [Increasing level of debug](https://discuss.elastic.co/t/increasing-level-of-debug/158783)
**Author:** [@Carlos\_Magalhaes](https://discuss.elastic.co/u/Carlos_Magalhaes)\ **Replies:** 0\ **Last updated:** [November 29, 2018, 2:49pm UTC](https://discuss.elastic.co/t/increasing-level-of-debug/158783 "2018-11-29T14:49:04Z")
Hi all, Trying to debug a issue I think I have on my pipeline config, usually using \[2018-11-29T16:22:20,520\]\[DEBUG\]\[logstash.runner \] \*config.debug: true (default: false) \[2018-11-29T16:22:20,521\]\[DEBUG\]\[logs… --- ## [Logstash Output to Cloudwatch Configuration](https://discuss.elastic.co/t/logstash-output-to-cloudwatch-configuration/158777)
**Author:** [@xtonehari](https://discuss.elastic.co/u/xtonehari)\ **Replies:** 6\ **Last updated:** [November 29, 2018, 2:46pm UTC](https://discuss.elastic.co/t/logstash-output-to-cloudwatch-configuration/158777 "2018-11-29T14:46:05Z")
We need to send the logs to CLoudwatch using Logstash. I could not find any site online which shows how to setup and configure in step by step or details. Could you please help on how to configure and send the logs to C… --- ## [Referencing an Environment Variable when existing field content is part of the name](https://discuss.elastic.co/t/referencing-an-environment-variable-when-existing-field-content-is-part-of-the-name/158709)
**Author:** [@Krobar](https://discuss.elastic.co/u/Krobar)\ **Replies:** 5\ **Last updated:** [November 29, 2018, 1:29pm UTC](https://discuss.elastic.co/t/referencing-an-environment-variable-when-existing-field-content-is-part-of-the-name/158709 "2018-11-29T13:29:21Z")
Is this possible? I have tried various configs including the one below: mutate { add\_field =\> { "BRAND\_VERSION\_NAME" =\> "BRAND\_VERSION\_%{Region}\_%{Language}" "BRAND\_VERSION" =\> "'${%{BRAND\_VERSION\_NAME}}'" } } BR… --- ## [Incorrect parsing /var/log/messages](https://discuss.elastic.co/t/incorrect-parsing-var-log-messages/158767)
**Author:** [@MrGangster](https://discuss.elastic.co/u/MrGangster)\ **Replies:** 0\ **Last updated:** [November 29, 2018, 1:09pm UTC](https://discuss.elastic.co/t/incorrect-parsing-var-log-messages/158767 "2018-11-29T13:09:38Z")
Incorrect parsing /var/log/messages from Centos 7 /var/log/messages have this alerts: Nov 29 15:49:27 vm logstash: "offset" =\> 902283 Filebeat paste date in Logstash. /etc/logstash/conf.d/logs.conf have this data: i… --- ## [Delete documents in elasticserach like DeleteByQuery](https://discuss.elastic.co/t/delete-documents-in-elasticserach-like-deletebyquery/158763)
**Author:** [@mwe](https://discuss.elastic.co/u/mwe)\ **Replies:** 0\ **Last updated:** [November 29, 2018, 12:50pm UTC](https://discuss.elastic.co/t/delete-documents-in-elasticserach-like-deletebyquery/158763 "2018-11-29T12:50:05Z")
Hi All, is it possible to deleted documents in elasticsearch in a way like DeletByQuery ? Thanks, and regards Michael --- ## [LS not ingesting syslog](https://discuss.elastic.co/t/ls-not-ingesting-syslog/158742)
**Author:** [@blueren](https://discuss.elastic.co/u/blueren)\ **Replies:** 5\ **Last updated:** [November 29, 2018, 12:29pm UTC](https://discuss.elastic.co/t/ls-not-ingesting-syslog/158742 "2018-11-29T12:29:46Z")
Hi, I've installed this on a docker container, this is the current docker-compose.yaml: version: '2' services: elasticsearch: image: elasticsearch-img:6.3.2 container\_name: elasticsearch-container volumes… --- ## [Create json object with logstash](https://discuss.elastic.co/t/create-json-object-with-logstash/158581)
**Author:** [@Shawcs](https://discuss.elastic.co/u/Shawcs)\ **Replies:** 2\ **Last updated:** [November 29, 2018, 10:42am UTC](https://discuss.elastic.co/t/create-json-object-with-logstash/158581 "2018-11-29T10:42:07Z")
Hi I face some trouble with JSON logs. I have a big JSON message comming in my Logstash and I made some parse in my file to create a new clean message. But now I want to create nested JSON field as I can have an array o… --- ## [How can I set the option for ls.cgroup.cpuacct.path.override and ls.cgroup.cpu.path.override in Docker?](https://discuss.elastic.co/t/how-can-i-set-the-option-for-ls-cgroup-cpuacct-path-override-and-ls-cgroup-cpu-path-override-in-docker/158612)
**Author:** [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\ **Replies:** 2\ **Last updated:** [November 29, 2018, 10:19am UTC](https://discuss.elastic.co/t/how-can-i-set-the-option-for-ls-cgroup-cpuacct-path-override-and-ls-cgroup-cpu-path-override-in-docker/158612 "2018-11-29T10:19:00Z")
Hi! I'm running Logstash as a docker container. In the debug logs, I get very frequent message like this; logstash\_1 | \[2018-11-28T15:30:32,132\]\[DEBUG\]\[logstash.instrument.periodicpoller.cgroup.cpuacctresource\] File … --- ## [Multiple logstash config files for different logic. input port is filebeat for both the files](https://discuss.elastic.co/t/multiple-logstash-config-files-for-different-logic-input-port-is-filebeat-for-both-the-files/158732)
**Author:** [@pkshara](https://discuss.elastic.co/u/pkshara)\ **Replies:** 0\ **Last updated:** [November 29, 2018, 10:15am UTC](https://discuss.elastic.co/t/multiple-logstash-config-files-for-different-logic-input-port-is-filebeat-for-both-the-files/158732 "2018-11-29T10:15:08Z")
Hi Team, I am using Logstash 5.6.1 version . I have 2 different configuration file, which as different logic in it. Is there anyway to run both the configuration in parallel. I read in some thread that we need to chan… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=639) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=641)