# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=530 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 531 --- ## [Logstash unbalanced persisted queues](https://discuss.elastic.co/t/logstash-unbalanced-persisted-queues/185401)
**Author:** [@nahiko2000](https://discuss.elastic.co/u/nahiko2000)\ **Replies:** 0\ **Last updated:** [June 12, 2019, 11:44am UTC](https://discuss.elastic.co/t/logstash-unbalanced-persisted-queues/185401 "2019-06-12T11:44:17Z")
Hi! I have a lot of Filebeats (one per computer I want logs to be inserted in ElasticSearch) Then I have a cluster of two Logstash nodes, both with the exact same configuration. They have a disk persisted queue, 1 GB … --- ## [Reloading logstash config files](https://discuss.elastic.co/t/reloading-logstash-config-files/185390)
**Author:** [@wadhah](https://discuss.elastic.co/u/wadhah)\ **Replies:** 0\ **Last updated:** [June 12, 2019, 10:38am UTC](https://discuss.elastic.co/t/reloading-logstash-config-files/185390 "2019-06-12T10:38:14Z")
Hello, I am working on picking up automatically changes in logstash config file: get the changes whenever i make modifications in logstash config file. In my kubernetes config file i used this : env: - name: LS\_OPTS … --- ## [Detected a 6.x and above cluster: the \`type\` event field won't be used to determine the document \_type {:es\_version=\>6}](https://discuss.elastic.co/t/detected-a-6-x-and-above-cluster-the-type-event-field-wont-be-used-to-determine-the-document-type-es-version-6/183471)
**Author:** [@shruti51](https://discuss.elastic.co/u/shruti51)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 10:33am UTC](https://discuss.elastic.co/t/detected-a-6-x-and-above-cluster-the-type-event-field-wont-be-used-to-determine-the-document-type-es-version-6/183471 "2019-06-12T10:33:07Z")
Hi, I am trying to parse my logs via filebeat but getting the error: Detected a 6.x and above cluster: the type event field won't be used to determine the document \_type {:es\_version=\>6}. Please suggest how to resolv… --- ## [How Filter Logs of Mdaemon?](https://discuss.elastic.co/t/how-filter-logs-of-mdaemon/183055)
**Author:** [@gerard.ramos](https://discuss.elastic.co/u/gerard.ramos)\ **Replies:** 4\ **Last updated:** [June 3, 2019, 2:39pm UTC](https://discuss.elastic.co/t/how-filter-logs-of-mdaemon/183055 "2019-06-03T14:39:01Z")
Hello I'm new in Logstash and I was trying to filter the logs of Mdaemon to visualise it in Kibana. The logs are in a file and are formed by multiple lines. They are separated with dashes like this "----------". I leave… --- ## [How to import millions of rows from Postgres to elastic](https://discuss.elastic.co/t/how-to-import-millions-of-rows-from-postgres-to-elastic/185332)
**Author:** [@Sayed\_Atif](https://discuss.elastic.co/u/Sayed_Atif)\ **Replies:** 0\ **Last updated:** [June 12, 2019, 6:22am UTC](https://discuss.elastic.co/t/how-to-import-millions-of-rows-from-postgres-to-elastic/185332 "2019-06-12T06:22:03Z")
Hello guys, I am trying to import millions of rows from Postgres to elastic using Logstash and jdbc-input-plugin. But I have to pull data from 15+ tables so join query is not a good solution as it takes too long to proce… --- ## [Logstash Stucks in org.logstash.beats.ConnectionHandler](https://discuss.elastic.co/t/logstash-stucks-in-org-logstash-beats-connectionhandler/185316)
**Author:** [@Gelinski](https://discuss.elastic.co/u/Gelinski)\ **Replies:** 0\ **Last updated:** [June 12, 2019, 3:40am UTC](https://discuss.elastic.co/t/logstash-stucks-in-org-logstash-beats-connectionhandler/185316 "2019-06-12T03:40:50Z")
My Logstash instances are stuck in "org.logstash.beats.ConnectionHandler" consuming a lot of CPU and not processing any input anymore. Trace bellow: \[2019-06-12T00:37:20,751\]\[TRACE\]\[org.logstash.beats.ConnectionHandler\]… --- ## [Logstash jmx input plugin](https://discuss.elastic.co/t/logstash-jmx-input-plugin/183232)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 3:38am UTC](https://discuss.elastic.co/t/logstash-jmx-input-plugin/183232 "2019-06-12T03:38:10Z")
hi all, I am using jmx plugin to ship WebLogic's MBeans into elasticsearch. for this purpose, following configuration has been considered in the logstash: input { jmx { path =\> "D:\\logstash\\jmx\_config" polling\_frequ… --- ## [コピーしたフィールドから〇文字抜き出す方法](https://discuss.elastic.co/t/topic/185143)
**Author:** [@Yuichi\_Takagi](https://discuss.elastic.co/u/Yuichi_Takagi)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 1:50am UTC](https://discuss.elastic.co/t/topic/185143 "2019-06-12T01:50:39Z")
logstashでパースしたデータのうち、特定フィールドの値をコピーし指定した文字数だけを抜き出す方法を探しています。 ご存知の方がいればご教示いただけますでしょうか。 以下は取得したい結果の例です。 user-nameの値から5文字分抜き出したuser-name5フィールドを作成。 // "user-name":"hogehoge","user-name5":"hogeh" 以下の様にmutateのadd\_fieldで単純… --- ## [Is there a for loop to iterate from %{header1} to %{header40}](https://discuss.elastic.co/t/is-there-a-for-loop-to-iterate-from-header1-to-header40/185078)
**Author:** [@TsuWeiQuan](https://discuss.elastic.co/u/TsuWeiQuan)\ **Replies:** 2\ **Last updated:** [June 11, 2019, 9:59am UTC](https://discuss.elastic.co/t/is-there-a-for-loop-to-iterate-from-header1-to-header40/185078 "2019-06-11T09:59:12Z")
Hello, I was wondering if there exists a more elegant way to do this. Currently, in my pipeline to process .log files, I used grok filter to filter field header and field data. This allows me to achieve 40 fields & 40… --- ## [Logstash keystore elasticsearch password](https://discuss.elastic.co/t/logstash-keystore-elasticsearch-password/185085)
**Author:** [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\ **Replies:** 2\ **Last updated:** [June 11, 2019, 11:33pm UTC](https://discuss.elastic.co/t/logstash-keystore-elasticsearch-password/185085 "2019-06-11T23:33:00Z")
How can I use the keystore for password instead of specifying it in the logstash.yml? xpack.monitoring.elasticsearch.password: I tried adding the ES\_PWD and listing it from the keystore /usr/share/logstash/bin/logstas… --- ## [Logstash fails to auto generate column names from CSV file](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 3\ **Last updated:** [June 11, 2019, 9:06pm UTC](https://discuss.elastic.co/t/logstash-fails-to-auto-generate-column-names-from-csv-file/185265 "2019-06-11T21:06:06Z")
Hi everyone, I used the following Logstash configuration, but sometimes Logstash auto generates the column names from the CSV file, and sometimes its not (According to the pictures). Could you please clarify what is the… --- ## [Logstash Translate Plugin - Dynamic dictionary path](https://discuss.elastic.co/t/logstash-translate-plugin-dynamic-dictionary-path/185275)
**Author:** [@myerasi](https://discuss.elastic.co/u/myerasi)\ **Replies:** 1\ **Last updated:** [June 11, 2019, 8:27pm UTC](https://discuss.elastic.co/t/logstash-translate-plugin-dynamic-dictionary-path/185275 "2019-06-11T20:27:44Z")
Hi all, We have a use-case to populate dictionary path dynamically for the Logstash translate plugin. We have lookup files for each location and are ingesting data from more than 30 locations. So, we would like to auto-… --- ## [Using Grok: How To Parse Multiple Entries With The Same Field Name?](https://discuss.elastic.co/t/using-grok-how-to-parse-multiple-entries-with-the-same-field-name/184819)
**Author:** [@rudyamid](https://discuss.elastic.co/u/rudyamid)\ **Replies:** 2\ **Last updated:** [June 11, 2019, 8:00pm UTC](https://discuss.elastic.co/t/using-grok-how-to-parse-multiple-entries-with-the-same-field-name/184819 "2019-06-11T20:00:27Z")
I've been searching for a pre-made Grok pattern for Apache's mod\_security error log, and couldn't find any. An example of a log entry: Mon Jun 03 15:07:12.453090 2019\] \[:error\] \[pid 15595\] \[client 192.168.0.254:57318\] … --- ## [Examples for setting up Beats (modules) \> Logstash \> ES with ILM?](https://discuss.elastic.co/t/examples-for-setting-up-beats-modules-logstash-es-with-ilm/182045)
**Author:** [@pdizz](https://discuss.elastic.co/u/pdizz)\ **Replies:** 9\ **Last updated:** [June 11, 2019, 7:43pm UTC](https://discuss.elastic.co/t/examples-for-setting-up-beats-modules-logstash-es-with-ilm/182045 "2019-06-11T19:43:00Z")
Are there any basic configuration examples showing how to set up a Beats (with modules and dashboards) \> Logstash \> ES pipeline using Index Lifecycle Management? Currently I'm familiar with the "classic" setup with Metri… --- ## [Grok Parse Failure - Filebeat -\> Logstash](https://discuss.elastic.co/t/grok-parse-failure-filebeat-logstash/185259)
**Author:** [@bdobsonbcm](https://discuss.elastic.co/u/bdobsonbcm)\ **Replies:** 3\ **Last updated:** [June 11, 2019, 6:50pm UTC](https://discuss.elastic.co/t/grok-parse-failure-filebeat-logstash/185259 "2019-06-11T18:50:44Z")
I apologize in that I may have missed the correct part in the documentation on this but have been getting nowhere so far this week. I have a device which I believe is writing fairly simple JSON entries to a log. Filebe… --- ## [Logstash 7.1.1 grokparsefailure despite match pattern](https://discuss.elastic.co/t/logstash-7-1-1-grokparsefailure-despite-match-pattern/185075)
**Author:** [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\ **Replies:** 18\ **Last updated:** [June 11, 2019, 5:10pm UTC](https://discuss.elastic.co/t/logstash-7-1-1-grokparsefailure-despite-match-pattern/185075 "2019-06-11T17:10:58Z")
I have upgraded logstash to 7.1.1 and I am receiving a grokparsefailure despite my pattern matching. See below. {"log":{"file":{"path":"/var/log/cassandra/system.log"},"offset":18875190},"input":{"type":"log"},"tags":\[… --- ## [Use environment variable to control output](https://discuss.elastic.co/t/use-environment-variable-to-control-output/185052)
**Author:** [@nobound](https://discuss.elastic.co/u/nobound)\ **Replies:** 9\ **Last updated:** [June 11, 2019, 4:45pm UTC](https://discuss.elastic.co/t/use-environment-variable-to-control-output/185052 "2019-06-11T16:45:47Z")
Hi, From the logstash.conf, I want to control the output based on the environment variable. I have tried the following, but it does not work. I find the use of the environment variable is not consistent throughout the c… --- ## [If path containing some string then field value is "somthing"](https://discuss.elastic.co/t/if-path-containing-some-string-then-field-value-is-somthing/184938)
**Author:** [@nitzanm](https://discuss.elastic.co/u/nitzanm)\ **Replies:** 4\ **Last updated:** [June 11, 2019, 4:30pm UTC](https://discuss.elastic.co/t/if-path-containing-some-string-then-field-value-is-somthing/184938 "2019-06-11T16:30:36Z")
hi I'm trying to fill a field based on path substring that is coming for filebeat here's an example for what i tried: input { beats { port =\> 5044 } } filter { if \[fields\]\[logtype\] == "l… --- ## [Failed to parse error for certain records](https://discuss.elastic.co/t/failed-to-parse-error-for-certain-records/184784)
**Author:** [@bkasrai](https://discuss.elastic.co/u/bkasrai)\ **Replies:** 4\ **Last updated:** [June 11, 2019, 4:29pm UTC](https://discuss.elastic.co/t/failed-to-parse-error-for-certain-records/184784 "2019-06-11T16:29:10Z")
Good morning all, I am ingesting some logs and while specific records make it over, I am seeing errors in the logs and they are not making it over to Kibana outside of your standard fields (doc\_id, index, time, etc) I ha… --- ## [Question about filter effects on event handling order](https://discuss.elastic.co/t/question-about-filter-effects-on-event-handling-order/185223)
**Author:** [@jdmcalee](https://discuss.elastic.co/u/jdmcalee)\ **Replies:** 4\ **Last updated:** [June 11, 2019, 4:26pm UTC](https://discuss.elastic.co/t/question-about-filter-effects-on-event-handling-order/185223 "2019-06-11T16:26:51Z")
I am trying to make a config that does a batch of indexing, then an elasticsearch filter query to find documents that need an update to be marked as stale (via a new event created in a subsequent ruby filter). It's impor… --- ## [Array not getting accepted for "filters" parameter? Plugin:logstash-input-cloudwatch](https://discuss.elastic.co/t/array-not-getting-accepted-for-filters-parameter-plugin-logstash-input-cloudwatch/185194)
**Author:** [@skerkar](https://discuss.elastic.co/u/skerkar)\ **Replies:** 4\ **Last updated:** [June 11, 2019, 4:24pm UTC](https://discuss.elastic.co/t/array-not-getting-accepted-for-filters-parameter-plugin-logstash-input-cloudwatch/185194 "2019-06-11T16:24:38Z")
Hi, I am trying to fetch metrics for ES using "logstash-input-cloudwatch" plugin. As per docs, "filters" configuration is of 'array' type. For fetching ES metrics, I need to pass DomainName and ClientId both, but plugi… --- ## [How to set the document id dynamically with the column value from the table](https://discuss.elastic.co/t/how-to-set-the-document-id-dynamically-with-the-column-value-from-the-table/185064)
**Author:** [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\ **Replies:** 2\ **Last updated:** [June 11, 2019, 4:18pm UTC](https://discuss.elastic.co/t/how-to-set-the-document-id-dynamically-with-the-column-value-from-the-table/185064 "2019-06-11T16:18:57Z")
I want to set the id value for each document in the elastic search. I am trying to set the account\_no as an id. However, when I tried to do, the id is set as the string value "%{account\_no}". The statement document\_id … --- ## [Define format/type of specific number fields](https://discuss.elastic.co/t/define-format-type-of-specific-number-fields/185229)
**Author:** [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)\ **Replies:** 2\ **Last updated:** [June 11, 2019, 4:11pm UTC](https://discuss.elastic.co/t/define-format-type-of-specific-number-fields/185229 "2019-06-11T16:11:35Z")
Hi here, Thanks in advance to take the time to read it and I hope help me out. I'm using a logstash filter with a ruby code to subtract two fields and create a 3rd one. Like below : input { jdbc { db details \[...\] } … --- ## [Lots of 429s, can't connect](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047)
**Author:** [@topher](https://discuss.elastic.co/u/topher)\ **Replies:** 3\ **Last updated:** [June 11, 2019, 4:09pm UTC](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047 "2019-06-11T16:09:23Z")
A little over a week ago, I updated from 7.0 to 7.1.1. I didn't realize it until late last week, but going back through logs I see I am getting a ton of 429 errors that started about the same time. Now it sends message… --- ## [Many Conditional Checks in Logstash Conf](https://discuss.elastic.co/t/many-conditional-checks-in-logstash-conf/183934)
**Author:** [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Replies:** 6\ **Last updated:** [June 11, 2019, 3:50pm UTC](https://discuss.elastic.co/t/many-conditional-checks-in-logstash-conf/183934 "2019-06-11T15:50:06Z")
Hi, Will it be a performance impact If I have many conditional statements in Logstash config ? My idea was to create a single Logstash setup whose input is FileBeat, WinLogBeat and HeartBeat. Based on the beat type an… --- ## [Xml parcing problem](https://discuss.elastic.co/t/xml-parcing-problem/184725)
**Author:** [@taku](https://discuss.elastic.co/u/taku)\ **Replies:** 4\ **Last updated:** [June 11, 2019, 3:05pm UTC](https://discuss.elastic.co/t/xml-parcing-problem/184725 "2019-06-11T15:05:46Z")
Hello everyone, I'm trying to parse the xml file, but the logstach configuration fails below, their hml needs to pull out only the value of the web container, but for some reason everything gets into the message field x… --- ## [Aggregate filter not working correctly iwth logstash](https://discuss.elastic.co/t/aggregate-filter-not-working-correctly-iwth-logstash/185088)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 1\ **Last updated:** [June 11, 2019, 2:58pm UTC](https://discuss.elastic.co/t/aggregate-filter-not-working-correctly-iwth-logstash/185088 "2019-06-11T14:58:31Z")
Here is my input log file names as : cadence-cdslmd-dlhl0939\_dlhl0940\_dlhl0941-5.11.2019-event.log 15:56:24 (cdslmd) OUT: "PKC0603" mnarasim@dlhsx00012 15:56:32 (cdslmd) OUT: "Analog\_Design\_Environment\_XL" gautamd@dlhs… --- ## [Http header data extract](https://discuss.elastic.co/t/http-header-data-extract/185213)
**Author:** [@muntazirabbas](https://discuss.elastic.co/u/muntazirabbas)\ **Replies:** 1\ **Last updated:** [June 11, 2019, 2:56pm UTC](https://discuss.elastic.co/t/http-header-data-extract/185213 "2019-06-11T14:56:00Z")
Hi everyone, I am new to ELK and have started with Logstash already. One of my Task it to use http as an Input and extract the Header data out of it and send it to Elasticsearch and then to Kibana in later stage. Here is… --- ## [Sophos UTM syslog message parsing with Logstash](https://discuss.elastic.co/t/sophos-utm-syslog-message-parsing-with-logstash/185199)
**Author:** [@skills](https://discuss.elastic.co/u/skills)\ **Replies:** 1\ **Last updated:** [June 11, 2019, 2:21pm UTC](https://discuss.elastic.co/t/sophos-utm-syslog-message-parsing-with-logstash/185199 "2019-06-11T14:21:37Z")
Hi all, I am struggling to parse Sophos Firewall Syslog messages with logstash. Log format is as follows: Jun 11 14:23:57 10.11.223.44 2019:06:11-14:23:57 bewg-t2-ff2-2 ulogd\[23094\]: id="2001" severity="info" sys="Secu… --- ## [Should Logstash be off if no pipelines are on?](https://discuss.elastic.co/t/should-logstash-be-off-if-no-pipelines-are-on/185196)
**Author:** [@System\_Even](https://discuss.elastic.co/u/System_Even)\ **Replies:** 0\ **Last updated:** [June 11, 2019, 1:20pm UTC](https://discuss.elastic.co/t/should-logstash-be-off-if-no-pipelines-are-on/185196 "2019-06-11T13:20:33Z")
I work at a company with multiple teams and we work on a closed network (on premise). Each team has their own Logstash server on Production and Pre-Production enviorments, while I want them to always have the servers op… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=529) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=531)