# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=767 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 768 --- ## [Multiline problem with python stack trace](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329)
**Author:** [@romanskz](https://discuss.elastic.co/u/romanskz)\ **Replies:** 5\ **Last updated:** [May 4, 2018, 9:47am UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329 "2018-05-04T09:47:23Z")
Hello, I'm using an elk stack. I have a log file from a python program. Some logs have stacktrace. I'm trying to use the multiline option in filebeat to get all stacktrace in addition to the error log. However, nothing … --- ## [Multiple logstash config files](https://discuss.elastic.co/t/multiple-logstash-config-files/130471)
**Author:** [@Hamza\_Dhahri](https://discuss.elastic.co/u/Hamza_Dhahri)\ **Replies:** 3\ **Last updated:** [May 4, 2018, 9:27am UTC](https://discuss.elastic.co/t/multiple-logstash-config-files/130471 "2018-05-04T09:27:44Z")
hello everyone i have a log in which i have to parse some informations i have did 7 patterns (7 config files ) and i have to stock this informations in elasticsearch well when i use logstash to run each config file … --- ## [Azureblob plugin not resuming where it's supposed to](https://discuss.elastic.co/t/azureblob-plugin-not-resuming-where-its-supposed-to/130586)
**Author:** [@Martijn](https://discuss.elastic.co/u/Martijn)\ **Replies:** 0\ **Last updated:** [May 4, 2018, 8:35am UTC](https://discuss.elastic.co/t/azureblob-plugin-not-resuming-where-its-supposed-to/130586 "2018-05-04T08:35:19Z")
Hi all, We are experiencing very strange behaviour from the Azureblob plugin. I am not sure if it's an issue with the plugin, or with Logstash. The data is being parsed correctly, but Logstash seems to lag behind quite… --- ## [No such file to load -- org/apache/httpcomponents/httpclient/4.5/httpclient-4.5 (LoadError)](https://discuss.elastic.co/t/no-such-file-to-load-org-apache-httpcomponents-httpclient-4-5-httpclient-4-5-loaderror/130042)
**Author:** [@dmccuk](https://discuss.elastic.co/u/dmccuk)\ **Replies:** 2\ **Last updated:** [May 4, 2018, 8:29am UTC](https://discuss.elastic.co/t/no-such-file-to-load-org-apache-httpcomponents-httpclient-4-5-httpclient-4-5-loaderror/130042 "2018-05-04T08:29:13Z")
Hi, I'm returning to ELK after a break for a few years. I've got this error when i start logstash up and include my remote elasticsearch node in the config on a very basic setup: # bin/logstash -f conf.d/inputs.conf ##… --- ## [Configuration difference in elk5 and elk6](https://discuss.elastic.co/t/configuration-difference-in-elk5-and-elk6/130577)
**Author:** [@nik9](https://discuss.elastic.co/u/nik9)\ **Replies:** 0\ **Last updated:** [May 4, 2018, 7:44am UTC](https://discuss.elastic.co/t/configuration-difference-in-elk5-and-elk6/130577 "2018-05-04T07:44:09Z")
Hi, In recent version of filebeat6.2.4 "document\_type" is missing. and on ELK server i have stetted the filter for logstash with reference to type is : filter { if \[type\] == "syslog" { grok { match =\> { "message" =\>… --- ## [Error registering plugin, Pipeline aborted due to error (\), Failed to execute action](https://discuss.elastic.co/t/error-registering-plugin-pipeline-aborted-due-to-error-typeerror-cant-dup-fixnum-failed-to-execute-action/128987)
**Author:** [@chu](https://discuss.elastic.co/u/chu)\ **Replies:** 10\ **Last updated:** [May 4, 2018, 7:43am UTC](https://discuss.elastic.co/t/error-registering-plugin-pipeline-aborted-due-to-error-typeerror-cant-dup-fixnum-failed-to-execute-action/128987 "2018-05-04T07:43:50Z")
Hi, everyone: I'm a beginner on ELK and trying to load data from mysql to elasticsearch(for next step i want query them via javarestclient), so i used logstash6.2.4 and elasticsearch6.2.4. and followed a example here. … --- ## [Tag input name file](https://discuss.elastic.co/t/tag-input-name-file/130308)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 4\ **Last updated:** [May 4, 2018, 7:39am UTC](https://discuss.elastic.co/t/tag-input-name-file/130308 "2018-05-04T07:39:00Z")
It 's possible to tag multiple input file name ? I have many txt file and I want to tag all file maybe with name file if it's possible. input { file { path =\> "/var/log/testing/\*.txt" start\_position =\> "begi… --- ## [Logstash with jdbc mssql-jdbc-6.4.0.jre9 error](https://discuss.elastic.co/t/logstash-with-jdbc-mssql-jdbc-6-4-0-jre9-error/130371)
**Author:** [@sislandtw](https://discuss.elastic.co/u/sislandtw)\ **Replies:** 2\ **Last updated:** [May 4, 2018, 1:22am UTC](https://discuss.elastic.co/t/logstash-with-jdbc-mssql-jdbc-6-4-0-jre9-error/130371 "2018-05-04T01:22:32Z")
need your help ------------ env -------------------------------------- Logstash connect to MSSQL 2016 DB JDBC = \> mssql-jdbc-6.4.0.jre9， JRE =\> jre1.8.0\_172 windows 10 ----------------- config ---------------------… --- ## [Translate two fields in one event](https://discuss.elastic.co/t/translate-two-fields-in-one-event/130516)
**Author:** [@john.bennett](https://discuss.elastic.co/u/john.bennett)\ **Replies:** 2\ **Last updated:** [May 3, 2018, 10:53pm UTC](https://discuss.elastic.co/t/translate-two-fields-in-one-event/130516 "2018-05-03T22:53:10Z")
HI, I have two translate blocks I am trying to translate two different fields from two dictionary yaml files. One for user name, one for IP. So far, the first translate block is the only one that works. If IP is first… --- ## [\[SOLVED\] Split filter question a.k.a flatten json sub array](https://discuss.elastic.co/t/solved-split-filter-question-a-k-a-flatten-json-sub-array/130481)
**Author:** [@blastik](https://discuss.elastic.co/u/blastik)\ **Replies:** 14\ **Last updated:** [May 3, 2018, 7:14pm UTC](https://discuss.elastic.co/t/solved-split-filter-question-a-k-a-flatten-json-sub-array/130481 "2018-05-03T19:14:54Z")
Hi there! I have a json input which looks like this: \[{ "Action": "COUNT", "Timestamp": "2018-05-02T13:09:58Z", "Request": { "Country": "ES", "URI": "/uploadMultiplePhotos.aspx", "Headers": \[{ \> "Name": "… --- ## [Can Logstash run update operation in MongoDb](https://discuss.elastic.co/t/can-logstash-run-update-operation-in-mongodb/130467)
**Author:** [@Hederson](https://discuss.elastic.co/u/Hederson)\ **Replies:** 2\ **Last updated:** [May 3, 2018, 5:50pm UTC](https://discuss.elastic.co/t/can-logstash-run-update-operation-in-mongodb/130467 "2018-05-03T17:50:16Z")
I am using Logstash 6.2.4 for replicate data from a MS Sql Server table to a collection MongoDB. Since the records are inserted in the source table, they can be updated with new information. I need to update these new i… --- ## [Grok field extract: One pattern, multiple matches](https://discuss.elastic.co/t/grok-field-extract-one-pattern-multiple-matches/130497)
**Author:** [@jonrust](https://discuss.elastic.co/u/jonrust)\ **Replies:** 6\ **Last updated:** [May 3, 2018, 6:11pm UTC](https://discuss.elastic.co/t/grok-field-extract-one-pattern-multiple-matches/130497 "2018-05-03T18:11:46Z")
Oracle alert logs contain multiple ORA- codes. I can't figure how to tell grok to match all and return an array of the matches. if \[msg\] =~ /ORA-/ { grok { match =\> \[ "msg","(?\ORA-\[0-9\]+)" \] } … --- ## [Bug on kv remove\_char\_value =\> "\\\\"](https://discuss.elastic.co/t/bug-on-kv-remove-char-value/130503)
**Author:** [@junnester](https://discuss.elastic.co/u/junnester)\ **Replies:** 0\ **Last updated:** [May 3, 2018, 5:33pm UTC](https://discuss.elastic.co/t/bug-on-kv-remove-char-value/130503 "2018-05-03T17:33:20Z")
Version: logstash-6.2.4 -- filter plugin kv Operating System: CentOS Linux release 7.4.1708 (Core) Linux 3.10.0-693.2.2.el7.x86\_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86\_64 x86\_64 x86\_64 GNU/Linux Config File… --- ## [Logstash translate filter dynamic substitution](https://discuss.elastic.co/t/logstash-translate-filter-dynamic-substitution/130381)
**Author:** [@Priya\_Darshini](https://discuss.elastic.co/u/Priya_Darshini)\ **Replies:** 1\ **Last updated:** [May 3, 2018, 5:24pm UTC](https://discuss.elastic.co/t/logstash-translate-filter-dynamic-substitution/130381 "2018-05-03T17:24:08Z")
Hi, I have a yml file that has input like "John" : "yes" In logstash conf file I should check if the incoming event has name as John and if the value of John is "yes" then I should drop the message. My conf file look… --- ## [How logstash can get the total number of line in a file (count))](https://discuss.elastic.co/t/how-logstash-can-get-the-total-number-of-line-in-a-file-count/130233)
**Author:** [@alicia1](https://discuss.elastic.co/u/alicia1)\ **Replies:** 10\ **Last updated:** [May 3, 2018, 4:37pm UTC](https://discuss.elastic.co/t/how-logstash-can-get-the-total-number-of-line-in-a-file-count/130233 "2018-05-03T16:37:34Z")
Hi, I have 3 files CSV in windows, one is to store logs errors Error.csv, one is for all succes events success.csv and the last for all incomming events totalEvent.csv (= logs errors + succes events). I want to have a … --- ## [Logstash vs Spark vs something else](https://discuss.elastic.co/t/logstash-vs-spark-vs-something-else/130066)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 6\ **Last updated:** [May 3, 2018, 4:35pm UTC](https://discuss.elastic.co/t/logstash-vs-spark-vs-something-else/130066 "2018-05-03T16:35:12Z")
I have been using Logstash to connect to Postgres to parse XML and extract certain xpaths and then push them to Elasticsearch. However, it's been fairly apparent that Logstash isn't meant for parsing entire files. Is th… --- ## [Logstash 6.2.3 Failed to create monitoring event](https://discuss.elastic.co/t/logstash-6-2-3-failed-to-create-monitoring-event/130464)
**Author:** [@roketyyang](https://discuss.elastic.co/u/roketyyang)\ **Replies:** 1\ **Last updated:** [May 3, 2018, 2:55pm UTC](https://discuss.elastic.co/t/logstash-6-2-3-failed-to-create-monitoring-event/130464 "2018-05-03T14:55:52Z")
I can't get any help from similar topics. \[2018-05-03T21:45:28,088\]\[ERROR\]\[logstash.inputs.metrics \] Failed to create monitoring event {:message=\>"For path: events. Map keys: \[:pipelines, :reloads\]", :error=\>"LogStash:… --- ## [How to get checkpoint logs in logstash](https://discuss.elastic.co/t/how-to-get-checkpoint-logs-in-logstash/128403)
**Author:** [@addanuj](https://discuss.elastic.co/u/addanuj)\ **Replies:** 2\ **Last updated:** [May 3, 2018, 1:54pm UTC](https://discuss.elastic.co/t/how-to-get-checkpoint-logs-in-logstash/128403 "2018-05-03T13:54:58Z")
in our environment we have 2 checkpoint firewalls, I need to to know how I can push or pull checkpoint firewall logs to logstash so that i can create some index and create Kibana visualization? i am new to ELK, please h… --- ## [Logstash CPU usage 150%](https://discuss.elastic.co/t/logstash-cpu-usage-150/130463)
**Author:** [@Akaren](https://discuss.elastic.co/u/Akaren)\ **Replies:** 0\ **Last updated:** [May 3, 2018, 1:51pm UTC](https://discuss.elastic.co/t/logstash-cpu-usage-150/130463 "2018-05-03T13:51:26Z")
Hi, we are having logstash 5.6.3, our logstash config contains many filters, add field, geoip and also aggregate. It's long and complicated. We had to set pipeline.workers to 1 because of aggregate module. But sometimes… --- ## [Conditional match for numeric type in list?](https://discuss.elastic.co/t/conditional-match-for-numeric-type-in-list/130447)
**Author:** [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Replies:** 2\ **Last updated:** [May 3, 2018, 1:36pm UTC](https://discuss.elastic.co/t/conditional-match-for-numeric-type-in-list/130447 "2018-05-03T13:36:46Z")
Hi, Please help, I have been struggling with this for a few days. How can I match a numeric field against a long list of numbers? Would it be best to use grok or ruby? I can do it for one but there must be a more el… --- ## [How logstash throttle filter works with auto scaling and pipeline workers?](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161)
**Author:** [@Debarghya\_Chakrabort](https://discuss.elastic.co/u/Debarghya_Chakrabort)\ **Replies:** 4\ **Last updated:** [May 3, 2018, 1:14pm UTC](https://discuss.elastic.co/t/how-logstash-throttle-filter-works-with-auto-scaling-and-pipeline-workers/130161 "2018-05-03T13:14:34Z")
Hello, I am using ELK stack to process our logs. Now we are going to introduce throttling by using logstash throttle filter. Right now we have 2 logstash with 2 pipeline workers each and this can can be auto scaled. Ca… --- ## [Logstash Not Receiving TCP Data](https://discuss.elastic.co/t/logstash-not-receiving-tcp-data/128402)
**Author:** [@clannadqs](https://discuss.elastic.co/u/clannadqs)\ **Replies:** 6\ **Last updated:** [May 3, 2018, 12:36pm UTC](https://discuss.elastic.co/t/logstash-not-receiving-tcp-data/128402 "2018-05-03T12:36:09Z")
I am shipping monitored data as JSON from Python. I have tested whether the data is actually being sent outside of Logstash and it is successfully sending and being received. With Logstash the input is showing no signs o… --- ## [Autodetection is not working in ELK 6.2.3](https://discuss.elastic.co/t/autodetection-is-not-working-in-elk-6-2-3/130224)
**Author:** [@DSak](https://discuss.elastic.co/u/DSak)\ **Replies:** 4\ **Last updated:** [May 3, 2018, 12:33pm UTC](https://discuss.elastic.co/t/autodetection-is-not-working-in-elk-6-2-3/130224 "2018-05-03T12:33:35Z")
Hello, On Linux, autodetection property is not working for version 6.2.3? The same property is working fine with windows. Also, we are playing with csv & JSON object. Same is working fine with version 5.6.3. Can somebod… --- ## [How can i add a field to the original message (json) that was sent to logstash?](https://discuss.elastic.co/t/how-can-i-add-a-field-to-the-original-message-json-that-was-sent-to-logstash/130276)
**Author:** [@awatkins](https://discuss.elastic.co/u/awatkins)\ **Replies:** 14\ **Last updated:** [May 3, 2018, 12:18pm UTC](https://discuss.elastic.co/t/how-can-i-add-a-field-to-the-original-message-json-that-was-sent-to-logstash/130276 "2018-05-03T12:18:43Z")
I have a log message that is a json file. I would like to add a field to that original message within the json structure. I've tried multiple ways of doing it and none of them get the job done. I have trouble with the… --- ## [@timestamp - Epoch - Illegal Date Format exception](https://discuss.elastic.co/t/timestamp-epoch-illegal-date-format-exception/130261)
**Author:** [@SKumarMN](https://discuss.elastic.co/u/SKumarMN)\ **Replies:** 2\ **Last updated:** [May 3, 2018, 12:00pm UTC](https://discuss.elastic.co/t/timestamp-epoch-illegal-date-format-exception/130261 "2018-05-03T12:00:33Z")
Hi All, ES index is having a field with the mapping as below. "@timestamp": { "format": "epoch\_millis", "type": "date" } From logstash i want to populate this field with custom timestamp obtained from input documen… --- ## [Use path.config in pipelines.yml and and remove from logstash.yml for mulltipipeline configuration](https://discuss.elastic.co/t/use-path-config-in-pipelines-yml-and-and-remove-from-logstash-yml-for-mulltipipeline-configuration/130443)
**Author:** [@santoshgupta](https://discuss.elastic.co/u/santoshgupta)\ **Replies:** 0\ **Last updated:** [May 3, 2018, 11:32am UTC](https://discuss.elastic.co/t/use-path-config-in-pipelines-yml-and-and-remove-from-logstash-yml-for-mulltipipeline-configuration/130443 "2018-05-03T11:32:52Z")
I am using multi-pipeline feature. Now "path.config" in logstash.yaml is of no use to me, as I am mentioning that in my pipelines.yaml. So, I removed "path.config" from logstash.yaml. My logstashs.yml looks like this … --- ## [Grok parse issue parsing previously parsed string](https://discuss.elastic.co/t/grok-parse-issue-parsing-previously-parsed-string/130246)
**Author:** [@No\_Executable](https://discuss.elastic.co/u/No_Executable)\ **Replies:** 5\ **Last updated:** [May 3, 2018, 11:18am UTC](https://discuss.elastic.co/t/grok-parse-issue-parsing-previously-parsed-string/130246 "2018-05-03T11:18:45Z")
Hi. We are having issues with our syntax writing, where we try to extract a length of numbers from a long string that continues on both ends of the wanted piece of numbers. Our theory is that the problem lies within our… --- ## [How to limit the http output from logstash?](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099)
**Author:** [@santoshgupta](https://discuss.elastic.co/u/santoshgupta)\ **Replies:** 5\ **Last updated:** [May 3, 2018, 11:21am UTC](https://discuss.elastic.co/t/how-to-limit-the-http-output-from-logstash/130099 "2018-05-03T11:21:47Z")
I have a pipeline which uses kafka as i/p and HTTP as o/p. I want to limit the HTTP requests that are sent from logstash o/p. Is there a way to do that? --- ## [Cache optimization for ip2location filter problem](https://discuss.elastic.co/t/cache-optimization-for-ip2location-filter-problem/130434)
**Author:** [@deyanskiq](https://discuss.elastic.co/u/deyanskiq)\ **Replies:** 0\ **Last updated:** [May 3, 2018, 11:00am UTC](https://discuss.elastic.co/t/cache-optimization-for-ip2location-filter-problem/130434 "2018-05-03T11:00:32Z")
Hello, I have implemented cache optimization for ip2location filter ( which is similar to the default geoip filter). When event (a line of log file) is send for the first time everything is ok ( the ip2location fields a… --- ## [Logstash mutate split but maintain the separator character](https://discuss.elastic.co/t/logstash-mutate-split-but-maintain-the-separator-character/130338)
**Author:** [@safehouse](https://discuss.elastic.co/u/safehouse)\ **Replies:** 2\ **Last updated:** [May 3, 2018, 10:37am UTC](https://discuss.elastic.co/t/logstash-mutate-split-but-maintain-the-separator-character/130338 "2018-05-03T10:37:01Z")
I want to split a field into an array, but I want the array values to keep the separator characters. It seems that the way mutate split works is to remove the separator characters. Is there another way to accomplish thi… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=766) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=768)