# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=268 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 269 --- ## [Logstash 6.8 template gives 400 HTTP response code](https://discuss.elastic.co/t/logstash-6-8-template-gives-400-http-response-code/259699)
**Author:** [@mlpn](https://discuss.elastic.co/u/mlpn)\ **Replies:** 0\ **Last updated:** [December 27, 2020, 11:04pm UTC](https://discuss.elastic.co/t/logstash-6-8-template-gives-400-http-response-code/259699 "2020-12-27T23:04:36Z")
Hello, I'm in the middle of process of migrating ELK Stack from 5.4 to 7.1. I'm currently on 6.8 and my Logstash gives me HTTP response code 400 after start when I'm trying to import my template. I've renamed string valu… --- ## [Logstash kafka input plugin not retrieving data](https://discuss.elastic.co/t/logstash-kafka-input-plugin-not-retrieving-data/259686)
**Author:** [@LogstashDeveloper](https://discuss.elastic.co/u/LogstashDeveloper)\ **Replies:** 0\ **Last updated:** [December 27, 2020, 1:39pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-plugin-not-retrieving-data/259686 "2020-12-27T13:39:21Z")
Im using logstash kafka input plugin, but it does not retrieve data from kafka topic. Logstash log shows the following messages: Adding newly assigned partitions Setting offset for partition to the committed offset Fet… --- ## [Logstash kv field split problem](https://discuss.elastic.co/t/logstash-kv-field-split-problem/259552)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 7\ **Last updated:** [December 27, 2020, 10:00am UTC](https://discuss.elastic.co/t/logstash-kv-field-split-problem/259552 "2020-12-27T10:00:10Z")
Logs like this: topic = abc, partition = 2, offset = 4386, serialized key size = -1, serialized value size = 1139, key = null, value = {"Name":"pc-1","test":false,"message":"elk test - Your pc-1 better. Thank you.","Cou… --- ## [Aggregate filter dilemma](https://discuss.elastic.co/t/aggregate-filter-dilemma/259657)
**Author:** [@tommys](https://discuss.elastic.co/u/tommys)\ **Replies:** 4\ **Last updated:** [December 26, 2020, 10:58pm UTC](https://discuss.elastic.co/t/aggregate-filter-dilemma/259657 "2020-12-26T22:58:43Z")
Hi all, hoping to get some feedback on this as it took me 3 days to resolve which I think is potentially a knowledge gap on my side or a BUG. In summary: Two logs, I want to aggregate using the aggregate filter. One… --- ## [Can I get only one data in logstash's output?](https://discuss.elastic.co/t/can-i-get-only-one-data-in-logstashs-output/259626)
**Author:** [@iooi](https://discuss.elastic.co/u/iooi)\ **Replies:** 1\ **Last updated:** [December 25, 2020, 4:09pm UTC](https://discuss.elastic.co/t/can-i-get-only-one-data-in-logstashs-output/259626 "2020-12-25T16:09:15Z")
Can I get only one data in logstash's output? If the data after filter like: "a" =\> 1, "b" =\> 2, "c" =\> 3, in the output, send it to elasticsearch. output { elasticsearch { protocol =\> http host =\>… --- ## [Logsource On-boarding](https://discuss.elastic.co/t/logsource-on-boarding/259590)
**Author:** [@ameermane](https://discuss.elastic.co/u/ameermane)\ **Replies:** 2\ **Last updated:** [December 25, 2020, 2:49pm UTC](https://discuss.elastic.co/t/logsource-on-boarding/259590 "2020-12-25T14:49:13Z")
Hello All, Hope you are doing good!!! ELK is new for me. I have deployed ELK in my lab and wondering how to onboard different logs source like paloalto , checkpoint , cisco firewalls. Please share any leadings that wi… --- ## [Can logstash's JDBC input plugin do multiple sql tasks?](https://discuss.elastic.co/t/can-logstashs-jdbc-input-plugin-do-multiple-sql-tasks/259573)
**Author:** [@iooi](https://discuss.elastic.co/u/iooi)\ **Replies:** 6\ **Last updated:** [December 25, 2020, 9:17am UTC](https://discuss.elastic.co/t/can-logstashs-jdbc-input-plugin-do-multiple-sql-tasks/259573 "2020-12-25T09:17:12Z")
Now there is a JDBC task read from db as input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://${MYSQL\_MAIN\_HOST}/${MYSQL\_DATABASE}" jdbc\_driver\_class =\> "com.mysql.cj.jdbc.Driver" jdbc\_page\_size =\> 10000… --- ## [Logstash doesnt update the data when mongodb data gets updated](https://discuss.elastic.co/t/logstash-doesnt-update-the-data-when-mongodb-data-gets-updated/259562)
**Author:** [@Siddhanta\_Das](https://discuss.elastic.co/u/Siddhanta_Das)\ **Replies:** 1\ **Last updated:** [December 25, 2020, 8:42am UTC](https://discuss.elastic.co/t/logstash-doesnt-update-the-data-when-mongodb-data-gets-updated/259562 "2020-12-25T08:42:18Z")
I am using this configuration input{ mongodb{ uri=\>"mongodb://localhost:27017/ebyat" placeholder\_db\_dir =\> "/opt/logstash-mongodb/" placeholder\_db\_name =\> "logstash\_sqlite.db" collection =\> "amenities" batch\_size =\>… --- ## [Logstash filter plugin needs certificate with dns name](https://discuss.elastic.co/t/logstash-filter-plugin-needs-certificate-with-dns-name/259281)
**Author:** [@Moni\_Kherajani](https://discuss.elastic.co/u/Moni_Kherajani)\ **Replies:** 13\ **Last updated:** [December 24, 2020, 10:48am UTC](https://discuss.elastic.co/t/logstash-filter-plugin-needs-certificate-with-dns-name/259281 "2020-12-24T10:48:05Z")
Hi i tried to configure security in elkstack and wanted a common certificate in all nodes environments but logstash fillter plugin needs dns name to speicified else wont work, any other solution? --- ## [Logstash takes all the collections from mongodb that has the collection string](https://discuss.elastic.co/t/logstash-takes-all-the-collections-from-mongodb-that-has-the-collection-string/259564)
**Author:** [@Siddhanta\_Das](https://discuss.elastic.co/u/Siddhanta_Das)\ **Replies:** 0\ **Last updated:** [December 24, 2020, 8:25am UTC](https://discuss.elastic.co/t/logstash-takes-all-the-collections-from-mongodb-that-has-the-collection-string/259564 "2020-12-24T08:25:20Z")
I have a issue with the collection string in config file input. I have two collection with "properties" and "favourite\_properties". When I run the mongo conf file it retrive both the collection but I wan only one input{… --- ## [Logstash Failed to load the sqljdbc\_auth.dll cause : no sqljdbc\_auth in java.library.path](https://discuss.elastic.co/t/logstash-failed-to-load-the-sqljdbc-auth-dll-cause-no-sqljdbc-auth-in-java-library-path/259530)
**Author:** [@jjfellers](https://discuss.elastic.co/u/jjfellers)\ **Replies:** 0\ **Last updated:** [December 23, 2020, 9:27pm UTC](https://discuss.elastic.co/t/logstash-failed-to-load-the-sqljdbc-auth-dll-cause-no-sqljdbc-auth-in-java-library-path/259530 "2020-12-23T21:27:58Z")
I have pasted the required dll in about 20 different places. Always the same error. With the default logstash installation on CentOS, where is the correct place to place the sqljdbc42.jar (version 4.2.8112) and dll? I pl… --- ## [Logstash parse object](https://discuss.elastic.co/t/logstash-parse-object/259513)
**Author:** [@bbwolf](https://discuss.elastic.co/u/bbwolf)\ **Replies:** 4\ **Last updated:** [December 23, 2020, 6:13pm UTC](https://discuss.elastic.co/t/logstash-parse-object/259513 "2020-12-23T18:13:52Z")
Hello, I have in a field which I need to split thé next values FieldA : "\[{" FieldA1" :"A", "FieldA2" :"B"}{"FieldA1" :"C", "FieldA2" :"D"\]}" I wan't to split FieldA as two documents {" FieldA1" :"A", "FieldA2" :"B"}… --- ## [ERROR: Something went wrong when installing logstash-input-websocket,](https://discuss.elastic.co/t/error-something-went-wrong-when-installing-logstash-input-websocket/259518)
**Author:** [@srigayathri](https://discuss.elastic.co/u/srigayathri)\ **Replies:** 0\ **Last updated:** [December 23, 2020, 6:02pm UTC](https://discuss.elastic.co/t/error-something-went-wrong-when-installing-logstash-input-websocket/259518 "2020-12-23T18:02:04Z")
Trying to install the logstash-input-websocket plugin in a corporate environment and been receiving the following error. Could anyone please help me out. WARNING: A maven settings file already exist at C:\\Users\\\ **Author:** [@raghavendra186](https://discuss.elastic.co/u/raghavendra186)\ **Replies:** 0\ **Last updated:** [December 23, 2020, 4:39pm UTC](https://discuss.elastic.co/t/removed-field-mapping-is-still-shown-in-index/259511 "2020-12-23T16:39:21Z") Hi, I faced error saying "could not index to elastic limit of 9k exceeded". i planned to reduce unwanted fields using remove\_field in logstash. my incoming json gets 9k fields. i have applied mutate filter in logstash t… --- ## [How this elasticsearch input plugin works in logstash?](https://discuss.elastic.co/t/how-this-elasticsearch-input-plugin-works-in-logstash/259371)
**Author:** [@deshpas](https://discuss.elastic.co/u/deshpas)\ **Replies:** 3\ **Last updated:** [December 23, 2020, 3:58pm UTC](https://discuss.elastic.co/t/how-this-elasticsearch-input-plugin-works-in-logstash/259371 "2020-12-23T15:58:51Z")
I am trying to transfer data from one AWS ES cluster to another AWS ES cluster and in this case, i am going to use the schedule option to pull the data from input every time. I would like to understand how this elastic… --- ## [External Ruby, more than one value in a hash](https://discuss.elastic.co/t/external-ruby-more-than-one-value-in-a-hash/259434)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 3\ **Last updated:** [December 23, 2020, 3:53pm UTC](https://discuss.elastic.co/t/external-ruby-more-than-one-value-in-a-hash/259434 "2020-12-23T15:53:13Z")
I have been using this external ruby code to add data to the documents, when a document has a planet\_name equal to the key in the hash add the value to a field in the document def register(params) end def filter(event) … --- ## [Timestamp not mapping with event time](https://discuss.elastic.co/t/timestamp-not-mapping-with-event-time/258976)
**Author:** [@bharat1](https://discuss.elastic.co/u/bharat1)\ **Replies:** 6\ **Last updated:** [December 23, 2020, 3:00pm UTC](https://discuss.elastic.co/t/timestamp-not-mapping-with-event-time/258976 "2020-12-23T15:00:20Z")
My logs are exported in csv format and uploaded. The first column name called "Event Time" has following format event time followed by rest of the comma separated columns - "2020/11/10 00:00:00 CET" I am not able to ma… --- ## [The shutdown process appears to be stalled due to busy or blocked plugins](https://discuss.elastic.co/t/the-shutdown-process-appears-to-be-stalled-due-to-busy-or-blocked-plugins/259487)
**Author:** [@Suddhasil\_Sarkar](https://discuss.elastic.co/u/Suddhasil_Sarkar)\ **Replies:** 0\ **Last updated:** [December 23, 2020, 1:06pm UTC](https://discuss.elastic.co/t/the-shutdown-process-appears-to-be-stalled-due-to-busy-or-blocked-plugins/259487 "2020-12-23T13:06:05Z")
Hi Team, Need help- we have ELK cluster which was down for few months. we nor brought the cluster up . Now having issues in logstash log as below.. \[ERROR\]\[logstash.shutdownwatcher \] The shutdown process appears to be s… --- ## [Event attribute not resolved in elasticsearh output](https://discuss.elastic.co/t/event-attribute-not-resolved-in-elasticsearh-output/259472)
**Author:** [@KarlosA](https://discuss.elastic.co/u/KarlosA)\ **Replies:** 0\ **Last updated:** [December 23, 2020, 10:33am UTC](https://discuss.elastic.co/t/event-attribute-not-resolved-in-elasticsearh-output/259472 "2020-12-23T10:33:02Z")
Hello, I'm using helm chart with image: image: "docker.elastic.co/logstash/logstash-oss" imageTag: "7.10.0" I provide the following configuration in values.yml: logstashPipeline: logstash.conf: | input { … --- ## [How can convert unstructured log string to json format using logstatsh](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202)
**Author:** [@abhi.coerian](https://discuss.elastic.co/u/abhi.coerian)\ **Replies:** 5\ **Last updated:** [December 23, 2020, 6:45am UTC](https://discuss.elastic.co/t/how-can-convert-unstructured-log-string-to-json-format-using-logstatsh/259202 "2020-12-23T06:45:43Z")
INFO handlers.DrivelRequestHandler: 2020-12-14 00:00:15.486 - JOB job\_1603918538928\_4026468 QUEUE queue\_test USER test AUTHORIZED\_SCHEMA message student {\\n optional int64 name;\\n optional double score;\\n required bin… --- ## [Filebeat-logstash issue](https://discuss.elastic.co/t/filebeat-logstash-issue/259272)
**Author:** [@mustafa.husny](https://discuss.elastic.co/u/mustafa.husny)\ **Replies:** 3\ **Last updated:** [December 23, 2020, 3:16am UTC](https://discuss.elastic.co/t/filebeat-logstash-issue/259272 "2020-12-23T03:16:37Z")
I am using ELK 7.6.2 I have three piplines first for the heartbeat second for the tcp to forward logs from QRadar Third is for filebeat I am receiving logs from heartbeat and qradar, but when I installed filebeat on… --- ## [Logstash doesn't remove proceed gz files](https://discuss.elastic.co/t/logstash-doesnt-remove-proceed-gz-files/259347)
**Author:** [@snowline](https://discuss.elastic.co/u/snowline)\ **Replies:** 4\ **Last updated:** [December 22, 2020, 9:46pm UTC](https://discuss.elastic.co/t/logstash-doesnt-remove-proceed-gz-files/259347 "2020-12-22T21:46:11Z")
I've setup logstash as file reader for \*.gz log files in "read" mode. These files are copied from remote server to local folders every 15 minutes under user logstash. Files and folders have owner "logstash" and mode - … --- ## [When a JDBC input start, drops the previous if not completed?](https://discuss.elastic.co/t/when-a-jdbc-input-start-drops-the-previous-if-not-completed/259416)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 3\ **Last updated:** [December 22, 2020, 8:01pm UTC](https://discuss.elastic.co/t/when-a-jdbc-input-start-drops-the-previous-if-not-completed/259416 "2020-12-22T20:01:31Z")
Lets say I got to index data from a week ago, and I use a schedule of 5 minutes in JDB input, and the data of a week cant be indexed in those 5 minutes, the data ingestion will be stopped at five minutes, and start agai… --- ## [GROK Help This setting must be a hash](https://discuss.elastic.co/t/grok-help-this-setting-must-be-a-hash/259408)
**Author:** [@fastxl](https://discuss.elastic.co/u/fastxl)\ **Replies:** 1\ **Last updated:** [December 22, 2020, 4:55pm UTC](https://discuss.elastic.co/t/grok-help-this-setting-must-be-a-hash/259408 "2020-12-22T16:55:34Z")
Need help trying to break down some data. This is the data being sent {"timestamp": "2020-11-17T19:02:19.352Z","sequence": 2460137,"deviceName": "OKUMA.MachiningCenterMA650-EAST","deviceUUID": "OKUMA.MachiningCenterMA65… --- ## [Logstash crashing](https://discuss.elastic.co/t/logstash-crashing/259403)
**Author:** [@lquin1978](https://discuss.elastic.co/u/lquin1978)\ **Replies:** 1\ **Last updated:** [December 22, 2020, 4:19pm UTC](https://discuss.elastic.co/t/logstash-crashing/259403 "2020-12-22T16:19:06Z")
We have just started getting the following error \[2020-12-22T15:57:08,056\]\[ERROR\]\[logstash.javapipeline \]\[main\] Pipeline worker error, the pipeline will be stopped {:pipeline\_id=\>"main", :error=\>"", :exception=\>Java:… --- ## [Nested for loop in logstash / calling logstash filters from ruby](https://discuss.elastic.co/t/nested-for-loop-in-logstash-calling-logstash-filters-from-ruby/259404)
**Author:** [@aerodynamic](https://discuss.elastic.co/u/aerodynamic)\ **Replies:** 0\ **Last updated:** [December 22, 2020, 3:59pm UTC](https://discuss.elastic.co/t/nested-for-loop-in-logstash-calling-logstash-filters-from-ruby/259404 "2020-12-22T15:59:43Z")
Hello, Currenly I'm dealing with a development roadblock to shorten a pipeline config. The logic in any other programming language would be a nested for loop. An iteration over values on array A for each value in array… --- ## [Merge data from several csv sources](https://discuss.elastic.co/t/merge-data-from-several-csv-sources/259381)
**Author:** [@MickD](https://discuss.elastic.co/u/MickD)\ **Replies:** 4\ **Last updated:** [December 22, 2020, 3:16pm UTC](https://discuss.elastic.co/t/merge-data-from-several-csv-sources/259381 "2020-12-22T15:16:36Z")
Hello, I'm new to ELK so please apologies if my question is trivial. I have 3 CSV files with data that I would like to merge. In each file, each row represent a data which can be father or child of a row in another fi… --- ## [Overwrite message field issue in grok match](https://discuss.elastic.co/t/overwrite-message-field-issue-in-grok-match/259300)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 3\ **Last updated:** [December 22, 2020, 2:22pm UTC](https://discuss.elastic.co/t/overwrite-message-field-issue-in-grok-match/259300 "2020-12-22T14:22:03Z")
i want to overwrite message field , but it only overwrites last match i.e. IP and is not overwriting first match i.e. number \`input { stdin{} } filter { grok { match =\> { "message" =\> "%{DATA}(?\(\\d{9,12})|… --- ## [Logstash configuration advice needed](https://discuss.elastic.co/t/logstash-configuration-advice-needed/259376)
**Author:** [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Replies:** 2\ **Last updated:** [December 22, 2020, 12:17pm UTC](https://discuss.elastic.co/t/logstash-configuration-advice-needed/259376 "2020-12-22T12:17:08Z")
Dears, I need your advice in case of configuration of Logstash. Currently the configuration looks like this one: cat /etc/logstash/conf.d/logstash.conf input { beats { port =\> 5044 ssl =\> true ssl\_certif… --- ## [Logstash replace string and insert new line](https://discuss.elastic.co/t/logstash-replace-string-and-insert-new-line/259377)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 0\ **Last updated:** [December 22, 2020, 11:49am UTC](https://discuss.elastic.co/t/logstash-replace-string-and-insert-new-line/259377 "2020-12-22T11:49:35Z")
hi guys, need to replace all occurrences of | with newline in field , gsub does not accept \\n or \\r...any idea as it will replace it with actual \\n mutate { gsub =\> \[ "message", "|", "\\n" \]}\` [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=267) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=269)