# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=718 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 719 --- ## [S3 Output role\_arn parameter](https://discuss.elastic.co/t/s3-output-role-arn-parameter/140956)
**Author:** [@bmaloney](https://discuss.elastic.co/u/bmaloney)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 7:51pm UTC](https://discuss.elastic.co/t/s3-output-role-arn-parameter/140956 "2018-07-20T19:51:35Z")
I see that the S3 output has had the role\_arn parameter documented since plugin version 4.1.0 (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#plugins-outputs-s3-role\_arn) however I can't find an… --- ## [Logstash can not receive data from kafka](https://discuss.elastic.co/t/logstash-can-not-receive-data-from-kafka/140357)
**Author:** [@lsy](https://discuss.elastic.co/u/lsy)\ **Replies:** 7\ **Last updated:** [July 20, 2018, 6:18pm UTC](https://discuss.elastic.co/t/logstash-can-not-receive-data-from-kafka/140357 "2018-07-20T18:18:18Z")
The structure I used is filebeat/metricbeat to kafka to logstash to es to kibana. At the first time logstash actually received data from kafka. However, after I trid to develop the kafka cluster, logstash no longer rece… --- ## [Log File is not being created with logstash output config](https://discuss.elastic.co/t/log-file-is-not-being-created-with-logstash-output-config/140110)
**Author:** [@DGMurthy](https://discuss.elastic.co/u/DGMurthy)\ **Replies:** 23\ **Last updated:** [July 20, 2018, 6:01pm UTC](https://discuss.elastic.co/t/log-file-is-not-being-created-with-logstash-output-config/140110 "2018-07-20T18:01:59Z")
Hi All, I am trying to create a log file(txt) by configuring logstash output section with codec =\> json\_lines, logstash is running successful but file is not been created . Mean time i am not doing any filters, once i s… --- ## [Multiple source of input in logstash config file](https://discuss.elastic.co/t/multiple-source-of-input-in-logstash-config-file/139891)
**Author:** [@pkshara](https://discuss.elastic.co/u/pkshara)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 5:53pm UTC](https://discuss.elastic.co/t/multiple-source-of-input-in-logstash-config-file/139891 "2018-07-20T17:53:00Z")
Hi Team, Can we have multiple source of inputs (filepath and jdbc) in input filter of logstash config file. Ex : input { file { path : } jdbc { select \* from table\_name; } } filter { } output { } TIA --- ## [Kafka messages not getting into elasticsearch/kibana](https://discuss.elastic.co/t/kafka-messages-not-getting-into-elasticsearch-kibana/139889)
**Author:** [@Jyothi\_Balla](https://discuss.elastic.co/u/Jyothi_Balla)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 5:52pm UTC](https://discuss.elastic.co/t/kafka-messages-not-getting-into-elasticsearch-kibana/139889 "2018-07-20T17:52:11Z")
When i start the filebeat the log is sending from the filebeat to the kafka-topic(read by kafka-consumer) then this message is a input to the logstash which is getting in and then not able to view the message in the kiba… --- ## [Logstash Multiline json format - Nagios XI Api](https://discuss.elastic.co/t/logstash-multiline-json-format-nagios-xi-api/139729)
**Author:** [@Alex\_Kefallonitis](https://discuss.elastic.co/u/Alex_Kefallonitis)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 5:47pm UTC](https://discuss.elastic.co/t/logstash-multiline-json-format-nagios-xi-api/139729 "2018-07-20T17:47:32Z")
Hi all, I have a problem parsing the below json format. Its Nagios XI api response . { "recordcount": "xxx", "logentry": \[ { "instance\_id": "1", "entry\_time": "YYYY-MM-DD HH:MM:SS", "logentry\_type": "8192", "log… --- ## [Can logstash write out the log files passed from filebeat?](https://discuss.elastic.co/t/can-logstash-write-out-the-log-files-passed-from-filebeat/140949)
**Author:** [@cholden14](https://discuss.elastic.co/u/cholden14)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 5:44pm UTC](https://discuss.elastic.co/t/can-logstash-write-out-the-log-files-passed-from-filebeat/140949 "2018-07-20T17:44:16Z")
Due to some regulator requirements we have to backup log files from our applications. I am lookin to having filebeat ship the logs to logstash and was hoping logstash would write out the logfiles to the server so we can… --- ## [Logstash how to end and start a new instance](https://discuss.elastic.co/t/logstash-how-to-end-and-start-a-new-instance/140712)
**Author:** [@Jack\_McAleese](https://discuss.elastic.co/u/Jack_McAleese)\ **Replies:** 6\ **Last updated:** [July 20, 2018, 5:38pm UTC](https://discuss.elastic.co/t/logstash-how-to-end-and-start-a-new-instance/140712 "2018-07-20T17:38:07Z")
Hi! I am new to ELK, hopefully someone can understand my problem here. I've ingested data via logstash into elasticsearch, but I need to clear all of that data and re-ingest it. My goal is to time how long it takes to … --- ## [Grok Pattern to add missing characters to a mac address](https://discuss.elastic.co/t/grok-pattern-to-add-missing-characters-to-a-mac-address/140865)
**Author:** [@martinrw](https://discuss.elastic.co/u/martinrw)\ **Replies:** 2\ **Last updated:** [July 20, 2018, 4:22pm UTC](https://discuss.elastic.co/t/grok-pattern-to-add-missing-characters-to-a-mac-address/140865 "2018-07-20T16:22:33Z")
Hi I'm attempting to use grok patterns in my logstash config to tokenise output from dhcpd. I have successfully managed it apart form one bit... DHCPD sometimes removes leading 0's from Mac addresses, specifically on t… --- ## [Ruby exception occurred: undefined method \`\[\]' for nil:NilClass](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-for-nil-nilclass/140630)
**Author:** [@Hanneu](https://discuss.elastic.co/u/Hanneu)\ **Replies:** 8\ **Last updated:** [July 20, 2018, 3:58pm UTC](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-for-nil-nilclass/140630 "2018-07-20T15:58:00Z")
Hello experts, I have been getting very many below error and warning messages. 1.) \[ERROR\]\[logstash.filters.ruby \] Ruby exception occurred: undefined method \`\[\]' for nil:NilClass 2.) \]\[WARN \]\[logstash.outputs.elast… --- ## [Aggregate filter plugin - need help to make it work](https://discuss.elastic.co/t/aggregate-filter-plugin-need-help-to-make-it-work/140901)
**Author:** [@Jevgenij](https://discuss.elastic.co/u/Jevgenij)\ **Replies:** 2\ **Last updated:** [July 20, 2018, 2:22pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-need-help-to-make-it-work/140901 "2018-07-20T14:22:16Z")
Hello! I have some data in joined db tables. To collect it I use the following query: SELECT \`item\`.\`id\`, \`item\`.\`name\`, \`colour\`.\`name\` AS \`colour\` FROM \`item\` LEFT JOIN \`item\_has\_colour\` AS \`ic\` ON \`ic\`.\`item\` = \`it… --- ## [Problem with field mapping](https://discuss.elastic.co/t/problem-with-field-mapping/140902)
**Author:** [@eegor](https://discuss.elastic.co/u/eegor)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 2:18pm UTC](https://discuss.elastic.co/t/problem-with-field-mapping/140902 "2018-07-20T14:18:04Z")
Hi all I use ruby web call inside a logstash filter in order to enrich log data. I call a URL with the content of an existing field and I would like to put the response into a new field. The code works and I can find th… --- ## [Logstash consuming more memory](https://discuss.elastic.co/t/logstash-consuming-more-memory/140905)
**Author:** [@Virenderdubey](https://discuss.elastic.co/u/Virenderdubey)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 1:57pm UTC](https://discuss.elastic.co/t/logstash-consuming-more-memory/140905 "2018-07-20T13:57:51Z")
We were running logstash 2.4 with 128MB and 256MB heap since ~2 years and it is running very fine. We are using input plugin as udp with port, then a grok fliter and output plugin as Elasticsearch of same version as log… --- ## [Jdbc streaming for mysql and oracle](https://discuss.elastic.co/t/jdbc-streaming-for-mysql-and-oracle/140868)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 10:21am UTC](https://discuss.elastic.co/t/jdbc-streaming-for-mysql-and-oracle/140868 "2018-07-20T10:21:55Z")
In jdbc streaming for mysql, it is possible. statement =\> "SELECT \* FROM (SELECT A.CID, ROW\_NUMBER() OVER (ORDER BY 1) AS NUM FROM TB\_REN016\_002 A WHERE LORA\_DEVID= :LoraID AND PORT= :port AND MEAIN\_STAT\_CODE\_NO != 700… --- ## [How to identify logstash pipeline status as complete?](https://discuss.elastic.co/t/how-to-identify-logstash-pipeline-status-as-complete/140515)
**Author:** [@abhisheksimion](https://discuss.elastic.co/u/abhisheksimion)\ **Replies:** 3\ **Last updated:** [July 20, 2018, 6:39am UTC](https://discuss.elastic.co/t/how-to-identify-logstash-pipeline-status-as-complete/140515 "2018-07-20T06:39:53Z")
I have below 2 queries Is there a specific key in logstash API response which can identify that a pipeline (incremental) completed successfully? I checked this where the Pipeline stats give an event response ...in, fi… --- ## [How to get shards from elasticsearch to logstash](https://discuss.elastic.co/t/how-to-get-shards-from-elasticsearch-to-logstash/140825)
**Author:** [@deok](https://discuss.elastic.co/u/deok)\ **Replies:** 1\ **Last updated:** [July 20, 2018, 3:09am UTC](https://discuss.elastic.co/t/how-to-get-shards-from-elasticsearch-to-logstash/140825 "2018-07-20T03:09:12Z")
first i am sorry than i poored at english i want to get a shards... and print out the log file i used elasticsearch for search engine.... this result is an api using the postman get : xxx.x.xx.xxx:9800/searchlog\_2018… --- ## [Problem with transfer Filebeat 6.1.3 \> Logstash 6.1.3 \> Elasticsearch 6.1.3](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264)
**Author:** [@artful](https://discuss.elastic.co/u/artful)\ **Replies:** 8\ **Last updated:** [July 20, 2018, 3:03am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264 "2018-07-20T03:03:55Z")
Hello! Transfer log Filebeat \> Elasticsearch without Logstash work fine. When I add Logstash, I see in log: \[2018-06-18T08:42:00,801\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:st… --- ## [Build failing for Logstash 6.3.1](https://discuss.elastic.co/t/build-failing-for-logstash-6-3-1/140823)
**Author:** [@nick-george](https://discuss.elastic.co/u/nick-george)\ **Replies:** 0\ **Last updated:** [July 20, 2018, 2:14am UTC](https://discuss.elastic.co/t/build-failing-for-logstash-6-3-1/140823 "2018-07-20T02:14:34Z")
Hi, I'm attempting to build logstash into an RPM. The reason I'm doing this is because I want to include some additional input/output/filter plugins at build time, rather than install them later. My ruby, rake, bundler… --- ## [Kafka Input - Queuing Beats](https://discuss.elastic.co/t/kafka-input-queuing-beats/140806)
**Author:** [@ms\_sec](https://discuss.elastic.co/u/ms_sec)\ **Replies:** 0\ **Last updated:** [July 19, 2018, 9:09pm UTC](https://discuss.elastic.co/t/kafka-input-queuing-beats/140806 "2018-07-19T21:09:07Z")
We have been having an issue for a few months where some of our larger elasticsearch clusters will get behind on winlogbeats data showing in kibana. In our architecture, we forward winlogbeats to kafka then use the kafk… --- ## [Aggregate filter push map](https://discuss.elastic.co/t/aggregate-filter-push-map/140686)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 1\ **Last updated:** [July 19, 2018, 8:41pm UTC](https://discuss.elastic.co/t/aggregate-filter-push-map/140686 "2018-07-19T20:41:18Z")
Hello ! I use the aggregate filter but and I want to push the map as event. I know that is possible to push the map as event on timeout but is that possible to push the map on end of task ? When the end event is detecte… --- ## [Filtre logstash "if not"](https://discuss.elastic.co/t/filtre-logstash-if-not/140739)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 3\ **Last updated:** [July 19, 2018, 6:43pm UTC](https://discuss.elastic.co/t/filtre-logstash-if-not/140739 "2018-07-19T18:43:27Z")
Hi, Can someone help me please : When test my configuration, I have this error : \[2018-07-19T15:01:56,149\]\[ERROR\]\[org.logstash.Logstash \] java.lang.IllegalStateException: Logstash stopped processing because of an e… --- ## [Logstash output exec executed only once](https://discuss.elastic.co/t/logstash-output-exec-executed-only-once/140755)
**Author:** [@mintalicious](https://discuss.elastic.co/u/mintalicious)\ **Replies:** 2\ **Last updated:** [July 19, 2018, 6:40pm UTC](https://discuss.elastic.co/t/logstash-output-exec-executed-only-once/140755 "2018-07-19T18:40:04Z")
Hi, is there a possibility to run an output exec command only once per logfile and not with every processed document? My data comes from CSV-files and I want to delete them after they have been processed by Logstash. I… --- ## [Doing a double split on json files](https://discuss.elastic.co/t/doing-a-double-split-on-json-files/140780)
**Author:** [@alex3](https://discuss.elastic.co/u/alex3)\ **Replies:** 0\ **Last updated:** [July 19, 2018, 5:19pm UTC](https://discuss.elastic.co/t/doing-a-double-split-on-json-files/140780 "2018-07-19T17:19:54Z")
So I have a nested list of object represented in a json format, where each json object contains a list I also want to split. When I had my codec as "json" in my file input, it would automatically split the json object an… --- ## [How to migrate MySQL data with many-to-many associations?](https://discuss.elastic.co/t/how-to-migrate-mysql-data-with-many-to-many-associations/140746)
**Author:** [@Jevgenij](https://discuss.elastic.co/u/Jevgenij)\ **Replies:** 1\ **Last updated:** [July 19, 2018, 5:07pm UTC](https://discuss.elastic.co/t/how-to-migrate-mysql-data-with-many-to-many-associations/140746 "2018-07-19T17:07:07Z")
Hi! I have db schema like this: I've succeded migration for city table data with LEFT JOIN: statement =\> "SELECT \`item\`.\`id\`, \`item\`.\`title\`, \`item\`.\`description\`, \`city\`.\`name\` AS \`city\` FROM \`item\` LEFT JOIN \`city… --- ## [How to configure filebeat and logstash? 3 VMs](https://discuss.elastic.co/t/how-to-configure-filebeat-and-logstash-3-vms/140748)
**Author:** [@saisimo02](https://discuss.elastic.co/u/saisimo02)\ **Replies:** 2\ **Last updated:** [July 19, 2018, 2:22pm UTC](https://discuss.elastic.co/t/how-to-configure-filebeat-and-logstash-3-vms/140748 "2018-07-19T14:22:00Z")
Hello, I have 3 VMs and I would like to install logstash locally and filebeat in each VM. I installed filebeat and my filebeat.yml is filebeat.prospectors: - input\_type: log paths: - /path/\*.xml # scan\_frequency: 60s … --- ## [Problem parsing irregular logs](https://discuss.elastic.co/t/problem-parsing-irregular-logs/140326)
**Author:** [@lbosec](https://discuss.elastic.co/u/lbosec)\ **Replies:** 4\ **Last updated:** [July 19, 2018, 1:36pm UTC](https://discuss.elastic.co/t/problem-parsing-irregular-logs/140326 "2018-07-19T13:36:38Z")
Hello everybody, I have a problem for parsing some logs that are very different according to the type of log, they contain spaces in the keys and in the values which makes the parsing really difficult to do. Here is a… --- ## [Grok pattern DATA (.\*?) not giving all matches from the line](https://discuss.elastic.co/t/grok-pattern-data-not-giving-all-matches-from-the-line/140468)
**Author:** [@laxman1](https://discuss.elastic.co/u/laxman1)\ **Replies:** 2\ **Last updated:** [July 19, 2018, 12:55pm UTC](https://discuss.elastic.co/t/grok-pattern-data-not-giving-all-matches-from-the-line/140468 "2018-07-19T12:55:28Z")
Text : Beginning Master Job Beginning child1 Job Beginning child2 Job when I used the below grok pattern temp variable is capturing only "Master" where as I need "child1" and "child2" matches also from text grok{ … --- ## [GitLab API Pagination](https://discuss.elastic.co/t/gitlab-api-pagination/139199)
**Author:** [@AurelienG](https://discuss.elastic.co/u/AurelienG)\ **Replies:** 11\ **Last updated:** [July 19, 2018, 12:46pm UTC](https://discuss.elastic.co/t/gitlab-api-pagination/139199 "2018-07-19T12:46:38Z")
Hello guys, I'm trying to get some global information about GitLab projects from the API. The problem is that this API is limited to 100 projects per page. I've managed to retrieve the number of pages with the following… --- ## [HELP! decoding](https://discuss.elastic.co/t/help-decoding/140469)
**Author:** [@Sunbro](https://discuss.elastic.co/u/Sunbro)\ **Replies:** 3\ **Last updated:** [July 19, 2018, 12:25pm UTC](https://discuss.elastic.co/t/help-decoding/140469 "2018-07-19T12:25:48Z")
I’m using ELK for centralized logging There is one field in my logs that needs to be decoded from base64 and then converted to ucs-2be How can I do that? --- ## [Logstash configuration is not giving value to add\_field](https://discuss.elastic.co/t/logstash-configuration-is-not-giving-value-to-add-field/140291)
**Author:** [@satendr](https://discuss.elastic.co/u/satendr)\ **Replies:** 6\ **Last updated:** [July 19, 2018, 11:24am UTC](https://discuss.elastic.co/t/logstash-configuration-is-not-giving-value-to-add-field/140291 "2018-07-19T11:24:08Z")
I am trying to Parse XMl file , and adding extra field for test status . file is getting parsed but for New field , instead of getting value , its displaying filter itself. please suggest.. logstash.config file: input {… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=717) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=719)