# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=326 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 327 --- ## [How to query logstash for all incoming or outgoing records](https://discuss.elastic.co/t/how-to-query-logstash-for-all-incoming-or-outgoing-records/241977)
**Author:** [@rustyweise](https://discuss.elastic.co/u/rustyweise)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 7:40am UTC](https://discuss.elastic.co/t/how-to-query-logstash-for-all-incoming-or-outgoing-records/241977 "2020-07-21T07:40:37Z")
We are using logstash as a proxy between beats, syslog, fluentbit, etc and logz.io. I would like to inspect ('tee') the input stream, specifically from a vCSA syslog stream, or query logstash for in-memory records, or ('… --- ## [How to access / address nested fields on Logstash](https://discuss.elastic.co/t/how-to-access-address-nested-fields-on-logstash/241861)
**Author:** [@Yoorae\_Kim](https://discuss.elastic.co/u/Yoorae_Kim)\ **Replies:** 3\ **Last updated:** [July 21, 2020, 6:17am UTC](https://discuss.elastic.co/t/how-to-access-address-nested-fields-on-logstash/241861 "2020-07-21T06:17:18Z")
Hello, I am currently trying to convert a nested sub field that contains hexadecimal string to an int type field using Logstash filter ruby{ code =\> 'event.set("\[transactions\]\[gasprice\_int\]", event.get("\[transaction… --- ## [How to aggregate multiple events into single output](https://discuss.elastic.co/t/how-to-aggregate-multiple-events-into-single-output/239536)
**Author:** [@pvaithee07](https://discuss.elastic.co/u/pvaithee07)\ **Replies:** 4\ **Last updated:** [July 21, 2020, 4:42am UTC](https://discuss.elastic.co/t/how-to-aggregate-multiple-events-into-single-output/239536 "2020-07-21T04:42:25Z")
Hi All - I am trying to create a logstash pipeline, that reads all the events in that pipeline, aggregates and produces one single output to elastic search. i/p looks like this - ''' { 'entity': '1212', ... }, { … --- ## [Input file path wildcard is not working](https://discuss.elastic.co/t/input-file-path-wildcard-is-not-working/241617)
**Author:** [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Replies:** 3\ **Last updated:** [July 21, 2020, 4:16am UTC](https://discuss.elastic.co/t/input-file-path-wildcard-is-not-working/241617 "2020-07-21T04:16:25Z")
Hello all, input file path wildcard is not working, As I remember, logstash 6.5.4 don't have the proble, while currently 7.6.2 encounter this problem, what could be the reason? Not working: input { file { … --- ## [Pull data periodically from an Elasticsearch index](https://discuss.elastic.co/t/pull-data-periodically-from-an-elasticsearch-index/241962)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 3:52am UTC](https://discuss.elastic.co/t/pull-data-periodically-from-an-elasticsearch-index/241962 "2020-07-21T03:52:07Z")
Hi, We have some log indices and now there is a requirement to extract some pieces of information from them and put them in another index where we can run some machine learning jobs on them. Initally I was thinking of … --- ## [Json\_lines codec is not able to parse certain jsons correctly](https://discuss.elastic.co/t/json-lines-codec-is-not-able-to-parse-certain-jsons-correctly/241593)
**Author:** [@buch11](https://discuss.elastic.co/u/buch11)\ **Replies:** 3\ **Last updated:** [July 21, 2020, 3:56am UTC](https://discuss.elastic.co/t/json-lines-codec-is-not-able-to-parse-certain-jsons-correctly/241593 "2020-07-21T03:56:51Z")
I am using json\_lines codec along with http\_input in my pipeline. One of the elements in my JSON is querystring. Usually, this codec is able to handle all the variants of valid jsons except below "querystring": { … --- ## [Logstash 7.8 doesn't re-parse existing file](https://discuss.elastic.co/t/logstash-7-8-doesnt-re-parse-existing-file/240669)
**Author:** [@Alex\_Liu](https://discuss.elastic.co/u/Alex_Liu)\ **Replies:** 7\ **Last updated:** [July 21, 2020, 3:34am UTC](https://discuss.elastic.co/t/logstash-7-8-doesnt-re-parse-existing-file/240669 "2020-07-21T03:34:53Z")
Hi team, I'm using Elastic Stack 7.8, below logstash doesn't read old files in /usr/data/nsg/ folder. input { file { path =\> "/usr/data/nsg/\*" start\_position =\> "beginning" codec =\> "json" … --- ## [Performance of running ELK Stack on Docker or VM?](https://discuss.elastic.co/t/performance-of-running-elk-stack-on-docker-or-vm/241595)
**Author:** [@eva\_cch](https://discuss.elastic.co/u/eva_cch)\ **Replies:** 2\ **Last updated:** [July 21, 2020, 2:38am UTC](https://discuss.elastic.co/t/performance-of-running-elk-stack-on-docker-or-vm/241595 "2020-07-21T02:38:49Z")
I have tried running ELK Stack on Docker in a CentOS 7 VM with 3 different Docker containers - Elasticsearch, Logstash and Kibana containers. It took me about an hour to run 625MB as per screenshot below whereby I have a… --- ## [Logstash Connectivity Issue with Elastic Search on Elastic Cloud](https://discuss.elastic.co/t/logstash-connectivity-issue-with-elastic-search-on-elastic-cloud/241585)
**Author:** [@dhirajagarwal2010](https://discuss.elastic.co/u/dhirajagarwal2010)\ **Replies:** 3\ **Last updated:** [July 21, 2020, 12:51am UTC](https://discuss.elastic.co/t/logstash-connectivity-issue-with-elastic-search-on-elastic-cloud/241585 "2020-07-21T00:51:25Z")
I am not able to connect my logstash instance on my local server with Elastic Search on Elastic Cloud . The user tag in logstash encodes the @ symbol in the email address twice to %2540. I guess because of this Elastic C… --- ## [Fluentd to External ELK stack](https://discuss.elastic.co/t/fluentd-to-external-elk-stack/241880)
**Author:** [@Lilth](https://discuss.elastic.co/u/Lilth)\ **Replies:** 1\ **Last updated:** [July 20, 2020, 11:58pm UTC](https://discuss.elastic.co/t/fluentd-to-external-elk-stack/241880 "2020-07-20T23:58:35Z")
Hello, I have a problem with Logstash. FluentD is sending logs to external ELK stack. Whichever option I have tried logstash is not creating index in ES. Config: input { tcp { port =\> codec =\> fluent } } outpu… --- ## [Importing Historical Log Data](https://discuss.elastic.co/t/importing-historical-log-data/241913)
**Author:** [@phinkel](https://discuss.elastic.co/u/phinkel)\ **Replies:** 1\ **Last updated:** [July 20, 2020, 11:41pm UTC](https://discuss.elastic.co/t/importing-historical-log-data/241913 "2020-07-20T23:41:58Z")
Hello currently we are transitioning to Elastic from LogRhythm SIEM. The current logs from LR are in .LCA format. Due to compliance reasons we will need to have access to this historical log data. I assume as LR is built… --- ## [Flush elasticsearch filter in output](https://discuss.elastic.co/t/flush-elasticsearch-filter-in-output/241825)
**Author:** [@Charlie\_Liang](https://discuss.elastic.co/u/Charlie_Liang)\ **Replies:** 0\ **Last updated:** [July 20, 2020, 2:13am UTC](https://discuss.elastic.co/t/flush-elasticsearch-filter-in-output/241825 "2020-07-20T02:13:25Z")
does pipeline.batch.delay control elasticsearch in output? it seems to be working on plugins in filter section. I've set the value to 5, but not able to see indices got updated in a min. Are there other controls for ela… --- ## [Need help with ILM when using logstash](https://discuss.elastic.co/t/need-help-with-ilm-when-using-logstash/241927)
**Author:** [@SimonR](https://discuss.elastic.co/u/SimonR)\ **Replies:** 1\ **Last updated:** [July 20, 2020, 8:03pm UTC](https://discuss.elastic.co/t/need-help-with-ilm-when-using-logstash/241927 "2020-07-20T20:03:15Z")
Hi all, i am struggling to setup ILM when sending logs from beats to via Logstash to Elasticsearch. Im looking to do daily indices, with rollover like the following index naming scheme: "companyname-winlogbeat-7.7.0--0… --- ## [Logstash Kakfa is Slow and Rebalancing Often](https://discuss.elastic.co/t/logstash-kakfa-is-slow-and-rebalancing-often/241924)
**Author:** [@dhinesh](https://discuss.elastic.co/u/dhinesh)\ **Replies:** 0\ **Last updated:** [July 20, 2020, 6:43pm UTC](https://discuss.elastic.co/t/logstash-kakfa-is-slow-and-rebalancing-often/241924 "2020-07-20T18:43:17Z")
Hi., I have logstash reading from Kafka topic with 3 paritions. The logstash is slow in processing the data and kafka is rebalancing very often, which is causing duplication of records. I have tried to increase the max… --- ## [Grokfilter Parser error](https://discuss.elastic.co/t/grokfilter-parser-error/241236)
**Author:** [@Deepika\_Rawat](https://discuss.elastic.co/u/Deepika_Rawat)\ **Replies:** 4\ **Last updated:** [July 20, 2020, 5:10pm UTC](https://discuss.elastic.co/t/grokfilter-parser-error/241236 "2020-07-20T17:10:27Z")
I have a log file with data like this indent preformatted text by 4 spaces 172.16.3.254 Jun 22 11:00:40 date=2020-06-22 local7 notice time=11:00:39 devname="MIBLR\_FW\_1" devid="FG200ETK19907000" logid="0000000013" type… --- ## [Restarting logstash from where it last shutdown](https://discuss.elastic.co/t/restarting-logstash-from-where-it-last-shutdown/241915)
**Author:** [@sandeepsingh](https://discuss.elastic.co/u/sandeepsingh)\ **Replies:** 1\ **Last updated:** [July 20, 2020, 5:26pm UTC](https://discuss.elastic.co/t/restarting-logstash-from-where-it-last-shutdown/241915 "2020-07-20T17:26:43Z")
I am using Logstash to transfer data from Elasticsearch to MongoDB. Now whenever there is an error or a Shutdown of Logstash instance, upon restarting Logstash, it starts reading the indexes in Elasticsearch from the beg… --- ## [Logstash OOM from one host only](https://discuss.elastic.co/t/logstash-oom-from-one-host-only/241625)
**Author:** [@Mandraenke](https://discuss.elastic.co/u/Mandraenke)\ **Replies:** 3\ **Last updated:** [July 20, 2020, 5:24pm UTC](https://discuss.elastic.co/t/logstash-oom-from-one-host-only/241625 "2020-07-20T17:24:57Z")
Hi, I'm facing an issue I cant dig down. I have setup filebeat to ship logs from different hosts to logstash. I found that some logs were missing and I found an out of memory error in the logs: \[2020-07-17T09:37:03… --- ## [How to change default file permission of log files in /var/log/logstash/](https://discuss.elastic.co/t/how-to-change-default-file-permission-of-log-files-in-var-log-logstash/241868)
**Author:** [@chitreshg](https://discuss.elastic.co/u/chitreshg)\ **Replies:** 1\ **Last updated:** [July 20, 2020, 5:07pm UTC](https://discuss.elastic.co/t/how-to-change-default-file-permission-of-log-files-in-var-log-logstash/241868 "2020-07-20T17:07:37Z")
I want to change the default file permission of logstash's log files as 600, currently it is 644. In logstash.yml file I found only the file path and log level related information but I didn't get any file permission re… --- ## [mariadb.jdbc.Driver not loaded](https://discuss.elastic.co/t/mariadb-jdbc-driver-not-loaded/241866)
**Author:** [@Fahad\_Ibrahim](https://discuss.elastic.co/u/Fahad_Ibrahim)\ **Replies:** 2\ **Last updated:** [July 20, 2020, 3:31pm UTC](https://discuss.elastic.co/t/mariadb-jdbc-driver-not-loaded/241866 "2020-07-20T15:31:30Z")
Hi All, I am trying to establish connectivity between mariadb and ELK. PFB conf file. input { jdbc { jdbc\_validate\_connection =\> true jdbc\_driver\_library =\> "D:/ELK/logstash/logstash-core/lib/jars/mari… --- ## [Split array with key+value fields to key: value - Follow up](https://discuss.elastic.co/t/split-array-with-key-value-fields-to-key-value-follow-up/240469)
**Author:** [@Rudolf\_Reddy\_Macejka](https://discuss.elastic.co/u/Rudolf_Reddy_Macejka)\ **Replies:** 2\ **Last updated:** [July 20, 2020, 9:28am UTC](https://discuss.elastic.co/t/split-array-with-key-value-fields-to-key-value-follow-up/240469 "2020-07-20T09:28:47Z")
Hello, I am trying to convert this input { "request": { "time": "2020-04-30T07:32:13.997Z", "size": "0", "headers": \[ { "key": "Correlation-Id", "value": "124f4cdf-… --- ## [Only a fraction of the data get indexed when executing logstash via service](https://discuss.elastic.co/t/only-a-fraction-of-the-data-get-indexed-when-executing-logstash-via-service/241853)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [July 20, 2020, 8:03am UTC](https://discuss.elastic.co/t/only-a-fraction-of-the-data-get-indexed-when-executing-logstash-via-service/241853 "2020-07-20T08:03:13Z")
Im consuming data from a DB, When I run the pipelines separately they get all the data, but when I run them via logstash service or with the command /usr/share/logstash/bin/logstash --path.settings /etc/logstash to execu… --- ## [One pipeline fail restarts Logstash, disrupting all other pipelines defined in pipelines.yml](https://discuss.elastic.co/t/one-pipeline-fail-restarts-logstash-disrupting-all-other-pipelines-defined-in-pipelines-yml/241845)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 0\ **Last updated:** [July 20, 2020, 6:46am UTC](https://discuss.elastic.co/t/one-pipeline-fail-restarts-logstash-disrupting-all-other-pipelines-defined-in-pipelines-yml/241845 "2020-07-20T06:46:50Z")
Hello, We have around 20 different pipelines defined in our pipelines.yml. One of them is outputing elastic query result to a file in a dedicated folder. Last weekend the owner of this folder changed to root, making Log… --- ## [Concatenate two fields](https://discuss.elastic.co/t/concatenate-two-fields/241817)
**Author:** [@Jasmin](https://discuss.elastic.co/u/Jasmin)\ **Replies:** 1\ **Last updated:** [July 19, 2020, 10:39pm UTC](https://discuss.elastic.co/t/concatenate-two-fields/241817 "2020-07-19T22:39:00Z")
hi , i am using mutate add\_field to concatenate two fileds to make them a unique id , but ouptut give me this : { "\_index" : "sql", "\_type" : "\_doc", "\_id" : " %{GL\_ETABLISSEMENT} %{GL\_NUMERO} ", "\_score" : 1.0, … --- ## [Logstash and elasticsearch mismatch](https://discuss.elastic.co/t/logstash-and-elasticsearch-mismatch/241758)
**Author:** [@elk6](https://discuss.elastic.co/u/elk6)\ **Replies:** 5\ **Last updated:** [July 19, 2020, 6:17pm UTC](https://discuss.elastic.co/t/logstash-and-elasticsearch-mismatch/241758 "2020-07-19T18:17:50Z")
I'm trying to ship logs from a server with filebeat to another server that hosts logstash and elasticsearch. Everything is latest and greatest (7.8.0). Problem is, I'm getting an error from logstash. This is the error I… --- ## [Error in system log filter](https://discuss.elastic.co/t/error-in-system-log-filter/241788)
**Author:** [@MikyTVA](https://discuss.elastic.co/u/MikyTVA)\ **Replies:** 1\ **Last updated:** [July 19, 2020, 6:09pm UTC](https://discuss.elastic.co/t/error-in-system-log-filter/241788 "2020-07-19T18:09:47Z")
I'm creating the ELK stack and I'm new to ELK stack. I'm using Logstash 6.8.10 on CentOS 7. I want process system logs and I'm realized that it doesn't work. Seeing the log I found that error \[ERROR\]\[logstash.agent … --- ## [Getiing this error 403 forbidden erro in "logstash-plain.log](https://discuss.elastic.co/t/getiing-this-error-403-forbidden-erro-in-logstash-plain-log/241796)
**Author:** [@rama543](https://discuss.elastic.co/u/rama543)\ **Replies:** 1\ **Last updated:** [July 19, 2020, 6:04pm UTC](https://discuss.elastic.co/t/getiing-this-error-403-forbidden-erro-in-logstash-plain-log/241796 "2020-07-19T18:04:09Z")
Getting below Error message in logstash-plain. log I am using AWS managed ElasticSearch ,Kibana . Logstash i have installed in amazon ec2 instance . Please help to get resolve the issue Attempted to resurrect connect… --- ## [Unable to use kafka input with SASL\_SSL](https://discuss.elastic.co/t/unable-to-use-kafka-input-with-sasl-ssl/241785)
**Author:** [@madhu72](https://discuss.elastic.co/u/madhu72)\ **Replies:** 0\ **Last updated:** [July 19, 2020, 11:52am UTC](https://discuss.elastic.co/t/unable-to-use-kafka-input-with-sasl-ssl/241785 "2020-07-19T11:52:50Z")
I dont see any incoming data from Kafka to Kibana thru Logstash. I configured everything, and noticed that, it have security.protocol = PLAINTEXT even I configured SASL\_SSL. input { kafka { sasl\_mechanism =\> "SCRAM-S… --- ## [Logstash multiple pipeline on docker](https://discuss.elastic.co/t/logstash-multiple-pipeline-on-docker/241301)
**Author:** [@Abolurah](https://discuss.elastic.co/u/Abolurah)\ **Replies:** 4\ **Last updated:** [July 19, 2020, 9:09am UTC](https://discuss.elastic.co/t/logstash-multiple-pipeline-on-docker/241301 "2020-07-19T09:09:05Z")
Hi All I'm trying to build logstash on docker with multiple pipelines. After reading a lot over the internet and trying things, still can't start multiple pipelines on docker. I'm trying to run some basic configuratio… --- ## [TAB in grok filter and delete mismatched records](https://discuss.elastic.co/t/tab-in-grok-filter-and-delete-mismatched-records/241767)
**Author:** [@merlock](https://discuss.elastic.co/u/merlock)\ **Replies:** 2\ **Last updated:** [July 19, 2020, 5:49am UTC](https://discuss.elastic.co/t/tab-in-grok-filter-and-delete-mismatched-records/241767 "2020-07-19T05:49:06Z")
hi all, (sorry for my english) hello everyone I have the following problem, the log I'm trying to read comes with tabs and I can't filter correctly. I have tried several options but I cannot read it correctly 100% this… --- ## [Logstash Input from Firestore (Firebase) to Elasticsearch](https://discuss.elastic.co/t/logstash-input-from-firestore-firebase-to-elasticsearch/241764)
**Author:** [@estack](https://discuss.elastic.co/u/estack)\ **Replies:** 0\ **Last updated:** [July 18, 2020, 10:18pm UTC](https://discuss.elastic.co/t/logstash-input-from-firestore-firebase-to-elasticsearch/241764 "2020-07-18T22:18:48Z")
Hi Guys, I have tried to implement a logstash config by installing a custom logstash input from the link below: I have made necessary configs, but getting below error: \]\[FATAL\]\[logstash.runner \] An unexpe… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=325) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=327)