# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=78 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 79 --- ## [The problem with getting a lot of logs](https://discuss.elastic.co/t/the-problem-with-getting-a-lot-of-logs/332291)
**Author:** [@DenisGr](https://discuss.elastic.co/u/DenisGr)\ **Replies:** 5\ **Last updated:** [May 3, 2023, 6:30am UTC](https://discuss.elastic.co/t/the-problem-with-getting-a-lot-of-logs/332291 "2023-05-03T06:30:03Z")
Hey guys. I'm new here and I apologize in advance if I opened a thread in the wrong section. I have the task of handling a large number of logs, generating up to 100,000 per minute or even more. I currently have elk mas… --- ## [Json codec failed with data bigger then 4k size](https://discuss.elastic.co/t/json-codec-failed-with-data-bigger-then-4k-size/330687)
**Author:** [@arunporwal](https://discuss.elastic.co/u/arunporwal)\ **Replies:** 12\ **Last updated:** [May 3, 2023, 4:49am UTC](https://discuss.elastic.co/t/json-codec-failed-with-data-bigger-then-4k-size/330687 "2023-05-03T04:49:18Z")
I am trying to read the JSON files using logstash input plugin. For this I used codec as JSON, it is working fine until it encounters a file containing a size of more than 4k. It is giving a parsing error for this. I did… --- ## [Require Last 15 minutes from timestamp](https://discuss.elastic.co/t/require-last-15-minutes-from-timestamp/332039)
**Author:** [@rubhamra](https://discuss.elastic.co/u/rubhamra)\ **Replies:** 2\ **Last updated:** [May 3, 2023, 1:42am UTC](https://discuss.elastic.co/t/require-last-15-minutes-from-timestamp/332039 "2023-05-03T01:42:09Z")
I have REST API call and I need to pull records which were updated every last 15 minutes. I can pull last 24 records but I need just updated with in last 15 minutes based on the date field. This is the ruby code ruby … --- ## [Logs not pushed to elasticsearch](https://discuss.elastic.co/t/logs-not-pushed-to-elasticsearch/330590)
**Author:** [@aaronlbk](https://discuss.elastic.co/u/aaronlbk)\ **Replies:** 5\ **Last updated:** [May 2, 2023, 8:25pm UTC](https://discuss.elastic.co/t/logs-not-pushed-to-elasticsearch/330590 "2023-05-02T20:25:48Z")
I have an issue with logstash that is not pushing logs to elastic. Below is the config file filebeat.yml that is used. I have tried to check on kibana if the index is even created and it is not. # Sample Logstash config… --- ## [Logstash input plugin http\_poller only loading 300 records in a run](https://discuss.elastic.co/t/logstash-input-plugin-http-poller-only-loading-300-records-in-a-run/332281)
**Author:** [@abhisarika\_verma](https://discuss.elastic.co/u/abhisarika_verma)\ **Replies:** 0\ **Last updated:** [May 2, 2023, 1:10pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-http-poller-only-loading-300-records-in-a-run/332281 "2023-05-02T13:10:47Z")
I am trying to use the http\_poller input plugin to load data from sap CDC into Elasticsearch. But Logstash input plugin http\_poller is only loading 300 records in a run. How can I load bulk data using the http\_poller … --- ## [Reading append blob from azure storage account using logstash-input-azure\_blob\_storage](https://discuss.elastic.co/t/reading-append-blob-from-azure-storage-account-using-logstash-input-azure-blob-storage/332270)
**Author:** [@salramahi](https://discuss.elastic.co/u/salramahi)\ **Replies:** 0\ **Last updated:** [May 2, 2023, 12:20pm UTC](https://discuss.elastic.co/t/reading-append-blob-from-azure-storage-account-using-logstash-input-azure-blob-storage/332270 "2023-05-02T12:20:10Z")
Hello, I'm using the azure\_blob\_storage logstash input to read log files from storage account. Currently, I have a diagnostic setting on my ADF that pushes any pipeline/activity logs to storage account. for every hour, … --- ## [Put logstash behind httpd reverse proxy](https://discuss.elastic.co/t/put-logstash-behind-httpd-reverse-proxy/330916)
**Author:** [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Replies:** 4\ **Last updated:** [May 2, 2023, 10:48am UTC](https://discuss.elastic.co/t/put-logstash-behind-httpd-reverse-proxy/330916 "2023-05-02T10:48:36Z")
Hi, Can i put logstash behind httpd reverse proxy and use /logstash sub path? I have filebeat installe d on windows in one VPC and logstash in a private vpc. So if i put logstash behind httpd Reverse proxy which has a pu… --- ## [Unable to connect to database. Tried 1 times](https://discuss.elastic.co/t/unable-to-connect-to-database-tried-1-times/331547)
**Author:** [@m3bgwad](https://discuss.elastic.co/u/m3bgwad)\ **Replies:** 1\ **Last updated:** [May 2, 2023, 9:22am UTC](https://discuss.elastic.co/t/unable-to-connect-to-database-tried-1-times/331547 "2023-05-02T09:22:00Z")
Hello to All, I am trying to connect with the Database, and I configured the JDBC plugin in case there is a time-out connection. jdbc\_validate\_connection =\> "true" jdbc\_validation\_timeout =\> "60" In the first time-out… --- ## [How can I generate a CEF output](https://discuss.elastic.co/t/how-can-i-generate-a-cef-output/331867)
**Author:** [@CyberGuy](https://discuss.elastic.co/u/CyberGuy)\ **Replies:** 1\ **Last updated:** [May 1, 2023, 4:53pm UTC](https://discuss.elastic.co/t/how-can-i-generate-a-cef-output/331867 "2023-05-01T16:53:42Z")
I have created a logstash configuration that successfully parses CEF logs and applies certain logic to it. The filter configuration extracts the CEF with a grok filter and then uses the kv plugin to extract the different… --- ## [Logstash sometimes parsing sometime not, even though sending the Same message](https://discuss.elastic.co/t/logstash-sometimes-parsing-sometime-not-even-though-sending-the-same-message/331870)
**Author:** [@Prakash111](https://discuss.elastic.co/u/Prakash111)\ **Replies:** 0\ **Last updated:** [May 1, 2023, 12:40pm UTC](https://discuss.elastic.co/t/logstash-sometimes-parsing-sometime-not-even-though-sending-the-same-message/331870 "2023-05-01T12:40:01Z")
Log message 2023-05-01T05:22:20.154Z \[INFO\] "interSample" {"PID": 1, "Service": "XYZService", "Data": \[\]} some times log stash parsing successfully "\_message\_json\_parsed" some times parse failure "\_grokparsefailure" e… --- ## [How to extract the entire value of a complicated field?](https://discuss.elastic.co/t/how-to-extract-the-entire-value-of-a-complicated-field/330891)
**Author:** [@CyberGuy](https://discuss.elastic.co/u/CyberGuy)\ **Replies:** 6\ **Last updated:** [May 1, 2023, 12:34pm UTC](https://discuss.elastic.co/t/how-to-extract-the-entire-value-of-a-complicated-field/330891 "2023-05-01T12:34:29Z")
HI guys, I'm trying to create a logstash pipeline that parses incoming CEF logs, apply some logic and then outputs the log in JSON format to the console. Some logs are a bit complicated to parse since the key=value pai… --- ## [How to index audio/video files to kibana](https://discuss.elastic.co/t/how-to-index-audio-video-files-to-kibana/330834)
**Author:** [@AdityaKhajuria](https://discuss.elastic.co/u/AdityaKhajuria)\ **Replies:** 1\ **Last updated:** [April 30, 2023, 11:15pm UTC](https://discuss.elastic.co/t/how-to-index-audio-video-files-to-kibana/330834 "2023-04-30T23:15:08Z")
Hi, Im trying to index audio/video files to kibana. I am able to get audio in a field by setting Format-URL and type-Audio in index pattern. But i want my logstash to index my audio/video files to kibana. --- ## [Log stash behavior when output plug-in not reachable](https://discuss.elastic.co/t/log-stash-behavior-when-output-plug-in-not-reachable/331376)
**Author:** [@eth](https://discuss.elastic.co/u/eth)\ **Replies:** 0\ **Last updated:** [April 30, 2023, 6:14pm UTC](https://discuss.elastic.co/t/log-stash-behavior-when-output-plug-in-not-reachable/331376 "2023-04-30T18:14:03Z")
I am using logstash 7 with syslog as output plugin. The syslog server is not reachable for a quite a long time and persistent queue is growing as expected to the limit. But the persistent queue data size is growing bey… --- ## [Logstash gives OOM & CPU Usage too high when used with S3 Input plugin](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121)
**Author:** [@Utpal\_Brahma](https://discuss.elastic.co/u/Utpal_Brahma)\ **Replies:** 3\ **Last updated:** [April 29, 2023, 5:25pm UTC](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121 "2023-04-29T17:25:38Z")
Logstash gives out of Memory when S3 plugin is used for a bucket which has already existing tones of files. --- ## [Logstash filters for log file which is included some raw data and json data](https://discuss.elastic.co/t/logstash-filters-for-log-file-which-is-included-some-raw-data-and-json-data/330950)
**Author:** [@Harish1](https://discuss.elastic.co/u/Harish1)\ **Replies:** 3\ **Last updated:** [April 28, 2023, 5:24pm UTC](https://discuss.elastic.co/t/logstash-filters-for-log-file-which-is-included-some-raw-data-and-json-data/330950 "2023-04-28T17:24:50Z")
Hi Elastic team, I'm new to ELK, I'm trying to find out the filters for below log file but I'm not able to find the proper Logstash filter for below data 2023-01-19 15:38:31 INFO VCIPDownstreamController:138 - {"timest… --- ## [Multisource index on elasticsearch passing by logstash](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844)
**Author:** [@Abdeljalil\_El\_Yousso](https://discuss.elastic.co/u/Abdeljalil_El_Yousso)\ **Replies:** 10\ **Last updated:** [April 28, 2023, 3:57pm UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844 "2023-04-28T15:57:42Z")
hey , im trying to create multiple source input from Filebeat , than injecting them into logstash to apply filters , and finally transfer the sources to elasticsearch as indexes The problem i have , only one index is cr… --- ## [Rename nested field based on its data type](https://discuss.elastic.co/t/rename-nested-field-based-on-its-data-type/331044)
**Author:** [@aversecguy](https://discuss.elastic.co/u/aversecguy)\ **Replies:** 0\ **Last updated:** [April 28, 2023, 10:18am UTC](https://discuss.elastic.co/t/rename-nested-field-based-on-its-data-type/331044 "2023-04-28T10:18:21Z")
Hello, dear community, I am brand new to logstash, but have to fix a problem: We are gathering eks audit logs and have errors like illegal\_state\_exception error because of the field responseObject.status could be the t… --- ## [Logstash is not showing base64 encoded data for pdf's extracted from urls](https://discuss.elastic.co/t/logstash-is-not-showing-base64-encoded-data-for-pdfs-extracted-from-urls/330386)
**Author:** [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\ **Replies:** 5\ **Last updated:** [April 28, 2023, 6:58am UTC](https://discuss.elastic.co/t/logstash-is-not-showing-base64-encoded-data-for-pdfs-extracted-from-urls/330386 "2023-04-28T06:58:36Z")
Hi Team, I am using logstash http filter to get pdf from url and extract it. http filter has downloaded pdf and extracted its content on target\_field. But the contents are not proper and also its not base64 encoded. Ho… --- ## [How to create new array by using existing list of strings field in logstash ruby filter](https://discuss.elastic.co/t/how-to-create-new-array-by-using-existing-list-of-strings-field-in-logstash-ruby-filter/330761)
**Author:** [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\ **Replies:** 2\ **Last updated:** [April 28, 2023, 4:57am UTC](https://discuss.elastic.co/t/how-to-create-new-array-by-using-existing-list-of-strings-field-in-logstash-ruby-filter/330761 "2023-04-28T04:57:00Z")
Hi Team, I have three arrays created from xml in logstash content.REFERENCE: \[PXXXX, TECHNICAL\_SUPPORT\] content.ROOT: \[INTERNAL\_PRODUCT\_OR\_APPLICATION, TOPICS\] I have to create a result array from above inputs if roo… --- ## [Protobuf data decode issue](https://discuss.elastic.co/t/protobuf-data-decode-issue/330976)
**Author:** [@Nithingowda](https://discuss.elastic.co/u/Nithingowda)\ **Replies:** 0\ **Last updated:** [April 27, 2023, 3:21pm UTC](https://discuss.elastic.co/t/protobuf-data-decode-issue/330976 "2023-04-27T15:21:12Z")
Here is the code to read the protobuf data from pubsub and decode in logstash but we are unable to decode the protobuf data. Code: input { google\_pubsub { project\_id =\> "project\_id" topic =\> "topic… --- ## [Can logstash.yml can be reloaded?](https://discuss.elastic.co/t/can-logstash-yml-can-be-reloaded/330942)
**Author:** [@prashant1](https://discuss.elastic.co/u/prashant1)\ **Replies:** 2\ **Last updated:** [April 27, 2023, 2:42pm UTC](https://discuss.elastic.co/t/can-logstash-yml-can-be-reloaded/330942 "2023-04-27T14:42:34Z")
We have deployed logstash in kubernetes platform. For one usecase we want to update queue.page\_capacity: 64mb to 1mb. So if we update these changes it can't be reloaded until restart. So is there any way so that this ca… --- ## [Logstash startup error-) Could not load FFI Provider: (NotImplementedError) FFI not available](https://discuss.elastic.co/t/logstash-startup-error-could-not-load-ffi-provider-notimplementederror-ffi-not-available/330904)
**Author:** [@karthic](https://discuss.elastic.co/u/karthic)\ **Replies:** 1\ **Last updated:** [April 27, 2023, 2:40pm UTC](https://discuss.elastic.co/t/logstash-startup-error-could-not-load-ffi-provider-notimplementederror-ffi-not-available/330904 "2023-04-27T14:40:32Z")
Tried to load logstash in a Centos environment \[INFO \]\[logstash.runner \] JVM bootstrap flags: \[-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpO… --- ## [ESET Protect Cloud logs](https://discuss.elastic.co/t/eset-protect-cloud-logs/330925)
**Author:** [@rodmontgt](https://discuss.elastic.co/u/rodmontgt)\ **Replies:** 2\ **Last updated:** [April 27, 2023, 1:50pm UTC](https://discuss.elastic.co/t/eset-protect-cloud-logs/330925 "2023-04-27T13:50:24Z")
Hi everyone, I've been playing around with logtash for days but still have not found a solution for this, my ESET console is configured to send syslog/BSD logs but I am getting this odd character set in my logstash inst… --- ## [Failed to start Logstash - S3 output plugin is not working](https://discuss.elastic.co/t/failed-to-start-logstash-s3-output-plugin-is-not-working/330096)
**Author:** [@WonhyeongCho](https://discuss.elastic.co/u/WonhyeongCho)\ **Replies:** 4\ **Last updated:** [April 27, 2023, 1:46pm UTC](https://discuss.elastic.co/t/failed-to-start-logstash-s3-output-plugin-is-not-working/330096 "2023-04-27T13:46:17Z")
Hi. I'm using Logstash. I recently upgraded Logstash to version 8.7.0, but it's not working. I'm getting an error message. Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:OSQUERY, :excep… --- ## [File input plugin is treating the line as plain string eventhough the input is json](https://discuss.elastic.co/t/file-input-plugin-is-treating-the-line-as-plain-string-eventhough-the-input-is-json/330713)
**Author:** [@djrshn2346](https://discuss.elastic.co/u/djrshn2346)\ **Replies:** 10\ **Last updated:** [April 27, 2023, 12:48pm UTC](https://discuss.elastic.co/t/file-input-plugin-is-treating-the-line-as-plain-string-eventhough-the-input-is-json/330713 "2023-04-27T12:48:22Z")
File input plugin is treating the line as plain string eventhough the input is json. Output in Opensearch is : { "\_index" : "sample-logs-2023.04.24", "\_type" : "\_doc", "\_id" : "ui9PsocBICgSyzdw… --- ## [How to parse API HTTP output data](https://discuss.elastic.co/t/how-to-parse-api-http-output-data/330829)
**Author:** [@sonirajil](https://discuss.elastic.co/u/sonirajil)\ **Replies:** 2\ **Last updated:** [April 27, 2023, 9:00am UTC](https://discuss.elastic.co/t/how-to-parse-api-http-output-data/330829 "2023-04-27T09:00:03Z")
Hello Team, I am running an API to get servicestatus data which looks like : { "recordcount": 11906, "servicestatus": \[ { "host\_name": "unixteam.abc.com", "service\_description": … --- ## [Handling Kuberenets Labels Mapping Conflict](https://discuss.elastic.co/t/handling-kuberenets-labels-mapping-conflict/330915)
**Author:** [@Noa](https://discuss.elastic.co/u/Noa)\ **Replies:** 0\ **Last updated:** [April 27, 2023, 7:50am UTC](https://discuss.elastic.co/t/handling-kuberenets-labels-mapping-conflict/330915 "2023-04-27T07:50:35Z")
I have two kinds of labels which are causing a mapping conflict: app: \* vs. app.kubernetes.io/instance: \* (type text vs. type object) The second label is predefined by Kubernetes and is used to differentiate between d… --- ## [Why does this field exist in my output even though I have removed this?](https://discuss.elastic.co/t/why-does-this-field-exist-in-my-output-even-though-i-have-removed-this/330890)
**Author:** [@CyberGuy](https://discuss.elastic.co/u/CyberGuy)\ **Replies:** 0\ **Last updated:** [April 26, 2023, 11:15pm UTC](https://discuss.elastic.co/t/why-does-this-field-exist-in-my-output-even-though-i-have-removed-this/330890 "2023-04-26T23:15:39Z")
HI guys, I'm trying to create a logstash pipeline that parses incoming CEF logs, apply some logic and then outputs the log in JSON format to the console. For some reason, a field is generated with the name "Virtual Syst… --- ## [Logstash Output](https://discuss.elastic.co/t/logstash-output/330786)
**Author:** [@Dasher](https://discuss.elastic.co/u/Dasher)\ **Replies:** 1\ **Last updated:** [April 26, 2023, 3:15pm UTC](https://discuss.elastic.co/t/logstash-output/330786 "2023-04-26T15:15:22Z")
I have a logstash conf file with multiple output configured In both the outputs i'm using multiple if else statements.Is it possible for the data to get entered in the else statement of both output? --- ## [ELK stack in windows 11](https://discuss.elastic.co/t/elk-stack-in-windows-11/330832)
**Author:** [@Sardor](https://discuss.elastic.co/u/Sardor)\ **Replies:** 15\ **Last updated:** [April 26, 2023, 1:11pm UTC](https://discuss.elastic.co/t/elk-stack-in-windows-11/330832 "2023-04-26T13:11:08Z")
I tried to install ELK stack, Elasticsearch, Logstash, Kibana. Elasticsearch and Kibana run succesfully, but logstash returned some exceptions. How can I fix it? This is last logs from logstash : \[2023-04-26T16:45:34,3… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=77) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=79)