# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=236 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 237 --- ## [Adding logstash IP address field when processing filter](https://discuss.elastic.co/t/adding-logstash-ip-address-field-when-processing-filter/270055)
**Author:** [@yquirion](https://discuss.elastic.co/u/yquirion)\ **Replies:** 2\ **Last updated:** [April 22, 2021, 5:48pm UTC](https://discuss.elastic.co/t/adding-logstash-ip-address-field-when-processing-filter/270055 "2021-04-22T17:48:33Z")
Dear Elastic Community, We are using several logstash servers behind a loadbalancer. All you client are sending their logs through those VIP created on the load balancer, not directly to the logstash servers. For the "b… --- ## [How can I set heap memory for jruby in logstash 7.4?](https://discuss.elastic.co/t/how-can-i-set-heap-memory-for-jruby-in-logstash-7-4/270990)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 4:32pm UTC](https://discuss.elastic.co/t/how-can-i-set-heap-memory-for-jruby-in-logstash-7-4/270990 "2021-04-22T16:32:53Z")
Hi all, Please let me know how can I set jruby\_opts for controlling jruby memory which is default 500 mb, I need to control by setting it low and high. Thanks --- ## [Add time interval in between each document's timestamp](https://discuss.elastic.co/t/add-time-interval-in-between-each-documents-timestamp/270988)
**Author:** [@Felipe\_Fuller](https://discuss.elastic.co/u/Felipe_Fuller)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 4:24pm UTC](https://discuss.elastic.co/t/add-time-interval-in-between-each-documents-timestamp/270988 "2021-04-22T16:24:45Z")
Hi Community! I hope you're doing great :slight\_smile: I'm struggling to give a 5 seconds window/interval of time between each document inserted to elasticsearch in the timestamp field. Actually, I have this pipeline: … --- ## [How to trigger mail in watcher if date&time difference more than 2 sec](https://discuss.elastic.co/t/how-to-trigger-mail-in-watcher-if-date-time-difference-more-than-2-sec/270980)
**Author:** [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 3:16pm UTC](https://discuss.elastic.co/t/how-to-trigger-mail-in-watcher-if-date-time-difference-more-than-2-sec/270980 "2021-04-22T15:16:36Z")
Hello all, How I can trigger mail in watcher if date&time difference more than 2 sec . I have 2 different log line. First log line contain date & time for sent at and second log line contain date & time for received at… --- ## [Update of the Geolite database](https://discuss.elastic.co/t/update-of-the-geolite-database/270801)
**Author:** [@Pacous](https://discuss.elastic.co/u/Pacous)\ **Replies:** 12\ **Last updated:** [April 22, 2021, 10:32am UTC](https://discuss.elastic.co/t/update-of-the-geolite-database/270801 "2021-04-22T10:32:58Z")
I configured the Geoip and the Geolite database in logstash. How often do you update the Geolite database? Regards, --- ## [Problem with logstash input kinesis plugin while reading data from the stream](https://discuss.elastic.co/t/problem-with-logstash-input-kinesis-plugin-while-reading-data-from-the-stream/270969)
**Author:** [@dbElastic](https://discuss.elastic.co/u/dbElastic)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 1:58pm UTC](https://discuss.elastic.co/t/problem-with-logstash-input-kinesis-plugin-while-reading-data-from-the-stream/270969 "2021-04-22T13:58:23Z")
Caught exception while sync'ing Kinesis shards and leases com.amazonaws.services.kinesis.model.ResourceNotFoundException: Stream aws-xxx-xxx-xxxx-kinesis-xxxxxx under account xxxxxxxxxx not found. (Service: AmazonKinesi… --- ## [Redundancy for syslog](https://discuss.elastic.co/t/redundancy-for-syslog/270943)
**Author:** [@debrt](https://discuss.elastic.co/u/debrt)\ **Replies:** 1\ **Last updated:** [April 22, 2021, 2:56pm UTC](https://discuss.elastic.co/t/redundancy-for-syslog/270943 "2021-04-22T14:56:38Z")
Hello, I am trying to deploy a redundant solution where I have end devices logging over syslog to my logstash server. I already have a single logstash server running. Is is possible to send the syslog logs to multiple … --- ## [Splitting Data](https://discuss.elastic.co/t/splitting-data/270965)
**Author:** [@spike83](https://discuss.elastic.co/u/spike83)\ **Replies:** 1\ **Last updated:** [April 22, 2021, 2:17pm UTC](https://discuss.elastic.co/t/splitting-data/270965 "2021-04-22T14:17:48Z")
Hi, I'm consuming yum.log with filebeat, this is great but I want to be able to manipulate the data as it comes in to allow me to filter better and visulise the data. At the moment a message looks like this "message":… --- ## [Logstash shows parsing error for json files](https://discuss.elastic.co/t/logstash-shows-parsing-error-for-json-files/270890)
**Author:** [@pratyush\_elastic](https://discuss.elastic.co/u/pratyush_elastic)\ **Replies:** 0\ **Last updated:** [April 21, 2021, 5:54pm UTC](https://discuss.elastic.co/t/logstash-shows-parsing-error-for-json-files/270890 "2021-04-21T17:54:08Z")
Hi, I am getting json parsing error with logstash intermittently while it tries to read and load to Elastic. Below are the details. Any lead would really be helpful. I tried isolating the file and it works fine Error: … --- ## [Logstash plugin](https://discuss.elastic.co/t/logstash-plugin/270968)
**Author:** [@mbee](https://discuss.elastic.co/u/mbee)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 1:50pm UTC](https://discuss.elastic.co/t/logstash-plugin/270968 "2021-04-22T13:50:02Z")
Hello, I'm trying to install codec\_avro\_schema\_registry plugin in offline mode. i downloaded the zip file from github The installation doesn't work, it throw this error : The pack must contains at least one plugin I… --- ## [How do we get logstash to process a gzip file as input?](https://discuss.elastic.co/t/how-do-we-get-logstash-to-process-a-gzip-file-as-input/270934)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 1\ **Last updated:** [April 22, 2021, 1:49pm UTC](https://discuss.elastic.co/t/how-do-we-get-logstash-to-process-a-gzip-file-as-input/270934 "2021-04-22T13:49:47Z")
I have trouble getting logstash to process gzip file . i have installed the gzip codec and referenced it in the code as well . During execution , logstash says " Error: not in gzip format" even the file is gzip . This ex… --- ## [Does Logstash support wilcards for multiple empty fields](https://discuss.elastic.co/t/does-logstash-support-wilcards-for-multiple-empty-fields/270948)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 2\ **Last updated:** [April 22, 2021, 12:20pm UTC](https://discuss.elastic.co/t/does-logstash-support-wilcards-for-multiple-empty-fields/270948 "2021-04-22T12:20:17Z")
Hello, I know that in Logstash I can check if a field is empty with the following if !\[myfield\] { do something } Having said that I though that wildcards will match all fields that are empty but that was not the case. … --- ## [Logstash file Input Plugin to read all files from a directory where log files gets rotated every hour](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806)
**Author:** [@shadu88](https://discuss.elastic.co/u/shadu88)\ **Replies:** 4\ **Last updated:** [April 22, 2021, 11:40am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806 "2021-04-22T11:40:45Z")
Hello Dear ELkan's Hope all are doing good!!. here are details of my query: Logstash version7.10 on Centos7.8 Using File input plugin to read files and forward logs to other components Issue Statement : How can i rea… --- ## ["reason"=\>"failed to parse field \[host.name\] of type \[text\] in document with id](https://discuss.elastic.co/t/reason-failed-to-parse-field-host-name-of-type-text-in-document-with-id/270923)
**Author:** [@BlackCat](https://discuss.elastic.co/u/BlackCat)\ **Replies:** 2\ **Last updated:** [April 22, 2021, 10:18am UTC](https://discuss.elastic.co/t/reason-failed-to-parse-field-host-name-of-type-text-in-document-with-id/270923 "2021-04-22T10:18:04Z")
Logstash outputs below error log. \[2021-04-22T15:00:43,643\]\[WARN \]\[logstash.outputs.elasticsearch\]\[＜pipeline id＞\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"＜index na… --- ## [Added multiple config files in a directory and edited pipelines.yml. yet only one of the config file is being processed](https://discuss.elastic.co/t/added-multiple-config-files-in-a-directory-and-edited-pipelines-yml-yet-only-one-of-the-config-file-is-being-processed/270607)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 2\ **Last updated:** [April 22, 2021, 8:09am UTC](https://discuss.elastic.co/t/added-multiple-config-files-in-a-directory-and-edited-pipelines-yml-yet-only-one-of-the-config-file-is-being-processed/270607 "2021-04-22T08:09:12Z")
I'm running logstash on a MAC , i got two config file, which run fine when executed on their own . However i moved them to a custom directory and added the path onto the pipelines.yml, while execution via "logstash -f /… --- ## [Logstash KUSTOS output plugin](https://discuss.elastic.co/t/logstash-kustos-output-plugin/270922)
**Author:** [@shadu88](https://discuss.elastic.co/u/shadu88)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 6:15am UTC](https://discuss.elastic.co/t/logstash-kustos-output-plugin/270922 "2021-04-22T06:15:20Z")
Dear ELKan's Have ever tried kustos output plugin to ingest the data from Logstash to Azure Data explorer. I'm currently following Azure documentation for the same. If you can help me with additional information/refere… --- ## [How can I index the data in an array?](https://discuss.elastic.co/t/how-can-i-index-the-data-in-an-array/270664)
**Author:** [@xo1534](https://discuss.elastic.co/u/xo1534)\ **Replies:** 2\ **Last updated:** [April 20, 2021, 7:27am UTC](https://discuss.elastic.co/t/how-can-i-index-the-data-in-an-array/270664 "2021-04-20T07:27:44Z")
biz\_tag data is A,B,C,D,E i want result data.... biz\_tag: \[ A, B, C, D, E \] but... Actual Results biz\_tag: A What's the problem??? Thank you... --- ## [How to use multiple filter files in logstash-filter-verifier command](https://discuss.elastic.co/t/how-to-use-multiple-filter-files-in-logstash-filter-verifier-command/270908)
**Author:** [@Usman18](https://discuss.elastic.co/u/Usman18)\ **Replies:** 0\ **Last updated:** [April 21, 2021, 10:33pm UTC](https://discuss.elastic.co/t/how-to-use-multiple-filter-files-in-logstash-filter-verifier-command/270908 "2021-04-21T22:33:09Z")
I am using logstash-filter-verifier to execute my test cases for testing various logstash pipelines. From the git documentation I can see the command that will be used to run those test cases $ path/to/logstash-filter-v… --- ## [Replicating Logstash Fingerprint in elasticsearch\_dsl\_py](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905)
**Author:** [@Mark\_Parrish](https://discuss.elastic.co/u/Mark_Parrish)\ **Replies:** 5\ **Last updated:** [April 21, 2021, 10:46pm UTC](https://discuss.elastic.co/t/replicating-logstash-fingerprint-in-elasticsearch-dsl-py/270905 "2021-04-21T22:46:25Z")
I am trying replicate this piece of code in the python. ruby { code =\> ' physical = \[ event.get("address1").to\_s, event.get("address2").to\_s, event.get("city").to\_s, event.get… --- ## [Seeking advice on parsing Nagios logs using pre-built pattern files from herokuapp.com](https://discuss.elastic.co/t/seeking-advice-on-parsing-nagios-logs-using-pre-built-pattern-files-from-herokuapp-com/270850)
**Author:** [@kgeil](https://discuss.elastic.co/u/kgeil)\ **Replies:** 2\ **Last updated:** [April 21, 2021, 2:48pm UTC](https://discuss.elastic.co/t/seeking-advice-on-parsing-nagios-logs-using-pre-built-pattern-files-from-herokuapp-com/270850 "2021-04-21T14:48:09Z")
Good morning everyone. I'm early in the Elastic Stack learning curve, and I'm working on parsing some nagios logs this morning. I Have nagios logging to syslog, and can use the pattern file located on the herokuapp gro… --- ## [Logstash json parsing](https://discuss.elastic.co/t/logstash-json-parsing/270851)
**Author:** [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\ **Replies:** 1\ **Last updated:** [April 21, 2021, 2:01pm UTC](https://discuss.elastic.co/t/logstash-json-parsing/270851 "2021-04-21T14:01:52Z")
Hello. I have incoming JSON log. It has nested JSON. I want to parse main JSON and nested. Example: { "value": "123", "nested": { "value-two": "555" } } . I have a filter like this json { source =\> "messag… --- ## [Uploading csv file: Failed to parse date field dd/MM/yyyy HH:mm](https://discuss.elastic.co/t/uploading-csv-file-failed-to-parse-date-field-dd-mm-yyyy-hh-mm/270813)
**Author:** [@MARCO\_RAMBALDI](https://discuss.elastic.co/u/MARCO_RAMBALDI)\ **Replies:** 23\ **Last updated:** [April 21, 2021, 1:46pm UTC](https://discuss.elastic.co/t/uploading-csv-file-failed-to-parse-date-field-dd-mm-yyyy-hh-mm/270813 "2021-04-21T13:46:16Z")
Hi, I'm uploading a csv file (; as separator). I identified one of the columns as Machine Time. The format on excel is "customized" as seen in the image below: To parse the date as a date type on Elastic I used the … --- ## [File paths must be absolute, relative path specified error while running logstash on docker](https://discuss.elastic.co/t/file-paths-must-be-absolute-relative-path-specified-error-while-running-logstash-on-docker/270704)
**Author:** [@pradeep\_gadkari](https://discuss.elastic.co/u/pradeep_gadkari)\ **Replies:** 2\ **Last updated:** [April 21, 2021, 12:40pm UTC](https://discuss.elastic.co/t/file-paths-must-be-absolute-relative-path-specified-error-while-running-logstash-on-docker/270704 "2021-04-21T12:40:59Z")
I am running ELK stack on docker using deviantony/docker-elk from Github. When I try reading log files from my local machine, logstash stops with error \[2021-04-20T12:36:43,101\]\[ERROR\]\[logstash.javapipeline \]\[main\] Pipe… --- ## ["reason"=\>"object mapping for \[host\] tried to parse field \[host\] as object, but found a concrete value"](https://discuss.elastic.co/t/reason-object-mapping-for-host-tried-to-parse-field-host-as-object-but-found-a-concrete-value/270790)
**Author:** [@BlackCat](https://discuss.elastic.co/u/BlackCat)\ **Replies:** 5\ **Last updated:** [April 21, 2021, 10:09am UTC](https://discuss.elastic.co/t/reason-object-mapping-for-host-tried-to-parse-field-host-as-object-but-found-a-concrete-value/270790 "2021-04-21T10:09:36Z")
Please how to do resolve. Logstash outputs below error log. \[2021-04-21T15:31:47,803\]\[WARN \]\[logstash.outputs.elasticsearch\]\[＜pipeline.id＞\]\[b0981f4f69ba05c595e4bfce70b5e813c0b082207400c67bd72b366d2d36ba65\] Could not in… --- ## [Logstash installed on Windows based container using zip download. Not running using bin/logstash -f logstash.conf](https://discuss.elastic.co/t/logstash-installed-on-windows-based-container-using-zip-download-not-running-using-bin-logstash-f-logstash-conf/270816)
**Author:** [@omkarg81](https://discuss.elastic.co/u/omkarg81)\ **Replies:** 0\ **Last updated:** [April 21, 2021, 9:46am UTC](https://discuss.elastic.co/t/logstash-installed-on-windows-based-container-using-zip-download-not-running-using-bin-logstash-f-logstash-conf/270816 "2021-04-21T09:46:55Z")
Looking at the Logstash docker images, I went ahead and created a custom image, including base .net framework and adding the logstash installation folder to it, by virtue of downloading the zip file, extracting it and cu… --- ## [Logstash fails with not an SSL/TLS record](https://discuss.elastic.co/t/logstash-fails-with-not-an-ssl-tls-record/270812)
**Author:** [@Enrique\_Pedroza](https://discuss.elastic.co/u/Enrique_Pedroza)\ **Replies:** 0\ **Last updated:** [April 21, 2021, 9:20am UTC](https://discuss.elastic.co/t/logstash-fails-with-not-an-ssl-tls-record/270812 "2021-04-21T09:20:49Z")
I'm running Logstash with Elasticsearch and Kibana, and all configuration setted up. The problem is that the services is giving the following error \[2021-04-21T10:08:56,344\]\[INFO \]\[org.logstash.beats.BeatsHandler\]\[main\]… --- ## [Logstash throws out 'Host name does not match the certificate subject provided by the peer' error](https://discuss.elastic.co/t/logstash-throws-out-host-name-does-not-match-the-certificate-subject-provided-by-the-peer-error/270622)
**Author:** [@arunhk3](https://discuss.elastic.co/u/arunhk3)\ **Replies:** 2\ **Last updated:** [April 21, 2021, 8:11am UTC](https://discuss.elastic.co/t/logstash-throws-out-host-name-does-not-match-the-certificate-subject-provided-by-the-peer-error/270622 "2021-04-21T08:11:25Z")
Hi All, I recently installed a three-node cluster in a vagrant box to test out the security features of Elasticsearch: I followed the guides in the below order: Set up basic security for the Elastic Stack | Elasticse… --- ## [The issue of tcp connection of logstash](https://discuss.elastic.co/t/the-issue-of-tcp-connection-of-logstash/270780)
**Author:** [@iammanmale](https://discuss.elastic.co/u/iammanmale)\ **Replies:** 0\ **Last updated:** [April 21, 2021, 3:27am UTC](https://discuss.elastic.co/t/the-issue-of-tcp-connection-of-logstash/270780 "2021-04-21T03:27:49Z")
It seems to me that the limit of the no of tcp connection of logstash is about 110. If the no of connection is more than 110, filebeat will show something like handshake.....ERROR...i/o timeout when running test output c… --- ## [Ignore a specific or drop a field when using another Grok Pattern](https://discuss.elastic.co/t/ignore-a-specific-or-drop-a-field-when-using-another-grok-pattern/270522)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 3\ **Last updated:** [April 20, 2021, 1:35pm UTC](https://discuss.elastic.co/t/ignore-a-specific-or-drop-a-field-when-using-another-grok-pattern/270522 "2021-04-20T13:35:14Z")
HI, I have a filter that works except for when the log doesn't contains different data for a field that I am using an extra grok pattern on. I am extracting the username from DN (Distinguished Name) from the main log a… --- ## [Logstash jdbc\_streaming filter pipeline reload and database access](https://discuss.elastic.co/t/logstash-jdbc-streaming-filter-pipeline-reload-and-database-access/270619)
**Author:** [@rverchere](https://discuss.elastic.co/u/rverchere)\ **Replies:** 2\ **Last updated:** [April 20, 2021, 11:56am UTC](https://discuss.elastic.co/t/logstash-jdbc-streaming-filter-pipeline-reload-and-database-access/270619 "2021-04-20T11:56:05Z")
Hello, I'm using logstash with the jdbc\_streaming filter, that connects to an external MariaDB database. When I stop and restart the database, the pipeline using the jdbc\_streaming filter fails, and there is no automat… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=235) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=237)