# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=187 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 188 --- ## [Logstash GeoIP filter plugin not working after update to 7.15.1](https://discuss.elastic.co/t/logstash-geoip-filter-plugin-not-working-after-update-to-7-15-1/286989)
**Author:** [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Replies:** 2\ **Last updated:** [October 20, 2021, 5:23am UTC](https://discuss.elastic.co/t/logstash-geoip-filter-plugin-not-working-after-update-to-7-15-1/286989 "2021-10-20T05:23:13Z")
Hi, I am running a logstash server with 7.14.0 and one of my pipelines uses the geoip filter plugin. After upgrade to 7.15.1, the pipeline refuses to start with an error like \[2021-10-18T17:22:08,987\]\[ERROR\]\[logstash.j… --- ## [How to aggregate log transactions based on field value in logstash](https://discuss.elastic.co/t/how-to-aggregate-log-transactions-based-on-field-value-in-logstash/287134)
**Author:** [@venkataraman](https://discuss.elastic.co/u/venkataraman)\ **Replies:** 1\ **Last updated:** [October 20, 2021, 1:40am UTC](https://discuss.elastic.co/t/how-to-aggregate-log-transactions-based-on-field-value-in-logstash/287134 "2021-10-20T01:40:20Z")
My log file looks like this: start time: 2021-10-11T13:54:34Z category: commercial status: started end time: start time: 2021-10-11T13:54:34Z category: commercial status: Running end time: start time: 2021-10-11T… --- ## [Aggregate nested arrays](https://discuss.elastic.co/t/aggregate-nested-arrays/285934)
**Author:** [@Lyes\_Ouchene](https://discuss.elastic.co/u/Lyes_Ouchene)\ **Replies:** 6\ **Last updated:** [October 19, 2021, 9:17pm UTC](https://discuss.elastic.co/t/aggregate-nested-arrays/285934 "2021-10-19T21:17:33Z")
Hello everyone i am new to Elasticsearch, i need help with a use case, I want to create an array inside an array using the aggregate option The desired result is to have a list of services, for each service there will… --- ## [Translate error not working](https://discuss.elastic.co/t/translate-error-not-working/287118)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 1\ **Last updated:** [October 19, 2021, 5:22pm UTC](https://discuss.elastic.co/t/translate-error-not-working/287118 "2021-10-19T17:22:43Z")
Hello .. i am trying to use translate filter in my logstash config file but im getting below error here's is my logstash config file input { tcp { port =\> 9563 codec =\> plain { charset=\>"UTF-8" } } } fil… --- ## [Issues with Logstash reading csv file](https://discuss.elastic.co/t/issues-with-logstash-reading-csv-file/287117)
**Author:** [@ThePurple5merf](https://discuss.elastic.co/u/ThePurple5merf)\ **Replies:** 2\ **Last updated:** [October 19, 2021, 3:34pm UTC](https://discuss.elastic.co/t/issues-with-logstash-reading-csv-file/287117 "2021-10-19T15:34:39Z")
Hello! I'm pretty new to the ELK Stack so I'm not sure how to troubleshoot this error. I've set up my Elasticsearch and kibana successfully, but I'm having issues with getting logstash to read in my .csv file to the ELK… --- ## [\_grokparsefailure on kibana for record through logstash](https://discuss.elastic.co/t/grokparsefailure-on-kibana-for-record-through-logstash/287073)
**Author:** [@kish2010](https://discuss.elastic.co/u/kish2010)\ **Replies:** 1\ **Last updated:** [October 19, 2021, 12:12pm UTC](https://discuss.elastic.co/t/grokparsefailure-on-kibana-for-record-through-logstash/287073 "2021-10-19T12:12:36Z")
Hi All, I have created grok pattern and tested with log on grokdebugger but still it is failing with grokparsefailure error on kibana. PFB Config file:- input { file { path =\> "/tmp/test.csv" } } filter { grok { … --- ## [Logstash can't read the environment variable](https://discuss.elastic.co/t/logstash-cant-read-the-environment-variable/286949)
**Author:** [@ramilbermejo](https://discuss.elastic.co/u/ramilbermejo)\ **Replies:** 2\ **Last updated:** [October 19, 2021, 4:58am UTC](https://discuss.elastic.co/t/logstash-cant-read-the-environment-variable/286949 "2021-10-19T04:58:42Z")
Good day everyone! Need some help here. On my dev environment, checking the logstash configuration will show this error. root@devlogstash:~# /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t OpenJDK 64… --- ## [Logstash OutOfMemoryError: Direct buffer memory](https://discuss.elastic.co/t/logstash-outofmemoryerror-direct-buffer-memory/286473)
**Author:** [@Muthu\_Ramachandran\_K](https://discuss.elastic.co/u/Muthu_Ramachandran_K)\ **Replies:** 2\ **Last updated:** [October 19, 2021, 3:41am UTC](https://discuss.elastic.co/t/logstash-outofmemoryerror-direct-buffer-memory/286473 "2021-10-19T03:41:15Z")
Hi, we are using filebeat-\> logstash -\> elastic setup. The set up was working fine without any issues, but recently we noticed that logstash is going down frequently and we could see the below error. can anyone suggest … --- ## [S3-input-plugin; restart logstash on aws\_credentials\_file update](https://discuss.elastic.co/t/s3-input-plugin-restart-logstash-on-aws-credentials-file-update/287036)
**Author:** [@jnkroeker](https://discuss.elastic.co/u/jnkroeker)\ **Replies:** 2\ **Last updated:** [October 19, 2021, 2:07am UTC](https://discuss.elastic.co/t/s3-input-plugin-restart-logstash-on-aws-credentials-file-update/287036 "2021-10-19T02:07:49Z")
Hey there, How can I force logstash to restart when my aws\_credentials\_file is updated? --config.reload.automatic does not appear to work here as neither the config.yaml or pipelines.yaml files are changing. the aws\_cr… --- ## [Logstash stops receiving logs](https://discuss.elastic.co/t/logstash-stops-receiving-logs/286897)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 7\ **Last updated:** [October 18, 2021, 7:38pm UTC](https://discuss.elastic.co/t/logstash-stops-receiving-logs/286897 "2021-10-18T19:38:42Z")
Hi everyone. I am a beginner in logstash and ELK in general. Lately I have been facing an issue with logstash. I am receiving some logs on port 3014, and this is the second time that this is happening and I identified t… --- ## [Logstash in Kubernetes Not Using ServiceAccount with IAM Role](https://discuss.elastic.co/t/logstash-in-kubernetes-not-using-serviceaccount-with-iam-role/286081)
**Author:** [@xzxpurple2017](https://discuss.elastic.co/u/xzxpurple2017)\ **Replies:** 3\ **Last updated:** [October 18, 2021, 7:02pm UTC](https://discuss.elastic.co/t/logstash-in-kubernetes-not-using-serviceaccount-with-iam-role/286081 "2021-10-18T19:02:59Z")
I have a Logstash input to pull logs from S3. I attached a serviceaccount with IAM role to interact with S3 to the Logstash stateful set. I checked that the env variables on the container have AWS\_ROLE\_ARN and AWS\_WEB\_… --- ## [Parsing logs from text file using using logstash](https://discuss.elastic.co/t/parsing-logs-from-text-file-using-using-logstash/286995)
**Author:** [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Replies:** 1\ **Last updated:** [October 18, 2021, 6:08pm UTC](https://discuss.elastic.co/t/parsing-logs-from-text-file-using-using-logstash/286995 "2021-10-18T18:08:29Z")
Hi, I am having logs in the txt file.I checked with kv, grok and dissect filter to ingest logs using logstash, but as logs format is not the same. Maybe I need to assign field and particular value as this is unstructure… --- ## [Add field if based on another timestamps-field value](https://discuss.elastic.co/t/add-field-if-based-on-another-timestamps-field-value/286985)
**Author:** [@Cibot](https://discuss.elastic.co/u/Cibot)\ **Replies:** 2\ **Last updated:** [October 18, 2021, 5:44pm UTC](https://discuss.elastic.co/t/add-field-if-based-on-another-timestamps-field-value/286985 "2021-10-18T17:44:08Z")
I've been trying to add a field describing the status of the current rpm. Basically I've been using execbeats to execute a command which returns all the currently installed rpms. Now we have an internal tool which basi… --- ## [Calculating EPS? events.filtered / duration\_in\_millis is incorrect](https://discuss.elastic.co/t/calculating-eps-events-filtered-duration-in-millis-is-incorrect/286936)
**Author:** [@CamTheMan](https://discuss.elastic.co/u/CamTheMan)\ **Replies:** 1\ **Last updated:** [October 18, 2021, 3:10pm UTC](https://discuss.elastic.co/t/calculating-eps-events-filtered-duration-in-millis-is-incorrect/286936 "2021-10-18T15:10:01Z")
Hello All, I would like to calculate my events per second for logstash. Currently I'm grabbing data via the logstash API but when I calculate the EPS it doesn't match up to what the 'metrics' logstash filter is reporti… --- ## [Logstash CIDR Plugin not working](https://discuss.elastic.co/t/logstash-cidr-plugin-not-working/286267)
**Author:** [@mrodriguez](https://discuss.elastic.co/u/mrodriguez)\ **Replies:** 1\ **Last updated:** [October 18, 2021, 2:25pm UTC](https://discuss.elastic.co/t/logstash-cidr-plugin-not-working/286267 "2021-10-18T14:25:28Z")
Hi, I need to filter events by IP addresses. I would like to use CIDR plugin but it is not working for me. This is my code: filter { if \[type\] == "mytype" { kv{} cidr{ address =\> \[ "%{srcip}" \] network =\>… --- ## [Logstash cpu usage blocked at 50%](https://discuss.elastic.co/t/logstash-cpu-usage-blocked-at-50/286465)
**Author:** [@thomastatin](https://discuss.elastic.co/u/thomastatin)\ **Replies:** 2\ **Last updated:** [October 18, 2021, 2:13pm UTC](https://discuss.elastic.co/t/logstash-cpu-usage-blocked-at-50/286465 "2021-10-18T14:13:15Z")
Hi everybody ! I have a logstash pipeline which can receive about 10k e/s in event peaks My logstash usual CPU usage is in "nice" priority with an average 30-40% CPU consumption and without data delay in the data store… --- ## [Log forwarding issue after upgrading logstash](https://discuss.elastic.co/t/log-forwarding-issue-after-upgrading-logstash/286859)
**Author:** [@yankeesfan01x](https://discuss.elastic.co/u/yankeesfan01x)\ **Replies:** 1\ **Last updated:** [October 18, 2021, 2:01pm UTC](https://discuss.elastic.co/t/log-forwarding-issue-after-upgrading-logstash/286859 "2021-10-18T14:01:23Z")
I upgraded the logstash version from 7.12.1 to 7.15.1 and I'm not seeing logs in Kibana from the source host I'm testing with. Is this too much of a jump in versions or did something change in 7.15.1 that I need to upda… --- ## [Event.duration bug](https://discuss.elastic.co/t/event-duration-bug/286733)
**Author:** [@maltewhiite](https://discuss.elastic.co/u/maltewhiite)\ **Replies:** 4\ **Last updated:** [October 18, 2021, 1:45pm UTC](https://discuss.elastic.co/t/event-duration-bug/286733 "2021-10-18T13:45:53Z")
I have a field as a double in milliseconds. I want to turn it into nanoseconds and apply it to the event.duration field in logstash, so I can monitor how long certain events take for my machines to process. Here is the … --- ## [Logs from different sources in single txt file, to be stored in different indices using logstash](https://discuss.elastic.co/t/logs-from-different-sources-in-single-txt-file-to-be-stored-in-different-indices-using-logstash/286694)
**Author:** [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Replies:** 3\ **Last updated:** [October 18, 2021, 1:19pm UTC](https://discuss.elastic.co/t/logs-from-different-sources-in-single-txt-file-to-be-stored-in-different-indices-using-logstash/286694 "2021-10-18T13:19:35Z")
Hi, I am having txt file consist of 3 different logs coming from 3 sources collated into single text file. I want to separate out logs and stored them in a respective index. Sample Logs- 10/12/2021 8:54:00 AM,MSGID: \<… --- ## [Memory usage grows in (short-ish) time](https://discuss.elastic.co/t/memory-usage-grows-in-short-ish-time/286817)
**Author:** [@depesz](https://discuss.elastic.co/u/depesz)\ **Replies:** 4\ **Last updated:** [October 18, 2021, 10:15am UTC](https://discuss.elastic.co/t/memory-usage-grows-in-short-ish-time/286817 "2021-10-18T10:15:52Z")
We just recently started trying to use logstash. The problem is that memory usage grows, at least within first 2 days, to the point that we can't use it. I tried reducing Xms and Xmx from default of 1G to 512M. restarte… --- ## [Logstash can't collected logs of the current day](https://discuss.elastic.co/t/logstash-cant-collected-logs-of-the-current-day/286968)
**Author:** [@lisiyu](https://discuss.elastic.co/u/lisiyu)\ **Replies:** 0\ **Last updated:** [October 18, 2021, 10:10am UTC](https://discuss.elastic.co/t/logstash-cant-collected-logs-of-the-current-day/286968 "2021-10-18T10:10:34Z")
I set FileBeat to pull the log and LogStash to process the log format to es.But today's log does not collect the index. The logs from the last two days are here. filebeat.yaml filebeat.inputs: - type: container … --- ## [Jdbc config not working](https://discuss.elastic.co/t/jdbc-config-not-working/286680)
**Author:** [@tractor\_boy](https://discuss.elastic.co/u/tractor_boy)\ **Replies:** 2\ **Last updated:** [October 18, 2021, 8:29am UTC](https://discuss.elastic.co/t/jdbc-config-not-working/286680 "2021-10-18T08:29:11Z")
Simple issue in that I have a jdbc config set up, but no output so no idea whats going on. I have changed logstash config to go to debug and can see the config is being consumed without errors. The input is select "tes… --- ## [Invalid url](https://discuss.elastic.co/t/invalid-url/286814)
**Author:** [@Jakub\_Kaczmarek](https://discuss.elastic.co/u/Jakub_Kaczmarek)\ **Replies:** 2\ **Last updated:** [October 18, 2021, 12:46am UTC](https://discuss.elastic.co/t/invalid-url/286814 "2021-10-18T00:46:56Z")
Hi I'm getting invalid url error in Logstash. My config: http\_poller { urls =\> { executedsteps =\> { method =\> get cacert =\> "/elk-stack/cert.pem" url =\> "https://odata-trial.… --- ## [Logstash-keystore Java error](https://discuss.elastic.co/t/logstash-keystore-java-error/286925)
**Author:** [@Rysiu](https://discuss.elastic.co/u/Rysiu)\ **Replies:** 0\ **Last updated:** [October 17, 2021, 10:02am UTC](https://discuss.elastic.co/t/logstash-keystore-java-error/286925 "2021-10-17T10:02:28Z")
Hi, I have a problem when running logstash-keystore Windows command line: bin\\logstash-keystore create Does not work and returns the information: "Using boundled JDK:" Cannot find Java 1.5 or higher However... b… --- ## [JSON Parse error: Illegal unquoted character](https://discuss.elastic.co/t/json-parse-error-illegal-unquoted-character/286894)
**Author:** [@geetika\_gopi](https://discuss.elastic.co/u/geetika_gopi)\ **Replies:** 1\ **Last updated:** [October 16, 2021, 4:08pm UTC](https://discuss.elastic.co/t/json-parse-error-illegal-unquoted-character/286894 "2021-10-16T16:08:31Z")
Hello, I am using rsyslog to send my auditd logs to logstash. Before reaching logstash, an rsyslog template is applied for better formatting. I run into an error here jsonlines - JSON parse error, original data now in… --- ## [O365.audit to message field](https://discuss.elastic.co/t/o365-audit-to-message-field/286396)
**Author:** [@geetika\_gopi](https://discuss.elastic.co/u/geetika_gopi)\ **Replies:** 2\ **Last updated:** [October 16, 2021, 2:43pm UTC](https://discuss.elastic.co/t/o365-audit-to-message-field/286396 "2021-10-16T14:43:34Z")
Hello, I have O365 logs coming in to my logstash via the O365 filebeat module. I am trying to create two outputs : Elasticsearch output S3 bucket output The S3 bucket output requires a "message" field to be present. … --- ## [PARSE Logs having pip saperator](https://discuss.elastic.co/t/parse-logs-having-pip-saperator/286881)
**Author:** [@Nikparab](https://discuss.elastic.co/u/Nikparab)\ **Replies:** 7\ **Last updated:** [October 16, 2021, 3:33am UTC](https://discuss.elastic.co/t/parse-logs-having-pip-saperator/286881 "2021-10-16T03:33:50Z")
Hi I am having the log format as below. 2021-10-16 00:14:14|http-nio-2222-exec-1|DEBUG|c.v.a.ultimatevault.web.TraceFilter|01AD08F98FB24AB7A639383F44613AE4|Processing request:- id: 01AD08F98FB24AB7A639383F44613AE4 metho… --- ## [Logstash open file descriptors only increases](https://discuss.elastic.co/t/logstash-open-file-descriptors-only-increases/285571)
**Author:** [@N1k-3l](https://discuss.elastic.co/u/N1k-3l)\ **Replies:** 2\ **Last updated:** [October 15, 2021, 5:58pm UTC](https://discuss.elastic.co/t/logstash-open-file-descriptors-only-increases/285571 "2021-10-15T17:58:09Z")
Hello. I have a problem with logstash (logstash 7.12.1) file descriptors only increases. I set limits to 50000, but on another host, with the same config i have: may be somebody knows about this? :blush: --- ## [Certificate error on database update](https://discuss.elastic.co/t/certificate-error-on-database-update/286550)
**Author:** [@rugenl](https://discuss.elastic.co/u/rugenl)\ **Replies:** 5\ **Last updated:** [October 15, 2021, 5:51pm UTC](https://discuss.elastic.co/t/certificate-error-on-database-update/286550 "2021-10-15T17:51:04Z")
After upgrading to logstash 7.14.1, I'm getting this error daily in the logstash-plain.log: \[ERROR\]\[logstash.filters.geoip.databasemanager\] certificate verify failed {:cause=\>#\ **Author:** [@Catalina\_Boteanu](https://discuss.elastic.co/u/Catalina_Boteanu)\ **Replies:** 1\ **Last updated:** [October 15, 2021, 5:34pm UTC](https://discuss.elastic.co/t/parse-xml-log-with-xml-filter-and-grok-fiter/286712 "2021-10-15T17:34:46Z") Hello. I am currently trying to apply some filters to different logs in Logstash. I have a log that has a string in the beginning and then it is in xml format. I was able to take out the xml from the log with a regex. No… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=186) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=188)