# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=162 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 163 --- ## [Getting an error while upgrading logstash from 5.6 to 6.8](https://discuss.elastic.co/t/getting-an-error-while-upgrading-logstash-from-5-6-to-6-8/296722)
**Author:** [@rutika\_kamble](https://discuss.elastic.co/u/rutika_kamble)\ **Replies:** 0\ **Last updated:** [February 9, 2022, 12:36pm UTC](https://discuss.elastic.co/t/getting-an-error-while-upgrading-logstash-from-5-6-to-6-8/296722 "2022-02-09T12:36:33Z")
I'm getting this error while upgrading logstash 5.6 to 6.8. It would be very kind of you if you could suggest here something? Please find the snippet of the error- \[2022-02-09T17:39:49,294\]\[FATAL\]\[logstash.runner … --- ## [Add field when line contains EXCEPTION](https://discuss.elastic.co/t/add-field-when-line-contains-exception/296656)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 5\ **Last updated:** [February 9, 2022, 9:12am UTC](https://discuss.elastic.co/t/add-field-when-line-contains-exception/296656 "2022-02-09T09:12:34Z")
Hi, logstash read the file line by line and add each line to Elasticsearch ok but i just want to add it with a condition when he find "EXCEPTION". example of my log file: 01/01/2022 15:00:02 546- MAIN EXCEPTION : Sys… --- ## [Logstash exited with code 0](https://discuss.elastic.co/t/logstash-exited-with-code-0/296694)
**Author:** [@jeremd](https://discuss.elastic.co/u/jeremd)\ **Replies:** 0\ **Last updated:** [February 9, 2022, 8:54am UTC](https://discuss.elastic.co/t/logstash-exited-with-code-0/296694 "2022-02-09T08:54:44Z")
Hi, for some reasons my logstash shut down when I run docker-compose up. Anyone has an idea please ? version: "3.7" services: elasticsearch: image: elasticsearch:7.9.2 ports: - "9200:9200" environme… --- ## [Logstash 7.16.3 failed to start: "cannot load Java class org.logstash.util.JavaVersion"](https://discuss.elastic.co/t/logstash-7-16-3-failed-to-start-cannot-load-java-class-org-logstash-util-javaversion/296590)
**Author:** [@pikopl](https://discuss.elastic.co/u/pikopl)\ **Replies:** 0\ **Last updated:** [February 8, 2022, 12:10pm UTC](https://discuss.elastic.co/t/logstash-7-16-3-failed-to-start-cannot-load-java-class-org-logstash-util-javaversion/296590 "2022-02-08T12:10:56Z")
I installed Logstash OSS ver. 7.16.3 rpm: $ sudo rpm -ivh logstash-oss-7.16.3-x86\_64.rpm warning: logstash-oss-7.16.3-x86\_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing... … --- ## [Is there a queue between filter and output in logstash?](https://discuss.elastic.co/t/is-there-a-queue-between-filter-and-output-in-logstash/296682)
**Author:** [@JohnPhilip](https://discuss.elastic.co/u/JohnPhilip)\ **Replies:** 0\ **Last updated:** [February 9, 2022, 6:58am UTC](https://discuss.elastic.co/t/is-there-a-queue-between-filter-and-output-in-logstash/296682 "2022-02-09T06:58:09Z")
As the Execution Model mentioned, By default, Logstash uses in-memory bounded queues between pipeline stages (input → filter and filter → output) to buffer events. but there is another saying in Memory queue, By defau… --- ## [Split message logstash elasticsearch](https://discuss.elastic.co/t/split-message-logstash-elasticsearch/296291)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 2\ **Last updated:** [February 8, 2022, 7:28pm UTC](https://discuss.elastic.co/t/split-message-logstash-elasticsearch/296291 "2022-02-08T19:28:33Z")
Hi, my message it looks like : i want to add field when found the "|" it means to split it with "|" i did it but it looks with a wrong method : imagine that we have a lot of data in one message and in the same f… --- ## [Logstash holding deleted output file](https://discuss.elastic.co/t/logstash-holding-deleted-output-file/296113)
**Author:** [@Xiaochi\_Weng](https://discuss.elastic.co/u/Xiaochi_Weng)\ **Replies:** 7\ **Last updated:** [February 8, 2022, 7:06pm UTC](https://discuss.elastic.co/t/logstash-holding-deleted-output-file/296113 "2022-02-08T19:06:26Z")
I am currently using logstash with version 7.17 in the docker. Here is my output code: output{ file{ path=\>"/tmp/data/output.log" } } Meanwhile, I also have logrotate to keep it rotating: /tmp/data/output.log { si… --- ## [Logstash Ruby error when trying to rename a field the has a hyphen "-" in the value](https://discuss.elastic.co/t/logstash-ruby-error-when-trying-to-rename-a-field-the-has-a-hyphen-in-the-value/296618)
**Author:** [@Bryan\_Hamilton](https://discuss.elastic.co/u/Bryan_Hamilton)\ **Replies:** 4\ **Last updated:** [February 8, 2022, 6:51pm UTC](https://discuss.elastic.co/t/logstash-ruby-error-when-trying-to-rename-a-field-the-has-a-hyphen-in-the-value/296618 "2022-02-08T18:51:14Z")
I have events with nested fields and am trying and am trying the reduce the depth of some nested fields to match ecs and discarding the fields I don't need. Renaning of all other fields word except for one which causes a… --- ## [Removing non-numeric characters from a field not working in logstash](https://discuss.elastic.co/t/removing-non-numeric-characters-from-a-field-not-working-in-logstash/296648)
**Author:** [@curiousmind](https://discuss.elastic.co/u/curiousmind)\ **Replies:** 1\ **Last updated:** [February 8, 2022, 6:49pm UTC](https://discuss.elastic.co/t/removing-non-numeric-characters-from-a-field-not-working-in-logstash/296648 "2022-02-08T18:49:42Z")
need an urgent help.I am getting this error \[2022-02-08T18:25:01,995\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>"xjmdwt2/wbqlYsoFmJO31K… --- ## [Pipeline with id \[y\] does not exist](https://discuss.elastic.co/t/pipeline-with-id-y-does-not-exist/296625)
**Author:** [@Jarbird](https://discuss.elastic.co/u/Jarbird)\ **Replies:** 4\ **Last updated:** [February 8, 2022, 5:53pm UTC](https://discuss.elastic.co/t/pipeline-with-id-y-does-not-exist/296625 "2022-02-08T17:53:29Z")
Hello, I've found multiple topics about this very issue however I don't really see a solution. I'm also having a little difficulty understanding it all as I'm fairly new at this. I have each of the ELK components runni… --- ## [Extracting some JSON fields from the message](https://discuss.elastic.co/t/extracting-some-json-fields-from-the-message/296479)
**Author:** [@hs\_shaikh](https://discuss.elastic.co/u/hs_shaikh)\ **Replies:** 6\ **Last updated:** [February 8, 2022, 5:37pm UTC](https://discuss.elastic.co/t/extracting-some-json-fields-from-the-message/296479 "2022-02-08T17:37:21Z")
Hello there, I want to extract "applicationOwner" and "proxyResponseCode" JSON fields from my message My message filed look like this: message": "TID: \[-1234\] \[2022-02-07 18:59:15,667\] INFO {org.wso2.am.analytics.pub… --- ## [How to filter single line json?](https://discuss.elastic.co/t/how-to-filter-single-line-json/296242)
**Author:** [@jeromeat](https://discuss.elastic.co/u/jeromeat)\ **Replies:** 4\ **Last updated:** [February 8, 2022, 5:21pm UTC](https://discuss.elastic.co/t/how-to-filter-single-line-json/296242 "2022-02-08T17:21:28Z")
I am trying to figure out how to filter this single line of json data I have been able to pull via http\_poller. I've obfuscated the data, but this is the format it's coming in as. I've never utilized a filter before, so… --- ## [ConfigurationError: Could not connect to a compatible version of Elasticsearch](https://discuss.elastic.co/t/configurationerror-could-not-connect-to-a-compatible-version-of-elasticsearch/296624)
**Author:** [@marwan](https://discuss.elastic.co/u/marwan)\ **Replies:** 0\ **Last updated:** [February 8, 2022, 3:34pm UTC](https://discuss.elastic.co/t/configurationerror-could-not-connect-to-a-compatible-version-of-elasticsearch/296624 "2022-02-08T15:34:16Z")
Hi guys, I have a subscription to elastic cloud I'm trying to send logs from our Kubernetes on aws to our elastic cloud account. The main image i use is : docker.elastic.co/logstash/logstash-oss:7.17.0 my output o… --- ## [HA/redundancy on Logstash when pulling data](https://discuss.elastic.co/t/ha-redundancy-on-logstash-when-pulling-data/295754)
**Author:** [@jdswifty](https://discuss.elastic.co/u/jdswifty)\ **Replies:** 2\ **Last updated:** [February 8, 2022, 2:17pm UTC](https://discuss.elastic.co/t/ha-redundancy-on-logstash-when-pulling-data/295754 "2022-02-08T14:17:17Z")
Looking to see how others might have approached this topic in the past. For examples sake I have 2 logstash nodes running in a hot/cold standby mode that run the exact same (centrally managed) pipelines that pull data f… --- ## [Role of kafka between filebeat and logstash](https://discuss.elastic.co/t/role-of-kafka-between-filebeat-and-logstash/296566)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 0\ **Last updated:** [February 8, 2022, 9:20am UTC](https://discuss.elastic.co/t/role-of-kafka-between-filebeat-and-logstash/296566 "2022-02-08T09:20:12Z")
Hi, I have a lot of log files in each hour it will be created a new file log so i get it and i filter it with this architecture : filebeat -\> logstash -\> Elasticsearch -\> kibana I would to know there is any avantage t… --- ## [Logstash Scheduler](https://discuss.elastic.co/t/logstash-scheduler/296431)
**Author:** [@shharukh](https://discuss.elastic.co/u/shharukh)\ **Replies:** 4\ **Last updated:** [February 8, 2022, 6:12am UTC](https://discuss.elastic.co/t/logstash-scheduler/296431 "2022-02-08T06:12:28Z")
Hey Elasticworld, I'm current Logstash version 7.9 ,I have a encounter a problem in Logstash, in which have a 4 pipeline which are using "JBDC" driver for inserting data into Elasticsearch. \*\*Now I have to add another … --- ## [Logstash filter if conditional fails to match string ended with "\\u0000"](https://discuss.elastic.co/t/logstash-filter-if-conditional-fails-to-match-string-ended-with-u0000/296248)
**Author:** [@juowong](https://discuss.elastic.co/u/juowong)\ **Replies:** 3\ **Last updated:** [February 8, 2022, 5:49am UTC](https://discuss.elastic.co/t/logstash-filter-if-conditional-fails-to-match-string-ended-with-u0000/296248 "2022-02-08T05:49:39Z")
I am using logstash to receive network logs using syslog input module. I would like to drop the event if the log message is equal to A10. The pipeline is as follows: input { syslog { ecs\_compatibility =\> "v1" time… --- ## [Problem : Logstash filtering mutate/grok](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456)
**Author:** [@personal.riz](https://discuss.elastic.co/u/personal.riz)\ **Replies:** 5\ **Last updated:** [February 8, 2022, 5:00am UTC](https://discuss.elastic.co/t/problem-logstash-filtering-mutate-grok/296456 "2022-02-08T05:00:32Z")
Hi guys, iam new in elasticstack environment and i have some question below i have a problem with logstash filter parsing, i just config my firewall labs to sending json log to logstash using tcp port 5000. how to use … --- ## [CyberArk EPM (SaaS based) application integration with ELK](https://discuss.elastic.co/t/cyberark-epm-saas-based-application-integration-with-elk/296548)
**Author:** [@kirankatkar](https://discuss.elastic.co/u/kirankatkar)\ **Replies:** 0\ **Last updated:** [February 8, 2022, 4:53am UTC](https://discuss.elastic.co/t/cyberark-epm-saas-based-application-integration-with-elk/296548 "2022-02-08T04:53:30Z")
Hello Team, I am new to ELK, we are trying to integrate CyberArk EPM (SaaS based) application with ELK and vendor only provided APIs to pull the events / logs manually but hasn’t provided any script to automate the proc… --- ## [Unable to Ship logs through Grafana Loki plugin](https://discuss.elastic.co/t/unable-to-ship-logs-through-grafana-loki-plugin/296535)
**Author:** [@JPablo1982](https://discuss.elastic.co/u/JPablo1982)\ **Replies:** 2\ **Last updated:** [February 8, 2022, 1:26am UTC](https://discuss.elastic.co/t/unable-to-ship-logs-through-grafana-loki-plugin/296535 "2022-02-08T01:26:25Z")
Hello I am trying to get to work the Grafana loki Plugin for Logstash but I am getting a runtime error, here is the detail. Grafana Loki plugin was installed successfully by executing the following command: sudo /usr/… --- ## [Logstash multiple pipelines failure with persistent queues](https://discuss.elastic.co/t/logstash-multiple-pipelines-failure-with-persistent-queues/296520)
**Author:** [@proton](https://discuss.elastic.co/u/proton)\ **Replies:** 4\ **Last updated:** [February 7, 2022, 10:01pm UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-failure-with-persistent-queues/296520 "2022-02-07T22:01:15Z")
Hello! I need some help understanding pipeline to pipeline communication in logstash. I'm reading the output isolator pattern documentation and it says below: If any of the persistent queues of the downstream pipelines… --- ## [Noob Question: What Is Placing Specific Data Into Fields Called](https://discuss.elastic.co/t/noob-question-what-is-placing-specific-data-into-fields-called/296351)
**Author:** [@Jarbird](https://discuss.elastic.co/u/Jarbird)\ **Replies:** 5\ **Last updated:** [February 7, 2022, 5:03pm UTC](https://discuss.elastic.co/t/noob-question-what-is-placing-specific-data-into-fields-called/296351 "2022-02-07T17:03:54Z")
Hello, What is placing specific data, such as an IP address, into a field, perhaps called src.ip called? I'm asking so that I can begin to learn by searching. Thank you! --- ## [Logstash is Stopping automatically](https://discuss.elastic.co/t/logstash-is-stopping-automatically/296455)
**Author:** [@David\_Samm](https://discuss.elastic.co/u/David_Samm)\ **Replies:** 1\ **Last updated:** [February 7, 2022, 2:04pm UTC](https://discuss.elastic.co/t/logstash-is-stopping-automatically/296455 "2022-02-07T14:04:46Z")
Hi logstash is stopping automatically after some time I have increased xms and xmx to 4 GB in config in JVM.Options file even though Facing same issue --- ## [Logstash Persistant Queue](https://discuss.elastic.co/t/logstash-persistant-queue/296410)
**Author:** [@amitmahajan29](https://discuss.elastic.co/u/amitmahajan29)\ **Replies:** 0\ **Last updated:** [February 7, 2022, 4:18am UTC](https://discuss.elastic.co/t/logstash-persistant-queue/296410 "2022-02-07T04:18:17Z")
I'm trying to create a persistant Queue which will sit between logstash and Elasticsearch. My aim being, in case the cluster goes red due to any scenario, the logs won't be lost. But since the persistant queue typicall… --- ## [Logstash Listening on port 8080 got struck](https://discuss.elastic.co/t/logstash-listening-on-port-8080-got-struck/296130)
**Author:** [@naveen\_g](https://discuss.elastic.co/u/naveen_g)\ **Replies:** 5\ **Last updated:** [February 5, 2022, 4:32pm UTC](https://discuss.elastic.co/t/logstash-listening-on-port-8080-got-struck/296130 "2022-02-05T16:32:59Z")
Hi Everyone, I'm new to ELK and go-through handful of articles & videos , I setup ELK for our projects. When I do POC on setup ELK & Filebeat in same machine, it was working fine but when we planned to separate ELK to se… --- ## [How to dynamically recognize points via HTTPS ingestion?](https://discuss.elastic.co/t/how-to-dynamically-recognize-points-via-https-ingestion/296346)
**Author:** [@HC\_LW](https://discuss.elastic.co/u/HC_LW)\ **Replies:** 1\ **Last updated:** [February 4, 2022, 10:03pm UTC](https://discuss.elastic.co/t/how-to-dynamically-recognize-points-via-https-ingestion/296346 "2022-02-04T22:03:46Z")
We are sending https requests to ingest data. How can the request be formatted so that Elastic dynamically maps the 'geo\_point' field as type:point and not text or number? Thank you! curl -X POST "https://in-https.URL" … --- ## [Issue with a conditional statement](https://discuss.elastic.co/t/issue-with-a-conditional-statement/296327)
**Author:** [@JeremyP](https://discuss.elastic.co/u/JeremyP)\ **Replies:** 11\ **Last updated:** [February 4, 2022, 6:19pm UTC](https://discuss.elastic.co/t/issue-with-a-conditional-statement/296327 "2022-02-04T18:19:35Z")
Hello, I'm debugging my configuration and I'm having an issue comparing two fields, both are date fields and one eventually will become a multi-valued field. All my data is static and once a record is updated, a time st… --- ## [Input tcp codec fluent: invalid byte sequence in UTF-8 in regex](https://discuss.elastic.co/t/input-tcp-codec-fluent-invalid-byte-sequence-in-utf-8-in-regex/296290)
**Author:** [@Samuel\_Lima1](https://discuss.elastic.co/u/Samuel_Lima1)\ **Replies:** 0\ **Last updated:** [February 4, 2022, 12:45pm UTC](https://discuss.elastic.co/t/input-tcp-codec-fluent-invalid-byte-sequence-in-utf-8-in-regex/296290 "2022-02-04T12:45:40Z")
Hi folks, I'm using input tcp codec fluent and getting error " invalid byte sequence in UTF-8" when doing regex. It seems there is an invalid character in the message. Any idea how to solve it? Is there a way to simpl… --- ## [Split document in mongodb with logstash](https://discuss.elastic.co/t/split-document-in-mongodb-with-logstash/296271)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 0\ **Last updated:** [February 4, 2022, 9:43am UTC](https://discuss.elastic.co/t/split-document-in-mongodb-with-logstash/296271 "2022-02-04T09:43:45Z")
Hi, I would to know how to split document in mongodb with logstash because i send data from logstash to mongodb automtically so i want to split some document automatically so i think it's possible in bloc filter in logs… --- ## [Logstash elasticsearch input error](https://discuss.elastic.co/t/logstash-elasticsearch-input-error/296172)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 14\ **Last updated:** [February 4, 2022, 9:06am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-error/296172 "2022-02-04T09:06:31Z")
Hello I am extracting data from an http server through the apace pipeline, indicating the pipeline to use in the logstash output and it indexes correctly. On the other hand, I want to enrich the data already obtained an… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=161) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=163)