# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=84 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 85 --- ## [Logstash Kafka input - converting date to string format](https://discuss.elastic.co/t/logstash-kafka-input-converting-date-to-string-format/327967)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 6\ **Last updated:** [March 24, 2023, 4:20pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-converting-date-to-string-format/327967 "2023-03-24T16:20:27Z")
Hello, We are using Kafka plugin to get feed into Logstash. One of the fields that come with the message is in date format. I need to convert that date into string format. Please guide. My config file looks as follo… --- ## [Logstash nested json parsing,getting every nested json as seperate field](https://discuss.elastic.co/t/logstash-nested-json-parsing-getting-every-nested-json-as-seperate-field/327956)
**Author:** [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Replies:** 12\ **Last updated:** [March 24, 2023, 11:51am UTC](https://discuss.elastic.co/t/logstash-nested-json-parsing-getting-every-nested-json-as-seperate-field/327956 "2023-03-24T11:51:44Z")
Hello, After trying several times I'm unable to parse below json string data coming from oracle column called: package\_data.Kindly assist how to get data in elastic to show data like below from given data. {"status":"R… --- ## [2 Mongo DB collection how to merge in Logstash](https://discuss.elastic.co/t/2-mongo-db-collection-how-to-merge-in-logstash/328423)
**Author:** [@rachit](https://discuss.elastic.co/u/rachit)\ **Replies:** 0\ **Last updated:** [March 24, 2023, 6:29am UTC](https://discuss.elastic.co/t/2-mongo-db-collection-how-to-merge-in-logstash/328423 "2023-03-24T06:29:03Z")
Hi Team, Can anyone help me to merge 2 different collection of mongodb in single index in Elasticsearch with sync enable feature. Thanks --- ## [TypeError: no implicit conversion of nil into String](https://discuss.elastic.co/t/typeerror-no-implicit-conversion-of-nil-into-string/328416)
**Author:** [@kala\_y](https://discuss.elastic.co/u/kala_y)\ **Replies:** 0\ **Last updated:** [March 24, 2023, 3:38am UTC](https://discuss.elastic.co/t/typeerror-no-implicit-conversion-of-nil-into-string/328416 "2023-03-24T03:38:12Z")
2023-03-23T22:23:02.967-05:00 Copy \[2023-03-24T03:23:02,967\]\[WARN \]\[logstash.inputs.s3snssqs \]\[main\]\[97e0bbb90d3a28c08cdd88a484e0aa3737932f33f22b59839ba78170f15c7929\] Error in poller loop {:error=\>#\ **Author:** [@Whazaza](https://discuss.elastic.co/u/Whazaza)\ **Replies:** 0\ **Last updated:** [March 24, 2023, 12:55am UTC](https://discuss.elastic.co/t/json-logstash/328403 "2023-03-24T00:55:06Z") I have a problem when taking the data from my json suppose i need to take the data from this json { "header" : { "sendingApplicationNs" : "value", "sendingApplicationId" : "value", "sendingApplicationIdTy… --- ## [Error connecting to node kafka-broker:9092 (id: 1 rack: null) java.net.UnknownHostException: kafka-broker](https://discuss.elastic.co/t/error-connecting-to-node-kafka-broker-9092-id-1-rack-null-java-net-unknownhostexception-kafka-broker/327434)
**Author:** [@Rajesh\_R](https://discuss.elastic.co/u/Rajesh_R)\ **Replies:** 3\ **Last updated:** [March 23, 2023, 7:31pm UTC](https://discuss.elastic.co/t/error-connecting-to-node-kafka-broker-9092-id-1-rack-null-java-net-unknownhostexception-kafka-broker/327434 "2023-03-23T19:31:43Z")
When I use kafka plugin in logstash, I am getting the below error. \[2023-03-10T15:11:46,310\]\[WARN \]\[org.apache.kafka.clients.NetworkClient\]\[main\]\[289f84d5c44d64af9505535a5352178af25a608c83252a1c520ab39a4faa4855\] \[Consum… --- ## [Permission denied /usr/share/logstash/run](https://discuss.elastic.co/t/permission-denied-usr-share-logstash-run/327769)
**Author:** [@RJC](https://discuss.elastic.co/u/RJC)\ **Replies:** 3\ **Last updated:** [March 23, 2023, 7:29pm UTC](https://discuss.elastic.co/t/permission-denied-usr-share-logstash-run/327769 "2023-03-23T19:29:51Z")
I am getting the following error message when starting Logstash on a Linux server: \[ERROR\]\[logstash.java.pipeline \]\[main\] Pipeline worker error, the pipeline will be stopped (:pipeline\_id=\>"main", :error=\>" (EACCESS) Pe… --- ## [Logstash S3 input not deleting files](https://discuss.elastic.co/t/logstash-s3-input-not-deleting-files/328386)
**Author:** [@true64gurus](https://discuss.elastic.co/u/true64gurus)\ **Replies:** 0\ **Last updated:** [March 23, 2023, 6:17pm UTC](https://discuss.elastic.co/t/logstash-s3-input-not-deleting-files/328386 "2023-03-23T18:17:41Z")
I am using logstash S3 input plugin. The plugin is not deleting logs post ingestion. I am running logstsash docker on VM and have "delete =\> true". I have job running to delete logs older than 7 days and this is how I … --- ## [Error in parsing some logs from firewall due to object being returned](https://discuss.elastic.co/t/error-in-parsing-some-logs-from-firewall-due-to-object-being-returned/328349)
**Author:** [@viera120](https://discuss.elastic.co/u/viera120)\ **Replies:** 1\ **Last updated:** [March 23, 2023, 3:57pm UTC](https://discuss.elastic.co/t/error-in-parsing-some-logs-from-firewall-due-to-object-being-returned/328349 "2023-03-23T15:57:57Z")
Hi all, The setup is: Firewall --\> Filebeat --\> Logstash --\> Elasticsearch The following error keeps appearing in /var/log/logstash/logstash-plain.log \[2023-03-23T18:03:53,651\]\[WARN \]\[logstash.outputs.elasticsearch\]\[… --- ## [Wrong calculations - ruby code](https://discuss.elastic.co/t/wrong-calculations-ruby-code/328093)
**Author:** [@wedkarz014](https://discuss.elastic.co/u/wedkarz014)\ **Replies:** 1\ **Last updated:** [March 23, 2023, 3:13pm UTC](https://discuss.elastic.co/t/wrong-calculations-ruby-code/328093 "2023-03-23T15:13:40Z")
Hi all, logstash v.7.17.8 I have ~45 metrics to calculate, here is the example, code and results: ruby { code =\> " if !event.get('\[Package2VersionCreatesWithoutValidation\]\[Max\]').nil? and !e… --- ## [\[Logstash\] SSL TCP input certificate issue](https://discuss.elastic.co/t/logstash-ssl-tcp-input-certificate-issue/328220)
**Author:** [@perezdev](https://discuss.elastic.co/u/perezdev)\ **Replies:** 1\ **Last updated:** [March 23, 2023, 3:00pm UTC](https://discuss.elastic.co/t/logstash-ssl-tcp-input-certificate-issue/328220 "2023-03-23T15:00:44Z")
Hello, I'm trying to setup an SSL TCP input config file in Logstash to receive logs from other syslog server over TLS 1.2. For the certificates I have created the following files using openssl: openssl req -x509 -nodes… --- ## [Logstash MultiPipeline](https://discuss.elastic.co/t/logstash-multipipeline/328341)
**Author:** [@Julien069](https://discuss.elastic.co/u/Julien069)\ **Replies:** 2\ **Last updated:** [March 23, 2023, 1:39pm UTC](https://discuss.elastic.co/t/logstash-multipipeline/328341 "2023-03-23T13:39:59Z")
Hello , I want to use several pipeline . I had put them in the logstash directory conf.d . I named the files like this 01\_Input 02\_Filter 03\_Output Must I do anything else for work with the pipelines ? Does it wo… --- ## [Logstash error -"Could not load '.aprc' from ENV\['HOME'\]: couldn't find HOME environment -- expanding \`~"](https://discuss.elastic.co/t/logstash-error-could-not-load-aprc-from-env-home-couldnt-find-home-environment-expanding/328318)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 0\ **Last updated:** [March 23, 2023, 7:36am UTC](https://discuss.elastic.co/t/logstash-error-could-not-load-aprc-from-env-home-couldnt-find-home-environment-expanding/328318 "2023-03-23T07:36:38Z")
Hi Folks, I have a log file that logstash(8.6.2) is successfully parse, but has this error . i'm not sure whats causing it ? Could have a look ? , the data doesnt appear in kibana either Could not load '.aprc' from EN… --- ## [Split nested docs into a separate docs](https://discuss.elastic.co/t/split-nested-docs-into-a-separate-docs/328265)
**Author:** [@silverjoe](https://discuss.elastic.co/u/silverjoe)\ **Replies:** 0\ **Last updated:** [March 22, 2023, 3:02pm UTC](https://discuss.elastic.co/t/split-nested-docs-into-a-separate-docs/328265 "2023-03-22T15:02:21Z")
Hi, I have a simple Logstash pipeline that reads all docs from an ES index using an ES input plugin, then I have a filter that splits one doc into several, and finally the output plugin to index docs in the ES. My probl… --- ## [K8s multiple replicas pq](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887)
**Author:** [@liel\_bondy](https://discuss.elastic.co/u/liel_bondy)\ **Replies:** 3\ **Last updated:** [March 19, 2023, 1:10pm UTC](https://discuss.elastic.co/t/k8s-multiple-replicas-pq/327887 "2023-03-19T13:10:15Z")
Hey, quick question. If I want to scale my logstash (with PQ) horizontally in k8s, I would just increase the replica amount. Now that I have multiple nodes, I would like to understand how the PQ manages race conditions.… --- ## [Logstash TCP input pipeline performance issues](https://discuss.elastic.co/t/logstash-tcp-input-pipeline-performance-issues/328006)
**Author:** [@mread830](https://discuss.elastic.co/u/mread830)\ **Replies:** 9\ **Last updated:** [March 22, 2023, 9:44pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-pipeline-performance-issues/328006 "2023-03-22T21:44:22Z")
I’m having an issue when a particular pipeline and i’m not sure how to track it down or trouble shoot it further.. First, I have multiple cloud environments, configured the same, sending to the same endpoints. 2 of the… --- ## [Remove Specific Field matching pattern](https://discuss.elastic.co/t/remove-specific-field-matching-pattern/328260)
**Author:** [@rubhamra](https://discuss.elastic.co/u/rubhamra)\ **Replies:** 4\ **Last updated:** [March 22, 2023, 8:55pm UTC](https://discuss.elastic.co/t/remove-specific-field-matching-pattern/328260 "2023-03-22T20:55:44Z")
Hello I am trying to remove specific fields in logstash before it goes to elasticssearch, I tried below config. with drop option. if "\[response\]\[body\]\[entries\]\[values\]" == '^n1D.\*' { drop { } } I have a… --- ## [Datastream with upsert](https://discuss.elastic.co/t/datastream-with-upsert/328266)
**Author:** [@djkprojects](https://discuss.elastic.co/u/djkprojects)\ **Replies:** 1\ **Last updated:** [March 22, 2023, 3:28pm UTC](https://discuss.elastic.co/t/datastream-with-upsert/328266 "2023-03-22T15:28:45Z")
Hello, We are pulling data from MS SQL database into Elastic via Logstash which is working fine however some records get updated and so we want to update the existing entries in Elastic accordingly. The data is stored … --- ## [How to use user-defined plugin](https://discuss.elastic.co/t/how-to-use-user-defined-plugin/328224)
**Author:** [@wendywong0020](https://discuss.elastic.co/u/wendywong0020)\ **Replies:** 0\ **Last updated:** [March 22, 2023, 9:24am UTC](https://discuss.elastic.co/t/how-to-use-user-defined-plugin/328224 "2023-03-22T09:24:06Z")
logstash.conf: input { file { type =\> "\_doc" path =\> "/data/mysql\_\*\_log/slow.log" codec =\> multiline { … --- ## [Detected ambiguous Field Reference warning](https://discuss.elastic.co/t/detected-ambiguous-field-reference-warning/328218)
**Author:** [@parosio](https://discuss.elastic.co/u/parosio)\ **Replies:** 1\ **Last updated:** [March 22, 2023, 8:29am UTC](https://discuss.elastic.co/t/detected-ambiguous-field-reference-warning/328218 "2023-03-22T08:29:58Z")
Hello, I've got to ingest (logstash 6.7) documents which are stages of a workflow (queue\_in, start\_work, end\_work, queue\_out, etc.). I need to add various fields with elapsed times (looking for initial times in previou… --- ## [Extract logs from a file that start with a line and end with a known line do this for the whole file using logstash](https://discuss.elastic.co/t/extract-logs-from-a-file-that-start-with-a-line-and-end-with-a-known-line-do-this-for-the-whole-file-using-logstash/328216)
**Author:** [@chikugerson](https://discuss.elastic.co/u/chikugerson)\ **Replies:** 0\ **Last updated:** [March 22, 2023, 7:58am UTC](https://discuss.elastic.co/t/extract-logs-from-a-file-that-start-with-a-line-and-end-with-a-known-line-do-this-for-the-whole-file-using-logstash/328216 "2023-03-22T07:58:21Z")
input { file { path =\> "C:/Users/user/Documents/Logstash/mylogs/spa2.log" start\_position =\> "beginning" } } filter { if "SPAHGW:31 32 30 30 :004:: 1200" in \[message\] { … --- ## [Logstash ignores newly created template when importing index](https://discuss.elastic.co/t/logstash-ignores-newly-created-template-when-importing-index/328215)
**Author:** [@eeijlar](https://discuss.elastic.co/u/eeijlar)\ **Replies:** 0\ **Last updated:** [March 22, 2023, 7:40am UTC](https://discuss.elastic.co/t/logstash-ignores-newly-created-template-when-importing-index/328215 "2023-03-22T07:40:43Z")
I am using the following pipeline to do an import of an index exported from Elastic: - pipeline.id: import-process pipeline.workers: 4 config.string: | input { file { path =\>… --- ## [Logstash stop working priodicly!](https://discuss.elastic.co/t/logstash-stop-working-priodicly/328201)
**Author:** [@Siavash\_Fazli](https://discuss.elastic.co/u/Siavash_Fazli)\ **Replies:** 5\ **Last updated:** [March 22, 2023, 1:38am UTC](https://discuss.elastic.co/t/logstash-stop-working-priodicly/328201 "2023-03-22T01:38:42Z")
hi guys. my logstash stopped working several times as you can see in the picture. I have 15 pipelines on one docker logstash node can anyone guess what happened? --- ## [Handle space in a field](https://discuss.elastic.co/t/handle-space-in-a-field/328190)
**Author:** [@rubhamra](https://discuss.elastic.co/u/rubhamra)\ **Replies:** 2\ **Last updated:** [March 21, 2023, 7:43pm UTC](https://discuss.elastic.co/t/handle-space-in-a-field/328190 "2023-03-21T19:43:41Z")
I am trying to remove a space from a field in logstash but it's not working, because there is space in the field, I can't even rename the field or not able to do replacement is with gsub. Request ID to be renamed to Req… --- ## [Ingest RESTAPI response into Elasticsearch as separate document through Logstash](https://discuss.elastic.co/t/ingest-restapi-response-into-elasticsearch-as-separate-document-through-logstash/328117)
**Author:** [@rubhamra](https://discuss.elastic.co/u/rubhamra)\ **Replies:** 2\ **Last updated:** [March 21, 2023, 5:30pm UTC](https://discuss.elastic.co/t/ingest-restapi-response-into-elasticsearch-as-separate-document-through-logstash/328117 "2023-03-21T17:30:28Z")
I am working http\_poller and using http plugin to ingest RestApi Array response output into Elasticsearch. using Logstash , I need help to split the output and store each as a separate document in Elasticsearch. Below… --- ## [Extract from ElasticSearch, into Kafka, continuously add any new ES updates using logstash](https://discuss.elastic.co/t/extract-from-elasticsearch-into-kafka-continuously-add-any-new-es-updates-using-logstash/328172)
**Author:** [@aniketdatir](https://discuss.elastic.co/u/aniketdatir)\ **Replies:** 0\ **Last updated:** [March 21, 2023, 2:21pm UTC](https://discuss.elastic.co/t/extract-from-elasticsearch-into-kafka-continuously-add-any-new-es-updates-using-logstash/328172 "2023-03-21T14:21:20Z")
Hi Team, My objective is to add latest ES index documents to kafka Below is my logstash conf -\> ''' input { elasticsearch { hosts =\> \["IP"\] index =\> "Index\_name" query =\> '{"query":{"range":{"@timestamp":{"gte": … --- ## [Logstash Parse stingyfied json to seperate json fieldsl](https://discuss.elastic.co/t/logstash-parse-stingyfied-json-to-seperate-json-fieldsl/327097)
**Author:** [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Replies:** 0\ **Last updated:** [March 6, 2023, 3:13pm UTC](https://discuss.elastic.co/t/logstash-parse-stingyfied-json-to-seperate-json-fieldsl/327097 "2023-03-06T15:13:24Z")
Hello All, I've a column in oracle table PACKAGE\_DATA and it has json like string in it and I would like to get every fileds and its value seperate: PACKAGE\_DATA Column data {"status":"READY\_FOR\_PROCESSING","errorData… --- ## [Logstash filter to process jason array fileds as seperate fileds in elastic indexl](https://discuss.elastic.co/t/logstash-filter-to-process-jason-array-fileds-as-seperate-fileds-in-elastic-indexl/326874)
**Author:** [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Replies:** 9\ **Last updated:** [March 15, 2023, 3:56pm UTC](https://discuss.elastic.co/t/logstash-filter-to-process-jason-array-fileds-as-seperate-fileds-in-elastic-indexl/326874 "2023-03-15T15:56:53Z")
Hello All, After trying several time,I'm unable to process one column in oracle table that contains json data and I would require every field in that as seperate filed created in elastic index.Could someone guide what a… --- ## [Updating an existing field using path data](https://discuss.elastic.co/t/updating-an-existing-field-using-path-data/328110)
**Author:** [@Jeferson\_Schiavinato](https://discuss.elastic.co/u/Jeferson_Schiavinato)\ **Replies:** 5\ **Last updated:** [March 21, 2023, 1:07pm UTC](https://discuss.elastic.co/t/updating-an-existing-field-using-path-data/328110 "2023-03-21T13:07:36Z")
Hello Guys, I am using a path which is formed by /dir/subdir/filename\_log.gz. I want to extract the filename and update an existent Field called Hostname with this information. I have tried to use this code, but I had… --- ## [Offline plugin needed elf\_elk](https://discuss.elastic.co/t/offline-plugin-needed-elf-elk/328163)
**Author:** [@UsmanNiazi](https://discuss.elastic.co/u/UsmanNiazi)\ **Replies:** 4\ **Last updated:** [March 21, 2023, 12:48pm UTC](https://discuss.elastic.co/t/offline-plugin-needed-elf-elk/328163 "2023-03-21T12:48:39Z")
Hi I need to install Salesforce Event Log File on ELK Stack. But it is giving error. Can you please provide the offline plugin. I have tried to download the plugin from below but its not working. Giving errors when try … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=83) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=85)