# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=56 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 57 --- ## [Unable to see logs in Elasticsearch from logstash](https://discuss.elastic.co/t/unable-to-see-logs-in-elasticsearch-from-logstash/344288)
**Author:** [@AnyThink\_A](https://discuss.elastic.co/u/AnyThink_A)\ **Replies:** 8\ **Last updated:** [October 11, 2023, 5:23pm UTC](https://discuss.elastic.co/t/unable-to-see-logs-in-elasticsearch-from-logstash/344288 "2023-10-11T17:23:44Z")
Hello All, I am trying to deploy ELK stack in lab. I was able to get elastic and kibana up and running with XPAC enabled. But now struggling to get logs in elasticsearch from logstash. Refer config as below elasticsear… --- ## [Migrating data from Elastic Search 2.X to 8.X using logstash](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760)
**Author:** [@gslp456](https://discuss.elastic.co/u/gslp456)\ **Replies:** 3\ **Last updated:** [October 11, 2023, 3:48pm UTC](https://discuss.elastic.co/t/migrating-data-from-elastic-search-2-x-to-8-x-using-logstash/344760 "2023-10-11T15:48:38Z")
I have an on-premise elastic cluster running on version 2.3.3. (aka old) I have another on-premise elastic cluster running on version 8.9.0 (aka new) I want to migrate data from one index of old to new using logstash … --- ## [Object mapping error for common field name "error"](https://discuss.elastic.co/t/object-mapping-error-for-common-field-name-error/344768)
**Author:** [@true64gurus](https://discuss.elastic.co/u/true64gurus)\ **Replies:** 4\ **Last updated:** [October 11, 2023, 3:43pm UTC](https://discuss.elastic.co/t/object-mapping-error-for-common-field-name-error/344768 "2023-10-11T15:43:21Z")
Hello, I am getting the object mapping error. The logs are coming from Kubernetes cluster . On same index , the field "error" comes from App1 as json object , while comes as number from app2, comes as "one word strin… --- ## [Logstash exec input plugin issue](https://discuss.elastic.co/t/logstash-exec-input-plugin-issue/344834)
**Author:** [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\ **Replies:** 0\ **Last updated:** [October 11, 2023, 3:32pm UTC](https://discuss.elastic.co/t/logstash-exec-input-plugin-issue/344834 "2023-10-11T15:32:13Z")
Hi All, I am using logstash's exec input plugin to pull data from ManageEngine via the REST API using a Python script. The script takes around 2 mins to run and return json lines. The pipeline runs fine for a few iterat… --- ## [Overwrite the value for String field](https://discuss.elastic.co/t/overwrite-the-value-for-string-field/344813)
**Author:** [@My\_Google\_Account](https://discuss.elastic.co/u/My_Google_Account)\ **Replies:** 5\ **Last updated:** [October 11, 2023, 1:43pm UTC](https://discuss.elastic.co/t/overwrite-the-value-for-string-field/344813 "2023-10-11T13:43:05Z")
Good day ! could you please help me with the following question: how can i overwrite or (what plugin should i use?) the following value to another one: from /server/oauth2/userinfo/slaves/\* to /server/oauth2/userinfo … --- ## [Where is logstash log](https://discuss.elastic.co/t/where-is-logstash-log/344800)
**Author:** [@ChiMu\_Yuan](https://discuss.elastic.co/u/ChiMu_Yuan)\ **Replies:** 2\ **Last updated:** [October 11, 2023, 11:10am UTC](https://discuss.elastic.co/t/where-is-logstash-log/344800 "2023-10-11T11:10:21Z")
Hello everyone, I installed logstash with yum. But I can't start it. And the log is empty. How can i find the error log Thank you. --- ## [I want to read key value kafka headers](https://discuss.elastic.co/t/i-want-to-read-key-value-kafka-headers/344711)
**Author:** [@Ayushi\_bhardwaj](https://discuss.elastic.co/u/Ayushi_bhardwaj)\ **Replies:** 3\ **Last updated:** [October 11, 2023, 11:00am UTC](https://discuss.elastic.co/t/i-want-to-read-key-value-kafka-headers/344711 "2023-10-11T11:00:47Z")
Hello I want to read key value kafka headers but it is not giving me any output Can anyone please help me with the configuration or piece of code to read key value from kafka headers. My old ticket reference - --- ## [A question around logstash S3 input plugin](https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 2\ **Last updated:** [October 11, 2023, 1:04am UTC](https://discuss.elastic.co/t/a-question-around-logstash-s3-input-plugin/344769 "2023-10-11T01:04:04Z")
Hi All, We run logstash on multiple EC2 instances behind a loadbalancer for reliability purposes. We are thinking of using the S3 input plugin. Since the servers are created by auto-scaling process of AWS, they are exac… --- ## [Split my json input in logstash and push to ES](https://discuss.elastic.co/t/split-my-json-input-in-logstash-and-push-to-es/344767)
**Author:** [@shdasgupta](https://discuss.elastic.co/u/shdasgupta)\ **Replies:** 1\ **Last updated:** [October 11, 2023, 12:17am UTC](https://discuss.elastic.co/t/split-my-json-input-in-logstash-and-push-to-es/344767 "2023-10-11T00:17:44Z")
Hi, I have a gzipped json coming from kafka and I need to push it to ES after some transformations. With the help of this forum, I was able solve some of my problem. Right now i need to split the decompressed json into s… --- ## [Logstash HTTP output plugin](https://discuss.elastic.co/t/logstash-http-output-plugin/344553)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 4\ **Last updated:** [October 10, 2023, 7:12pm UTC](https://discuss.elastic.co/t/logstash-http-output-plugin/344553 "2023-10-10T19:12:26Z")
I'm trying to use the HTTP output plugin to send a PUT request to a URL but getting a 400 message, and I can't figure out what I'm messing up. I'm using Logstash v8 running in a Docker container. I'm able to successful… --- ## [Which filter(s) to extract array of JSON values, then use array to look up nested JSON objects?](https://discuss.elastic.co/t/which-filter-s-to-extract-array-of-json-values-then-use-array-to-look-up-nested-json-objects/343814)
**Author:** [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Replies:** 11\ **Last updated:** [October 10, 2023, 3:08pm UTC](https://discuss.elastic.co/t/which-filter-s-to-extract-array-of-json-values-then-use-array-to-look-up-nested-json-objects/343814 "2023-10-10T15:08:32Z")
I have a Logstash pipeline which is processing JSON data from a flat file. The data is somewhat structured like this: { "name": "job1", "tasks": { "75fc": { "name": "restAction", "variables": { "incoming"… --- ## [Send data from Elastic Agents through Logstash into ElasticSearch](https://discuss.elastic.co/t/send-data-from-elastic-agents-through-logstash-into-elasticsearch/344664)
**Author:** [@Gio\_27](https://discuss.elastic.co/u/Gio_27)\ **Replies:** 1\ **Last updated:** [October 10, 2023, 1:29pm UTC](https://discuss.elastic.co/t/send-data-from-elastic-agents-through-logstash-into-elasticsearch/344664 "2023-10-10T13:29:29Z")
Hey everyone I have some Fleet-managed Elastic Agents deployed. I would like those agents to send data into logstash for filtering / enrichment and then from logstash to Elasticsearch. I found this piece of doc: Elastic… --- ## [Multilined csv error while parsing](https://discuss.elastic.co/t/multilined-csv-error-while-parsing/344715)
**Author:** [@younes-gr](https://discuss.elastic.co/u/younes-gr)\ **Replies:** 0\ **Last updated:** [October 10, 2023, 9:00am UTC](https://discuss.elastic.co/t/multilined-csv-error-while-parsing/344715 "2023-10-10T09:00:06Z")
I am trying to process a csv file containing logs from several applications. Containing 10 columns like shown in the mapping at the end. I run into errors while trying to parse it because the'field\_8\_request' that usual… --- ## [All pipelines shutdown after one is not working](https://discuss.elastic.co/t/all-pipelines-shutdown-after-one-is-not-working/344653)
**Author:** [@Keremcan\_Seker](https://discuss.elastic.co/u/Keremcan_Seker)\ **Replies:** 12\ **Last updated:** [October 10, 2023, 9:48am UTC](https://discuss.elastic.co/t/all-pipelines-shutdown-after-one-is-not-working/344653 "2023-10-10T09:48:35Z")
i have 2 pipelines one is for kafka and other one is for http when kafka is not working and logstash can not create connection to kafka it terminates http pipeline too i asked to chat gpt it said Run each pipeline as a… --- ## [Is @timestamp get value automatically from field timestamp?](https://discuss.elastic.co/t/is-timestamp-get-value-automatically-from-field-timestamp/343952)
**Author:** [@waitspring](https://discuss.elastic.co/u/waitspring)\ **Replies:** 4\ **Last updated:** [October 10, 2023, 3:16am UTC](https://discuss.elastic.co/t/is-timestamp-get-value-automatically-from-field-timestamp/343952 "2023-10-10T03:16:45Z")
When we use this configure: filter { grok { match =\> { "message" =\> \[ "(?\%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}\\.\\d{3}) %{LOGLEVEL:level} \\\[%{DAT… --- ## [Documentation on in\_event and expect in Ruby filter test framework?](https://discuss.elastic.co/t/documentation-on-in-event-and-expect-in-ruby-filter-test-framework/344551)
**Author:** [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\ **Replies:** 6\ **Last updated:** [October 9, 2023, 9:32pm UTC](https://discuss.elastic.co/t/documentation-on-in-event-and-expect-in-ruby-filter-test-framework/344551 "2023-10-09T21:32:07Z")
I started working on a Ruby script to be called from my Logstash ruby filter. The official documentation mentions a test framework here There is an example test provided test "drop percentage 100%" do parameters do … --- ## [How to parse multiple nested arrays](https://discuss.elastic.co/t/how-to-parse-multiple-nested-arrays/344671)
**Author:** [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\ **Replies:** 0\ **Last updated:** [October 9, 2023, 2:14pm UTC](https://discuss.elastic.co/t/how-to-parse-multiple-nested-arrays/344671 "2023-10-09T14:14:05Z")
Hello everyone, I am trying to parse a json document using logstash version 8.3.3. The json document has multiple nested arrays, to flatten the document split is being used inside the filter. The issue is that the split… --- ## [Oracle to ElasticSearch using logstash](https://discuss.elastic.co/t/oracle-to-elasticsearch-using-logstash/344662)
**Author:** [@K\_Nguy\_n\_Kh\_c](https://discuss.elastic.co/u/K_Nguy_n_Kh_c)\ **Replies:** 0\ **Last updated:** [October 9, 2023, 1:38pm UTC](https://discuss.elastic.co/t/oracle-to-elasticsearch-using-logstash/344662 "2023-10-09T13:38:19Z")
Hi, i'm new to Elastic Stack and i try to push data from Oracle database to my Elasticsearch via Logstash but i'm stuck at this error Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil my p… --- ## [Duplicate logs whenever server reboots](https://discuss.elastic.co/t/duplicate-logs-whenever-server-reboots/344635)
**Author:** [@ShubhamKumarJena](https://discuss.elastic.co/u/ShubhamKumarJena)\ **Replies:** 0\ **Last updated:** [October 9, 2023, 9:40am UTC](https://discuss.elastic.co/t/duplicate-logs-whenever-server-reboots/344635 "2023-10-09T09:40:02Z")
Hi All, I have my datascience logs on a linux machine and I use Filebeat installed on the client linux machine which forwards logs to Logstash, we parse it and forward those logs to Elasticsearch. The issue is that when… --- ## [Config logstash for using kafka and jdbc input](https://discuss.elastic.co/t/config-logstash-for-using-kafka-and-jdbc-input/344612)
**Author:** [@azar\_uac](https://discuss.elastic.co/u/azar_uac)\ **Replies:** 0\ **Last updated:** [October 9, 2023, 3:50am UTC](https://discuss.elastic.co/t/config-logstash-for-using-kafka-and-jdbc-input/344612 "2023-10-09T03:50:00Z")
How to config Logstash to get table name , index name from kafka input message and use them in jdbc input for get table data of mssql database and send table data to index of elasticsearch --- ## [Convert a csv file column to array of json](https://discuss.elastic.co/t/convert-a-csv-file-column-to-array-of-json/344584)
**Author:** [@Sudharsan\_Aravind](https://discuss.elastic.co/u/Sudharsan_Aravind)\ **Replies:** 1\ **Last updated:** [October 8, 2023, 8:45pm UTC](https://discuss.elastic.co/t/convert-a-csv-file-column-to-array-of-json/344584 "2023-10-08T20:45:16Z")
I have a csv file, where there is this one column with array of json. column1, column2, array\_json\_col, column4, ... item1, item2, "\[{'type': 'StillImage', 'format': 'image/jpeg', 'url',: 'https://sample.url'}, {'type':… --- ## [Using Date plugin to parse apache2 error log datetime](https://discuss.elastic.co/t/using-date-plugin-to-parse-apache2-error-log-datetime/344547)
**Author:** [@lobart78](https://discuss.elastic.co/u/lobart78)\ **Replies:** 2\ **Last updated:** [October 6, 2023, 11:00pm UTC](https://discuss.elastic.co/t/using-date-plugin-to-parse-apache2-error-log-datetime/344547 "2023-10-06T23:00:07Z")
Hi all ! I am trying to use Logstash to parse apache2 error logs. These logs contain a dattime in a format e.g. Fri Oct 03 09:07:41.570 2023. I have already successfully transfered this string into a field "eventfire" … --- ## [Deserialising Avro data in losgstash](https://discuss.elastic.co/t/deserialising-avro-data-in-losgstash/344531)
**Author:** [@DivyaDileep](https://discuss.elastic.co/u/DivyaDileep)\ **Replies:** 0\ **Last updated:** [October 6, 2023, 10:22am UTC](https://discuss.elastic.co/t/deserialising-avro-data-in-losgstash/344531 "2023-10-06T10:22:07Z")
Continuing the discussion from Unable to Parse AVRO using Kafka Input and Avro Codec: --- ## [Migration from OpenSearch1.1 to Elasticsearch7.18 using logstash](https://discuss.elastic.co/t/migration-from-opensearch1-1-to-elasticsearch7-18-using-logstash/343535)
**Author:** [@gaurav\_jain](https://discuss.elastic.co/u/gaurav_jain)\ **Replies:** 12\ **Last updated:** [October 5, 2023, 9:19pm UTC](https://discuss.elastic.co/t/migration-from-opensearch1-1-to-elasticsearch7-18-using-logstash/343535 "2023-10-05T21:19:25Z")
Hi Experts, I am trying to migrate my Opensearch cluster version 1.1 to elastic cloud 7.18. I have created a logstash pipeline for the same who's configuration looks like this : input { opensearch { hosts … --- ## [@Timestamp is not matching event timestamp \_dateparsefailure](https://discuss.elastic.co/t/timestamp-is-not-matching-event-timestamp-dateparsefailure/344506)
**Author:** [@Cara410](https://discuss.elastic.co/u/Cara410)\ **Replies:** 2\ **Last updated:** [October 5, 2023, 8:01pm UTC](https://discuss.elastic.co/t/timestamp-is-not-matching-event-timestamp-dateparsefailure/344506 "2023-10-05T20:01:48Z")
Hello All, I am having filebeat send data through logstash and I have been unable to get the @timestamp to match the event time. I get a \_dateparsefailure tag in Kibana. Everything else is ingesting as intended. I have … --- ## [Logstash Metrics unavailable on Kibana Stack Monitoring UI](https://discuss.elastic.co/t/logstash-metrics-unavailable-on-kibana-stack-monitoring-ui/343328)
**Author:** [@gsekar](https://discuss.elastic.co/u/gsekar)\ **Replies:** 8\ **Last updated:** [October 5, 2023, 3:01pm UTC](https://discuss.elastic.co/t/logstash-metrics-unavailable-on-kibana-stack-monitoring-ui/343328 "2023-10-05T15:01:56Z")
Hi all Have installed metricbeat to monitor Logstash Nodes. The data stream - .monitoring-logstash-8-mb does get created and am seeing the data in the discover tab. But in the Stack Monitoring page for some reason the d… --- ## [Optimal way to handle log with multiple format?](https://discuss.elastic.co/t/optimal-way-to-handle-log-with-multiple-format/344470)
**Author:** [@Tanin\_Imanothai](https://discuss.elastic.co/u/Tanin_Imanothai)\ **Replies:** 0\ **Last updated:** [October 5, 2023, 10:33am UTC](https://discuss.elastic.co/t/optimal-way-to-handle-log-with-multiple-format/344470 "2023-10-05T10:33:57Z")
I try to parse this dataset: https://github.com/logpai/loghub/tree/master/Android using logstash. I have tried using grok filter but some parts of the log contains multiple templates. example of log: 03-17 16:13:38.81… --- ## [Salesforce input logstash - add filter](https://discuss.elastic.co/t/salesforce-input-logstash-add-filter/344217)
**Author:** [@Samuele\_Lolli](https://discuss.elastic.co/u/Samuele_Lolli)\ **Replies:** 4\ **Last updated:** [October 5, 2023, 8:17am UTC](https://discuss.elastic.co/t/salesforce-input-logstash-add-filter/344217 "2023-10-05T08:17:50Z")
I have a input configuration like that and i wonder if is possible to filter document like with a query or something like that. salesforce{ use\_test\_sandbox =\> true client\_id =\> '' client… --- ## [Error with net/smtp in Logstash (Docker)](https://discuss.elastic.co/t/error-with-net-smtp-in-logstash-docker/344312)
**Author:** [@Samuele\_Lolli](https://discuss.elastic.co/u/Samuele_Lolli)\ **Replies:** 2\ **Last updated:** [October 5, 2023, 8:08am UTC](https://discuss.elastic.co/t/error-with-net-smtp-in-logstash-docker/344312 "2023-10-05T08:08:47Z")
Hi everyone, im having some issue with docker and logstash. I have the following error: 2023-10-03 14:46:24 warning: thread "\[main\]-pipeline-manager" terminated with exception (report\_on\_exception is true): 2023-10-03 … --- ## [Optimizing Elasticsearch Cluster Setup: Merging Logs Across Two Node](https://discuss.elastic.co/t/optimizing-elasticsearch-cluster-setup-merging-logs-across-two-node/344371)
**Author:** [@Priyaansh\_Dwivedi](https://discuss.elastic.co/u/Priyaansh_Dwivedi)\ **Replies:** 0\ **Last updated:** [October 4, 2023, 9:57am UTC](https://discuss.elastic.co/t/optimizing-elasticsearch-cluster-setup-merging-logs-across-two-node/344371 "2023-10-04T09:57:08Z")
"I have always valued the support of this community, and I find myself in need of assistance once again. Here's the situation: I currently have Elasticsearch installed on VM1, but I'm facing disk space issues, and the cl… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=55) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=57)