# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=16 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 17 --- ## [Why I am getting this manage\_inference error when using the elasticsearch python library?](https://discuss.elastic.co/t/why-i-am-getting-this-manage-inference-error-when-using-the-elasticsearch-python-library/370289)
**Author:** [@977db4bf8694e7df28b0](https://discuss.elastic.co/u/977db4bf8694e7df28b0)\ **Replies:** 3\ **Last updated:** [November 10, 2024, 7:44pm UTC](https://discuss.elastic.co/t/why-i-am-getting-this-manage-inference-error-when-using-the-elasticsearch-python-library/370289 "2024-11-10T19:44:20Z")
I am using elasticsearch python library with api access to the cluster. The api has manage previliges for the indices I am processing but seems to fail for 8.15.1 cluster but works fine for all the others. I tried it wit… --- ## [Mutate lowercase example uses array/list](https://discuss.elastic.co/t/mutate-lowercase-example-uses-array-list/370265)
**Author:** [@holobolo0815](https://discuss.elastic.co/u/holobolo0815)\ **Replies:** 1\ **Last updated:** [November 9, 2024, 10:23pm UTC](https://discuss.elastic.co/t/mutate-lowercase-example-uses-array-list/370265 "2024-11-09T22:23:46Z")
Hello, Concerning In the example it tells you to use a list as the argument. However that makes a list out of what might just be a string. In my case what worked as expected is: add\_field =\> { "qname\_lc" =\> "%{qn… --- ## [Using part of parsed date as selected output index](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170)
**Author:** [@nilsen](https://discuss.elastic.co/u/nilsen)\ **Replies:** 5\ **Last updated:** [November 8, 2024, 12:41pm UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170 "2024-11-08T12:41:15Z")
I need to select output index based on the parsed date instead of using current date %{+YYYY.MM.dd}, because the log file does not roll at midnight. I have tried to research, but only solution I find is using ruby code: … --- ## [How to handle fields with mixed field types in logstash?](https://discuss.elastic.co/t/how-to-handle-fields-with-mixed-field-types-in-logstash/370199)
**Author:** [@vwu\_su](https://discuss.elastic.co/u/vwu_su)\ **Replies:** 1\ **Last updated:** [November 8, 2024, 11:51am UTC](https://discuss.elastic.co/t/how-to-handle-fields-with-mixed-field-types-in-logstash/370199 "2024-11-08T11:51:20Z")
Hello, I am using filebeat to collect Diagnostic Logs for Entra ID on Azure. I found that the properties.location field has different types in different logs. In some logs it should be configured as object type, in othe… --- ## [Using global method and headers for http\_poller multiple urls](https://discuss.elastic.co/t/using-global-method-and-headers-for-http-poller-multiple-urls/370210)
**Author:** [@dogee](https://discuss.elastic.co/u/dogee)\ **Replies:** 1\ **Last updated:** [November 8, 2024, 11:40am UTC](https://discuss.elastic.co/t/using-global-method-and-headers-for-http-poller-multiple-urls/370210 "2024-11-08T11:40:05Z")
Hello, Having trouble to use global method GET and headers for multiple urls call. Copilot told me that it is possible, but it doesn't work in my case. This is my configuration : input { http\_poller { urls =\> { … --- ## [Logstash DLQ: sincedb never gets created](https://discuss.elastic.co/t/logstash-dlq-sincedb-never-gets-created/370205)
**Author:** [@hilsonp](https://discuss.elastic.co/u/hilsonp)\ **Replies:** 0\ **Last updated:** [November 8, 2024, 8:13am UTC](https://discuss.elastic.co/t/logstash-dlq-sincedb-never-gets-created/370205 "2024-11-08T08:13:15Z")
Hello, I'm trying to configure dead letter queue on a setup which was installed by another team a few years ago. Documents refused by elastic are well going to the dlq. The dlq pipeline ingests them and send them corr… --- ## [Error parsing json and truncated error in JSON logs](https://discuss.elastic.co/t/error-parsing-json-and-truncated-error-in-json-logs/370079)
**Author:** [@Seemant\_Bind](https://discuss.elastic.co/u/Seemant_Bind)\ **Replies:** 4\ **Last updated:** [November 7, 2024, 7:49pm UTC](https://discuss.elastic.co/t/error-parsing-json-and-truncated-error-in-json-logs/370079 "2024-11-07T19:49:48Z")
Hi, I am pushing logs through Filebeat and using logstash into Elastic. I have JSON logs and for some of the logs I am getting the error Error parsing json , \[truncated 4301 bytes\]; line: 1, column: 16\] but some of … --- ## [Logstash plugins](https://discuss.elastic.co/t/logstash-plugins/370175)
**Author:** [@DOkuwa](https://discuss.elastic.co/u/DOkuwa)\ **Replies:** 1\ **Last updated:** [November 7, 2024, 4:29pm UTC](https://discuss.elastic.co/t/logstash-plugins/370175 "2024-11-07T16:29:45Z")
Hello, I am writing a new plugin using grpc from a cisco router to logstash here I have checked the website for this info as i am bit lost and not a ruby expert The information from the cisco router is just like this … --- ## [Rename fields in logstash](https://discuss.elastic.co/t/rename-fields-in-logstash/369812)
**Author:** [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\ **Replies:** 6\ **Last updated:** [November 6, 2024, 1:50pm UTC](https://discuss.elastic.co/t/rename-fields-in-logstash/369812 "2024-11-06T13:50:44Z")
Hello team, from below log i am trying to remove below fields from logstash but it is not working as expected. Can you please help me on this: This is how field is coming in kibana: value.com.softwareag.um.server:… --- ## [Logstash pods not pushing logs to Elastic](https://discuss.elastic.co/t/logstash-pods-not-pushing-logs-to-elastic/370120)
**Author:** [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Replies:** 0\ **Last updated:** [November 6, 2024, 11:18am UTC](https://discuss.elastic.co/t/logstash-pods-not-pushing-logs-to-elastic/370120 "2024-11-06T11:18:50Z")
There is no error in log file (logstash-plain.log). Sincedb is zero in size and I restarted the logstash pods. Then the sincedb file is 81 bytes and later it resets to zero. I m using file input. This was working set an… --- ## [Drop json array data and read first event only](https://discuss.elastic.co/t/drop-json-array-data-and-read-first-event-only/370106)
**Author:** [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\ **Replies:** 0\ **Last updated:** [November 6, 2024, 7:14am UTC](https://discuss.elastic.co/t/drop-json-array-data-and-read-first-event-only/370106 "2024-11-06T07:14:27Z")
Hello team, Can we drop json array values and send data to elasticsearch only for latest values. Sample log line: { "@ad.context": "mydata", "value": \[ { "app\_name": "facebook", … --- ## [Logstash elastic\_app\_search output plugin returning 404 page](https://discuss.elastic.co/t/logstash-elastic-app-search-output-plugin-returning-404-page/370097)
**Author:** [@brad-g](https://discuss.elastic.co/u/brad-g)\ **Replies:** 0\ **Last updated:** [November 5, 2024, 10:44pm UTC](https://discuss.elastic.co/t/logstash-elastic-app-search-output-plugin-returning-404-page/370097 "2024-11-05T22:44:31Z")
I have used Logstash to pass documents to App Search since 2022, but after a recent Elasticsearch update (to 8.15) my systemctl logs are showing a 404 error, with the html for this page returned in the response: This… --- ## [Logstash "block in start\_workers" on shutdown](https://discuss.elastic.co/t/logstash-block-in-start-workers-on-shutdown/370015)
**Author:** [@nilsen](https://discuss.elastic.co/u/nilsen)\ **Replies:** 15\ **Last updated:** [November 5, 2024, 3:55pm UTC](https://discuss.elastic.co/t/logstash-block-in-start-workers-on-shutdown/370015 "2024-11-05T15:55:05Z")
~All of our Logstash instances gets tons of warnings on shutdown. Warning seems to tell us that something is blocked: "block in start\_workers" Our Logstash applications run with pipeline.yml: pipeline.yml example- pip… --- ## [How to drop any fields containing the word "PublicKey" from logstash](https://discuss.elastic.co/t/how-to-drop-any-fields-containing-the-word-publickey-from-logstash/369809)
**Author:** [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\ **Replies:** 4\ **Last updated:** [November 5, 2024, 12:27pm UTC](https://discuss.elastic.co/t/how-to-drop-any-fields-containing-the-word-publickey-from-logstash/369809 "2024-11-05T12:27:34Z")
Hello team, I wanted to drop all fields which containaing PublicKey word in the field name. date=2019-05-10 time=11:37:47 PublicKey="abc" type.PublicKey="xyz" PublicKey.subtype="pqr" level="notice" vd="vdom1" eventtime… --- ## [How can i count the entries of duplicate data when deduplicating in fingerprint and provide feedback in some form (such as in log form)?](https://discuss.elastic.co/t/how-can-i-count-the-entries-of-duplicate-data-when-deduplicating-in-fingerprint-and-provide-feedback-in-some-form-such-as-in-log-form/370058)
**Author:** [@WeirdorPersist](https://discuss.elastic.co/u/WeirdorPersist)\ **Replies:** 2\ **Last updated:** [November 5, 2024, 12:21pm UTC](https://discuss.elastic.co/t/how-can-i-count-the-entries-of-duplicate-data-when-deduplicating-in-fingerprint-and-provide-feedback-in-some-form-such-as-in-log-form/370058 "2024-11-05T12:21:00Z")
I am using the fingerprint plugin to deduplicate data. I hope to obtain the duplicate entries of each piece of data. What should I do?This is my configuration file. input { beats { port =\> 5044 } } filter{ g… --- ## [How Logstash sync work?](https://discuss.elastic.co/t/how-logstash-sync-work/369868)
**Author:** [@Coftbred](https://discuss.elastic.co/u/Coftbred)\ **Replies:** 1\ **Last updated:** [November 5, 2024, 12:09pm UTC](https://discuss.elastic.co/t/how-logstash-sync-work/369868 "2024-11-05T12:09:32Z")
I am new to Elasticsearch and Logstash, I want to know how Logstash can Sync data with Elasticsearch, for example, what mechanism does Logstash have to recognize a changed data field or just scan the entire database ever… --- ## [Logstash connectivity through TrustStore](https://discuss.elastic.co/t/logstash-connectivity-through-truststore/370052)
**Author:** [@ekta.thakur](https://discuss.elastic.co/u/ekta.thakur)\ **Replies:** 0\ **Last updated:** [November 5, 2024, 7:12am UTC](https://discuss.elastic.co/t/logstash-connectivity-through-truststore/370052 "2024-11-05T07:12:05Z")
Hi, I am connecting to logstash HTTP plugin of 8.12.0 version using java application where we are trying to pass a file to logstash and it will get collected from there and index it into elastcisearch as a output plugin… --- ## [Logstash doesn't parse message into separate parts](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958)
**Author:** [@Xavier\_24314](https://discuss.elastic.co/u/Xavier_24314)\ **Replies:** 5\ **Last updated:** [November 3, 2024, 4:56pm UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958 "2024-11-03T16:56:12Z")
I'm trying to parse my error.log file through logstash into elasticsearch but the message shows up in kibana as a whole without being split. I am not sure if my logstash conf file is wrong or if the error is elsewhere. … --- ## [Not able to connect docker logstash to docker elasticsearch using basic-auth using docker-compose](https://discuss.elastic.co/t/not-able-to-connect-docker-logstash-to-docker-elasticsearch-using-basic-auth-using-docker-compose/369941)
**Author:** [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Replies:** 11\ **Last updated:** [November 2, 2024, 7:02pm UTC](https://discuss.elastic.co/t/not-able-to-connect-docker-logstash-to-docker-elasticsearch-using-basic-auth-using-docker-compose/369941 "2024-11-02T19:02:07Z")
Hi team, facing the following issue-: While running the following docker-compose.yml, rest of the containers apart from logstash container are up and running , logstash-container fails and is exited, while trying to con… --- ## [Logstash agent failed to execute action](https://discuss.elastic.co/t/logstash-agent-failed-to-execute-action/369943)
**Author:** [@Xavier\_24314](https://discuss.elastic.co/u/Xavier_24314)\ **Replies:** 2\ **Last updated:** [November 2, 2024, 3:31pm UTC](https://discuss.elastic.co/t/logstash-agent-failed-to-execute-action/369943 "2024-11-02T15:31:15Z")
I'm trying to configure logstash to take in logs from my apache error.log file but it keeps throwing this error \[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id… --- ## [Got response code '401' contacting Elasticsearch](https://discuss.elastic.co/t/got-response-code-401-contacting-elasticsearch/369890)
**Author:** [@aldosilva6](https://discuss.elastic.co/u/aldosilva6)\ **Replies:** 3\ **Last updated:** [November 1, 2024, 6:33pm UTC](https://discuss.elastic.co/t/got-response-code-401-contacting-elasticsearch/369890 "2024-11-01T18:33:51Z")
I was working with Logstash, Elasticsearch and Kibana without security and everything was working, but I needed to put a basic authentication on Elasticsearch to go to production, and Logstash stoped to send messages to… --- ## [Avaya CDR Data in Elastic](https://discuss.elastic.co/t/avaya-cdr-data-in-elastic/369837)
**Author:** [@nitesh.srivastava](https://discuss.elastic.co/u/nitesh.srivastava)\ **Replies:** 8\ **Last updated:** [October 31, 2024, 10:47pm UTC](https://discuss.elastic.co/t/avaya-cdr-data-in-elastic/369837 "2024-10-31T22:47:25Z")
Hello everyone, We are ingesting some Avaya CDR data in our Elastic 8.11 and this is how the data looks like: \\u0001\\u0001\\u0001\\u0002\\a\\u0001\\u0000\\u0006\\u0001\\u0001\\u0002\\u0001\\u0006\\b\\u0001\\u001E\\x81 I see the belo… --- ## [Logstash TCP input not being processed?](https://discuss.elastic.co/t/logstash-tcp-input-not-being-processed/369871)
**Author:** [@pselvini](https://discuss.elastic.co/u/pselvini)\ **Replies:** 2\ **Last updated:** [October 31, 2024, 1:35pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-not-being-processed/369871 "2024-10-31T13:35:03Z")
Hi there, I have a very simple pipeline in logstash: input { tcp { port =\> 4560 codec =\> json\_lines } } output { stdout { codec =\> rubydebug } } I am sending JSON lines to Logstash from… --- ## [Slow SQL with timestamp on Oracle](https://discuss.elastic.co/t/slow-sql-with-timestamp-on-oracle/369819)
**Author:** [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\ **Replies:** 2\ **Last updated:** [October 31, 2024, 8:44am UTC](https://discuss.elastic.co/t/slow-sql-with-timestamp-on-oracle/369819 "2024-10-31T08:44:10Z")
Hello all, We have a Logstash Pipeline with jdbc input that ran really slow. I investigated and found something where I am not sure if this could be improved on Logstash side or if this is something we need to handle so… --- ## [Logstash delete all documents before http\_poller](https://discuss.elastic.co/t/logstash-delete-all-documents-before-http-poller/368470)
**Author:** [@dogee](https://discuss.elastic.co/u/dogee)\ **Replies:** 2\ **Last updated:** [October 30, 2024, 11:46pm UTC](https://discuss.elastic.co/t/logstash-delete-all-documents-before-http-poller/368470 "2024-10-30T23:46:23Z")
Hi, How can I remove old data of an index before http\_polling ? I call Jira API every day with http\_poller. Every call fetches data from Jira and creates documents. My problem is that I have finally too many duplicated… --- ## [Logstash writing to older index](https://discuss.elastic.co/t/logstash-writing-to-older-index/369754)
**Author:** [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Replies:** 5\ **Last updated:** [October 30, 2024, 10:59pm UTC](https://discuss.elastic.co/t/logstash-writing-to-older-index/369754 "2024-10-30T22:59:57Z")
Below is the error message from logstash. The Elasticsearch is designed with hot, warm and cold nodes. Index is created on daily basis. \[2024-10-29T11:14:22,632\]\[INFO \]\[logstash.outputs.elasticsearch\]\[main\]\[1dde6b9e906… --- ## [Grok from end of line](https://discuss.elastic.co/t/grok-from-end-of-line/369801)
**Author:** [@johnwood](https://discuss.elastic.co/u/johnwood)\ **Replies:** 2\ **Last updated:** [October 30, 2024, 10:50pm UTC](https://discuss.elastic.co/t/grok-from-end-of-line/369801 "2024-10-30T22:50:15Z")
Hi I am struggling to extract the correct bit of a log message. what I want to extract is the AgentID which sits at the end of the string and comprises 8 alphanumerics, a hyphen and then 8 more alphnumerics. In this c… --- ## [Java Heap logstash](https://discuss.elastic.co/t/java-heap-logstash/365601)
**Author:** [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Replies:** 3\ **Last updated:** [October 30, 2024, 5:20am UTC](https://discuss.elastic.co/t/java-heap-logstash/365601 "2024-10-30T05:20:57Z")
\[2024-08-16T07:54:27,150\]\[FATAL\]\[org.logstash.Logstash \] uncaught error (in thread pool-701-thread-1) java.lang.OutOfMemoryError: Java heap space I will be running one pipeline. Haven't changed any default settings. … --- ## [SNMP input plugin integration with logstash](https://discuss.elastic.co/t/snmp-input-plugin-integration-with-logstash/369702)
**Author:** [@Mohan.k](https://discuss.elastic.co/u/Mohan.k)\ **Replies:** 1\ **Last updated:** [October 29, 2024, 3:20pm UTC](https://discuss.elastic.co/t/snmp-input-plugin-integration-with-logstash/369702 "2024-10-29T15:20:56Z")
Hi All, I am new to the ELK monitoring system. I have requirement to implement SNMP input plugin integration with one of logstash server. I referred the document and unfortunately I am not able to understand how can I… --- ## [Logstash cannot connect to elasticsearch](https://discuss.elastic.co/t/logstash-cannot-connect-to-elasticsearch/369510)
**Author:** [@saqibmushtaq](https://discuss.elastic.co/u/saqibmushtaq)\ **Replies:** 9\ **Last updated:** [October 28, 2024, 4:55pm UTC](https://discuss.elastic.co/t/logstash-cannot-connect-to-elasticsearch/369510 "2024-10-28T16:55:48Z")
I am not able to connect logstash with elasticsearch, below are my configuration files version: "1.0" services: elasticsearch: container\_name: elasticsearch-container image: docker.elastic.co/elasticsearch/ela… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=15) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=17)