# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=71 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 72 --- ## [Creating Headers (with key value) in kafka offset using logstash Configuration](https://discuss.elastic.co/t/creating-headers-with-key-value-in-kafka-offset-using-logstash-configuration/336089)
**Author:** [@Ayushi\_bhardwaj](https://discuss.elastic.co/u/Ayushi_bhardwaj)\ **Replies:** 4\ **Last updated:** [June 16, 2023, 12:54pm UTC](https://discuss.elastic.co/t/creating-headers-with-key-value-in-kafka-offset-using-logstash-configuration/336089 "2023-06-16T12:54:28Z")
Below is my config i had written to create headers (key & value) in my logstash config , but still it does not reflect to me in headers under offset tool. input { elasticsearch { hosts =\> \["localhost"\] index… --- ## [Logstash: Input File Plugin Showing Zero Events](https://discuss.elastic.co/t/logstash-input-file-plugin-showing-zero-events/336097)
**Author:** [@ksaimohan2k](https://discuss.elastic.co/u/ksaimohan2k)\ **Replies:** 2\ **Last updated:** [June 16, 2023, 5:21am UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-showing-zero-events/336097 "2023-06-16T05:21:34Z")
Hello, I am new to elastic. I am trying to parse XML logs using Logstash. As a result, I am using an input file plugin, and for the filtering process, I am using an XML plugin. Pipeline is running successfully, but show… --- ## [Convert normal logstash output to json output for adx ingestion](https://discuss.elastic.co/t/convert-normal-logstash-output-to-json-output-for-adx-ingestion/336138)
**Author:** [@ashokkrishna99\_Vemur](https://discuss.elastic.co/u/ashokkrishna99_Vemur)\ **Replies:** 2\ **Last updated:** [June 15, 2023, 8:51pm UTC](https://discuss.elastic.co/t/convert-normal-logstash-output-to-json-output-for-adx-ingestion/336138 "2023-06-15T20:51:23Z")
I have been working on transferring Palo Alto firewall logs(syslog format) to ADX. To achieve this, I developed grok filters and incorporated kv and mutate filters as well. However, I encountered an issue where the outpu… --- ## [Apply grok pattern based on the log file path](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395)
**Author:** [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Replies:** 40\ **Last updated:** [June 15, 2023, 4:29pm UTC](https://discuss.elastic.co/t/apply-grok-pattern-based-on-the-log-file-path/334395 "2023-06-15T16:29:21Z")
Hi Here is my logstash config file input { beats { port =\> 5044 } } output { elasticsearch { hosts =\> "http://ip:9200" index =\> "%{type}-%{+YYYY.MM.dd}" user =\> "elastic" password =\> "pwd" } … --- ## [How can I identify new elements in an array via Logstash/Elasticsearch?](https://discuss.elastic.co/t/how-can-i-identify-new-elements-in-an-array-via-logstash-elasticsearch/336105)
**Author:** [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)\ **Replies:** 0\ **Last updated:** [June 15, 2023, 2:01pm UTC](https://discuss.elastic.co/t/how-can-i-identify-new-elements-in-an-array-via-logstash-elasticsearch/336105 "2023-06-15T14:01:38Z")
For the context, I have an API request that returns a few devices and the apps installed in them. here is an example of what the result looks like in the elasticsearch: "hits": \[ { "\_index": "devices\_xxxx", "\_… --- ## [Collecting logs from Azure EH](https://discuss.elastic.co/t/collecting-logs-from-azure-eh/336088)
**Author:** [@wedkarz014](https://discuss.elastic.co/u/wedkarz014)\ **Replies:** 1\ **Last updated:** [June 15, 2023, 12:05pm UTC](https://discuss.elastic.co/t/collecting-logs-from-azure-eh/336088 "2023-06-15T12:05:00Z")
Hello, My question is, which tool should i use to collect data from Eh, logstash: Azure Event Hubs plugin | Logstash Reference \[8.8\] | Elastic or filebeat: Azure eventhub input | Filebeat Reference \[8.8\] | Elastic ? Whi… --- ## [Does versions of logstash\>7.3.0 support addKeyValue() in the slf4j fluent API?](https://discuss.elastic.co/t/does-versions-of-logstash-7-3-0-support-addkeyvalue-in-the-slf4j-fluent-api/336058)
**Author:** [@Steinar\_Bang](https://discuss.elastic.co/u/Steinar_Bang)\ **Replies:** 0\ **Last updated:** [June 15, 2023, 8:23am UTC](https://discuss.elastic.co/t/does-versions-of-logstash-7-3-0-support-addkeyvalue-in-the-slf4j-fluent-api/336058 "2023-06-15T08:23:38Z")
I just discovered slf4j's fluent API yesterday, and tried to use it. Unfortunately the key/value pairs I added with addKeyValue() in slf4j 2.0.7/logstash 7.3.0 were lost from the output. I had hoped for at least the de… --- ## [Enriching log data with database or other sources](https://discuss.elastic.co/t/enriching-log-data-with-database-or-other-sources/336016)
**Author:** [@Tomahawk](https://discuss.elastic.co/u/Tomahawk)\ **Replies:** 1\ **Last updated:** [June 15, 2023, 2:27am UTC](https://discuss.elastic.co/t/enriching-log-data-with-database-or-other-sources/336016 "2023-06-15T02:27:02Z")
Hey, I could use some help please. I have two questions I have log data coming in (Log4J, Log4Net etc etc) and this contains a user ID, for example “12345” I want to enrich each log line with more information about th… --- ## [How to create multiple separate index using single conf file in logstash through filebeat?](https://discuss.elastic.co/t/how-to-create-multiple-separate-index-using-single-conf-file-in-logstash-through-filebeat/335949)
**Author:** [@KRISHNA\_KUMAR1](https://discuss.elastic.co/u/KRISHNA_KUMAR1)\ **Replies:** 1\ **Last updated:** [June 15, 2023, 12:49am UTC](https://discuss.elastic.co/t/how-to-create-multiple-separate-index-using-single-conf-file-in-logstash-through-filebeat/335949 "2023-06-15T00:49:24Z")
Hi, I want to create separate indices based on the condition of the logs, for example if my log consist of api1 then it should create index named "api1" and if it consist api2 then create another index named "api2". Pl… --- ## [Removing text qualifier double quotes from Logstash CSV output](https://discuss.elastic.co/t/removing-text-qualifier-double-quotes-from-logstash-csv-output/335990)
**Author:** [@mhoward](https://discuss.elastic.co/u/mhoward)\ **Replies:** 2\ **Last updated:** [June 14, 2023, 6:35pm UTC](https://discuss.elastic.co/t/removing-text-qualifier-double-quotes-from-logstash-csv-output/335990 "2023-06-14T18:35:18Z")
Not sure if possible but I'm creating a CSV using Logstash. When I open the CSV in Notepad++ it adds double quotations around one specific field. The field itself is a city state zip code field that is created using a … --- ## [Is there a way to convert a unicode escape sequence within the Logstash pipeline so that the actual emoji icon is show within Elastic?](https://discuss.elastic.co/t/is-there-a-way-to-convert-a-unicode-escape-sequence-within-the-logstash-pipeline-so-that-the-actual-emoji-icon-is-show-within-elastic/336002)
**Author:** [@farnazpatel](https://discuss.elastic.co/u/farnazpatel)\ **Replies:** 0\ **Last updated:** [June 14, 2023, 5:04pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-convert-a-unicode-escape-sequence-within-the-logstash-pipeline-so-that-the-actual-emoji-icon-is-show-within-elastic/336002 "2023-06-14T17:04:38Z")
I am sending messages from Kafka in to Logstash and then through to Elastic, some of the messages contain emojis, these emojis are converted to Unicode escape characters when being stored in Kafka e.g. :blush: ---\> is c… --- ## [Logstash not applying correct system time to ingestion timestamp](https://discuss.elastic.co/t/logstash-not-applying-correct-system-time-to-ingestion-timestamp/335884)
**Author:** [@Anthony\_Zottola](https://discuss.elastic.co/u/Anthony_Zottola)\ **Replies:** 3\ **Last updated:** [June 14, 2023, 4:53pm UTC](https://discuss.elastic.co/t/logstash-not-applying-correct-system-time-to-ingestion-timestamp/335884 "2023-06-14T16:53:50Z")
Hello, I live in the NA East timezone so currently we are 4 hours behind UTC, I understand that logstash puts the @timestamp in UTC but it is putting in the wrong time. Logstash parsed a log at 10:30 am in my timezone … --- ## [Kafka plugin with every new group id it is pointing to old offset and not able to consume events](https://discuss.elastic.co/t/kafka-plugin-with-every-new-group-id-it-is-pointing-to-old-offset-and-not-able-to-consume-events/335994)
**Author:** [@Selim\_Hassan](https://discuss.elastic.co/u/Selim_Hassan)\ **Replies:** 0\ **Last updated:** [June 14, 2023, 3:36pm UTC](https://discuss.elastic.co/t/kafka-plugin-with-every-new-group-id-it-is-pointing-to-old-offset-and-not-able-to-consume-events/335994 "2023-06-14T15:36:18Z")
I have pipeline created as input { kafka { group\_id =\> "3fixed1" client\_id =\> "2fixed1" codec =\> avro{ schema\_uri =\> "C:\\LogStash\\KafkaClient\\topology.avsc" encoding =\> "binary" } bootstrap\_servers =\> "obootstap… --- ## [Need help shipping stdout and stderr logs of docker container to different Elasticsearch/kibana](https://discuss.elastic.co/t/need-help-shipping-stdout-and-stderr-logs-of-docker-container-to-different-elasticsearch-kibana/335978)
**Author:** [@Shobana\_Nagarajan](https://discuss.elastic.co/u/Shobana_Nagarajan)\ **Replies:** 0\ **Last updated:** [June 14, 2023, 12:30pm UTC](https://discuss.elastic.co/t/need-help-shipping-stdout-and-stderr-logs-of-docker-container-to-different-elasticsearch-kibana/335978 "2023-06-14T12:30:31Z")
Hi, I need to separate stdout and stderr streams from my docker container app and ship them to different elasticsearch/kibana hosts. Is it possible? With beats+elasticsearch+kibana, i was not able to get it working. Re… --- ## [Getting HIgh s3 cost on LISTBUCKET OPERATION using logstash s3 pipeline](https://discuss.elastic.co/t/getting-high-s3-cost-on-listbucket-operation-using-logstash-s3-pipeline/335970)
**Author:** [@Dharampal\_Singh](https://discuss.elastic.co/u/Dharampal_Singh)\ **Replies:** 0\ **Last updated:** [June 14, 2023, 9:53am UTC](https://discuss.elastic.co/t/getting-high-s3-cost-on-listbucket-operation-using-logstash-s3-pipeline/335970 "2023-06-14T09:53:45Z")
Hi elastic Team, We have 3 logstash s3 pipeine from buckets(elb,cloudflare,cloudtrail) .Currently we are getting high s3 list bucket operation cost on these buckets.We want to know is there any way so we can minimize … --- ## [Replacing certificates on the server](https://discuss.elastic.co/t/replacing-certificates-on-the-server/335967)
**Author:** [@lolkerz](https://discuss.elastic.co/u/lolkerz)\ **Replies:** 0\ **Last updated:** [June 14, 2023, 9:42am UTC](https://discuss.elastic.co/t/replacing-certificates-on-the-server/335967 "2023-06-14T09:42:38Z")
Hello everyone. Previously, I had a certificate on the Logstash server. At the moment I have created a new certificate. Is it enough for me to simply replace it on the server, or does something need to be done beforehand… --- ## [Unable to execute commands in Logstash pipeline](https://discuss.elastic.co/t/unable-to-execute-commands-in-logstash-pipeline/335199)
**Author:** [@mr\_ph](https://discuss.elastic.co/u/mr_ph)\ **Replies:** 2\ **Last updated:** [June 14, 2023, 9:25am UTC](https://discuss.elastic.co/t/unable-to-execute-commands-in-logstash-pipeline/335199 "2023-06-14T09:25:28Z")
I have some pipelines in my logstash. In that pipelines i am executing some commands as per some conditions. After creating the pipeline i used the following command to test the pipeline /usr/share/logstash/bin/lo… --- ## [How to synchronise data (PostgreSQL + MongoDB) in ES](https://discuss.elastic.co/t/how-to-synchronise-data-postgresql-mongodb-in-es/335876)
**Author:** [@stephane\_chan](https://discuss.elastic.co/u/stephane_chan)\ **Replies:** 3\ **Last updated:** [June 14, 2023, 6:32am UTC](https://discuss.elastic.co/t/how-to-synchronise-data-postgresql-mongodb-in-es/335876 "2023-06-14T06:32:37Z")
Hi, I'm new in Elasticsearch. I have Logstash configurations with postgresql and mongodb as data source (data.postgresql.conf, data.mongodb.conf), my problem is that I have to launch the logstash configuration of postg… --- ## [Logstash in elasticcloud without elastic agent](https://discuss.elastic.co/t/logstash-in-elasticcloud-without-elastic-agent/335840)
**Author:** [@Sivaramakrishhna\_Amb](https://discuss.elastic.co/u/Sivaramakrishhna_Amb)\ **Replies:** 1\ **Last updated:** [June 13, 2023, 11:50pm UTC](https://discuss.elastic.co/t/logstash-in-elasticcloud-without-elastic-agent/335840 "2023-06-13T23:50:18Z")
Hi, we are looking for logstash without elastic agent inetgration in elastic cloud. I'm not able to find simple logstash (without elasticagent) in elastic cloud. As our network devices will send logs to logstash th… --- ## [Logstash Kafka input DNS lookup for Kafka is not working](https://discuss.elastic.co/t/logstash-kafka-input-dns-lookup-for-kafka-is-not-working/335862)
**Author:** [@youngin.son.naver](https://discuss.elastic.co/u/youngin.son.naver)\ **Replies:** 0\ **Last updated:** [June 13, 2023, 10:53am UTC](https://discuss.elastic.co/t/logstash-kafka-input-dns-lookup-for-kafka-is-not-working/335862 "2023-06-13T10:53:45Z")
Using Logstash 7.12.1 and Kafka input option, Logstash does not automatically do DNS lookup when Kafka cluster has been restarted. \[org.apache.kafka.clients.NetworkClient\]\[main\]\[kafka\_test\] \[Consumer clientId=\*\*\*\*\*\*.lo… --- ## [LogStash setting Date error](https://discuss.elastic.co/t/logstash-setting-date-error/335733)
**Author:** [@yy\_isam](https://discuss.elastic.co/u/yy_isam)\ **Replies:** 8\ **Last updated:** [June 13, 2023, 4:46am UTC](https://discuss.elastic.co/t/logstash-setting-date-error/335733 "2023-06-13T04:46:20Z")
Hello guys! I want to run my logtstash connect to elastic using input jdbc. Then i create conf file to run logstash but i always get this error, i try any changes in conf file but still not working. Anyone can help me t… --- ## [Logstash filter issue](https://discuss.elastic.co/t/logstash-filter-issue/335314)
**Author:** [@namdev](https://discuss.elastic.co/u/namdev)\ **Replies:** 4\ **Last updated:** [June 12, 2023, 5:34pm UTC](https://discuss.elastic.co/t/logstash-filter-issue/335314 "2023-06-12T17:34:55Z")
Hi team, I am using logstash filter to get the duration between two dates. StartDate =2023-05-23T 10:25:53.123Z EndDate =2023-05-23T 18:25:43.123Z using the code below:-- match =\> \[ "Start Date", "ISO86… --- ## [Install Logstash-jdbc-integration plugin offline](https://discuss.elastic.co/t/install-logstash-jdbc-integration-plugin-offline/335785)
**Author:** [@ztzy1907](https://discuss.elastic.co/u/ztzy1907)\ **Replies:** 1\ **Last updated:** [June 12, 2023, 3:07pm UTC](https://discuss.elastic.co/t/install-logstash-jdbc-integration-plugin-offline/335785 "2023-06-12T15:07:32Z")
Hi, this is Richard. I'm trying to install logstash-jdbc-integration plugin on Logstash 7.8.0 on a machine that does not have internet access. What I'm trying to do is like below which is similar to install plugin on e… --- ## [How to use JSON filter correctly to parse my data?](https://discuss.elastic.co/t/how-to-use-json-filter-correctly-to-parse-my-data/335777)
**Author:** [@Chel\_Db](https://discuss.elastic.co/u/Chel_Db)\ **Replies:** 0\ **Last updated:** [June 12, 2023, 2:08pm UTC](https://discuss.elastic.co/t/how-to-use-json-filter-correctly-to-parse-my-data/335777 "2023-06-12T14:08:30Z")
Below is the JSON format of the current data and I'm using JSON filter to handle the nested JSON construct but it is not working as expected. log field is again a JSON field, which I would like to expand further as Mess… --- ## [Metrics do nothing in my file](https://discuss.elastic.co/t/metrics-do-nothing-in-my-file/335700)
**Author:** [@javierelastic](https://discuss.elastic.co/u/javierelastic)\ **Replies:** 2\ **Last updated:** [June 11, 2023, 10:35am UTC](https://discuss.elastic.co/t/metrics-do-nothing-in-my-file/335700 "2023-06-11T10:35:19Z")
I am trying to count the number of logs that appear in my file. Now I am using a file with logs as an example, but later I will use a syslog, and I want it to count the logs that arrive in 2 minutes. if \[msgFinal\] =~… --- ## [How to add hostname to logs from syslog or snmp source if they don't include only IP, no hostname](https://discuss.elastic.co/t/how-to-add-hostname-to-logs-from-syslog-or-snmp-source-if-they-dont-include-only-ip-no-hostname/335691)
**Author:** [@PackElend](https://discuss.elastic.co/u/PackElend)\ **Replies:** 2\ **Last updated:** [June 10, 2023, 2:54pm UTC](https://discuss.elastic.co/t/how-to-add-hostname-to-logs-from-syslog-or-snmp-source-if-they-dont-include-only-ip-no-hostname/335691 "2023-06-10T14:54:40Z")
Hello, I'm aware of How to add hostname to logs that normally do not contain hostname? but that is not applicable to my case. My router's firewall sends syslog message but they only contain the IP of the host causing t… --- ## [Runninning two configs in parallel without conflict with schedule](https://discuss.elastic.co/t/runninning-two-configs-in-parallel-without-conflict-with-schedule/335575)
**Author:** [@geothomas](https://discuss.elastic.co/u/geothomas)\ **Replies:** 3\ **Last updated:** [June 9, 2023, 3:19pm UTC](https://discuss.elastic.co/t/runninning-two-configs-in-parallel-without-conflict-with-schedule/335575 "2023-06-09T15:19:55Z")
I am trying to create elasticsearch index from oracle table. I have two configs, one delta.conf \*input {\* \* jdbc\* \*{\* \* jdbc\_driver\_library =\> "\/ojdbc10.jar"\* \* jdbc\_driver\_class =\> "Java::oracle.jdbc.dri… --- ## [Sending logs to syslog using logstash](https://discuss.elastic.co/t/sending-logs-to-syslog-using-logstash/334952)
**Author:** [@mariya](https://discuss.elastic.co/u/mariya)\ **Replies:** 16\ **Last updated:** [June 9, 2023, 2:43pm UTC](https://discuss.elastic.co/t/sending-logs-to-syslog-using-logstash/334952 "2023-06-09T14:43:43Z")
I installed winlogbeat and Logstash on my WInodows and I want to send logs to Logstash that will forward the logs to pfSense,I mean using Logstash as an aggregator with the logstash-output-tcp to send events to Syslog. a… --- ## [Removing extra characters in Grok](https://discuss.elastic.co/t/removing-extra-characters-in-grok/335375)
**Author:** [@Datt\_Mamon](https://discuss.elastic.co/u/Datt_Mamon)\ **Replies:** 5\ **Last updated:** [June 8, 2023, 9:27pm UTC](https://discuss.elastic.co/t/removing-extra-characters-in-grok/335375 "2023-06-08T21:27:11Z")
Hello, I am converting an original windows event log from json to syslog at the Logstash server. Here is a partial output: \<13\>May 31 14:27:55 {"name":'TEST'} LOGSTASH\[-\]: 2023-05-31T14:27:55.283Z {name=TEST} Permissi… --- ## [Logstash, remove all fields that contain a specific value](https://discuss.elastic.co/t/logstash-remove-all-fields-that-contain-a-specific-value/335569)
**Author:** [@kaismax](https://discuss.elastic.co/u/kaismax)\ **Replies:** 1\ **Last updated:** [June 8, 2023, 9:21pm UTC](https://discuss.elastic.co/t/logstash-remove-all-fields-that-contain-a-specific-value/335569 "2023-06-08T21:21:54Z")
how to remove all fields that contain a specific value or reg expression [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=70) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=72)