# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=147 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 148 --- ## [How to parse json logs in logstash](https://discuss.elastic.co/t/how-to-parse-json-logs-in-logstash/302358)
**Author:** [@hassnain](https://discuss.elastic.co/u/hassnain)\ **Replies:** 1\ **Last updated:** [April 13, 2022, 7:02pm UTC](https://discuss.elastic.co/t/how-to-parse-json-logs-in-logstash/302358 "2022-04-13T19:02:10Z")
{"className":"HomePageBusiness.java","msisdn":"912345678902","method":"HOMEPAGE","externalAPI":"My project info","message":"Delta this is successfull","timestamp":"2022/04/06 21:49:14","timeElapsed":"0:0:0:109"} This is… --- ## [How to configure syslog output plugin to useboth tcp and udp protocols based on Remote syslog protocol](https://discuss.elastic.co/t/how-to-configure-syslog-output-plugin-to-useboth-tcp-and-udp-protocols-based-on-remote-syslog-protocol/302312)
**Author:** [@Nikhitha](https://discuss.elastic.co/u/Nikhitha)\ **Replies:** 1\ **Last updated:** [April 13, 2022, 5:58pm UTC](https://discuss.elastic.co/t/how-to-configure-syslog-output-plugin-to-useboth-tcp-and-udp-protocols-based-on-remote-syslog-protocol/302312 "2022-04-13T17:58:01Z")
logstash syslog output plugin protocol should be configured to use tcp or udp dynamically instead of being hardcoded to any one protocol. Can someone help me with this. --- ## [Handling Logstash with RabbitMQ Input receiving basic.cancel](https://discuss.elastic.co/t/handling-logstash-with-rabbitmq-input-receiving-basic-cancel/302339)
**Author:** [@Ben-G](https://discuss.elastic.co/u/Ben-G)\ **Replies:** 0\ **Last updated:** [April 13, 2022, 1:29pm UTC](https://discuss.elastic.co/t/handling-logstash-with-rabbitmq-input-receiving-basic-cancel/302339 "2022-04-13T13:29:31Z")
I have deployed a logstash pod on our k8s cluster that reads from a RabbitMQ deployment. Recently we saw lag suddenly start to increase on RabbitMQ as the number of ready messages grew linearly without stopping. The Rab… --- ## [Need help how to remove unwanted fields logstash](https://discuss.elastic.co/t/need-help-how-to-remove-unwanted-fields-logstash/302315)
**Author:** [@holpa](https://discuss.elastic.co/u/holpa)\ **Replies:** 1\ **Last updated:** [April 13, 2022, 1:08pm UTC](https://discuss.elastic.co/t/need-help-how-to-remove-unwanted-fields-logstash/302315 "2022-04-13T13:08:40Z")
Please tell me how to remove unnecessary fields. Type: agent.ephemeral\_id agent.id winlog.provider\_guid I tried, But kibana stops showing logs at all drop\_fields: fields: \["date\_created", "ecs.version", "agent.version… --- ## [Logstash stopped processing because of an error](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error/302291)
**Author:** [@anushka1203](https://discuss.elastic.co/u/anushka1203)\ **Replies:** 1\ **Last updated:** [April 13, 2022, 11:39am UTC](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error/302291 "2022-04-13T11:39:23Z")
Hi, I am trying to send multiple log files to different ES indices. However, Logstash shuts down due to an error. I am a beginner and I could use some help with the same. here is my logstash conf file - input { beats… --- ## [Separate file(Lookup file) for grok pattern](https://discuss.elastic.co/t/separate-file-lookup-file-for-grok-pattern/302320)
**Author:** [@vikas.maske](https://discuss.elastic.co/u/vikas.maske)\ **Replies:** 0\ **Last updated:** [April 13, 2022, 11:15am UTC](https://discuss.elastic.co/t/separate-file-lookup-file-for-grok-pattern/302320 "2022-04-13T11:15:40Z")
Hello Team, Thanks for your valuable time. In my entire current ELK setup everyday some new changes needs to be performed on grok patterns and to avoid any human mistakes during changes , I want to use lookup file whi… --- ## [Unable to get new field with mutate filter](https://discuss.elastic.co/t/unable-to-get-new-field-with-mutate-filter/302228)
**Author:** [@zubair\_aftab](https://discuss.elastic.co/u/zubair_aftab)\ **Replies:** 4\ **Last updated:** [April 13, 2022, 10:35am UTC](https://discuss.elastic.co/t/unable-to-get-new-field-with-mutate-filter/302228 "2022-04-13T10:35:10Z")
i split my array temp as below if \[temp\] { split { field =\> "\[temp\]" } } i get different fields, one of the is : temp.isup\_isup\_message\_type Then i want to add a new field based on values in this field : temp.isup… --- ## [Using dynamic value in google\_pubsub filter plugin attributes field](https://discuss.elastic.co/t/using-dynamic-value-in-google-pubsub-filter-plugin-attributes-field/302184)
**Author:** [@Altiano\_Gerung](https://discuss.elastic.co/u/Altiano_Gerung)\ **Replies:** 3\ **Last updated:** [April 13, 2022, 3:30am UTC](https://discuss.elastic.co/t/using-dynamic-value-in-google-pubsub-filter-plugin-attributes-field/302184 "2022-04-13T03:30:56Z")
Hi all, I don't know how to give a dynamic value to this attributes field in google\_pubsub output plugin. Currently this is what I do, which is not DRY output { if \[name\] == "userRegistered" { google\_pubsub { … --- ## [How to show the pipeline conf source filename of error in logs](https://discuss.elastic.co/t/how-to-show-the-pipeline-conf-source-filename-of-error-in-logs/302201)
**Author:** [@Mahf](https://discuss.elastic.co/u/Mahf)\ **Replies:** 8\ **Last updated:** [April 12, 2022, 4:52pm UTC](https://discuss.elastic.co/t/how-to-show-the-pipeline-conf-source-filename-of-error-in-logs/302201 "2022-04-12T16:52:19Z")
Hello, We have multiple logstash conf files that we run simultaneously, the problem is that when we have many errors come from several pipelines, so we can't tell which one is the source of the error, sometimes it's eas… --- ## [Logstash reading recursively directories](https://discuss.elastic.co/t/logstash-reading-recursively-directories/302213)
**Author:** [@Andrea\_Bozzano](https://discuss.elastic.co/u/Andrea_Bozzano)\ **Replies:** 1\ **Last updated:** [April 12, 2022, 3:30pm UTC](https://discuss.elastic.co/t/logstash-reading-recursively-directories/302213 "2022-04-12T15:30:46Z")
Hi everyone, in my environment I'm trying to read log files that are saved on the file system like this: /home/andrea/mylogs/\[subdir1....subdirN\]/\*.gz /home/andrea/mylogs/\[subdir1...subdirM\]/\*.gz /home/andrea/mylogs/\[s… --- ## [Failed to parse date field](https://discuss.elastic.co/t/failed-to-parse-date-field/302248)
**Author:** [@Bavaria](https://discuss.elastic.co/u/Bavaria)\ **Replies:** 1\ **Last updated:** [April 12, 2022, 3:01pm UTC](https://discuss.elastic.co/t/failed-to-parse-date-field/302248 "2022-04-12T15:01:16Z")
I want to send Logs from Winlogbeat, via Logstash into Elasticsearch. From Winlogbeat directly to Elasticsearch is it working. However if I put the output to Logstash, I am getting this warning, that the date field cou… --- ## [Output to Azure Blob Storage](https://discuss.elastic.co/t/output-to-azure-blob-storage/302209)
**Author:** [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Replies:** 2\ **Last updated:** [April 12, 2022, 1:19pm UTC](https://discuss.elastic.co/t/output-to-azure-blob-storage/302209 "2022-04-12T13:19:51Z")
Hi, everyone I have been looking over Internet in order to find a plugin for Logstash or FIlebeat in order to send data to Azure Blob Storage. I have found a plugin on GitHub: Plugin However, it is an unofficial plug… --- ## [Need to create a grok pattern](https://discuss.elastic.co/t/need-to-create-a-grok-pattern/302123)
**Author:** [@amsmzn](https://discuss.elastic.co/u/amsmzn)\ **Replies:** 1\ **Last updated:** [April 12, 2022, 1:18pm UTC](https://discuss.elastic.co/t/need-to-create-a-grok-pattern/302123 "2022-04-12T13:18:42Z")
Hi there, I am new to logstash. I am trying to create a grok pattern for the below line, which would create a field in the Elasticsearch mvn clean deploy -P build -gs ./sd\_tools/config/maven/settings.xml -s ./sd\_tools/… --- ## [Logstash skips first lines and start reading from second line](https://discuss.elastic.co/t/logstash-skips-first-lines-and-start-reading-from-second-line/302121)
**Author:** [@Pranjal\_Sett](https://discuss.elastic.co/u/Pranjal_Sett)\ **Replies:** 3\ **Last updated:** [April 12, 2022, 12:23pm UTC](https://discuss.elastic.co/t/logstash-skips-first-lines-and-start-reading-from-second-line/302121 "2022-04-12T12:23:49Z")
Hi All, This is below my sample logs. \[2022-03-09T22:38:03.296+01:00\] \[String\] \[NOTIFICATION:16\] \[ODL-52001\] \[FileLogWriter\] \[Oracle\] \[host: \*\] \[nwaddr: \*\] \[tid: \[ACTIVE\].ExecuteThread: '157' for queue: 'weblogic.kerne… --- ## [Can't get value from hash in XML document](https://discuss.elastic.co/t/cant-get-value-from-hash-in-xml-document/301897)
**Author:** [@duanra22](https://discuss.elastic.co/u/duanra22)\ **Replies:** 4\ **Last updated:** [April 12, 2022, 7:06am UTC](https://discuss.elastic.co/t/cant-get-value-from-hash-in-xml-document/301897 "2022-04-12T07:06:02Z")
Hello, I'm struggling with an XML document that I am processing The part that is annoying me is something like \ \ \ \code1\ \value1\ \ \ \c… --- ## [Can logstash parsing port number with service name?](https://discuss.elastic.co/t/can-logstash-parsing-port-number-with-service-name/302080)
**Author:** [@111401](https://discuss.elastic.co/u/111401)\ **Replies:** 2\ **Last updated:** [April 12, 2022, 8:39am UTC](https://discuss.elastic.co/t/can-logstash-parsing-port-number-with-service-name/302080 "2022-04-12T08:39:40Z")
Hi, I wonder that is there any plugin or tools to help logstash use the port field to parsing corresponding service name? Like 53/tcp it will parsing the service name "DNS". Or I need to create a list of port/service, … --- ## [Unable to create geo points](https://discuss.elastic.co/t/unable-to-create-geo-points/302130)
**Author:** [@Dhinesh\_Kumar\_N](https://discuss.elastic.co/u/Dhinesh_Kumar_N)\ **Replies:** 2\ **Last updated:** [April 12, 2022, 7:56am UTC](https://discuss.elastic.co/t/unable-to-create-geo-points/302130 "2022-04-12T07:56:10Z")
Hi, Can any one help to configure geo points for the data which I have, I'm using live data streaming(filebeat), I the logstash configuration I have added location feed, config: geoip { database =\> "GeoIP2-City.mmdb… --- ## [How to use the date filter plugin in logstash](https://discuss.elastic.co/t/how-to-use-the-date-filter-plugin-in-logstash/302137)
**Author:** [@lolo\_prime](https://discuss.elastic.co/u/lolo_prime)\ **Replies:** 1\ **Last updated:** [April 11, 2022, 6:05pm UTC](https://discuss.elastic.co/t/how-to-use-the-date-filter-plugin-in-logstash/302137 "2022-04-11T18:05:30Z")
Hi Please find the below format of the date: 2022-04-12T05:11:47 my config date { match =\> \[ "time", "ISO8601" \] } but it giving the data parser error what is the format of the above date any one help me out --- ## [How to Parse the XML logs in Logstash and Convert into Json](https://discuss.elastic.co/t/how-to-parse-the-xml-logs-in-logstash-and-convert-into-json/302035)
**Author:** [@lolo\_prime](https://discuss.elastic.co/u/lolo_prime)\ **Replies:** 3\ **Last updated:** [April 11, 2022, 5:02pm UTC](https://discuss.elastic.co/t/how-to-parse-the-xml-logs-in-logstash-and-convert-into-json/302035 "2022-04-11T17:02:44Z")
i want to convert xml input to json data. The xml format is shown below --- ## [Throttling plugin not working with lot of messages](https://discuss.elastic.co/t/throttling-plugin-not-working-with-lot-of-messages/300360)
**Author:** [@elmar.vonlanthen](https://discuss.elastic.co/u/elmar.vonlanthen)\ **Replies:** 10\ **Last updated:** [April 11, 2022, 2:45pm UTC](https://discuss.elastic.co/t/throttling-plugin-not-working-with-lot-of-messages/300360 "2022-04-11T14:45:02Z")
Hi all We use the plugin logstash-filter-throttle to reduce the number of logs of a certain type to one message per 5 minutes, to use them for some kind of health messages. filter { if \[labels\]\[input\_type\] and \[l… --- ## [Logstash crashes out with the following message](https://discuss.elastic.co/t/logstash-crashes-out-with-the-following-message/301215)
**Author:** [@ksobon](https://discuss.elastic.co/u/ksobon)\ **Replies:** 7\ **Last updated:** [April 11, 2022, 2:25pm UTC](https://discuss.elastic.co/t/logstash-crashes-out-with-the-following-message/301215 "2022-04-11T14:25:26Z")
Logstash crashes out with the following log message: PS C:\\logstash\\bin\> .\\logstash.bat "Using bundled JDK: ." OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be… --- ## [Unable to remove lines from message filed](https://discuss.elastic.co/t/unable-to-remove-lines-from-message-filed/301866)
**Author:** [@amsmzn](https://discuss.elastic.co/u/amsmzn)\ **Replies:** 11\ **Last updated:** [April 11, 2022, 12:31pm UTC](https://discuss.elastic.co/t/unable-to-remove-lines-from-message-filed/301866 "2022-04-11T12:31:27Z")
I am trying to remove the below lines from my jenkins build logs with mutate, gsub filter but unable to do so. Can anybody please help me in this. Downloaded from snapshots: https://abc.com/abc.pom 16:39:44 Progress (1)… --- ## [Unable to execute custom mapping in Elasticsearch 7.10.2 Using Logstash 7.10.2](https://discuss.elastic.co/t/unable-to-execute-custom-mapping-in-elasticsearch-7-10-2-using-logstash-7-10-2/302068)
**Author:** [@Satya\_N](https://discuss.elastic.co/u/Satya_N)\ **Replies:** 0\ **Last updated:** [April 11, 2022, 5:55am UTC](https://discuss.elastic.co/t/unable-to-execute-custom-mapping-in-elasticsearch-7-10-2-using-logstash-7-10-2/302068 "2022-04-11T05:55:15Z")
Hi All, I am trying to sync data from mysql to Elasticsearch using logstash 7.10.2..Elasticsearch version I am using is : 7.10.2. When I am executing template using logstash , it is not creating in Elasticsearch so ev… --- ## [No Structured Logs Found in Stack Monitoring when using Logstash](https://discuss.elastic.co/t/no-structured-logs-found-in-stack-monitoring-when-using-logstash/302005)
**Author:** [@zx8086](https://discuss.elastic.co/u/zx8086)\ **Replies:** 6\ **Last updated:** [April 11, 2022, 2:23am UTC](https://discuss.elastic.co/t/no-structured-logs-found-in-stack-monitoring-when-using-logstash/302005 "2022-04-11T02:23:59Z")
Same issue Running filebeat -\> Elasticsearch = Works, can see Server, GC, Kibana etc logs... filebeat -\> Logstash = No Logs displayed --- ## [No Structured Logs Found in Stack Monitoring when using Logstashs](https://discuss.elastic.co/t/no-structured-logs-found-in-stack-monitoring-when-using-logstashs/296776)
**Author:** [@akballow](https://discuss.elastic.co/u/akballow)\ **Replies:** 5\ **Last updated:** [April 10, 2022, 11:13pm UTC](https://discuss.elastic.co/t/no-structured-logs-found-in-stack-monitoring-when-using-logstashs/296776 "2022-04-10T23:13:54Z")
Hello All, Hope someone can help as I spent the past week trying to resolve this issue but couldn't. So some backstory. If I setup Elasticsearch with filebeat pointing to Elasticsearch, the Logs section in Stack Monito… --- ## [How to configure Filebeat and Logstash 8.1 with TLS](https://discuss.elastic.co/t/how-to-configure-filebeat-and-logstash-8-1-with-tls/301534)
**Author:** [@fim](https://discuss.elastic.co/u/fim)\ **Replies:** 2\ **Last updated:** [April 10, 2022, 1:15pm UTC](https://discuss.elastic.co/t/how-to-configure-filebeat-and-logstash-8-1-with-tls/301534 "2022-04-10T13:15:01Z")
We are trying to secure the connectivity between Filebeat 8.1.1 \>\> Logstash 8.1.1 \>\> Elasticsearch 8.1.1 The documentation says we have to add a .p12 certificate in the output section of filebeat.yml and logstash pipeli… --- ## [Mapper parsing exception - failed to parse field](https://discuss.elastic.co/t/mapper-parsing-exception-failed-to-parse-field/302040)
**Author:** [@Robert777](https://discuss.elastic.co/u/Robert777)\ **Replies:** 9\ **Last updated:** [April 10, 2022, 9:44am UTC](https://discuss.elastic.co/t/mapper-parsing-exception-failed-to-parse-field/302040 "2022-04-10T09:44:23Z")
Hello, could you please advise what this problem is related to and how to solve it? Below is a snippet of the warning I am getting from logstash. A little situational description, we have a large number of microservice… --- ## [Logstash ruby code error](https://discuss.elastic.co/t/logstash-ruby-code-error/302030)
**Author:** [@leela](https://discuss.elastic.co/u/leela)\ **Replies:** 2\ **Last updated:** [April 9, 2022, 5:35pm UTC](https://discuss.elastic.co/t/logstash-ruby-code-error/302030 "2022-04-09T17:35:37Z")
ruby { code =\> " byt = event.get('\[network\]\[bytes\]'); bytes = byt\*8; microsec = ((event.get('\[netflow\]\[flow\_end\_sys\_up\_time\]'))-(event.get('\[netflow\]\[flow\_start\_sys\_up\_time\]')))/1000; avg = bytes/microsec; event.se… --- ## [Filebeat -\> kafka -\> logstash multiple brokers](https://discuss.elastic.co/t/filebeat-kafka-logstash-multiple-brokers/301978)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 1\ **Last updated:** [April 9, 2022, 9:06am UTC](https://discuss.elastic.co/t/filebeat-kafka-logstash-multiple-brokers/301978 "2022-04-09T09:06:21Z")
Hi, I have 3 brokers in one kafka cluster with zookeeper server and i create some topics with --replication-factor 3 --partitions 3 I run the zookeeper and the 3 brokers it works fine but when i kill the broker 1 it di… --- ## [Error when using elasticsearch logstash filter](https://discuss.elastic.co/t/error-when-using-elasticsearch-logstash-filter/301952)
**Author:** [@vladislav](https://discuss.elastic.co/u/vladislav)\ **Replies:** 1\ **Last updated:** [April 8, 2022, 2:29pm UTC](https://discuss.elastic.co/t/error-when-using-elasticsearch-logstash-filter/301952 "2022-04-08T14:29:32Z")
Good day! Please advice. I have an error when using Elasticsearch logstash filter. I need to make a query among index to find a value of ip, and if there is write it in the field 'match' Here is code of my .conf file: … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=146) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=148)