# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=222 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 223 --- ## [Logstach output data to mongodb handshake error](https://discuss.elastic.co/t/logstach-output-data-to-mongodb-handshake-error/275099)
**Author:** [@zhidong\_yuan](https://discuss.elastic.co/u/zhidong_yuan)\ **Replies:** 1\ **Last updated:** [June 7, 2021, 4:35pm UTC](https://discuss.elastic.co/t/logstach-output-data-to-mongodb-handshake-error/275099 "2021-06-07T16:35:12Z")
I just output data to monodb with Mongodb output plugin(v3.1.6). However, somethong worng happens when logstash straring up . The logs like this: \[INFO \] 2021-06-07 13:20:09.220 \[\[main\]-pipeline-manager\] javapipeline… --- ## [Logstash/grok to match only first occurrence and stop parsing repeatedly for same values](https://discuss.elastic.co/t/logstash-grok-to-match-only-first-occurrence-and-stop-parsing-repeatedly-for-same-values/274201)
**Author:** [@theirfan](https://discuss.elastic.co/u/theirfan)\ **Replies:** 23\ **Last updated:** [June 7, 2021, 12:10pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-only-first-occurrence-and-stop-parsing-repeatedly-for-same-values/274201 "2021-06-07T12:10:28Z")
Hi all, I have this issue with logstash/grok. Stated Working with ELK some days back. A newbie, please help. I have a log (sample below) Timestamp temperature 20 Timestamp temperature 20 Timestamp temperature 21 N… --- ## [Import JSON with nested fields](https://discuss.elastic.co/t/import-json-with-nested-fields/275010)
**Author:** [@aamrankw](https://discuss.elastic.co/u/aamrankw)\ **Replies:** 4\ **Last updated:** [June 7, 2021, 11:06am UTC](https://discuss.elastic.co/t/import-json-with-nested-fields/275010 "2021-06-07T11:06:23Z")
Hello all, I would like your support in defining a proper filter for importing the following JSON object: { "status": "ok", "message-type": "work-list", "message-version": "1.0.0", "message": { … --- ## [Why log are not show in elastic?](https://discuss.elastic.co/t/why-log-are-not-show-in-elastic/274857)
**Author:** [@Salim\_Adnan](https://discuss.elastic.co/u/Salim_Adnan)\ **Replies:** 1\ **Last updated:** [June 7, 2021, 5:01am UTC](https://discuss.elastic.co/t/why-log-are-not-show-in-elastic/274857 "2021-06-07T05:01:47Z")
--- ## [Error authentication](https://discuss.elastic.co/t/error-authentication/275005)
**Author:** [@Ilya\_21bA](https://discuss.elastic.co/u/Ilya_21bA)\ **Replies:** 1\ **Last updated:** [June 7, 2021, 4:17am UTC](https://discuss.elastic.co/t/error-authentication/275005 "2021-06-07T04:17:18Z")
\[2021-06-04T22:36:16,657\]\[WARN \]\[logstash.config.source.multilocal\] Ignoring the 'pipelines.yml' file because modules or command line options are specified \[2021-06-04T22:36:16,704\]\[INFO \]\[logstash.runner \] Star… --- ## [Processing large log files in logstash failing with \_jsonparsefailure error](https://discuss.elastic.co/t/processing-large-log-files-in-logstash-failing-with-jsonparsefailure-error/275077)
**Author:** [@tejpatil](https://discuss.elastic.co/u/tejpatil)\ **Replies:** 2\ **Last updated:** [June 7, 2021, 12:54am UTC](https://discuss.elastic.co/t/processing-large-log-files-in-logstash-failing-with-jsonparsefailure-error/275077 "2021-06-07T00:54:06Z")
The setup is like this. We send the files from mainframe to logstash server through CFI(Tibco MFT) basically a file transfer. The log file have around 700k lines in it and it is in JSON format. This is what I think is ha… --- ## [Logstash unable to Access AWS elasticsearch getting 403 role permissions seems fine](https://discuss.elastic.co/t/logstash-unable-to-access-aws-elasticsearch-getting-403-role-permissions-seems-fine/275059)
**Author:** [@Akhil\_Beeravalli05](https://discuss.elastic.co/u/Akhil_Beeravalli05)\ **Replies:** 1\ **Last updated:** [June 6, 2021, 11:43am UTC](https://discuss.elastic.co/t/logstash-unable-to-access-aws-elasticsearch-getting-403-role-permissions-seems-fine/275059 "2021-06-06T11:43:34Z")
\[2021-06-06T08:38:50,640\]\[WARN \]\[logstash.outputs.amazonelasticsearch\] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"https://search-xxxxxx-elasticsearch-n2wtsxzzf23iyifadn4mi2fmfy.us-ea… --- ## [Logstash doesn't delete nested fileds from json field](https://discuss.elastic.co/t/logstash-doesnt-delete-nested-fileds-from-json-field/275022)
**Author:** [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\ **Replies:** 0\ **Last updated:** [June 5, 2021, 10:25am UTC](https://discuss.elastic.co/t/logstash-doesnt-delete-nested-fileds-from-json-field/275022 "2021-06-05T10:25:03Z")
Hello there. I have an issue. In my logs there is raw string that contains json. I apply the json Logstash filter to it to get json-structured field. json { source =\> "response\_body" target =\> "respons… --- ## [Last Event not uploaded to kibana](https://discuss.elastic.co/t/last-event-not-uploaded-to-kibana/275017)
**Author:** [@a\_k2](https://discuss.elastic.co/u/a_k2)\ **Replies:** 0\ **Last updated:** [June 5, 2021, 8:11am UTC](https://discuss.elastic.co/t/last-event-not-uploaded-to-kibana/275017 "2021-06-05T08:11:58Z")
Logstash config input { file { path =\> "C:/elk/\*.log" start\_position =\> beginning sincedb\_path =\> "NUL" codec =\> multiline { pattern =\> "^%{TIMESTAMP\_ISO8601:timestamp} " negate =\> true what =\> previous auto\_f… --- ## [Grok Pattern/Expression for Multiline log from kibana (icingabeat index)](https://discuss.elastic.co/t/grok-pattern-expression-for-multiline-log-from-kibana-icingabeat-index/273059)
**Author:** [@shailesh](https://discuss.elastic.co/u/shailesh)\ **Replies:** 4\ **Last updated:** [June 4, 2021, 10:30pm UTC](https://discuss.elastic.co/t/grok-pattern-expression-for-multiline-log-from-kibana-icingabeat-index/273059 "2021-06-04T22:30:00Z")
Hello Team - I am trying to process some data from icinagbeat index from Kibana- so that i can visualize the output from icinagbeat - for that reason i am writing the Grok pattern - but the problem is when i write the pa… --- ## [Logstash with heavy Syslog input](https://discuss.elastic.co/t/logstash-with-heavy-syslog-input/275007)
**Author:** [@Kfiro](https://discuss.elastic.co/u/Kfiro)\ **Replies:** 1\ **Last updated:** [June 4, 2021, 10:28pm UTC](https://discuss.elastic.co/t/logstash-with-heavy-syslog-input/275007 "2021-06-04T22:28:19Z")
Hi friends, I have a SIEM system that sends 10,000 events per second to Logstash over Syslog TCP. It seems there is a bottleneck on logstash and i get only 1500 events per second. Is there a nice solution for that? A… --- ## [Logstash Kinesis Plugin - with\_kinesisEndpoint - Undefined Method](https://discuss.elastic.co/t/logstash-kinesis-plugin-with-kinesisendpoint-undefined-method/274922)
**Author:** [@cosruss](https://discuss.elastic.co/u/cosruss)\ **Replies:** 2\ **Last updated:** [June 4, 2021, 8:00pm UTC](https://discuss.elastic.co/t/logstash-kinesis-plugin-with-kinesisendpoint-undefined-method/274922 "2021-06-04T20:00:14Z")
Trying to use the Logstash Kinesis Plugin with a custom endpoint. In the config, I've got the following line: additional\_settings =\> {"kinesisEndpoint" =\> "https://\"} but I'm getting a strange error… --- ## [Update Existing Log Via Logstash](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 17\ **Last updated:** [June 4, 2021, 6:49pm UTC](https://discuss.elastic.co/t/update-existing-log-via-logstash/274796 "2021-06-04T18:49:05Z")
I want to update an existing document in Elasticsearch based on certain conditions. However, I am facing some errors. Here is my filter plugin configuration filter { json { source =\> "message" } if \[baseValueUnitA… --- ## [How to extract response and key-values value from response time string of apache logs](https://discuss.elastic.co/t/how-to-extract-response-and-key-values-value-from-response-time-string-of-apache-logs/274991)
**Author:** [@learningelk](https://discuss.elastic.co/u/learningelk)\ **Replies:** 1\ **Last updated:** [June 4, 2021, 4:02pm UTC](https://discuss.elastic.co/t/how-to-extract-response-and-key-values-value-from-response-time-string-of-apache-logs/274991 "2021-06-04T16:02:19Z")
My logs are as follows : ::ffff:10.67.0.179 - - \[23/Feb/2021:13:55:18 +0000\] "GET /files/77570035-bc7e-4be7-9554-e2164dd9397e.otf HTTP/1.1" 200 211 "-" "-" "c37004e0-75de-11eb-b6d4-cb790f9fe1ad" "40.324 ms" "serviceName… --- ## [Configuration problem](https://discuss.elastic.co/t/configuration-problem/274358)
**Author:** [@farciarz121](https://discuss.elastic.co/u/farciarz121)\ **Replies:** 20\ **Last updated:** [June 4, 2021, 12:12am UTC](https://discuss.elastic.co/t/configuration-problem/274358 "2021-06-04T00:12:43Z")
I have cloud elastic activated. I want to set up my environment in following way: \[beats - on prem\] ----\> \[logstash-on prem\] ------\> \[elastic cloud\] So, basically install beats on servers, send logs to logstash on prem… --- ## [Conditional parsing for the logs using logstash](https://discuss.elastic.co/t/conditional-parsing-for-the-logs-using-logstash/274892)
**Author:** [@learningelk](https://discuss.elastic.co/u/learningelk)\ **Replies:** 3\ **Last updated:** [June 4, 2021, 2:11pm UTC](https://discuss.elastic.co/t/conditional-parsing-for-the-logs-using-logstash/274892 "2021-06-04T14:11:52Z")
I am able to parse the logs and send to elk but I have this requirement and would request some suggestions on it . Application and Access logs are being written to the stdout of K8 cluster. I am able to setup logshippe… --- ## [Check filed value](https://discuss.elastic.co/t/check-filed-value/274976)
**Author:** [@royalE](https://discuss.elastic.co/u/royalE)\ **Replies:** 0\ **Last updated:** [June 4, 2021, 11:18am UTC](https://discuss.elastic.co/t/check-filed-value/274976 "2021-06-04T11:18:15Z")
I want to check the value of the field if its contain a number or string how can i do that ? --- ## [Logstash monitoring Encountered a retryable error. Will Retry with exponential backoff](https://discuss.elastic.co/t/logstash-monitoring-encountered-a-retryable-error-will-retry-with-exponential-backoff/274958)
**Author:** [@vaclav1](https://discuss.elastic.co/u/vaclav1)\ **Replies:** 0\ **Last updated:** [June 4, 2021, 8:18am UTC](https://discuss.elastic.co/t/logstash-monitoring-encountered-a-retryable-error-will-retry-with-exponential-backoff/274958 "2021-06-04T08:18:43Z")
Hello, I have just swapped to use TLS on http and configured api\_key as per Configuring Security in Logstash | Logstash Reference \[7.9\] | Elastic My logstash.yml file: # X-Pack Monitoring xpack.monitoring.enabled: tru… --- ## [Logstash configuration- How to do dynamic mapping?](https://discuss.elastic.co/t/logstash-configuration-how-to-do-dynamic-mapping/274749)
**Author:** [@Nisha2297](https://discuss.elastic.co/u/Nisha2297)\ **Replies:** 5\ **Last updated:** [June 4, 2021, 4:16am UTC](https://discuss.elastic.co/t/logstash-configuration-how-to-do-dynamic-mapping/274749 "2021-06-04T04:16:13Z")
I have one requirement to create dynamic mapping in my Logstash configuration to map two different columns. Please find below details: - Below is the ruby code I'm using in my logstash config: !\[image|690x351\](upload:/… --- ## [Update field value based on reference variable](https://discuss.elastic.co/t/update-field-value-based-on-reference-variable/274826)
**Author:** [@san2597](https://discuss.elastic.co/u/san2597)\ **Replies:** 2\ **Last updated:** [June 4, 2021, 12:04am UTC](https://discuss.elastic.co/t/update-field-value-based-on-reference-variable/274826 "2021-06-04T00:04:40Z")
Using Logstash, I'm parsing a log file that contains a timestamp value every time a new process starts. Each process starts and logs its own run time starting from 0 to the time the process completes. I'm trying to visua… --- ## [Increase logstash performance](https://discuss.elastic.co/t/increase-logstash-performance/274923)
**Author:** [@Kfiro](https://discuss.elastic.co/u/Kfiro)\ **Replies:** 0\ **Last updated:** [June 3, 2021, 8:57pm UTC](https://discuss.elastic.co/t/increase-logstash-performance/274923 "2021-06-03T20:57:38Z")
Hi, I'm sending events from my SIEM system to logstash. sending SYSLOG CEF over tcp 5000-10000 EPS (events per second) my logstash hardware: 16 Giga RAM 8 CPU 100 Giga SSD My logstash don't ingest all data and cra… --- ## [Logstash date filter](https://discuss.elastic.co/t/logstash-date-filter/274864)
**Author:** [@Kamikaze\_K](https://discuss.elastic.co/u/Kamikaze_K)\ **Replies:** 1\ **Last updated:** [June 3, 2021, 3:49pm UTC](https://discuss.elastic.co/t/logstash-date-filter/274864 "2021-06-03T15:49:40Z")
I have logs that I am parsing and the date does not contain the year which should be default current year. The field that I get after grok etc is 06/25 11:33:19.9 PM and 06/25 11:33:19.9 PM etc I am trying to use the d… --- ## [Parsing json logs using logstash](https://discuss.elastic.co/t/parsing-json-logs-using-logstash/274240)
**Author:** [@learningelk](https://discuss.elastic.co/u/learningelk)\ **Replies:** 10\ **Last updated:** [June 3, 2021, 2:15pm UTC](https://discuss.elastic.co/t/parsing-json-logs-using-logstash/274240 "2021-06-03T14:15:03Z")
Hi I want to parse the json logs using logstash and send them to elastic .There are multiple nested fields in my logs but I want very specific fields for eg , here is my log format : { "\_index": "ekslogs-2021.05.27", … --- ## [Metrcbeat withlogstach](https://discuss.elastic.co/t/metrcbeat-withlogstach/274877)
**Author:** [@mohamed\_el\_mannouti](https://discuss.elastic.co/u/mohamed_el_mannouti)\ **Replies:** 0\ **Last updated:** [June 3, 2021, 1:45pm UTC](https://discuss.elastic.co/t/metrcbeat-withlogstach/274877 "2021-06-03T13:45:33Z")
logs not found to kibana --- ## [Scheduler Java Input plugin Logstash](https://discuss.elastic.co/t/scheduler-java-input-plugin-logstash/274876)
**Author:** [@aalagia](https://discuss.elastic.co/u/aalagia)\ **Replies:** 0\ **Last updated:** [June 3, 2021, 1:37pm UTC](https://discuss.elastic.co/t/scheduler-java-input-plugin-logstash/274876 "2021-06-03T13:37:17Z")
Hi, I try to write a java plugin for Logstash, i need to schedule my program with a parameter passed in the configuration file. I tried to use the Java Timer for this but doesn't work. The pipeline execute the code but … --- ## [Xml filter Xpath not Working](https://discuss.elastic.co/t/xml-filter-xpath-not-working/274847)
**Author:** [@venmaniselvan](https://discuss.elastic.co/u/venmaniselvan)\ **Replies:** 1\ **Last updated:** [June 3, 2021, 12:21pm UTC](https://discuss.elastic.co/t/xml-filter-xpath-not-working/274847 "2021-06-03T12:21:20Z")
Hello, I have the following below XML: \ \CUSTINQ2104001\ \MOB\ \abcfinance:referenceNumConsumer\ \ **Author:** [@George\_Kossionis](https://discuss.elastic.co/u/George_Kossionis)\ **Replies:** 0\ **Last updated:** [June 3, 2021, 11:19am UTC](https://discuss.elastic.co/t/logstash-beats-input-nullpointerexception/274862 "2021-06-03T11:19:28Z") Using the following filebeat config in OCP: filebeat.autodiscover: providers: - type: kubernetes node: ${NODE\_NAME} templates: - condition: equals: … --- ## [Automatic indexing data from MySQL to Kibana](https://discuss.elastic.co/t/automatic-indexing-data-from-mysql-to-kibana/274849)
**Author:** [@pablixelastic](https://discuss.elastic.co/u/pablixelastic)\ **Replies:** 0\ **Last updated:** [June 3, 2021, 10:18am UTC](https://discuss.elastic.co/t/automatic-indexing-data-from-mysql-to-kibana/274849 "2021-06-03T10:18:26Z")
Hi there, I want to know which tool do i have to use to automatically index data from my MySQL table (where i also do changes in the existing data) to my index in Kibana. I've been trying modifying the mysql.conf and t… --- ## [Some logs missing in Kibana](https://discuss.elastic.co/t/some-logs-missing-in-kibana/274819)
**Author:** [@fsiu](https://discuss.elastic.co/u/fsiu)\ **Replies:** 1\ **Last updated:** [June 3, 2021, 3:39am UTC](https://discuss.elastic.co/t/some-logs-missing-in-kibana/274819 "2021-06-03T03:39:55Z")
I have an api application that uses Logstash to send logs to Kibana, all logs are successfully sent. However some logs are missing when viewing the index using Discovery - Kibana. Can you help me to identify the issue … --- ## [Elasticsearch appears to be unreachable or down](https://discuss.elastic.co/t/elasticsearch-appears-to-be-unreachable-or-down/274814)
**Author:** [@abdallah](https://discuss.elastic.co/u/abdallah)\ **Replies:** 0\ **Last updated:** [June 3, 2021, 2:53am UTC](https://discuss.elastic.co/t/elasticsearch-appears-to-be-unreachable-or-down/274814 "2021-06-03T02:53:32Z")
Hi, we suddenly lost the connection between our two logstash servers and the Elasticsearch nodes. The odd thing is that from logstash you can curl, telnet and ping the ElasticSearch nodes, but there is no sending logsta… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=221) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=223)