# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=254 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 255 --- ## [Central pipeline management - beats strategy](https://discuss.elastic.co/t/central-pipeline-management-beats-strategy/264680)
**Author:** [@Waxman](https://discuss.elastic.co/u/Waxman)\ **Replies:** 0\ **Last updated:** [February 18, 2021, 10:06am UTC](https://discuss.elastic.co/t/central-pipeline-management-beats-strategy/264680 "2021-02-18T10:06:18Z")
Hi Everybody, we'd like to have some strategy for logstash and beats and we're using centralized pipeline management. Rather we'd like to use it :slight\_smile: The idea is to not configure anything on logstash but only… --- ## [Logstash Jdbc Metadata path not creating](https://discuss.elastic.co/t/logstash-jdbc-metadata-path-not-creating/264652)
**Author:** [@Bryce\_Fernandes](https://discuss.elastic.co/u/Bryce_Fernandes)\ **Replies:** 0\ **Last updated:** [February 18, 2021, 6:41am UTC](https://discuss.elastic.co/t/logstash-jdbc-metadata-path-not-creating/264652 "2021-02-18T06:41:11Z")
Hi, I have a database and using jdbc plugin to fetch data from it. I have used sql last value and given metadata path it was getting created at first. I deleted and tried to create again but not creating since then. I m… --- ## [Filters](https://discuss.elastic.co/t/filters/264644)
**Author:** [@Anu6006](https://discuss.elastic.co/u/Anu6006)\ **Replies:** 0\ **Last updated:** [February 18, 2021, 5:12am UTC](https://discuss.elastic.co/t/filters/264644 "2021-02-18T05:12:18Z")
Hi Team, In logstash the filters are not working. The data is coming from azure blob storage. filter { split { field =\> "\[records\]" } split { field =\> "\[records\]\[properties\]\[flows\]" } split { fiel… --- ## [Parse Error json object](https://discuss.elastic.co/t/parse-error-json-object/264534)
**Author:** [@ljeganathan](https://discuss.elastic.co/u/ljeganathan)\ **Replies:** 1\ **Last updated:** [February 17, 2021, 5:05pm UTC](https://discuss.elastic.co/t/parse-error-json-object/264534 "2021-02-17T17:05:47Z")
Hi, I am getting parse error like below JSON parse error, original data now in message field {:error=\>#\ **Author:** [@arclinear](https://discuss.elastic.co/u/arclinear)\ **Replies:** 1\ **Last updated:** [February 17, 2021, 1:40pm UTC](https://discuss.elastic.co/t/how-to-not-log-extra-metadata-from-logstash-to-elasticsearch/264524 "2021-02-17T13:40:22Z") Hello, currently I have a working example of an ELK stack, however, in my indices I do not want to log the following fields : Ideally, in Elasticsearch, I only want to see "message" and "@timestamp", the other 4 is se… --- ## [Parse json in string in json (nested json)](https://discuss.elastic.co/t/parse-json-in-string-in-json-nested-json/264569)
**Author:** [@Romanian\_Coder](https://discuss.elastic.co/u/Romanian_Coder)\ **Replies:** 3\ **Last updated:** [February 17, 2021, 1:03pm UTC](https://discuss.elastic.co/t/parse-json-in-string-in-json-nested-json/264569 "2021-02-17T13:03:42Z")
I have next json in input { "partitionId": 3, "value": { "name": "updatedPaymentInfo", "value": "{\\"reference\\":\\"Z091702210000015\\",\\"transactionId\\":\\"CC11702210000020\\",\\"amountDTO\\":{\\"value\\":10000,\\"c… --- ## [Persistent queue Issue on logstash](https://discuss.elastic.co/t/persistent-queue-issue-on-logstash/264546)
**Author:** [@iammanmale](https://discuss.elastic.co/u/iammanmale)\ **Replies:** 0\ **Last updated:** [February 17, 2021, 10:10am UTC](https://discuss.elastic.co/t/persistent-queue-issue-on-logstash/264546 "2021-02-17T10:10:12Z")
We have three logstash nodes to collect log in parallel. All three logstash have 6 output pipelines and 1 intake pipeline. After i try to enable persistent queue on one of the logstash, no matter how little the queue\_max… --- ## [Multiple Elapsed](https://discuss.elastic.co/t/multiple-elapsed/264459)
**Author:** [@artobstrel](https://discuss.elastic.co/u/artobstrel)\ **Replies:** 2\ **Last updated:** [February 17, 2021, 10:05am UTC](https://discuss.elastic.co/t/multiple-elapsed/264459 "2021-02-17T10:05:11Z")
I want to measure the duration of several events within a single transaction. I use a elapsed filter for this. My data and logstash config file is below: data file2021-02-16 16:00:00 016cbeb4 Input 2021-02-16 1… --- ## [Logstash can´t index event to Elasticsearch](https://discuss.elastic.co/t/logstash-can-t-index-event-to-elasticsearch/264398)
**Author:** [@Martin\_perez](https://discuss.elastic.co/u/Martin_perez)\ **Replies:** 6\ **Last updated:** [February 17, 2021, 7:28am UTC](https://discuss.elastic.co/t/logstash-can-t-index-event-to-elasticsearch/264398 "2021-02-17T07:28:17Z")
Hello everyone, First of all, I want to introduce you to my problem, what I want to do, and how I have it installed. I have ELK Stack installed this docker, and this docker, at the same time, on a virtualized ubuntu wi… --- ## [Incorrect type of field in index from Logstash, why?](https://discuss.elastic.co/t/incorrect-type-of-field-in-index-from-logstash-why/264335)
**Author:** [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Replies:** 8\ **Last updated:** [February 17, 2021, 6:10am UTC](https://discuss.elastic.co/t/incorrect-type-of-field-in-index-from-logstash-why/264335 "2021-02-17T06:10:37Z")
Dears, I loading tomcat logs to Elastic. My logstash config include such grok pattern: if "tomcat" in \[tags\] { grok { match =\> \["message", "%{IPV4} (?:-|%{USER:ident}) (?:-|%{USER:auth}) \\\[%{HTTPDATE:timesta… --- ## [X-pack for Logstash](https://discuss.elastic.co/t/x-pack-for-logstash/264466)
**Author:** [@testsemd\_email](https://discuss.elastic.co/u/testsemd_email)\ **Replies:** 3\ **Last updated:** [February 17, 2021, 5:50am UTC](https://discuss.elastic.co/t/x-pack-for-logstash/264466 "2021-02-17T05:50:19Z")
I have enabled the x-pack in my cluster. Now I want to install Logstash on ingest node. Enabling x-pack for Logstash is optional or compulsory? --- ## [Logstash self restart without error](https://discuss.elastic.co/t/logstash-self-restart-without-error/264501)
**Author:** [@cling1988](https://discuss.elastic.co/u/cling1988)\ **Replies:** 0\ **Last updated:** [February 17, 2021, 4:42am UTC](https://discuss.elastic.co/t/logstash-self-restart-without-error/264501 "2021-02-17T04:42:52Z")
After a few days I upgrade the Logstash from 7.4 to Logstash 7.10.1, the Logstash kill itself and restart. I only can found the log in journalctl -u logstash.service Feb 16 03:44:23 xxx systemd\[1\]: logstash.service: ma… --- ## [Match multiple 4 digit numbers in field](https://discuss.elastic.co/t/match-multiple-4-digit-numbers-in-field/264494)
**Author:** [@Deny7](https://discuss.elastic.co/u/Deny7)\ **Replies:** 1\ **Last updated:** [February 17, 2021, 12:28am UTC](https://discuss.elastic.co/t/match-multiple-4-digit-numbers-in-field/264494 "2021-02-17T00:28:35Z")
I need to match only 4 digit number from field "created:". If there are multiple 4 digit numbers I need to take the biggest and save it back to the field. The problem is that the field can contain strings, dashes, dots, … --- ## [A JSON array consists of objects - how to deal with?](https://discuss.elastic.co/t/a-json-array-consists-of-objects-how-to-deal-with/264464)
**Author:** [@awer1967](https://discuss.elastic.co/u/awer1967)\ **Replies:** 5\ **Last updated:** [February 16, 2021, 9:33pm UTC](https://discuss.elastic.co/t/a-json-array-consists-of-objects-how-to-deal-with/264464 "2021-02-16T21:33:42Z")
Hi ! I strongly apologize for a possibly stupid, kinda "it-has-been-asked-a-lot-of-time", question. Nevertheless I have to ask for help because I have completely stuck with . So that I have a JSON array consist of ob… --- ## [Extract nested json](https://discuss.elastic.co/t/extract-nested-json/264062)
**Author:** [@Chris\_Clifton](https://discuss.elastic.co/u/Chris_Clifton)\ **Replies:** 1\ **Last updated:** [February 16, 2021, 7:13pm UTC](https://discuss.elastic.co/t/extract-nested-json/264062 "2021-02-16T19:13:23Z")
I have the following nested json that I'm trying to pull out into top level fields so I can use mutate to add and modify, These events are picked from an awscloudwatch stream with filebeat v7.10.2 then fed to a logstash… --- ## [Duplicate Issue - document\_id, how to prevent overwriting of entries](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404)
**Author:** [@Sampson\_Light](https://discuss.elastic.co/u/Sampson_Light)\ **Replies:** 5\ **Last updated:** [February 16, 2021, 1:44pm UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404 "2021-02-16T13:44:57Z")
Hi All, Some background information: I have duplicate entries in my elasticsearch indexes. Have used document\_id which prevented duplicates from appearing. But the issue with this is that it overwrites and updates the… --- ## [Logstash pipeline output and elastic cloud](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659)
**Author:** [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Replies:** 10\ **Last updated:** [February 16, 2021, 12:17pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659 "2021-02-16T12:17:29Z")
Hi all. I have elastic cloud 7.10.2 and we are setting up logstash with a pipilne with input/filter/output. Also we are using helm to deploy. Of course I have cloud.id and cloud.auth and not sure how to configure the … --- ## [Syslog output date format](https://discuss.elastic.co/t/syslog-output-date-format/264366)
**Author:** [@gunlomboy](https://discuss.elastic.co/u/gunlomboy)\ **Replies:** 7\ **Last updated:** [February 16, 2021, 3:38am UTC](https://discuss.elastic.co/t/syslog-output-date-format/264366 "2021-02-16T03:38:39Z")
Hi, I am running Windows logs successfully to our SIEM using a kafka input and a logstash syslog output. The logs are parsing perfectly in the SIEM. When running a virtually identical pipeline to poll a different topi… --- ## [LogStash::ConfigurationError", :message=\>"Expected one of \[A-Za-z0-9\_-\]](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of-a-za-z0-9/264315)
**Author:** [@muthug](https://discuss.elastic.co/u/muthug)\ **Replies:** 15\ **Last updated:** [February 16, 2021, 3:27am UTC](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of-a-za-z0-9/264315 "2021-02-16T03:27:30Z")
Hi Guys, Good Day!! I am facing the issue while staring the logstash. LogStash::ConfigurationError", :message=\>"Expected one of \[A-Za-z0-9\_-\] \[ \\t\\r\\n\], "#", "{", \[A-Za-z0-9\_\], "}" at line 49, column 31 (byte 1290) af… --- ## [Grok timestamp (can't match pattern)](https://discuss.elastic.co/t/grok-timestamp-cant-match-pattern/263816)
**Author:** [@Frankie\_Braybon](https://discuss.elastic.co/u/Frankie_Braybon)\ **Replies:** 1\ **Last updated:** [February 15, 2021, 8:35pm UTC](https://discuss.elastic.co/t/grok-timestamp-cant-match-pattern/263816 "2021-02-15T20:35:24Z")
The log line I am trying to make patterns on: Sent from filebeat to logstash. \[Log Level- \[1\],\[2021-01-05 15:19:28:410 GMT+0000\], Thread ID: -858872064 HEALTH CHECK STATUS : 0, HealthCheck/Src/HealthCheckManager.cpp,… --- ## [Logstash not sending the logs to elk and kibana](https://discuss.elastic.co/t/logstash-not-sending-the-logs-to-elk-and-kibana/264357)
**Author:** [@Tugsan\_Tuncer](https://discuss.elastic.co/u/Tugsan_Tuncer)\ **Replies:** 0\ **Last updated:** [February 15, 2021, 7:55pm UTC](https://discuss.elastic.co/t/logstash-not-sending-the-logs-to-elk-and-kibana/264357 "2021-02-15T19:55:40Z")
Hİ all, i am trying configure filebeats send the logs to kibana but I was not successful.my setup is on kubernetes. I tried In My Spring Boot Project.Please help me logstash.conf: input { beats { port =\> 5000 typ… --- ## [Geo\_point in logstash](https://discuss.elastic.co/t/geo-point-in-logstash/264240)
**Author:** [@MABN](https://discuss.elastic.co/u/MABN)\ **Replies:** 2\ **Last updated:** [February 15, 2021, 7:05pm UTC](https://discuss.elastic.co/t/geo-point-in-logstash/264240 "2021-02-15T19:05:45Z")
Hi I used syslog to send log to logstash ( logstash 7.10) I used this grok in logstash.conf ##################### geoip { source =\> "source.ip" target =\> "source.geo" } geoip { source =\> "destination.ip" target… --- ## [Ingest-converter not mapping some fields properly?](https://discuss.elastic.co/t/ingest-converter-not-mapping-some-fields-properly/263644)
**Author:** [@Daniel314](https://discuss.elastic.co/u/Daniel314)\ **Replies:** 1\ **Last updated:** [February 15, 2021, 6:54pm UTC](https://discuss.elastic.co/t/ingest-converter-not-mapping-some-fields-properly/263644 "2021-02-15T18:54:45Z")
Background: at my organization, we heavily use Logstash and kafka to deal with buffering and spikes in the traffic flowing to ElasticSearch, as well as scaling processing horizontally. Because of this, the "ingest" pipe… --- ## [Logstash fails to restart after the queue size is full](https://discuss.elastic.co/t/logstash-fails-to-restart-after-the-queue-size-is-full/263278)
**Author:** [@Animesh\_Agarwal](https://discuss.elastic.co/u/Animesh_Agarwal)\ **Replies:** 1\ **Last updated:** [February 15, 2021, 4:56pm UTC](https://discuss.elastic.co/t/logstash-fails-to-restart-after-the-queue-size-is-full/263278 "2021-02-15T16:56:56Z")
Hi Team, We have enabled persistent queue on our logstash servers. queue.path is /var/lib/logstash queue.max\_bytes=30gb The logstash queue was full and I had restarted logstash service. Logstash fails to restart with… --- ## [Data migration from mongodb to elasticsearch](https://discuss.elastic.co/t/data-migration-from-mongodb-to-elasticsearch/264311)
**Author:** [@Pankaj1](https://discuss.elastic.co/u/Pankaj1)\ **Replies:** 0\ **Last updated:** [February 15, 2021, 12:42pm UTC](https://discuss.elastic.co/t/data-migration-from-mongodb-to-elasticsearch/264311 "2021-02-15T12:42:27Z")
I am doing data migration from mongodb to elasticsearch by using logstash . I am using jdbc mongodb jar as a plugin . I have aggregation query to get the records from mongodb and insert into elasticsearch . The issues… --- ## [Logstash dameon does not create index .... but manually running Logstash does](https://discuss.elastic.co/t/logstash-dameon-does-not-create-index-but-manually-running-logstash-does/262691)
**Author:** [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Replies:** 2\ **Last updated:** [February 15, 2021, 9:06am UTC](https://discuss.elastic.co/t/logstash-dameon-does-not-create-index-but-manually-running-logstash-does/262691 "2021-02-15T09:06:10Z")
Hello I believe this is more oriented to logstash but If I run as root: /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/01-winlogbeat-somefile.conf Kibana sees the index file, so it is created and as long as… --- ## [Logstash output - set output file name](https://discuss.elastic.co/t/logstash-output-set-output-file-name/264023)
**Author:** [@liavsh](https://discuss.elastic.co/u/liavsh)\ **Replies:** 3\ **Last updated:** [February 14, 2021, 5:46pm UTC](https://discuss.elastic.co/t/logstash-output-set-output-file-name/264023 "2021-02-14T17:46:03Z")
Hi I'm new to logstash... I'm using logstash to stream logfiles from AWS MSK Kafka to AWS S3 bucket. I FAIL setting the output filename inside the bucket. I was able to set the folder that will hold the output file. … --- ## [\_grokparsefailure with kv](https://discuss.elastic.co/t/grokparsefailure-with-kv/264195)
**Author:** [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\ **Replies:** 1\ **Last updated:** [February 13, 2021, 12:12am UTC](https://discuss.elastic.co/t/grokparsefailure-with-kv/264195 "2021-02-13T00:12:29Z")
Hi, I am using kv { } filter but output in Kibana have tag \_grokparsefailure, why? Output looks like is fine, without errors however still have this tag, when I even don't have grok (only input udp, filter kv and mutat… --- ## [Logstash condition to check if a field value starts with another field's value](https://discuss.elastic.co/t/logstash-condition-to-check-if-a-field-value-starts-with-another-fields-value/264168)
**Author:** [@mattinVS](https://discuss.elastic.co/u/mattinVS)\ **Replies:** 2\ **Last updated:** [February 12, 2021, 10:49pm UTC](https://discuss.elastic.co/t/logstash-condition-to-check-if-a-field-value-starts-with-another-fields-value/264168 "2021-02-12T22:49:28Z")
Hi, I'd like to know how I can check in an if-condition in Logstash, is a field's value (in the example below the field is named "complete") starts with the value of another field (in the example below the field is name… --- ## [Push Index to Cloud](https://discuss.elastic.co/t/push-index-to-cloud/264178)
**Author:** [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\ **Replies:** 2\ **Last updated:** [February 12, 2021, 9:54pm UTC](https://discuss.elastic.co/t/push-index-to-cloud/264178 "2021-02-12T21:54:19Z")
I am doing a proof-of-concept using my first instance of cloud. I have signed up for the trial but am having difficulty figuring out the best way to move my test index from my local server to the cloud instance. I unde… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=253) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=255)