# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=231 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 232 --- ## [Unable to connect logstash with a secured cluster](https://discuss.elastic.co/t/unable-to-connect-logstash-with-a-secured-cluster/272218)
**Author:** [@marone](https://discuss.elastic.co/u/marone)\ **Replies:** 4\ **Last updated:** [May 11, 2021, 4:58pm UTC](https://discuss.elastic.co/t/unable-to-connect-logstash-with-a-secured-cluster/272218 "2021-05-11T16:58:10Z")
Hi I'm using ELK 7.12.0 and I'm struggling to connect my logstash with my secured cluster, without xpack logstash send data to elasticsearch using twitter plugin, but when I enable xpack for elasticsearch and try to conf… --- ## [Recalculate times?](https://discuss.elastic.co/t/recalculate-times/272680)
**Author:** [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Replies:** 2\ **Last updated:** [May 11, 2021, 4:03pm UTC](https://discuss.elastic.co/t/recalculate-times/272680 "2021-05-11T16:03:00Z")
Hi, Due to a number of reasons we have servers logging with UTC en servers logging with timestamps in CEST. In kibana/es I would like to see everything in CESt to easily connect the dots with events in clientdevices. Is… --- ## [Parse very complicated logs with logstash (braces and random keys names)](https://discuss.elastic.co/t/parse-very-complicated-logs-with-logstash-braces-and-random-keys-names/272715)
**Author:** [@gmbynkrkrqz](https://discuss.elastic.co/u/gmbynkrkrqz)\ **Replies:** 1\ **Last updated:** [May 11, 2021, 3:59pm UTC](https://discuss.elastic.co/t/parse-very-complicated-logs-with-logstash-braces-and-random-keys-names/272715 "2021-05-11T15:59:30Z")
Hello, I'm trying to parse very complicated logs as below but it's very hard. Could someone help me ? 2020-05-29T16:28:49.051146+02:00 machinehostname APP\_NAME\[31161\] @cee: {"category": "USER", "context": {"authMode": … --- ## [ERROR: Pipelines YAML file must contain an array of pipeline configs. Found "Hash"](https://discuss.elastic.co/t/error-pipelines-yaml-file-must-contain-an-array-of-pipeline-configs-found-hash/272708)
**Author:** [@andcur](https://discuss.elastic.co/u/andcur)\ **Replies:** 2\ **Last updated:** [May 11, 2021, 3:33pm UTC](https://discuss.elastic.co/t/error-pipelines-yaml-file-must-contain-an-array-of-pipeline-configs-found-hash/272708 "2021-05-11T15:33:10Z")
Windows server: I have the following pipeline.yml file -pipeline.id: policylocalexposurehub path.config: "config/logstash-sample.conf" pipeline.workers: 1 error ERROR: Pipelines YAML file must contain an array of pip… --- ## [\_id like var in output email](https://discuss.elastic.co/t/id-like-var-in-output-email/272696)
**Author:** [@cassiopee](https://discuss.elastic.co/u/cassiopee)\ **Replies:** 4\ **Last updated:** [May 11, 2021, 2:09pm UTC](https://discuss.elastic.co/t/id-like-var-in-output-email/272696 "2021-05-11T14:09:46Z")
hi everyone, I would use \_id (metadata) like variable in my output mail. I make my output on this way : \`output { elasticsearch { hosts =\> \[ "https://192.168.1.160:9200" \] ssl =\> true ssl\_certificate\_ve… --- ## [Storage usage when maintaining data over 2 seperate indexes](https://discuss.elastic.co/t/storage-usage-when-maintaining-data-over-2-seperate-indexes/272257)
**Author:** [@Koren\_Molcho](https://discuss.elastic.co/u/Koren_Molcho)\ **Replies:** 1\ **Last updated:** [May 11, 2021, 11:54am UTC](https://discuss.elastic.co/t/storage-usage-when-maintaining-data-over-2-seperate-indexes/272257 "2021-05-11T11:54:02Z")
Hello, I am looking into a solution for producing a dedup version of an existing index, similar to the solution offered here: Little Logstash Lessons: Handling Duplicates | Elastic Blog I have an index containing all … --- ## [Logstash AWS code 403](https://discuss.elastic.co/t/logstash-aws-code-403/272499)
**Author:** [@Roy\_Levy](https://discuss.elastic.co/u/Roy_Levy)\ **Replies:** 4\ **Last updated:** [May 11, 2021, 12:35am UTC](https://discuss.elastic.co/t/logstash-aws-code-403/272499 "2021-05-11T00:35:02Z")
Hey, I'm trying to push documents from local to elastic server in AWS, and when trying to do so I get 403 error and logstash keeps on trying to establish connection with the server like so: \[2021-05-09T11:09:52,707\]\[TR… --- ## [Logstash conf output to elasticsearch with document\_id set only indexes one record](https://discuss.elastic.co/t/logstash-conf-output-to-elasticsearch-with-document-id-set-only-indexes-one-record/272604)
**Author:** [@andcur](https://discuss.elastic.co/u/andcur)\ **Replies:** 1\ **Last updated:** [May 10, 2021, 5:41pm UTC](https://discuss.elastic.co/t/logstash-conf-output-to-elasticsearch-with-document-id-set-only-indexes-one-record/272604 "2021-05-10T17:41:37Z")
Hi, I am using logstash to populate data from SQL server to elastic. When I specify document\_id (as below) I only get one document in my index. If I do not specify that document\_id field should be taken from PolicyId,… --- ## [Logstash no processing all my logs from Filebeat](https://discuss.elastic.co/t/logstash-no-processing-all-my-logs-from-filebeat/272593)
**Author:** [@calanon](https://discuss.elastic.co/u/calanon)\ **Replies:** 1\ **Last updated:** [May 10, 2021, 4:05pm UTC](https://discuss.elastic.co/t/logstash-no-processing-all-my-logs-from-filebeat/272593 "2021-05-10T16:05:20Z")
I have Filebeat transporting my logs to logstash and then to ES. The behaviour I have noticed is that some of my logs are processed and available in Kibana by not all. I see all the logs in the logstash debug log with … --- ## [Grok works in debugger but not in Logstash](https://discuss.elastic.co/t/grok-works-in-debugger-but-not-in-logstash/272509)
**Author:** [@therealjack404](https://discuss.elastic.co/u/therealjack404)\ **Replies:** 3\ **Last updated:** [May 10, 2021, 1:41pm UTC](https://discuss.elastic.co/t/grok-works-in-debugger-but-not-in-logstash/272509 "2021-05-10T13:41:39Z")
Hi. I am struggling to write a grok pattern for Modsecurity Logs from the apache error log. I have included a sample log and the grok pattern. I isolated the timestamp creation which works on its own. However the rest wo… --- ## [Logstash cannot find logstash.yml directory](https://discuss.elastic.co/t/logstash-cannot-find-logstash-yml-directory/272288)
**Author:** [@almotasim90](https://discuss.elastic.co/u/almotasim90)\ **Replies:** 4\ **Last updated:** [May 10, 2021, 10:53am UTC](https://discuss.elastic.co/t/logstash-cannot-find-logstash-yml-directory/272288 "2021-05-10T10:53:13Z")
Hi, I am running ELK stack in debian 10. I removed logstash with apt command and removed /etc/logstash , /var/lib/logstash directories. Now I am trying to install logstash again, but it does not install all confugrati… --- ## [Iterate through data in field logstash](https://discuss.elastic.co/t/iterate-through-data-in-field-logstash/272518)
**Author:** [@lusynda](https://discuss.elastic.co/u/lusynda)\ **Replies:** 1\ **Last updated:** [May 9, 2021, 9:33pm UTC](https://discuss.elastic.co/t/iterate-through-data-in-field-logstash/272518 "2021-05-09T21:33:45Z")
Hi all I have a field like this field: 10, 20, 30, 40 now that i want to make each of the data in the field to be parse like this: field.1: 10 field.2: 20 ... and then use the translate plugin on each of these field… --- ## [Send all Syslog messages that don't match output to another index?](https://discuss.elastic.co/t/send-all-syslog-messages-that-dont-match-output-to-another-index/272418)
**Author:** [@jeromeat](https://discuss.elastic.co/u/jeromeat)\ **Replies:** 2\ **Last updated:** [May 9, 2021, 9:22pm UTC](https://discuss.elastic.co/t/send-all-syslog-messages-that-dont-match-output-to-another-index/272418 "2021-05-09T21:22:02Z")
Noob here, so don't judge too hard. :wink: However, I am wanting to find out if it's possible to have an output as such that's looking for specific tags and sending those messages to the windows\* index but if they don'… --- ## [Logstash ambiguous port definition](https://discuss.elastic.co/t/logstash-ambiguous-port-definition/272494)
**Author:** [@Roy\_Levy](https://discuss.elastic.co/u/Roy_Levy)\ **Replies:** 2\ **Last updated:** [May 9, 2021, 2:15pm UTC](https://discuss.elastic.co/t/logstash-ambiguous-port-definition/272494 "2021-05-09T14:15:51Z")
Hey, I am using Logstash in order to push documents to elasticsearch, and when trying to run logstash I'm getting the error: \[2021-05-09T08:45:57,532\]\[DEBUG\]\[logstash.javapipeline \]\[main\] Pipeline terminated by work… --- ## [Goes through data in a field to parse](https://discuss.elastic.co/t/goes-through-data-in-a-field-to-parse/272429)
**Author:** [@lusynda](https://discuss.elastic.co/u/lusynda)\ **Replies:** 2\ **Last updated:** [May 9, 2021, 1:31pm UTC](https://discuss.elastic.co/t/goes-through-data-in-a-field-to-parse/272429 "2021-05-09T13:31:53Z")
Hi all I have some problems with parse data in logstash i have a field with value like this: testValue: 1,2,3,4 and i have an yaml file to map data like this: 1: some1 2: some2 3: some3 i want to parse the data to … --- ## [LogStash update fields received](https://discuss.elastic.co/t/logstash-update-fields-received/272480)
**Author:** [@Hamza\_El\_Aouane](https://discuss.elastic.co/u/Hamza_El_Aouane)\ **Replies:** 0\ **Last updated:** [May 8, 2021, 10:14pm UTC](https://discuss.elastic.co/t/logstash-update-fields-received/272480 "2021-05-08T22:14:48Z")
Hello guys. I hope everyone is doing well. I am a bit confused about ELK, specifically about LogStash and I hope you can help me to clarify this issue. A couple of weeks ago, one of our clients started forwarding syslo… --- ## [ELK Kafka integration](https://discuss.elastic.co/t/elk-kafka-integration/272435)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 4\ **Last updated:** [May 7, 2021, 8:28pm UTC](https://discuss.elastic.co/t/elk-kafka-integration/272435 "2021-05-07T20:28:28Z")
Hi All, Does ELK 7.6.2 support reading data directly from Kafka? --- ## [Is globbing not supported in S3 input plugin prefix?](https://discuss.elastic.co/t/is-globbing-not-supported-in-s3-input-plugin-prefix/272443)
**Author:** [@meatwad](https://discuss.elastic.co/u/meatwad)\ **Replies:** 1\ **Last updated:** [May 7, 2021, 8:26pm UTC](https://discuss.elastic.co/t/is-globbing-not-supported-in-s3-input-plugin-prefix/272443 "2021-05-07T20:26:35Z")
I've seen some mention of globbing not being supported in the S3 input plugin's prefix setting in threads from years ago but I just wanted to check in and see if that's still the case. At least it seems to be based on m… --- ## [Azure event hub input](https://discuss.elastic.co/t/azure-event-hub-input/271415)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 2\ **Last updated:** [May 7, 2021, 12:03pm UTC](https://discuss.elastic.co/t/azure-event-hub-input/271415 "2021-05-07T12:03:53Z")
I have an azure event hub i want to use the azure event input plugin to grab the data....I’ve gotten Logstash to connect in to the hub.. but im getting a strange error when the conf is running.... hoping someone has an i… --- ## [Unstable indexing rate using S3 input plugin](https://discuss.elastic.co/t/unstable-indexing-rate-using-s3-input-plugin/272410)
**Author:** [@hrochefort](https://discuss.elastic.co/u/hrochefort)\ **Replies:** 0\ **Last updated:** [May 7, 2021, 11:09am UTC](https://discuss.elastic.co/t/unstable-indexing-rate-using-s3-input-plugin/272410 "2021-05-07T11:09:18Z")
Hi, I am using logstash to ingest json files from s3 to elasticsearch and our current indexing speed does not match my needs. I noticed that the indexing rate dips regularly to 0. Sometimes for up to 5 minutes. And thi… --- ## [Filebeat no connection could be made because the target machine actively refused it](https://discuss.elastic.co/t/filebeat-no-connection-could-be-made-because-the-target-machine-actively-refused-it/272302)
**Author:** [@Rashmika\_Gamage](https://discuss.elastic.co/u/Rashmika_Gamage)\ **Replies:** 9\ **Last updated:** [May 7, 2021, 8:55am UTC](https://discuss.elastic.co/t/filebeat-no-connection-could-be-made-because-the-target-machine-actively-refused-it/272302 "2021-05-07T08:55:23Z")
Hi all, I am trying to read logs from beats to logstash in the same machine. I am getting an error no connection could be made because the target machine actively refused it My logstash conf file input { beats … --- ## [Help with grok pattern](https://discuss.elastic.co/t/help-with-grok-pattern/272351)
**Author:** [@Rahul\_A](https://discuss.elastic.co/u/Rahul_A)\ **Replies:** 1\ **Last updated:** [May 7, 2021, 6:32am UTC](https://discuss.elastic.co/t/help-with-grok-pattern/272351 "2021-05-07T06:32:36Z")
Hi friends, I'm a newbie with grok filter and need help parsing the log message. Here is my log message: 2016-07-11T23:56:42.000+00:00 INFO ALERT|ECE002|5 Error with transaction for session -464410bf-37bf-475a-afc0-49… --- ## [Logstash reading syslog from filebeat](https://discuss.elastic.co/t/logstash-reading-syslog-from-filebeat/272354)
**Author:** [@lachlan.simpson](https://discuss.elastic.co/u/lachlan.simpson)\ **Replies:** 1\ **Last updated:** [May 7, 2021, 4:22am UTC](https://discuss.elastic.co/t/logstash-reading-syslog-from-filebeat/272354 "2021-05-07T04:22:11Z")
Using RHEL8, ELK 7.12, I have filebeat 7.12 set up on a RH7 workstation, configured with the syslog plugin and delivering data to logstash. The data is coming into logstash, but it's raw - the message field isn't being … --- ## [\[WARN \] 2021-05-06 23:01:18.054 \[\[marketdata\_fix\_pipeline\]\>worker0\] split - Only String and Array types are splittable. field:\[layers\]\[fix\] is of type = Hash](https://discuss.elastic.co/t/warn-2021-05-06-2318-054-marketdata-fix-pipeline-worker0-split-only-string-and-array-types-are-splittable-field-layers-fix-is-of-type-hash/272344)
**Author:** [@wmei](https://discuss.elastic.co/u/wmei)\ **Replies:** 3\ **Last updated:** [May 7, 2021, 12:16am UTC](https://discuss.elastic.co/t/warn-2021-05-06-2318-054-marketdata-fix-pipeline-worker0-split-only-string-and-array-types-are-splittable-field-layers-fix-is-of-type-hash/272344 "2021-05-07T00:16:08Z")
Hi, I am getting this warning message and I believe it is because some of the events do not have a \[layers\]\[fix\]. \[WARN \] 2021-05-06 23:01:18.054 \[\[marketdata\_fix\_pipeline\]\>worker0\] split - Only String and Array types … --- ## [Logstash Elasticsearch Input](https://discuss.elastic.co/t/logstash-elasticsearch-input/272336)
**Author:** [@vishakh](https://discuss.elastic.co/u/vishakh)\ **Replies:** 3\ **Last updated:** [May 6, 2021, 9:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input/272336 "2021-05-06T21:52:21Z")
I'm trying to extract (or) export logs from Elasticsearch via Logstash I would like to write the extracted logs to a file input { elasticsearch { hosts =\> "elasticsearch.domain.com:9200" query =\> '{ "size": 5… --- ## [Create one custom document if no data is retrive from the JDBC input](https://discuss.elastic.co/t/create-one-custom-document-if-no-data-is-retrive-from-the-jdbc-input/272326)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [May 6, 2021, 6:12pm UTC](https://discuss.elastic.co/t/create-one-custom-document-if-no-data-is-retrive-from-the-jdbc-input/272326 "2021-05-06T18:12:39Z")
Hi, Its is posible to index one custom document if no data id retrieve from a database? how can logstash do this? --- ## [Parsing nested json not happening properly](https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 2\ **Last updated:** [May 6, 2021, 3:20pm UTC](https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282 "2021-05-06T15:20:51Z")
Hi Folks, I am trying to parse modsecurity audit logs which are natively being logged in JSON format. However when I am using logstash to ingest in elastic stack those appears like below and actual needed fields are not… --- ## [How to remove original message field after parsing using json filter](https://discuss.elastic.co/t/how-to-remove-original-message-field-after-parsing-using-json-filter/272306)
**Author:** [@Sandeep4](https://discuss.elastic.co/u/Sandeep4)\ **Replies:** 1\ **Last updated:** [May 6, 2021, 2:51pm UTC](https://discuss.elastic.co/t/how-to-remove-original-message-field-after-parsing-using-json-filter/272306 "2021-05-06T14:51:50Z")
First try:- filter { mutate { gsub =\> \[ "message", "^\\w+\`string\`", "" \] } json { source =\> "message" remove\_field =\> "message" } } Second try:- filter { mutate { gsub =\> \[ "message", "^\\w+\`string\`",… --- ## [JSON filter swallowing events with no errors](https://discuss.elastic.co/t/json-filter-swallowing-events-with-no-errors/272221)
**Author:** [@brian\_m](https://discuss.elastic.co/u/brian_m)\ **Replies:** 2\ **Last updated:** [May 6, 2021, 1:14pm UTC](https://discuss.elastic.co/t/json-filter-swallowing-events-with-no-errors/272221 "2021-05-06T13:14:45Z")
Hello, I'm trying to do some testing on ingesting JSON data, and having trouble with the JSON filter. My pipeline configuration is below (output filter has been modified to remove credentials and IPs) input { http {}… --- ## [Variable content from file](https://discuss.elastic.co/t/variable-content-from-file/272296)
**Author:** [@emilsvil](https://discuss.elastic.co/u/emilsvil)\ **Replies:** 0\ **Last updated:** [May 6, 2021, 12:50pm UTC](https://discuss.elastic.co/t/variable-content-from-file/272296 "2021-05-06T12:50:46Z")
Hi. Im using cidr filter to check if an IP si public or private. The list of cidrs to check is now hardcoded in the filter but I need to read it from a file or using a meta-variable loaded at runtime. cidr { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=230) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=232)