# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=284 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 285 --- ## [Multiple Logstash instances as Kafka inputs](https://discuss.elastic.co/t/multiple-logstash-instances-as-kafka-inputs/254784)
**Author:** [@Fran\_Raknic](https://discuss.elastic.co/u/Fran_Raknic)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 3:44pm UTC](https://discuss.elastic.co/t/multiple-logstash-instances-as-kafka-inputs/254784 "2020-11-09T15:44:24Z")
I am having trouble in making multiple Logstash instances read from a single Kafka topic. The Logstash instances are all on the same server, together with Kafka and Zookeeper. When I check the topic with kafka-console-c… --- ## [Pipeline aborted due to EOF in aggregate maps file](https://discuss.elastic.co/t/pipeline-aborted-due-to-eof-in-aggregate-maps-file/254783)
**Author:** [@lens](https://discuss.elastic.co/u/lens)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 3:42pm UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-eof-in-aggregate-maps-file/254783 "2020-11-09T15:42:49Z")
We have had several cases recently of logstash not being able to restart all of its configured pipelines. We are using the aggregation filter and it should write out any aggregation maps that are in-progress when logsta… --- ## [Monitoring logs from both inside the container and ELK](https://discuss.elastic.co/t/monitoring-logs-from-both-inside-the-container-and-elk/254772)
**Author:** [@nyquillus](https://discuss.elastic.co/u/nyquillus)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 2:49pm UTC](https://discuss.elastic.co/t/monitoring-logs-from-both-inside-the-container-and-elk/254772 "2020-11-09T14:49:58Z")
Hi, i have recently added a gelf input option to my elastic stack for docker container log monitoring. When i do this i can monitor logs from elk just fine but can't see them from inside the container. Is there a way to … --- ## [Logstash pipelines does not take filter](https://discuss.elastic.co/t/logstash-pipelines-does-not-take-filter/254769)
**Author:** [@Andex](https://discuss.elastic.co/u/Andex)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 2:34pm UTC](https://discuss.elastic.co/t/logstash-pipelines-does-not-take-filter/254769 "2020-11-09T14:34:48Z")
Hi, i have filebeat configured to send jboss log on Logstash. On logstash i have input.conf which gets me logs from filebeat, an output.conf to send log on elasticsearch and a jboss-filter.conf to parse the log. I see… --- ## [Grok regex too long](https://discuss.elastic.co/t/grok-regex-too-long/254763)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 1:40pm UTC](https://discuss.elastic.co/t/grok-regex-too-long/254763 "2020-11-09T13:40:02Z")
Hello there. I am trying to apply a really long regex to a field in a log. I keep getting an error and logstash doesn't even start. The regex is made up of several smaller regexes connected with |, like this: pattern\_de… --- ## [In logstash, How can we check if a data exists another elastic search index?](https://discuss.elastic.co/t/in-logstash-how-can-we-check-if-a-data-exists-another-elastic-search-index/254709)
**Author:** [@meghavinam](https://discuss.elastic.co/u/meghavinam)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 8:26am UTC](https://discuss.elastic.co/t/in-logstash-how-can-we-check-if-a-data-exists-another-elastic-search-index/254709 "2020-11-09T08:26:47Z")
I want to sync set of data from mysql to elastic search. That time i need to remove if same data exists in another index --- ## [Json file from logtash using fast API](https://discuss.elastic.co/t/json-file-from-logtash-using-fast-api/254716)
**Author:** [@abhishek1310](https://discuss.elastic.co/u/abhishek1310)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 8:18am UTC](https://discuss.elastic.co/t/json-file-from-logtash-using-fast-api/254716 "2020-11-09T08:18:42Z")
Can any one suggest me the approach to receive a json file through fast API from logtash --- ## [Logstash config error Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"{\\", \\",\\", \\"\]\\" at line 10, column 83](https://discuss.elastic.co/t/logstash-config-error-expected-one-of-t-r-n-at-line-10-column-83/254619)
**Author:** [@Tejas\_Bhosale](https://discuss.elastic.co/u/Tejas_Bhosale)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 7:39am UTC](https://discuss.elastic.co/t/logstash-config-error-expected-one-of-t-r-n-at-line-10-column-83/254619 "2020-11-09T07:39:52Z")
Hello, I am using Logstash to ingest nginx into ElasticSearch. I am using a Linux Machine but i am getting error when try to start logstatsh service elk logstash\[1233\]: \[2020-11-07T20:25:03,741\]\[ERROR\]\[logstash.agent … --- ## [How to make a automaticaly renew token authorization in file logstash configuration](https://discuss.elastic.co/t/how-to-make-a-automaticaly-renew-token-authorization-in-file-logstash-configuration/254712)
**Author:** [@salma\_widiarti](https://discuss.elastic.co/u/salma_widiarti)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 6:46am UTC](https://discuss.elastic.co/t/how-to-make-a-automaticaly-renew-token-authorization-in-file-logstash-configuration/254712 "2020-11-09T06:46:38Z")
Hi all, Do you know how to make a automaticaly renew token authorization in file logstash configuration? Thanks.. --- ## [Beginner question: Skipping the filtering of id into \[@metadata\]\[\_id\]](https://discuss.elastic.co/t/beginner-question-skipping-the-filtering-of-id-into-metadata-id/254508)
**Author:** [@pyrovoice](https://discuss.elastic.co/u/pyrovoice)\ **Replies:** 0\ **Last updated:** [November 6, 2020, 9:16am UTC](https://discuss.elastic.co/t/beginner-question-skipping-the-filtering-of-id-into-metadata-id/254508 "2020-11-06T09:16:53Z")
I followed the following tutorial: https://www.elastic.co/blog/how-to-keep-elasticsearch-synchronized-with-a-relational-database-using-logstash The configuration file for the pipeline contains the following: input … --- ## [Configure two outputs in Logstash](https://discuss.elastic.co/t/configure-two-outputs-in-logstash/254658)
**Author:** [@Falikou1](https://discuss.elastic.co/u/Falikou1)\ **Replies:** 3\ **Last updated:** [November 8, 2020, 10:19am UTC](https://discuss.elastic.co/t/configure-two-outputs-in-logstash/254658 "2020-11-08T10:19:56Z")
Logstash is a log aggregator that collects data from various input sources, performs various transformations and enhancements, and then sends the data to various supported output destinations. Is it possible to configur… --- ## [Huge dictionary in logstash translate filter](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857)
**Author:** [@heric](https://discuss.elastic.co/u/heric)\ **Replies:** 11\ **Last updated:** [November 8, 2020, 8:41am UTC](https://discuss.elastic.co/t/huge-dictionary-in-logstash-translate-filter/253857 "2020-11-08T08:41:25Z")
I have huge dictionary file to be used in logstash translate filter , around 180k entries. I tried to reduce it to around 80k but still always getting error when starting the pipeline , as below : \[2020-10-30T22:11:47,1… --- ## [Debugging geoip plugin - Ruby @logger statements appear but not Java logger](https://discuss.elastic.co/t/debugging-geoip-plugin-ruby-logger-statements-appear-but-not-java-logger/254644)
**Author:** [@hukel](https://discuss.elastic.co/u/hukel)\ **Replies:** 2\ **Last updated:** [November 8, 2020, 12:15am UTC](https://discuss.elastic.co/t/debugging-geoip-plugin-ruby-logger-statements-appear-but-not-java-logger/254644 "2020-11-08T00:15:56Z")
I know the geoip plugin has some limitations when handling private IP addresses so I am not trying to solve that problem. I am trying to get debug output from the plugin to explain where it is giving up when processi… --- ## [\[logstash-output-mongodb\] Drop collection before insert documents](https://discuss.elastic.co/t/logstash-output-mongodb-drop-collection-before-insert-documents/254631)
**Author:** [@brunorafaeI](https://discuss.elastic.co/u/brunorafaeI)\ **Replies:** 1\ **Last updated:** [November 7, 2020, 9:06pm UTC](https://discuss.elastic.co/t/logstash-output-mongodb-drop-collection-before-insert-documents/254631 "2020-11-07T21:06:41Z")
Hi there, First of all, I would like to introduce my sinceres apologies, my ingles is not very well. I would to know if there are any way to delete the existing collection before inserting the output lines to mongodb? e… --- ## [Logstash not adding new field tag](https://discuss.elastic.co/t/logstash-not-adding-new-field-tag/254579)
**Author:** [@inuygoku](https://discuss.elastic.co/u/inuygoku)\ **Replies:** 2\ **Last updated:** [November 6, 2020, 9:04pm UTC](https://discuss.elastic.co/t/logstash-not-adding-new-field-tag/254579 "2020-11-06T21:04:13Z")
I am trying to create a filter that will for event in my logs and add it to a field called "detected\_event". The filter I created is the following: filter { if "Scan completed" in \[logline\] { mutate { … --- ## [Recreate new field from old field](https://discuss.elastic.co/t/recreate-new-field-from-old-field/254567)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [November 6, 2020, 7:20pm UTC](https://discuss.elastic.co/t/recreate-new-field-from-old-field/254567 "2020-11-06T19:20:24Z")
I have used add\_filed, remove\_field, rename etc.. many time over on logstash now I have something different that I can't quite figure out. runnign nvidiabeat and it gives me following { "clocks":{"mem":405,"sm… --- ## [Remove a pattern from json nested fields](https://discuss.elastic.co/t/remove-a-pattern-from-json-nested-fields/254542)
**Author:** [@nilubkal](https://discuss.elastic.co/u/nilubkal)\ **Replies:** 1\ **Last updated:** [November 6, 2020, 3:42pm UTC](https://discuss.elastic.co/t/remove-a-pattern-from-json-nested-fields/254542 "2020-11-06T15:42:57Z")
Hey guys, i have the following json structure which i would like to rename from : json.detail.info.processed\_event.instance\_id json.detail.info.processed\_event.invalidations.metadata json.detail.info.p… --- ## [Logstash doesn't process some logs](https://discuss.elastic.co/t/logstash-doesnt-process-some-logs/254436)
**Author:** [@pab1it0](https://discuss.elastic.co/u/pab1it0)\ **Replies:** 3\ **Last updated:** [November 6, 2020, 3:07pm UTC](https://discuss.elastic.co/t/logstash-doesnt-process-some-logs/254436 "2020-11-06T15:07:29Z")
Hi, I've encountered an issue where only a subset of the logs are being processed via ELK stack: Digesting logs from k8s pods using kubernetes Hints based autodiscover with logstash. Pod annotations: annotations: … --- ## [Logstash shutdown after startup](https://discuss.elastic.co/t/logstash-shutdown-after-startup/254530)
**Author:** [@boernd](https://discuss.elastic.co/u/boernd)\ **Replies:** 0\ **Last updated:** [November 6, 2020, 1:22pm UTC](https://discuss.elastic.co/t/logstash-shutdown-after-startup/254530 "2020-11-06T13:22:47Z")
Hi, Version: 7.8.1 Plattform: OpenShift 3.11 Sometimes during Logstash pod startups it happens that the container restarts after startup. We use the monitoring feature and the logs indicate that there is some problem … --- ## [Conditionals with nested fields in logstash not working](https://discuss.elastic.co/t/conditionals-with-nested-fields-in-logstash-not-working/254424)
**Author:** [@darlin](https://discuss.elastic.co/u/darlin)\ **Replies:** 2\ **Last updated:** [November 6, 2020, 11:50am UTC](https://discuss.elastic.co/t/conditionals-with-nested-fields-in-logstash-not-working/254424 "2020-11-06T11:50:33Z")
Hi, somehow when we use the input http plugin afterwards the conditionals with nested fields are not working. I've already tried with a non nested field and then it is working. With the below configuration the request … --- ## [Passing time parameter to logstash-jdbc-input](https://discuss.elastic.co/t/passing-time-parameter-to-logstash-jdbc-input/254487)
**Author:** [@blukit](https://discuss.elastic.co/u/blukit)\ **Replies:** 2\ **Last updated:** [November 6, 2020, 8:44am UTC](https://discuss.elastic.co/t/passing-time-parameter-to-logstash-jdbc-input/254487 "2020-11-06T08:44:48Z")
Hi guys... Noobs here, so be gentle:) Is it possible to pass time parameter from top-right kibana web UI into logstash-jdbc-input statement? Passing something like: startTime-endTime, or last 3 hours. Thanks in advan… --- ## [Visualize a value calculated from the per-bucket sum of one field divided by the per-bucket sum of another?](https://discuss.elastic.co/t/visualize-a-value-calculated-from-the-per-bucket-sum-of-one-field-divided-by-the-per-bucket-sum-of-another/254359)
**Author:** [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Replies:** 3\ **Last updated:** [November 6, 2020, 4:17am UTC](https://discuss.elastic.co/t/visualize-a-value-calculated-from-the-per-bucket-sum-of-one-field-divided-by-the-per-bucket-sum-of-another/254359 "2020-11-06T04:17:49Z")
I have data in JSON Lines format that contains statistics about the tasks performed by a system over an interval of time. (The specific definition of "task" here is not important.) Each line of JSON Lines contains, amon… --- ## [Add new fields](https://discuss.elastic.co/t/add-new-fields/254017)
**Author:** [@Sakthivanan](https://discuss.elastic.co/u/Sakthivanan)\ **Replies:** 1\ **Last updated:** [November 6, 2020, 4:27am UTC](https://discuss.elastic.co/t/add-new-fields/254017 "2020-11-06T04:27:58Z")
\*\*I'm trying to add new hardcoded field in logstash filter:\*\* My logstash config: input { elasticsearch { hosts =\> "http://localhost:9200" index =\> "ccee\_mathematics\_academic\_indicator\_2018"… --- ## [Logstash: manipulate json data](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198)
**Author:** [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\ **Replies:** 7\ **Last updated:** [November 6, 2020, 1:50am UTC](https://discuss.elastic.co/t/logstash-manipulate-json-data/254198 "2020-11-06T01:50:36Z")
Hi, i retrieved data from ES and would like to add/modify the json data before index it. My json data looks like below: { "john" : { "enabled" : true, "roles" : \[ "developer" \], "rules" : { … --- ## [Cron in http\_poller still triggering after changing the pipeline configuration](https://discuss.elastic.co/t/cron-in-http-poller-still-triggering-after-changing-the-pipeline-configuration/254367)
**Author:** [@fbenaven](https://discuss.elastic.co/u/fbenaven)\ **Replies:** 5\ **Last updated:** [November 5, 2020, 10:34pm UTC](https://discuss.elastic.co/t/cron-in-http-poller-still-triggering-after-changing-the-pipeline-configuration/254367 "2020-11-05T22:34:46Z")
Hi, We have a pipeline configuration using some http\_poller as inputs. After changing the config from "cron" format (at '15 of all hours) to "every" (hourly basis), we see that the poller triggers hourly as intended bu… --- ## [Logstash could not index event to Elasticsearch](https://discuss.elastic.co/t/logstash-could-not-index-event-to-elasticsearch/254234)
**Author:** [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\ **Replies:** 3\ **Last updated:** [November 5, 2020, 9:44pm UTC](https://discuss.elastic.co/t/logstash-could-not-index-event-to-elasticsearch/254234 "2020-11-05T21:44:31Z")
Hello eveybody, I am using Logstash to parse my firewall logs, and in some logs I am getting that errors: \[2020-11-03T16:07:22,361\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\]\[f05eea78ee20871f68357cbab5919471405decdd5… --- ## [Logstash cloudwatch plugin error](https://discuss.elastic.co/t/logstash-cloudwatch-plugin-error/254337)
**Author:** [@deep9300](https://discuss.elastic.co/u/deep9300)\ **Replies:** 3\ **Last updated:** [November 5, 2020, 8:54pm UTC](https://discuss.elastic.co/t/logstash-cloudwatch-plugin-error/254337 "2020-11-05T20:54:52Z")
I'm trying to connect cloudwatch to logstash but get an error. this is my logstash conf file: input { cloudwatch { namespace =\> "AWS/Transfer" metrics =\> \[ "BytesIn", "BytesOut" \] filters … --- ## [How to add up values from list and send only that value to elasticserach](https://discuss.elastic.co/t/how-to-add-up-values-from-list-and-send-only-that-value-to-elasticserach/254441)
**Author:** [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Replies:** 1\ **Last updated:** [November 5, 2020, 6:22pm UTC](https://discuss.elastic.co/t/how-to-add-up-values-from-list-and-send-only-that-value-to-elasticserach/254441 "2020-11-05T18:22:38Z")
Hello, I want to write logstash script that use snmpwalk to get cpu values form network devices average them and send one field to elasticsearch. I guess I can do this in ruby plugin but I have so fare only expiernce in… --- ## [Random number of messages dropping on logstash import](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658)
**Author:** [@Paul\_Shriner](https://discuss.elastic.co/u/Paul_Shriner)\ **Replies:** 16\ **Last updated:** [November 5, 2020, 5:18pm UTC](https://discuss.elastic.co/t/random-number-of-messages-dropping-on-logstash-import/251658 "2020-11-05T17:18:57Z")
I have Logstash 7.6 running on Ubuntu 20.04. I am importing a pretty simple CSV file. The data in the file is separated by commas. See below: email,send\_time,engaged,last\_touch\_utc,score,frequency,open\_count,click\_… --- ## [GROKPARSE Failure and after Successful debug](https://discuss.elastic.co/t/grokparse-failure-and-after-successful-debug/254324)
**Author:** [@Vinay\_Kumar2](https://discuss.elastic.co/u/Vinay_Kumar2)\ **Replies:** 2\ **Last updated:** [November 5, 2020, 4:19pm UTC](https://discuss.elastic.co/t/grokparse-failure-and-after-successful-debug/254324 "2020-11-05T16:19:10Z")
Hi, i have used https://grokdebug.herokuapp.com/ to create grok match patterns and which are worked fine. However when checking with logstash, some of fields are not extracted and getting grokparse failure error. Samp… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=283) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=285)