# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=189 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 190 --- ## [Metric and drop filter not working together](https://discuss.elastic.co/t/metric-and-drop-filter-not-working-together/286380)
**Author:** [@Rakesh\_Partapsing](https://discuss.elastic.co/u/Rakesh_Partapsing)\ **Replies:** 2\ **Last updated:** [October 12, 2021, 12:07pm UTC](https://discuss.elastic.co/t/metric-and-drop-filter-not-working-together/286380 "2021-10-12T12:07:05Z")
Hi, I would like throttle events using the logstash-filter-throttle plugin. After that to create a logstash-filter-metric to push it to prometheus using graphite exporter. And finally drop the events that are marked by … --- ## [How to replace field values from json external file?](https://discuss.elastic.co/t/how-to-replace-field-values-from-json-external-file/286475)
**Author:** [@padamrai](https://discuss.elastic.co/u/padamrai)\ **Replies:** 7\ **Last updated:** [October 12, 2021, 12:01pm UTC](https://discuss.elastic.co/t/how-to-replace-field-values-from-json-external-file/286475 "2021-10-12T12:01:36Z")
I am getting messages from RabbitMQ with field name ResponseCode. Response code consist of short codes like , 00, 01, 02 and so on. I want to replace those values will full form. I have another json key:values pair file … --- ## [Logstash write to kafka's nonexistent topic, but no errors/warns in debug log](https://discuss.elastic.co/t/logstash-write-to-kafkas-nonexistent-topic-but-no-errors-warns-in-debug-log/286193)
**Author:** [@silbertmonaphia](https://discuss.elastic.co/u/silbertmonaphia)\ **Replies:** 12\ **Last updated:** [October 12, 2021, 3:50am UTC](https://discuss.elastic.co/t/logstash-write-to-kafkas-nonexistent-topic-but-no-errors-warns-in-debug-log/286193 "2021-10-12T03:50:11Z")
Description: Logstash writes event to kafka, but I forgot to create topic on kafka. And then I check logstash logs, but I can't find any error or warn messages in logs. Is this a bug of logstash 5.5 that logstash isn't … --- ## [Elapsed time filter is not calculating the time correctly](https://discuss.elastic.co/t/elapsed-time-filter-is-not-calculating-the-time-correctly/286418)
**Author:** [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)\ **Replies:** 2\ **Last updated:** [October 12, 2021, 3:12am UTC](https://discuss.elastic.co/t/elapsed-time-filter-is-not-calculating-the-time-correctly/286418 "2021-10-12T03:12:00Z")
Hi, I am using an elapsed time plugin, as per the documentation the elapsed-time is calculated in seconds. I am calculating the values on a unique field basis "seq". Here is what i am getting, In the snapshot the second… --- ## [Error for logstash.conf](https://discuss.elastic.co/t/error-for-logstash-conf/286400)
**Author:** [@Johnny\_Cash\_Cow](https://discuss.elastic.co/u/Johnny_Cash_Cow)\ **Replies:** 0\ **Last updated:** [October 11, 2021, 6:44pm UTC](https://discuss.elastic.co/t/error-for-logstash-conf/286400 "2021-10-11T18:44:47Z")
The only non-commented out line of my logstash.conf is: path.config: "C:/ELK/logstash-7.15.0-windows-x86\_64/logstash-7.15.0/config/logstash-sample.conf" My only non commented out section is of my pipeline.yml: - pip… --- ## [Logstash- Create different logfiles for different pipeline](https://discuss.elastic.co/t/logstash-create-different-logfiles-for-different-pipeline/286370)
**Author:** [@praffuln](https://discuss.elastic.co/u/praffuln)\ **Replies:** 1\ **Last updated:** [October 11, 2021, 5:14pm UTC](https://discuss.elastic.co/t/logstash-create-different-logfiles-for-different-pipeline/286370 "2021-10-11T17:14:52Z")
Hi, I am using logstash log4j2.properties configuration for log controlling e.g. log rotation, log preservation, controlling file name for logging, location of logs etc. Based on below discussion I used to configure lo… --- ## [DNS filter, how to save IP to a separate field](https://discuss.elastic.co/t/dns-filter-how-to-save-ip-to-a-separate-field/286391)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 1\ **Last updated:** [October 11, 2021, 5:06pm UTC](https://discuss.elastic.co/t/dns-filter-how-to-save-ip-to-a-separate-field/286391 "2021-10-11T17:06:28Z")
Hi, I'm using the DNS filter in my Logstash conf and I would like to store the IP address that the filter is resolving to a separate field. I tried using the add\_field parameter but its not working. I end up storing the… --- ## [Timestamp ISSUE](https://discuss.elastic.co/t/timestamp-issue/286170)
**Author:** [@frank\_rib](https://discuss.elastic.co/u/frank_rib)\ **Replies:** 3\ **Last updated:** [October 11, 2021, 4:54pm UTC](https://discuss.elastic.co/t/timestamp-issue/286170 "2021-10-11T16:54:01Z")
Hello every body, I have a stand allone architecture. Logs are sent from syslog\_ng server to logstash - \> Elasticsearch -\> kibana. I have a probleme with the timestamp as the capture below shows The logs are received … --- ## [How to read local0 files in unix using filebeat](https://discuss.elastic.co/t/how-to-read-local0-files-in-unix-using-filebeat/286181)
**Author:** [@muralikrishna](https://discuss.elastic.co/u/muralikrishna)\ **Replies:** 1\ **Last updated:** [October 11, 2021, 3:02pm UTC](https://discuss.elastic.co/t/how-to-read-local0-files-in-unix-using-filebeat/286181 "2021-10-11T15:02:28Z")
Hello Everyone, I have some applications writing the logs to local0.info & local4.info file in UNIX servers. I have to read the log content from above files using filebeat.yml and send the same to Logstash. We can't r… --- ## [How to move nested key value to root level and then rename elements root name?](https://discuss.elastic.co/t/how-to-move-nested-key-value-to-root-level-and-then-rename-elements-root-name/286059)
**Author:** [@mguttula](https://discuss.elastic.co/u/mguttula)\ **Replies:** 4\ **Last updated:** [October 11, 2021, 3:01pm UTC](https://discuss.elastic.co/t/how-to-move-nested-key-value-to-root-level-and-then-rename-elements-root-name/286059 "2021-10-11T15:01:26Z")
How to move nested key value to root level and then rename elements root name I have 4 core cpu and below is a sample of details for 2 cpu core from the source. { "\_source": { "cpu\_usage": { "sys/host-info/… --- ## [How to test the result of a query and take an action based on that result?](https://discuss.elastic.co/t/how-to-test-the-result-of-a-query-and-take-an-action-based-on-that-result/286362)
**Author:** [@mfilipelopes](https://discuss.elastic.co/u/mfilipelopes)\ **Replies:** 0\ **Last updated:** [October 11, 2021, 11:47am UTC](https://discuss.elastic.co/t/how-to-test-the-result-of-a-query-and-take-an-action-based-on-that-result/286362 "2021-10-11T11:47:58Z")
Hi. Is there a way to test the result of a query and take an action based on that result? I need to do the following: For new coming documents, make a search on a index X to see if that document already exists and if … --- ## [RFC-5424 log parsing](https://discuss.elastic.co/t/rfc-5424-log-parsing/286129)
**Author:** [@Sunflower](https://discuss.elastic.co/u/Sunflower)\ **Replies:** 6\ **Last updated:** [October 11, 2021, 10:59am UTC](https://discuss.elastic.co/t/rfc-5424-log-parsing/286129 "2021-10-11T10:59:10Z")
Hi, I have logs that I'm sending to Logstash from SentinelOne in an RFC-5424 format(this is the way they called it) that I wasn't sure how to handle. I began by dividing the fields using Grok and now my issue is how to… --- ## [Logstash change port](https://discuss.elastic.co/t/logstash-change-port/285895)
**Author:** [@mangeshs](https://discuss.elastic.co/u/mangeshs)\ **Replies:** 8\ **Last updated:** [October 11, 2021, 1:06am UTC](https://discuss.elastic.co/t/logstash-change-port/285895 "2021-10-11T01:06:54Z")
Logstash is running on 9600 by default and I want to run another logstash on same system Is there opion to change default port for logstash I don't want to use beats for this. beacuse I want to run two logstash with d… --- ## [Could not find logstash.yml file despite the file being located in the default directory](https://discuss.elastic.co/t/could-not-find-logstash-yml-file-despite-the-file-being-located-in-the-default-directory/286084)
**Author:** [@Johnny\_Cash\_Cow](https://discuss.elastic.co/u/Johnny_Cash_Cow)\ **Replies:** 1\ **Last updated:** [October 11, 2021, 12:52am UTC](https://discuss.elastic.co/t/could-not-find-logstash-yml-file-despite-the-file-being-located-in-the-default-directory/286084 "2021-10-11T00:52:42Z")
Good Evening, Here is the layout of my directory: Within /etc/logstash/ is the following files and directories: conf.d log4j2.properties logstash.yml startup.options jvm.options logstash-sample.conf piple… --- ## [How to create an array in logstash with already defined fields](https://discuss.elastic.co/t/how-to-create-an-array-in-logstash-with-already-defined-fields/286086)
**Author:** [@Juan\_David\_Jaramillo](https://discuss.elastic.co/u/Juan_David_Jaramillo)\ **Replies:** 0\ **Last updated:** [October 7, 2021, 2:35am UTC](https://discuss.elastic.co/t/how-to-create-an-array-in-logstash-with-already-defined-fields/286086 "2021-10-07T02:35:28Z")
Good, I have a doubt and is that I have some specific oids already named and I would like to know how I can do to concatenate them, that is to say to put them in an array or table, so that it contains me all the oids tha… --- ## [Measure volume of events we are consuming(Size of all input events arrived)](https://discuss.elastic.co/t/measure-volume-of-events-we-are-consuming-size-of-all-input-events-arrived/286113)
**Author:** [@Dhananjay\_vispute](https://discuss.elastic.co/u/Dhananjay_vispute)\ **Replies:** 0\ **Last updated:** [October 7, 2021, 9:46am UTC](https://discuss.elastic.co/t/measure-volume-of-events-we-are-consuming-size-of-all-input-events-arrived/286113 "2021-10-07T09:46:55Z")
I have added metric to count number events as input, dropped and arrived at output. I also want to measure size of all input events bytes. Is there any way we achieve this through metrics ? --- ## [How to parse two timestamp fields from one single log line message](https://discuss.elastic.co/t/how-to-parse-two-timestamp-fields-from-one-single-log-line-message/286282)
**Author:** [@Patr123](https://discuss.elastic.co/u/Patr123)\ **Replies:** 6\ **Last updated:** [October 10, 2021, 3:33pm UTC](https://discuss.elastic.co/t/how-to-parse-two-timestamp-fields-from-one-single-log-line-message/286282 "2021-10-10T15:33:30Z")
Hello All, I have log lines coming from K8's to logstash and they have two timestamps back to back. How can I parse those fields in let's say @timestamp field and timestamp2 field? My log lines look like: 2021-10-08 1… --- ## [Parsing nested json with logstash](https://discuss.elastic.co/t/parsing-nested-json-with-logstash/286287)
**Author:** [@diaztech](https://discuss.elastic.co/u/diaztech)\ **Replies:** 3\ **Last updated:** [October 10, 2021, 12:44am UTC](https://discuss.elastic.co/t/parsing-nested-json-with-logstash/286287 "2021-10-10T00:44:57Z")
I'm trying to import a collection from mongo with the following input input { mongodb { uri =\> "uripath" placeholder\_db\_dir =\> "../opt/logstash-mongodb/" placeholder\_db\_name =\> "logstash\_sqli… --- ## [Import text file to logstash](https://discuss.elastic.co/t/import-text-file-to-logstash/286303)
**Author:** [@quyennguyen](https://discuss.elastic.co/u/quyennguyen)\ **Replies:** 1\ **Last updated:** [October 9, 2021, 6:45pm UTC](https://discuss.elastic.co/t/import-text-file-to-logstash/286303 "2021-10-09T18:45:24Z")
I had some text files with formats like this. 10/09-22:09:34.989323 \[\*\*\] \[1:100000015:3\] "OR SQL Injection Detected" \[\*\*\] \[Priority: 0\] {TCP} 192.168.186.25:44748 -\> 192.168.186.162:80 10/09-22:09:34.989323 \[\*\*\] \[1:1000… --- ## [Append nested variable's value to a string](https://discuss.elastic.co/t/append-nested-variables-value-to-a-string/285470)
**Author:** [@Voolkan](https://discuss.elastic.co/u/Voolkan)\ **Replies:** 6\ **Last updated:** [October 8, 2021, 3:10pm UTC](https://discuss.elastic.co/t/append-nested-variables-value-to-a-string/285470 "2021-10-08T15:10:49Z")
Hello, I'm trying to setup a bit of a dynamic solution to the logstash config, and especially the json filter part. When setting the json part, I need to provide a source and a target fields. With source it's easy, howe… --- ## [Pipeline error {:pipeline\_id=\>"main", :exception=\>#\ **Author:** [@Busra\_Duygu](https://discuss.elastic.co/u/Busra_Duygu)\ **Replies:** 1\ **Last updated:** [October 8, 2021, 2:35pm UTC](https://discuss.elastic.co/t/pipeline-error-pipeline-id-main-exception-logstash-unexpected-character-code-58-expected-a-valid-value-number-string-array-object-true-false-or-null/286252 "2021-10-08T14:35:30Z") Hello friends, I want to add some data to a new index according to a query from an existing index in elasticsearch. But I am facing a problem while doing this, I would be very happy if you could help me. my conf file: i… --- ## [Mutate gsub - re-using regex](https://discuss.elastic.co/t/mutate-gsub-re-using-regex/286243)
**Author:** [@JimJ](https://discuss.elastic.co/u/JimJ)\ **Replies:** 2\ **Last updated:** [October 8, 2021, 2:13pm UTC](https://discuss.elastic.co/t/mutate-gsub-re-using-regex/286243 "2021-10-08T14:13:06Z")
Hi everybody, I would like to parttially remove the end of a string like that: mutate { gsub =\> \[ "fieldname", "(:\[0-5\]\[0-9\].\\d{3})\\d\*$", "what\_is\_between\_brackets" \] } Context: I parse… --- ## [Parsed JSON object/hash requires a target configuration option](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/286217)
**Author:** [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\ **Replies:** 0\ **Last updated:** [October 8, 2021, 9:47am UTC](https://discuss.elastic.co/t/parsed-json-object-hash-requires-a-target-configuration-option/286217 "2021-10-08T09:47:06Z")
My cluster architecture is elk + Kafka + filebeat. Use the following configuration to parse JSON logs: input { kafka { bootstrap\_servers =\> "192.168.1.176:9092,192.168.1.178:9092,192.168.1.177:9092" cl… --- ## [How to parse different log in logstash configuration](https://discuss.elastic.co/t/how-to-parse-different-log-in-logstash-configuration/286142)
**Author:** [@Roccof97](https://discuss.elastic.co/u/Roccof97)\ **Replies:** 2\ **Last updated:** [October 8, 2021, 7:47am UTC](https://discuss.elastic.co/t/how-to-parse-different-log-in-logstash-configuration/286142 "2021-10-08T07:47:16Z")
Hello to all, i have a problem with activemq logos i have created different types of grok, but the problem is that sometimes the message is different with additional fields and consequently it is parsed incorrectly, i… --- ## [Extract concrete words with logstash](https://discuss.elastic.co/t/extract-concrete-words-with-logstash/286139)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 2\ **Last updated:** [October 8, 2021, 7:38am UTC](https://discuss.elastic.co/t/extract-concrete-words-with-logstash/286139 "2021-10-08T07:38:38Z")
Hi! I was interested in knowing if with logstash I could find words that match a pattern and extract them from the message, without parsing the whole message. For example: there has been an error on the \*\*s123s\*\* mach… --- ## [Logstash error 403 with a remote elasticsearch instance](https://discuss.elastic.co/t/logstash-error-403-with-a-remote-elasticsearch-instance/286161)
**Author:** [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)\ **Replies:** 3\ **Last updated:** [October 7, 2021, 10:42pm UTC](https://discuss.elastic.co/t/logstash-error-403-with-a-remote-elasticsearch-instance/286161 "2021-10-07T22:42:13Z")
Hi, I have a logstash config which works fine with my local Elasticsearch instance but when i try to push to a remote instance config gives error at startup as follows: Using default mapping template \[2021-10-07T13:17… --- ## [Add caracter in logstash custom patterns](https://discuss.elastic.co/t/add-caracter-in-logstash-custom-patterns/286158)
**Author:** [@Adixon\_Diaz](https://discuss.elastic.co/u/Adixon_Diaz)\ **Replies:** 2\ **Last updated:** [October 7, 2021, 8:45pm UTC](https://discuss.elastic.co/t/add-caracter-in-logstash-custom-patterns/286158 "2021-10-07T20:45:25Z")
Hi comunity, I want to add a caracter when i'm processing data with logstash, for example, add a space or : or -. I mean, i have the next log 2021-07-29122715960 grok custom pattern: FECHACORTA ((?\>\\d\\d){1,2}\[./-\]%{MO… --- ## [Getting error while importing csv file to logstash](https://discuss.elastic.co/t/getting-error-while-importing-csv-file-to-logstash/285954)
**Author:** [@SSirurmath](https://discuss.elastic.co/u/SSirurmath)\ **Replies:** 6\ **Last updated:** [October 7, 2021, 3:56pm UTC](https://discuss.elastic.co/t/getting-error-while-importing-csv-file-to-logstash/285954 "2021-10-07T15:56:27Z")
\[2021-10-05T16:59:04,940\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of \[ \\… --- ## [Logstash.conf error](https://discuss.elastic.co/t/logstash-conf-error/286088)
**Author:** [@Johnny\_Cash\_Cow](https://discuss.elastic.co/u/Johnny_Cash_Cow)\ **Replies:** 1\ **Last updated:** [October 7, 2021, 3:54pm UTC](https://discuss.elastic.co/t/logstash-conf-error/286088 "2021-10-07T15:54:02Z")
Here is the error: "LogStash::ConfigurationError", :message=\>"Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"input\\", \\"filter\\", \\"output\\" at line 6, column 1 (byte 132) after ", and here is the logstash.conf file itself in… --- ## [Logstash keeps restarting itself](https://discuss.elastic.co/t/logstash-keeps-restarting-itself/286124)
**Author:** [@grigala](https://discuss.elastic.co/u/grigala)\ **Replies:** 3\ **Last updated:** [October 7, 2021, 1:59pm UTC](https://discuss.elastic.co/t/logstash-keeps-restarting-itself/286124 "2021-10-07T13:59:17Z")
Hello everyone, For some reason my Logstash container keeps restarting itself and every time producing the output below: Using bundled JDK: /usr/share/logstash/jdk warning: no jvm.options file found /usr/share/logstash… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=188) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=190)