# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=341 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 342 --- ## [Logstash translate and replace filter not working](https://discuss.elastic.co/t/logstash-translate-and-replace-filter-not-working/237582)
**Author:** [@marco2005](https://discuss.elastic.co/u/marco2005)\ **Replies:** 1\ **Last updated:** [June 18, 2020, 12:18pm UTC](https://discuss.elastic.co/t/logstash-translate-and-replace-filter-not-working/237582 "2020-06-18T12:18:56Z")
I have 5 log files using the below Golang log pattern: 2020-06-17 11:50:04 CEST | CORE | INFO | (pkg/collector/corechecks/net/ntp.go:162 in Run) | Failed to get clock offset from any ntp host And below is my filter: f… --- ## [Count the number of rows(lines) in a CSV file](https://discuss.elastic.co/t/count-the-number-of-rows-lines-in-a-csv-file/237601)
**Author:** [@inchirah](https://discuss.elastic.co/u/inchirah)\ **Replies:** 0\ **Last updated:** [June 18, 2020, 9:02am UTC](https://discuss.elastic.co/t/count-the-number-of-rows-lines-in-a-csv-file/237601 "2020-06-18T09:02:12Z")
Hey guys, i'm trying to count the number of rows in my csv file (it contain 2631 row) and save it in a variable. I used ruby filter but it didn't work so i tried using the metric filter but i get an result that i even … --- ## [Close Csv File when Logstash is run as a windows service](https://discuss.elastic.co/t/close-csv-file-when-logstash-is-run-as-a-windows-service/235363)
**Author:** [@Bender](https://discuss.elastic.co/u/Bender)\ **Replies:** 4\ **Last updated:** [June 18, 2020, 8:50am UTC](https://discuss.elastic.co/t/close-csv-file-when-logstash-is-run-as-a-windows-service/235363 "2020-06-18T08:50:33Z")
Hello Community, iam very new in this topic. I start a windows service via nssm.exe to observe a file for a csv to start the logstash.bat and specified config. Now i have the problem that after the logstash is triggere… --- ## [Output condition filter on filebeat module](https://discuss.elastic.co/t/output-condition-filter-on-filebeat-module/237505)
**Author:** [@tvander](https://discuss.elastic.co/u/tvander)\ **Replies:** 3\ **Last updated:** [June 18, 2020, 8:46am UTC](https://discuss.elastic.co/t/output-condition-filter-on-filebeat-module/237505 "2020-06-18T08:46:11Z")
Hi, i have my kafka server with two filebeat module enabled: kafka and system, the filebeat config also include a "kafka" tags. On my logstash server i use multi pipeline, what i'm trying to do is separate kafka logs fr… --- ## [Stuck with timezone issue in logstash JDBC input](https://discuss.elastic.co/t/stuck-with-timezone-issue-in-logstash-jdbc-input/236707)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 6\ **Last updated:** [June 18, 2020, 6:10am UTC](https://discuss.elastic.co/t/stuck-with-timezone-issue-in-logstash-jdbc-input/236707 "2020-06-18T06:10:21Z")
I am a bit lost here. Situation: Some transaction happens in Sydney. And the record is stored in SQL database. The time stored is in UTC. 5 mins later, we pull that data using Logstash JDBC input. Kibana pulls up and … --- ## [Logstash 5.5 and pipelines](https://discuss.elastic.co/t/logstash-5-5-and-pipelines/237482)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 4\ **Last updated:** [June 18, 2020, 4:15am UTC](https://discuss.elastic.co/t/logstash-5-5-and-pipelines/237482 "2020-06-18T04:15:38Z")
Hi , I'm facing Logstash 5.5 , is it right to create multiple pipelines under conf.d directory ? I just created 2 pipelines with 2 different ports and it seems working but I get all messages into all of them unrelate… --- ## [Your socket connection to the sever was not read from or written to within the timeout period](https://discuss.elastic.co/t/your-socket-connection-to-the-sever-was-not-read-from-or-written-to-within-the-timeout-period/237542)
**Author:** [@Annette1](https://discuss.elastic.co/u/Annette1)\ **Replies:** 0\ **Last updated:** [June 17, 2020, 10:11pm UTC](https://discuss.elastic.co/t/your-socket-connection-to-the-sever-was-not-read-from-or-written-to-within-the-timeout-period/237542 "2020-06-17T22:11:13Z")
Logstash to s3 output - getting Your socket connection to the sever was not read from or written to within the timeout period. AWS - explanation The number of bytes set in the "Content-Length" header is more than the ac… --- ## [Geoip to geopoint in logstash kibana 7.7](https://discuss.elastic.co/t/geoip-to-geopoint-in-logstash-kibana-7-7/237032)
**Author:** [@Rushabh\_Doshi](https://discuss.elastic.co/u/Rushabh_Doshi)\ **Replies:** 3\ **Last updated:** [June 17, 2020, 9:03pm UTC](https://discuss.elastic.co/t/geoip-to-geopoint-in-logstash-kibana-7-7/237032 "2020-06-17T21:03:11Z")
Hi I am using the latest 7.7.1 version of elk stack and my logstash geoip snippet looks as below: if \[log\_type\] == "querylog" { grok{ match =\> {"message" =\> "%{TIMESTAMP\_ISO8601:Activity-Time}%{SPACE}USERID=%{QUOTEDS… --- ## [Use Mutate after KV filter](https://discuss.elastic.co/t/use-mutate-after-kv-filter/237460)
**Author:** [@ksremo](https://discuss.elastic.co/u/ksremo)\ **Replies:** 3\ **Last updated:** [June 17, 2020, 8:46pm UTC](https://discuss.elastic.co/t/use-mutate-after-kv-filter/237460 "2020-06-17T20:46:04Z")
Is it not possible to use mutate filter after kv filter on fields created by the kv filter? I have key value pairs which get divided by a kv filter. this is working great, the new fields appears on discover menu in kiba… --- ## [Need help Parsing json with grok](https://discuss.elastic.co/t/need-help-parsing-json-with-grok/237518)
**Author:** [@some\_IT\_guy](https://discuss.elastic.co/u/some_IT_guy)\ **Replies:** 0\ **Last updated:** [June 17, 2020, 5:22pm UTC](https://discuss.elastic.co/t/need-help-parsing-json-with-grok/237518 "2020-06-17T17:22:53Z")
I am trying to use grok to parse down json, I am new and have the following filter { if \[syslog5424\_app\] == "nginx-access" { json { source =\> "syslog5424\_msg" skip\_on\_invalid\_json =\> … --- ## [Reading JSON data from logstash](https://discuss.elastic.co/t/reading-json-data-from-logstash/237465)
**Author:** [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\ **Replies:** 2\ **Last updated:** [June 17, 2020, 2:28pm UTC](https://discuss.elastic.co/t/reading-json-data-from-logstash/237465 "2020-06-17T14:28:39Z")
Hi all, I have a lot of JSON data and I want to visualize it. I have seen the POST\_bulk method in Kibana, but it is not dynamic. We can not update the data whenever it is being generated. So, I thought of using logstash… --- ## [Issues while installing Logstash 7.7.1 on Windows 10](https://discuss.elastic.co/t/issues-while-installing-logstash-7-7-1-on-windows-10/237458)
**Author:** [@Inshu](https://discuss.elastic.co/u/Inshu)\ **Replies:** 0\ **Last updated:** [June 17, 2020, 11:09am UTC](https://discuss.elastic.co/t/issues-while-installing-logstash-7-7-1-on-windows-10/237458 "2020-06-17T11:09:44Z")
I was installing Logstash 7.7.1 on Windows 10. When I tried to configure it with the command bin/logstash -f logstash.conf it is showing following error ignoring JAVA\_OPTS= -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CM… --- ## [Multiline log parsing - Filebeat, Logstash](https://discuss.elastic.co/t/multiline-log-parsing-filebeat-logstash/237132)
**Author:** [@Gyanprakash\_Mishra](https://discuss.elastic.co/u/Gyanprakash_Mishra)\ **Replies:** 1\ **Last updated:** [June 17, 2020, 10:35am UTC](https://discuss.elastic.co/t/multiline-log-parsing-filebeat-logstash/237132 "2020-06-17T10:35:20Z")
Hi All, I am using filebeat to stash data into logstash. Need help in parsing log of following format: 11-Jun-2020 18:31:51.900 SEVERE \[http-nio-8080-exec-24\] org.apache.catalina.core.StandardWrapperValve.invoke Servl… --- ## [Response data from text file multiple folder to multiple index](https://discuss.elastic.co/t/response-data-from-text-file-multiple-folder-to-multiple-index/236694)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 2\ **Last updated:** [June 17, 2020, 10:33am UTC](https://discuss.elastic.co/t/response-data-from-text-file-multiple-folder-to-multiple-index/236694 "2020-06-17T10:33:42Z")
My logstash configuration is this input { beats { port =\> 5044 } } filter { xml { store\_xml =\> "false" source =\> "message" xpath =\>\["/propertyAvailability/hotelRates/hotel\[2\]/bookingChannel\[4\]/ratePlan\[3\]/miscInfo/tex… --- ## [Why logstash is not considering my ports?](https://discuss.elastic.co/t/why-logstash-is-not-considering-my-ports/237443)
**Author:** [@AngeloM](https://discuss.elastic.co/u/AngeloM)\ **Replies:** 2\ **Last updated:** [June 17, 2020, 10:25am UTC](https://discuss.elastic.co/t/why-logstash-is-not-considering-my-ports/237443 "2020-06-17T10:25:57Z")
Getting crazy here ! The idea is to import different log from different port and import then in different indexes. I got two different configuration files on "etc/logtash/init.d". The two configuration files has two d… --- ## [Extract hour and weekday from custom date time field](https://discuss.elastic.co/t/extract-hour-and-weekday-from-custom-date-time-field/237286)
**Author:** [@Rushabh\_Doshi](https://discuss.elastic.co/u/Rushabh_Doshi)\ **Replies:** 5\ **Last updated:** [June 17, 2020, 10:22am UTC](https://discuss.elastic.co/t/extract-hour-and-weekday-from-custom-date-time-field/237286 "2020-06-17T10:22:25Z")
for extracting hour and weekday from Activity-Time field the logstash snippet is as below: date{ match =\> \[ "Activity-Time", "yyyy-MM-dd HH:mm:ss" \] target =\> "Activity-Time-New" add\_field =\> {"hour" =\> "%{+HH}"} ad… --- ## [\[Logstash\] Replace the @timestamp with a rsyslog ISO8691 timestamp from Filebeat log lines](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344)
**Author:** [@Kuo\_Hugo](https://discuss.elastic.co/u/Kuo_Hugo)\ **Replies:** 8\ **Last updated:** [June 17, 2020, 6:53am UTC](https://discuss.elastic.co/t/logstash-replace-the-timestamp-with-a-rsyslog-iso8691-timestamp-from-filebeat-log-lines/237344 "2020-06-17T06:53:18Z")
Hi Elastic users, I'm forwarding logs from host to an ELK's logstash instance via filebeat. I'm using @timestamp as the time for the index pattern for my logs. The original @timestamp from filebeat while processing the … --- ## [Not able to retrieve new logs in real-time](https://discuss.elastic.co/t/not-able-to-retrieve-new-logs-in-real-time/237387)
**Author:** [@drjosephliu](https://discuss.elastic.co/u/drjosephliu)\ **Replies:** 0\ **Last updated:** [June 17, 2020, 5:13am UTC](https://discuss.elastic.co/t/not-able-to-retrieve-new-logs-in-real-time/237387 "2020-06-17T05:13:24Z")
This is the setup: I'm running ELK locally with a java spring boot app that writes logs to a file called elk-logs.log I'm able to see the logs in kibana when I first run the java spring boot app, but after the initial … --- ## [Unable to create new document/event in logstash Ruby filter using yield](https://discuss.elastic.co/t/unable-to-create-new-document-event-in-logstash-ruby-filter-using-yield/236995)
**Author:** [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Replies:** 2\ **Last updated:** [June 16, 2020, 6:38pm UTC](https://discuss.elastic.co/t/unable-to-create-new-document-event-in-logstash-ruby-filter-using-yield/236995 "2020-06-16T18:38:00Z")
I am trying to calculate elapsed time between the first and last line of a file using a Ruby filter and push the field in a new event using the below Ruby filter ruby { init =\> "@save\_the\_path = 'none'" code =\> " … --- ## [Reindex a subset of data to a different index](https://discuss.elastic.co/t/reindex-a-subset-of-data-to-a-different-index/237341)
**Author:** [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Replies:** 2\ **Last updated:** [June 16, 2020, 5:00pm UTC](https://discuss.elastic.co/t/reindex-a-subset-of-data-to-a-different-index/237341 "2020-06-16T17:00:44Z")
Hello all, we're using Elasticsearch to index Windows event logs and keeping them for 6 months; what I would like to do is reindex a subset of data such as: user creation/deletion events group creation/deletion events i… --- ## [Store messages that fail to index to elastisearch into file for later analysis](https://discuss.elastic.co/t/store-messages-that-fail-to-index-to-elastisearch-into-file-for-later-analysis/237298)
**Author:** [@petrkalina](https://discuss.elastic.co/u/petrkalina)\ **Replies:** 1\ **Last updated:** [June 16, 2020, 3:17pm UTC](https://discuss.elastic.co/t/store-messages-that-fail-to-index-to-elastisearch-into-file-for-later-analysis/237298 "2020-06-16T15:17:43Z")
Hello Community, in a logstash / elasticsearch pipeline I notice rare errors similar to this one: \[2020-06-08T17:02:05,621\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] Could not index event to Elasticsearch. {:status=\>… --- ## [How to configure logstash input to accept https post from a third party service](https://discuss.elastic.co/t/how-to-configure-logstash-input-to-accept-https-post-from-a-third-party-service/235472)
**Author:** [@rndee](https://discuss.elastic.co/u/rndee)\ **Replies:** 1\ **Last updated:** [June 16, 2020, 2:07pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-input-to-accept-https-post-from-a-third-party-service/235472 "2020-06-16T14:07:02Z")
Hi I have a third party service which posts json data to our logstash instance. It is working using http, now I want to harden the connection using https. The third-party service ssl certificate was already registered i… --- ## [Issue with filter configuration in logstash](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911)
**Author:** [@kittu9999](https://discuss.elastic.co/u/kittu9999)\ **Replies:** 4\ **Last updated:** [June 12, 2020, 5:18pm UTC](https://discuss.elastic.co/t/issue-with-filter-configuration-in-logstash/236911 "2020-06-12T17:18:40Z")
Hello, we are having an issue with filter configuration in the logstash. we have upgraded the logstash form 5 to logstash 6.8.1 version. the filter configuration was working fine in logstash 5.4 version, but not in the… --- ## [Multiline XML not parsing](https://discuss.elastic.co/t/multiline-xml-not-parsing/237293)
**Author:** [@DWbank](https://discuss.elastic.co/u/DWbank)\ **Replies:** 0\ **Last updated:** [June 16, 2020, 12:14pm UTC](https://discuss.elastic.co/t/multiline-xml-not-parsing/237293 "2020-06-16T12:14:35Z")
There are may different examples out here on how to take Nessus files and bring them in to Elastic via Logstash. I have read them and tried many different configs. Here is the XML file I am trying to read in. https://… --- ## [Issues with geoip data logstash and filebeat using iis logs](https://discuss.elastic.co/t/issues-with-geoip-data-logstash-and-filebeat-using-iis-logs/236933)
**Author:** [@genehunter29009](https://discuss.elastic.co/u/genehunter29009)\ **Replies:** 6\ **Last updated:** [June 16, 2020, 11:57am UTC](https://discuss.elastic.co/t/issues-with-geoip-data-logstash-and-filebeat-using-iis-logs/236933 "2020-06-16T11:57:56Z")
Having a hard time getting geoip to work. I have the latest current verion of filebeat and logstash. filebeat 7.7.7 template is loaded. one strange thing also I get like 6 ip addresses when I use geoip but nothing els… --- ## [How to get Elastic, Kibana and Logstash working with docker compose](https://discuss.elastic.co/t/how-to-get-elastic-kibana-and-logstash-working-with-docker-compose/236760)
**Author:** [@Magnuss](https://discuss.elastic.co/u/Magnuss)\ **Replies:** 1\ **Last updated:** [June 16, 2020, 10:00am UTC](https://discuss.elastic.co/t/how-to-get-elastic-kibana-and-logstash-working-with-docker-compose/236760 "2020-06-16T10:00:40Z")
Hi all, I'm trying to get a Elastic cluster working with docker-compose. I'm running into some issues. I would like to use volumes locally so i can edit config files and add new conf files for Logstash. Also i would li… --- ## [Got Invalid Frame Type Exception even Filebeat version is same which pointing to Logstash 5.6.14](https://discuss.elastic.co/t/got-invalid-frame-type-exception-even-filebeat-version-is-same-which-pointing-to-logstash-5-6-14/237255)
**Author:** [@ridwanbejo](https://discuss.elastic.co/u/ridwanbejo)\ **Replies:** 0\ **Last updated:** [June 16, 2020, 8:40am UTC](https://discuss.elastic.co/t/got-invalid-frame-type-exception-even-filebeat-version-is-same-which-pointing-to-logstash-5-6-14/237255 "2020-06-16T08:40:48Z")
Hi Magnus, I have problems with Filebeat and Logstash setup. So I assigned to move logstash service to dockerize one. Then, I use logstash Docker with version 5.6.14. But I have an exception when attempt to redirect one… --- ## [How to use memcached plugin in logstash](https://discuss.elastic.co/t/how-to-use-memcached-plugin-in-logstash/237256)
**Author:** [@AayushPatel](https://discuss.elastic.co/u/AayushPatel)\ **Replies:** 0\ **Last updated:** [June 16, 2020, 8:44am UTC](https://discuss.elastic.co/t/how-to-use-memcached-plugin-in-logstash/237256 "2020-06-16T08:44:21Z")
Hi, I am recently learning elk stack and paying more attention to logstash configs. most recently decided to use memcached plugin to get data from another server and write it to elasticsearch according to this url … --- ## [Distributed Pipeline - Can the send\_to address be resolved from a variable?](https://discuss.elastic.co/t/distributed-pipeline-can-the-send-to-address-be-resolved-from-a-variable/237204)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 2\ **Last updated:** [June 16, 2020, 8:33am UTC](https://discuss.elastic.co/t/distributed-pipeline-can-the-send-to-address-be-resolved-from-a-variable/237204 "2020-06-16T08:33:25Z")
I have a working setup, where events come in from a single filebeat node, hit the log stash main pipeline which sends the event to one of 20 different pipelines. To do this there is an ugly if statement that does 20 str… --- ## [Indexing Rate is fluctuating](https://discuss.elastic.co/t/indexing-rate-is-fluctuating/237252)
**Author:** [@iammanmale](https://discuss.elastic.co/u/iammanmale)\ **Replies:** 0\ **Last updated:** [June 16, 2020, 8:17am UTC](https://discuss.elastic.co/t/indexing-rate-is-fluctuating/237252 "2020-06-16T08:17:47Z")
The indexing rate is quite fluctuating. Is it normal? Is there any way to make it more stable? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=340) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=342)