# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=500 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 501 --- ## [ElasticSearch loosing data](https://discuss.elastic.co/t/elasticsearch-loosing-data/192873)
**Author:** [@Aymen\_Ben\_Moussa](https://discuss.elastic.co/u/Aymen_Ben_Moussa)\ **Replies:** 5\ **Last updated:** [August 1, 2019, 12:31pm UTC](https://discuss.elastic.co/t/elasticsearch-loosing-data/192873 "2019-08-01T12:31:35Z")
My architecture consists of Kafka logstash elasticsearch, i wrote a ruby code to calculate a KPI in logstash, everything was working fine but some days pass by and that KPI i made has been reset to 0 and only that KPI, i… --- ## [How to send all the alerts in one mail using logstash](https://discuss.elastic.co/t/how-to-send-all-the-alerts-in-one-mail-using-logstash/187589)
**Author:** [@irobot678](https://discuss.elastic.co/u/irobot678)\ **Replies:** 2\ **Last updated:** [August 1, 2019, 12:22pm UTC](https://discuss.elastic.co/t/how-to-send-all-the-alerts-in-one-mail-using-logstash/187589 "2019-08-01T12:22:36Z")
HI , I will get around 100 alerts in a day and all those come in different emails . So , is there any way i can get all the alerts only in one mail on a particular time everyday using logstash. Please help me solve it… --- ## [Ruby filter to execute code only once](https://discuss.elastic.co/t/ruby-filter-to-execute-code-only-once/193243)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [August 1, 2019, 12:14pm UTC](https://discuss.elastic.co/t/ruby-filter-to-execute-code-only-once/193243 "2019-08-01T12:14:06Z")
Hi All I have a scenario where I am using Ruby filter and I want to execute the code only once after reading the input. Basically after executing this code it should come out to output and should not process further I … --- ## [Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError"](https://discuss.elastic.co/t/failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror/193247)
**Author:** [@hitesh\_kumar](https://discuss.elastic.co/u/hitesh_kumar)\ **Replies:** 2\ **Last updated:** [August 1, 2019, 11:47am UTC](https://discuss.elastic.co/t/failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror/193247 "2019-08-01T11:47:22Z")
Hi I'm using logstash-7.0.1, my configuration file is: input { stdout{} } output { elasticsearch { hosts =\> {"localhost:9200"} } stdout { codec =\> rubydebug … --- ## [Multiline Codec logstash for log level](https://discuss.elastic.co/t/multiline-codec-logstash-for-log-level/193094)
**Author:** [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\ **Replies:** 5\ **Last updated:** [August 1, 2019, 11:23am UTC](https://discuss.elastic.co/t/multiline-codec-logstash-for-log-level/193094 "2019-08-01T11:23:48Z")
Hi I have a log entry sample as below. every line starts with i.e., log level. Few log entries would be on the same line but few would be in multiple lines. Can you please suggest the logstash file input multiline cod… --- ## [Failed to execute logstash.javapipeline \] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\](https://discuss.elastic.co/t/failed-to-execute-logstash-javapipeline-pipeline-aborted-due-to-error-pipeline-id-main-exception-argumenterror-the-sincedb-path-argument-must-point-to-a-file-received-a-directory-c-dev-null/193272)
**Author:** [@vikrant031](https://discuss.elastic.co/u/vikrant031)\ **Replies:** 1\ **Last updated:** [August 1, 2019, 10:57am UTC](https://discuss.elastic.co/t/failed-to-execute-logstash-javapipeline-pipeline-aborted-due-to-error-pipeline-id-main-exception-argumenterror-the-sincedb-path-argument-must-point-to-a-file-received-a-directory-c-dev-null/193272 "2019-08-01T10:57:13Z")
Config File: input{ file{ path =\>"C:\\logstash-7.2.0\\config\\cars.csv" start\_position =\> "beginning" sincedb\_path =\> "C:/dev/null" } } filter{ csv{ separator=\>"," columns=\> \["maker", "model", "mileage", "manufac… --- ## [Logstash Confuguration error](https://discuss.elastic.co/t/logstash-confuguration-error/193296)
**Author:** [@Sammeta\_David\_Raju](https://discuss.elastic.co/u/Sammeta_David_Raju)\ **Replies:** 0\ **Last updated:** [August 1, 2019, 10:25am UTC](https://discuss.elastic.co/t/logstash-confuguration-error/193296 "2019-08-01T10:25:24Z")
\*error: java::org.postgressql.Driver not loaded input { jdbc { # Postgres jdbc connection string to our database, mydb jdbc\_connection\_string =\> "jdbc:postgresql:/localhost:5432/mydb" # The user we wish to execute … --- ## [Issue when deployingin remote server as a docker image: logstastash log -\> " Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties" - afterwards no logs from logstash](https://discuss.elastic.co/t/issue-when-deployingin-remote-server-as-a-docker-image-logstastash-log-sending-logstash-logs-to-usr-share-logstash-logs-which-is-now-configured-via-log4j2-properties-afterwards-no-logs-from-logstash/193133)
**Author:** [@jobin](https://discuss.elastic.co/u/jobin)\ **Replies:** 3\ **Last updated:** [August 1, 2019, 10:05am UTC](https://discuss.elastic.co/t/issue-when-deployingin-remote-server-as-a-docker-image-logstastash-log-sending-logstash-logs-to-usr-share-logstash-logs-which-is-now-configured-via-log4j2-properties-afterwards-no-logs-from-logstash/193133 "2019-08-01T10:05:09Z")
ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp \[::1\]:5044: connect: cannot assign requested address 2019-07-31T13:43:31.419Z INFO pipeline/output.go:93 At… --- ## [Cannot export Date fields to MongoDB](https://discuss.elastic.co/t/cannot-export-date-fields-to-mongodb/192943)
**Author:** [@HFolly](https://discuss.elastic.co/u/HFolly)\ **Replies:** 5\ **Last updated:** [August 1, 2019, 9:35am UTC](https://discuss.elastic.co/t/cannot-export-date-fields-to-mongodb/192943 "2019-08-01T09:35:52Z")
Hello. I'm facing an issue when I try to write date fields to MongoDB, I've tried to convert it in many ways but with no success. In my input JSON I have a collection like this: "Colecs": \[ { "Value": { "Va… --- ## [How CPU and memory and disk usage are consumed when we keep running Logstash as a log receiver/compressor/transmitter and increase the amount of log?](https://discuss.elastic.co/t/how-cpu-and-memory-and-disk-usage-are-consumed-when-we-keep-running-logstash-as-a-log-receiver-compressor-transmitter-and-increase-the-amount-of-log/192851)
**Author:** [@kensaku-okada](https://discuss.elastic.co/u/kensaku-okada)\ **Replies:** 6\ **Last updated:** [August 1, 2019, 9:26am UTC](https://discuss.elastic.co/t/how-cpu-and-memory-and-disk-usage-are-consumed-when-we-keep-running-logstash-as-a-log-receiver-compressor-transmitter-and-increase-the-amount-of-log/192851 "2019-08-01T09:26:48Z")
How CPU and memory and disk usage are utilized when we keep running Logstash as a log receiver/compressor/transmitter and increase the amount of input log? I watch/maintain a server where only one logstash runs and recei… --- ## [Logstash Error: Pipeline Failed to execute action](https://discuss.elastic.co/t/logstash-error-pipeline-failed-to-execute-action/193277)
**Author:** [@vikrant031](https://discuss.elastic.co/u/vikrant031)\ **Replies:** 0\ **Last updated:** [August 1, 2019, 8:38am UTC](https://discuss.elastic.co/t/logstash-error-pipeline-failed-to-execute-action/193277 "2019-08-01T08:38:51Z")
Config File: input{ file{ path =\>"C:\\logstash-7.2.0\\config\\cars.csv" start\_position =\> "beginning" sincedb\_path =\> "C:\\dev\\null" } } filter{ csv{ separator=\> "," columns=\> \["maker", "model", "mileage", "manufa… --- ## [Logstash keystore encoding error](https://discuss.elastic.co/t/logstash-keystore-encoding-error/193135)
**Author:** [@tombar](https://discuss.elastic.co/u/tombar)\ **Replies:** 4\ **Last updated:** [August 1, 2019, 7:14am UTC](https://discuss.elastic.co/t/logstash-keystore-encoding-error/193135 "2019-08-01T07:14:17Z")
I try to set up a password protected Logstash keystore. It works when I run Logstash from the command line (as root, with set environment variable for the keystore), but when running as a systemd service (installed from … --- ## [Logstash configuration issue](https://discuss.elastic.co/t/logstash-configuration-issue/193140)
**Author:** [@ch0k](https://discuss.elastic.co/u/ch0k)\ **Replies:** 2\ **Last updated:** [August 1, 2019, 7:08am UTC](https://discuss.elastic.co/t/logstash-configuration-issue/193140 "2019-08-01T07:08:13Z")
Hi, I have problems to configure my logstash config file (output) and I would like to know if someone can help me with that? I think my problem is related to the SSL encryption but I'm not sure... I created an encrypt… --- ## [How to handle \_kvParsefailure](https://discuss.elastic.co/t/how-to-handle-kvparsefailure/193262)
**Author:** [@Sajal](https://discuss.elastic.co/u/Sajal)\ **Replies:** 0\ **Last updated:** [August 1, 2019, 7:04am UTC](https://discuss.elastic.co/t/how-to-handle-kvparsefailure/193262 "2019-08-01T07:04:16Z")
How to handle \_kvParsefailure as we do in \_grokParsefailure ? --- ## [Build logstash image with self-defined config failed](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344)
**Author:** [@NashMiao](https://discuss.elastic.co/u/NashMiao)\ **Replies:** 4\ **Last updated:** [August 1, 2019, 2:46am UTC](https://discuss.elastic.co/t/build-logstash-image-with-self-defined-config-failed/192344 "2019-08-01T02:46:37Z")
Here is my Dockerfile: FROM docker.elastic.co/logstash/logstash:7.2.0 RUN rm -f /usr/share/logstash/pipeline/logstash.conf ADD ./pipeline/ /usr/share/logstash/pipeline/ ADD ./config/ /usr/share/logstash/config/ In fact… --- ## [S3 Input unable to download remote file](https://discuss.elastic.co/t/s3-input-unable-to-download-remote-file/193238)
**Author:** [@placidlife](https://discuss.elastic.co/u/placidlife)\ **Replies:** 0\ **Last updated:** [August 1, 2019, 2:44am UTC](https://discuss.elastic.co/t/s3-input-unable-to-download-remote-file/193238 "2019-08-01T02:44:46Z")
Hello, Currently, I am facing an issue when I am trying to use the S3 input plugin. The resulting error is : S3 input: Unable to download remote file {:remote\_key=\>"XXXX/XXXX/XXXX/XXXX/XXXX/XXXX/2019-05-15T11:48:15.55… --- ## [Unknown setting 'skip\_header' for csv](https://discuss.elastic.co/t/unknown-setting-skip-header-for-csv/193077)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 2\ **Last updated:** [August 1, 2019, 1:24am UTC](https://discuss.elastic.co/t/unknown-setting-skip-header-for-csv/193077 "2019-08-01T01:24:47Z")
I got this error when I add skip\_header property in csv filter filter { csv { columns =\> \[ "Date", "Product Categories", "Geo", "Revenue" \] separator =\> "," remove\_field =\> \["message"\] skip\_header =\> "true" … --- ## [Logstash XML parsing issues - trying to send to Graylog, part 2](https://discuss.elastic.co/t/logstash-xml-parsing-issues-trying-to-send-to-graylog-part-2/193209)
**Author:** [@eclipsed450](https://discuss.elastic.co/u/eclipsed450)\ **Replies:** 4\ **Last updated:** [July 31, 2019, 8:46pm UTC](https://discuss.elastic.co/t/logstash-xml-parsing-issues-trying-to-send-to-graylog-part-2/193209 "2019-07-31T20:46:51Z")
So, in this thread, I was able to get an initial config working, but ran into an issue parsing the data in that there are different events, which results in different data fields. Is there a way to handle these fields dy… --- ## [How to write to currently managed index from logstash (via Elasticsearch output)](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183)
**Author:** [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\ **Replies:** 4\ **Last updated:** [July 31, 2019, 6:16pm UTC](https://discuss.elastic.co/t/how-to-write-to-currently-managed-index-from-logstash-via-elasticsearch-output/193183 "2019-07-31T18:16:12Z")
My current deployment setup is as below. I want to write to filebeat indices with hostnames while ILM is in place to manage the indices EG: filebeat-7.2.0-host-name-1-000001 In order to achieve this, I'm trying to m… --- ## [Logstash started to crash trying to connect to Kafka](https://discuss.elastic.co/t/logstash-started-to-crash-trying-to-connect-to-kafka/193196)
**Author:** [@cyxop](https://discuss.elastic.co/u/cyxop)\ **Replies:** 0\ **Last updated:** [July 31, 2019, 6:03pm UTC](https://discuss.elastic.co/t/logstash-started-to-crash-trying-to-connect-to-kafka/193196 "2019-07-31T18:03:36Z")
Version: logstash 5.6.5 Plugin Version logstash-output-kafka Operating System: RHEL7 {code} \[2019-07-31T17:43:04,243\]\[INFO \]\[org.apache.kafka.clients.producer.KafkaProducer\] Closing the Kafka producer with timeoutMil… --- ## [Apache log not able to load into elastic through stashlog](https://discuss.elastic.co/t/apache-log-not-able-to-load-into-elastic-through-stashlog/192493)
**Author:** [@Gopal\_Chauhan](https://discuss.elastic.co/u/Gopal_Chauhan)\ **Replies:** 5\ **Last updated:** [July 31, 2019, 5:57pm UTC](https://discuss.elastic.co/t/apache-log-not-able-to-load-into-elastic-through-stashlog/192493 "2019-07-31T17:57:38Z")
Hi I am trying to load apache (dummy log) log into elastic through log stash. Below is my config file. I am not any index created in kibana. Am i doing something wrong ? input { file { path =\> "C:\\Elastic\\Data\\log" … --- ## [Logstash parsing Nginx logs failed](https://discuss.elastic.co/t/logstash-parsing-nginx-logs-failed/192864)
**Author:** [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\ **Replies:** 9\ **Last updated:** [July 31, 2019, 3:58pm UTC](https://discuss.elastic.co/t/logstash-parsing-nginx-logs-failed/192864 "2019-07-31T15:58:28Z")
I'm running Logstash-oss 7.2.0 (docker) and Filebeat 7.2.0 (rpm). Filebeat collecting Nginx access/error logs using Nginx module and sending to Logstash. My Logstash pipeline was taken from the official documentation: … --- ## [Logstash, ELB access logs, client port displays with a comma](https://discuss.elastic.co/t/logstash-elb-access-logs-client-port-displays-with-a-comma/193136)
**Author:** [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\ **Replies:** 2\ **Last updated:** [July 31, 2019, 3:31pm UTC](https://discuss.elastic.co/t/logstash-elb-access-logs-client-port-displays-with-a-comma/193136 "2019-07-31T15:31:06Z")
Hi, I'm using logstash for parsing AWS ELB access logs, and noticed that client port displays with a comma in Kibana but in JSON tab it displays correctly, could you please explain, why it happens? , thanks "backend… --- ## [Reloading pipeline fails with "Setting \\"modules.cli\\" hasn't been registered"](https://discuss.elastic.co/t/reloading-pipeline-fails-with-setting-modules-cli-hasnt-been-registered/193156)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 0\ **Last updated:** [July 31, 2019, 3:21pm UTC](https://discuss.elastic.co/t/reloading-pipeline-fails-with-setting-modules-cli-hasnt-been-registered/193156 "2019-07-31T15:21:52Z")
I have a logstash pipeline pointed at this configuration input {} input { generator { count =\> 1 lines =\> \[ '' \] } } filter { } output { stdout { codec =\> rubydebug { metadata =\> false } } } I then updated my pipelines… --- ## [Logstash not able to read input file](https://discuss.elastic.co/t/logstash-not-able-to-read-input-file/193018)
**Author:** [@Sachin\_Nerurkar](https://discuss.elastic.co/u/Sachin_Nerurkar)\ **Replies:** 4\ **Last updated:** [July 31, 2019, 2:19pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-read-input-file/193018 "2019-07-31T14:19:50Z")
I have following code. input { file { path =\> "C:\\logstash-7.2.0\\samples\\test\_10.log" start\_position =\> "beginning" ignore\_older =\> 0 } } filter { } output { stdout { codec =\> rubydebug} } I ran in debug mod… --- ## [Error in copying fields](https://discuss.elastic.co/t/error-in-copying-fields/193057)
**Author:** [@Vineeth\_Ananthula1](https://discuss.elastic.co/u/Vineeth_Ananthula1)\ **Replies:** 1\ **Last updated:** [July 31, 2019, 1:42pm UTC](https://discuss.elastic.co/t/error-in-copying-fields/193057 "2019-07-31T13:42:25Z")
I have created the logstash conf file properly and there is no error. My problem is I am unable to copy , split the fields properly. In the below log \<185\>Jul 30 2019 10:15:13 BR-RTS-BHB-T3-VM-NCN-BST-US57-01-BAB IFNE… --- ## [\[SOLVED\]Prune Filter Question](https://discuss.elastic.co/t/solved-prune-filter-question/193108)
**Author:** [@kolten](https://discuss.elastic.co/u/kolten)\ **Replies:** 7\ **Last updated:** [July 31, 2019, 1:36pm UTC](https://discuss.elastic.co/t/solved-prune-filter-question/193108 "2019-07-31T13:36:50Z")
Hi, I want to prune my logs and only take some fields. Here is my logstash config file but this config didnt work. How can solve this issue? input { beats { port =\> 5044 } } filter { json { source… --- ## [Failed to install template for suricata](https://discuss.elastic.co/t/failed-to-install-template-for-suricata/193010)
**Author:** [@Michael\_Tsikerdekis](https://discuss.elastic.co/u/Michael_Tsikerdekis)\ **Replies:** 1\ **Last updated:** [July 31, 2019, 1:26pm UTC](https://discuss.elastic.co/t/failed-to-install-template-for-suricata/193010 "2019-07-31T13:26:08Z")
I am not sure why but I cant get logstash 7 to install my template that work in 6. Here is the error: \[2019-07-31T01:31:52,296\]\[INFO \]\[logstash.outputs.elasticsearch\] Installing elasticsearch template to \_template/suri… --- ## [Is it possible detect duplicate message in Kafka topic with logstash](https://discuss.elastic.co/t/is-it-possible-detect-duplicate-message-in-kafka-topic-with-logstash/193026)
**Author:** [@arp220](https://discuss.elastic.co/u/arp220)\ **Replies:** 1\ **Last updated:** [July 31, 2019, 1:17pm UTC](https://discuss.elastic.co/t/is-it-possible-detect-duplicate-message-in-kafka-topic-with-logstash/193026 "2019-07-31T13:17:01Z")
we use logstash and we want read one table from oracle database and send this message to kafka, we format message in kafka is: Topic1: message1: {"name":"name-1", "id":"fbd89256-12gh-10og-etdgn1234njF", "site":"site-1",… --- ## [Logstash 7.2.0 \_mutate\_error](https://discuss.elastic.co/t/logstash-7-2-0-mutate-error/193086)
**Author:** [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\ **Replies:** 2\ **Last updated:** [July 31, 2019, 1:14pm UTC](https://discuss.elastic.co/t/logstash-7-2-0-mutate-error/193086 "2019-07-31T13:14:12Z")
hi, I'm using Logstash 6.8.0 for parsing logs, It works fine. I'm splitting "path" field using current pipeline: if \[path\] and !(\[path\] == "/") { mutate { copy =\> {"path" =\> "path\_split"} } mutate { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=499) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=501)