# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=312 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 313 --- ## [Use Logstash to Ingest Entire .txt file](https://discuss.elastic.co/t/use-logstash-to-ingest-entire-txt-file/246141)
**Author:** [@spyderman4g63](https://discuss.elastic.co/u/spyderman4g63)\ **Replies:** 1\ **Last updated:** [August 24, 2020, 3:07pm UTC](https://discuss.elastic.co/t/use-logstash-to-ingest-entire-txt-file/246141 "2020-08-24T15:07:07Z")
Is it possible to ingest the entire contents of a txt file into a field using logstash? I don't want a multiline delimeter. --- ## [Logstash and Grok filter problem with username](https://discuss.elastic.co/t/logstash-and-grok-filter-problem-with-username/246130)
**Author:** [@mcoa](https://discuss.elastic.co/u/mcoa)\ **Replies:** 1\ **Last updated:** [August 24, 2020, 2:16pm UTC](https://discuss.elastic.co/t/logstash-and-grok-filter-problem-with-username/246130 "2020-08-24T14:16:30Z")
Hi, I try parser apache (access\_log) log but i've problem with username section (not capture the complete name), log example: 10.13.252.185 - - \[23/Aug/2020:23:56:14 -0400\] "PROPFIND /remote.php/dav/files/JF009-01/PAP… --- ## [Logstash to Elasticsearch](https://discuss.elastic.co/t/logstash-to-elasticsearch/246034)
**Author:** [@headtea](https://discuss.elastic.co/u/headtea)\ **Replies:** 3\ **Last updated:** [August 24, 2020, 2:12pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch/246034 "2020-08-24T14:12:43Z")
I'm trying to send data from logstash to ES and I get an error. I can't make much sense of it. This is my logstash output: output{ elasticsearch { hosts =\> \["https://localhost:9200"\] cacert =\> "/etc/elasticsearch/http.c… --- ## [Logstash can't send to elastic higher 3500 rps on one node](https://discuss.elastic.co/t/logstash-cant-send-to-elastic-higher-3500-rps-on-one-node/246126)
**Author:** [@karvet](https://discuss.elastic.co/u/karvet)\ **Replies:** 0\ **Last updated:** [August 24, 2020, 1:38pm UTC](https://discuss.elastic.co/t/logstash-cant-send-to-elastic-higher-3500-rps-on-one-node/246126 "2020-08-24T13:38:19Z")
Hi i tested ELK stack to use in our enviroment. I send about 5k and more for one logstash node but always saw in Kibana about 3000-3500 reqest. All node is Ubuntu Os 18.04 used 1.9 version of ELK stack. When i used 4 no… --- ## [How to add a plugin to ELK in a docker-compose file?](https://discuss.elastic.co/t/how-to-add-a-plugin-to-elk-in-a-docker-compose-file/246091)
**Author:** [@nb03briceno](https://discuss.elastic.co/u/nb03briceno)\ **Replies:** 0\ **Last updated:** [August 24, 2020, 10:17am UTC](https://discuss.elastic.co/t/how-to-add-a-plugin-to-elk-in-a-docker-compose-file/246091 "2020-08-24T10:17:16Z")
Hello everyone, i have an important question. If I would like to add a logstash plugin to my docker-compose file, how I could do it? This is the section of logstash I have in my docker-compose (it's not working because… --- ## [Logstash script for both insert and update](https://discuss.elastic.co/t/logstash-script-for-both-insert-and-update/246088)
**Author:** [@Vamshi3](https://discuss.elastic.co/u/Vamshi3)\ **Replies:** 0\ **Last updated:** [August 24, 2020, 10:11am UTC](https://discuss.elastic.co/t/logstash-script-for-both-insert-and-update/246088 "2020-08-24T10:11:36Z")
Newbie with Elastic Stack I have an oracle database table where rows gets inserted daily at 12 AM and later on those column values will get updated if there is any change in their values. I tried with doc\_as\_upsert but… --- ## [Logstash configuration input error with if statement](https://discuss.elastic.co/t/logstash-configuration-input-error-with-if-statement/246019)
**Author:** [@anon56147639](https://discuss.elastic.co/u/anon56147639)\ **Replies:** 10\ **Last updated:** [August 24, 2020, 9:11am UTC](https://discuss.elastic.co/t/logstash-configuration-input-error-with-if-statement/246019 "2020-08-24T09:11:25Z")
Hi everyone, I'm trying to create a pipeline in logstash which receives CEF events and other input from Filebeat. Since not all input is in CEF format, I created an IF-statement. My logstash.conf looks like this: b… --- ## [Send data to logstash from filebeat using HTTP/HTTPS](https://discuss.elastic.co/t/send-data-to-logstash-from-filebeat-using-http-https/246078)
**Author:** [@aaleman](https://discuss.elastic.co/u/aaleman)\ **Replies:** 0\ **Last updated:** [August 24, 2020, 9:07am UTC](https://discuss.elastic.co/t/send-data-to-logstash-from-filebeat-using-http-https/246078 "2020-08-24T09:07:23Z")
Hello, Do you know how can I send my filebeat data to logstash through HTTP/HTTPS?¿ I think that beats are not compatible with this, is correct? I have: Filebeat machines outside my private cloud -- NGINX 80/443 loadba… --- ## [Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create\, action\_result: false", :backtrace=\>nil}](https://discuss.elastic.co/t/failed-to-execute-action-id-main-action-type-logstash-failedaction-message-could-not-execute-action-pipelineaction-create-main-action-result-false-backtrace-nil/245372)
**Author:** [@joel.ng](https://discuss.elastic.co/u/joel.ng)\ **Replies:** 5\ **Last updated:** [August 24, 2020, 3:29am UTC](https://discuss.elastic.co/t/failed-to-execute-action-id-main-action-type-logstash-failedaction-message-could-not-execute-action-pipelineaction-create-main-action-result-false-backtrace-nil/245372 "2020-08-24T03:29:12Z")
Trying out logstash for the first time, but I'm getting this error on a very basic localhost:5044 to S3 configuration: \[2020-08-18T08:15:53,638\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"… --- ## [My first centralized log server](https://discuss.elastic.co/t/my-first-centralized-log-server/246002)
**Author:** [@leostereo](https://discuss.elastic.co/u/leostereo)\ **Replies:** 5\ **Last updated:** [August 23, 2020, 10:52pm UTC](https://discuss.elastic.co/t/my-first-centralized-log-server/246002 "2020-08-23T22:52:56Z")
Hi guys , im learning elk. Just asking for some advices building my first loggin server. I need to gather al logs incoming from my mikrotik devices. Question is: What do I need to accomplish that ? (I already have m… --- ## [Rollover Indeces using Logstash and Elastic](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 13\ **Last updated:** [August 23, 2020, 10:52pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362 "2020-08-23T22:52:14Z")
Hi guys, I have following use case: I want to create new index every day, therefore my logstash output configuraiton looks as this: output { elasticsearch { index =\> "metricbeat-linux-vms-%{+YYYY.MM.dd}" I also ha… --- ## [Filtering two different time pattern in logstash](https://discuss.elastic.co/t/filtering-two-different-time-pattern-in-logstash/244370)
**Author:** [@Altair](https://discuss.elastic.co/u/Altair)\ **Replies:** 6\ **Last updated:** [August 10, 2020, 12:30pm UTC](https://discuss.elastic.co/t/filtering-two-different-time-pattern-in-logstash/244370 "2020-08-10T12:30:42Z")
Hello, I have two different time format in my logs like "Aug 10 09:18:10" and "2020-08-10T08:59:59.415319+02:00" I tried many different ways to have them in one grok but I fail date { match =\> \[ "syslog\_timestamp", "… --- ## [Ansi encoding for output of logstash](https://discuss.elastic.co/t/ansi-encoding-for-output-of-logstash/246014)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 0\ **Last updated:** [August 23, 2020, 11:04am UTC](https://discuss.elastic.co/t/ansi-encoding-for-output-of-logstash/246014 "2020-08-23T11:04:41Z")
Hi all, I want to save data as a txt file based on the ansi encoding using logstash. Logstash output is as following which ANSI\_X3.4-1968 codec is used to encode output in Ansi. But, when i check the encoding of data202… --- ## [Logstash error - bootstrap broker disconnected](https://discuss.elastic.co/t/logstash-error-bootstrap-broker-disconnected/245965)
**Author:** [@chandramouli\_sriniva](https://discuss.elastic.co/u/chandramouli_sriniva)\ **Replies:** 1\ **Last updated:** [August 21, 2020, 11:31pm UTC](https://discuss.elastic.co/t/logstash-error-bootstrap-broker-disconnected/245965 "2020-08-21T23:31:53Z")
we are sending logs from file.txt to Kafka(SASL\_SSL with kerberos). Below are the parm being set in conf file: bootstrap\_servers =\> "broker:9093" codec =\> json topic\_id =\> "topic" jaas\_path =\> "jaas.conf" kerberos\_conf… --- ## [Using output Kafka Codec Avro with KafkaAvroSerializer (schema\_registry\_url issue)](https://discuss.elastic.co/t/using-output-kafka-codec-avro-with-kafkaavroserializer-schema-registry-url-issue/245953)
**Author:** [@belguith\_chachia](https://discuss.elastic.co/u/belguith_chachia)\ **Replies:** 1\ **Last updated:** [August 21, 2020, 9:19pm UTC](https://discuss.elastic.co/t/using-output-kafka-codec-avro-with-kafkaavroserializer-schema-registry-url-issue/245953 "2020-08-21T21:19:54Z")
Hi guys, Our purpose is to post messages to event bus Kafka using Avro. We added all required jars libraries in LS\_HOME/vendor/jruby/lib/ And Logstash does recognize KafkaAvroSerializer and all other required classes… --- ## [System has not been booted with systemd as init system (PID 1). Can't operate](https://discuss.elastic.co/t/system-has-not-been-booted-with-systemd-as-init-system-pid-1-cant-operate/245954)
**Author:** [@Yanyan\_Liang](https://discuss.elastic.co/u/Yanyan_Liang)\ **Replies:** 0\ **Last updated:** [August 21, 2020, 7:20pm UTC](https://discuss.elastic.co/t/system-has-not-been-booted-with-systemd-as-init-system-pid-1-cant-operate/245954 "2020-08-21T19:20:17Z")
I can't start logstash in Ubuntu (wsl 2). My command is as follows: Anyone has the same problem and how to solve it? Many thanks! --- ## [S3 Input Plugin Following Errors Execution Expired and Net::OpenTimeout](https://discuss.elastic.co/t/s3-input-plugin-following-errors-execution-expired-and-net-opentimeout/245951)
**Author:** [@daniel\_a](https://discuss.elastic.co/u/daniel_a)\ **Replies:** 0\ **Last updated:** [August 21, 2020, 6:46pm UTC](https://discuss.elastic.co/t/s3-input-plugin-following-errors-execution-expired-and-net-opentimeout/245951 "2020-08-21T18:46:06Z")
I've been trying to use s3 input plugin in Logstash to pull data from the Tencent COS (cloud object storage). They're claiming they're using the AWS SDK. It works fine, I'm able to pull the COS data just fine, but it tim… --- ## [Logstash map id always detected as long, even it string](https://discuss.elastic.co/t/logstash-map-id-always-detected-as-long-even-it-string/245843)
**Author:** [@bayucandra](https://discuss.elastic.co/u/bayucandra)\ **Replies:** 2\ **Last updated:** [August 21, 2020, 5:21pm UTC](https://discuss.elastic.co/t/logstash-map-id-always-detected-as-long-even-it-string/245843 "2020-08-21T17:21:57Z")
Hi, Based on logs below, I assume that my logstsh always consider any record with field named "id" as a "long", even it "string". \[WARN \]\[logstash.outputs.elasticsearch\]\[main\]\[f4110a8e1e28ebbac18f43191bfa4dea9a1b050d31… --- ## [Docker ELK (multipipelines and pipeline error)](https://discuss.elastic.co/t/docker-elk-multipipelines-and-pipeline-error/245936)
**Author:** [@nb03briceno](https://discuss.elastic.co/u/nb03briceno)\ **Replies:** 1\ **Last updated:** [August 21, 2020, 5:05pm UTC](https://discuss.elastic.co/t/docker-elk-multipipelines-and-pipeline-error/245936 "2020-08-21T17:05:25Z")
Hello everyone, I'm working with ELK in docker. I was able to run Elasticsearch, KIbana, Logstash, and Filebeat with docker containers through a docker-compose file. Ingesting appear to works when I use simple pipelines… --- ## [Logstash Date Filter not working on Kibana](https://discuss.elastic.co/t/logstash-date-filter-not-working-on-kibana/245822)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 9\ **Last updated:** [August 21, 2020, 4:52pm UTC](https://discuss.elastic.co/t/logstash-date-filter-not-working-on-kibana/245822 "2020-08-21T16:52:48Z")
date { match =\> \["logdate", "YYYY.MM.dd HH:mm:ss,SSS"\] target =\> "logdate" } Above is my date filter configuration on my logstash config file. On kibana, when I create an index pattern, the only date option that sh… --- ## [Logstash restarts continuously with below error when enabling ilm](https://discuss.elastic.co/t/logstash-restarts-continuously-with-below-error-when-enabling-ilm/245934)
**Author:** [@sai\_kiran1](https://discuss.elastic.co/u/sai_kiran1)\ **Replies:** 1\ **Last updated:** [August 21, 2020, 3:48pm UTC](https://discuss.elastic.co/t/logstash-restarts-continuously-with-below-error-when-enabling-ilm/245934 "2020-08-21T15:48:20Z")
Hello, Am working on ilm settings in elasticsearch output plugin(logstash plugin) and I see Logstash restarts continuously with below error. \[2020-08-21T16:35:08,665\]\[FATAL\]\[logstash.runner \] An unexpected err… --- ## [Log queries from elasticsearch output](https://discuss.elastic.co/t/log-queries-from-elasticsearch-output/245882)
**Author:** [@SpaceRaven](https://discuss.elastic.co/u/SpaceRaven)\ **Replies:** 3\ **Last updated:** [August 21, 2020, 3:32pm UTC](https://discuss.elastic.co/t/log-queries-from-elasticsearch-output/245882 "2020-08-21T15:32:16Z")
Hi, Is there any way to log somehow queries from elasticsearch output plugin for es updates? i mean i want to inspect json queris which will go es cluster for update/delete/insert. Thanks --- ## [How to run a verification bloc (one time) after scheduled output](https://discuss.elastic.co/t/how-to-run-a-verification-bloc-one-time-after-scheduled-output/245898)
**Author:** [@RiadhBH](https://discuss.elastic.co/u/RiadhBH)\ **Replies:** 0\ **Last updated:** [August 21, 2020, 9:57am UTC](https://discuss.elastic.co/t/how-to-run-a-verification-bloc-one-time-after-scheduled-output/245898 "2020-08-21T09:57:00Z")
Hi, I'm using this configuration: SqlServer --\> LogStash --\> {ElasticSearch, Mysql}. I need to verify the number of input data (from logstash) and the output data (from mysql). It is possible to query elasticsearch on… --- ## [Logstash: How to pass value from one Aggregate Filter Task ID to another](https://discuss.elastic.co/t/logstash-how-to-pass-value-from-one-aggregate-filter-task-id-to-another/245921)
**Author:** [@mastersmit](https://discuss.elastic.co/u/mastersmit)\ **Replies:** 0\ **Last updated:** [August 21, 2020, 1:06pm UTC](https://discuss.elastic.co/t/logstash-how-to-pass-value-from-one-aggregate-filter-task-id-to-another/245921 "2020-08-21T13:06:48Z")
I have a requirement in which I need flow data from one aggreage filter of id different\_%{customer} to another %{traceid} logstash.conf code snippet: aggregate { task\_id =\> "different\_%{customer}" c… --- ## [Help with custom pattern](https://discuss.elastic.co/t/help-with-custom-pattern/245917)
**Author:** [@Jonathan\_Morin](https://discuss.elastic.co/u/Jonathan_Morin)\ **Replies:** 1\ **Last updated:** [August 21, 2020, 1:06pm UTC](https://discuss.elastic.co/t/help-with-custom-pattern/245917 "2020-08-21T13:06:02Z")
Hi guys, i'm trying to put parse a JDE log but i keep having a \_grokparsefailure and i really don't see where my problem is. Here's what i have filter { grok { pattern\_definitions =\> { "JDEPRO… --- ## [Can not parse bash output as json](https://discuss.elastic.co/t/can-not-parse-bash-output-as-json/245809)
**Author:** [@Yevhen\_Baidiuk](https://discuss.elastic.co/u/Yevhen_Baidiuk)\ **Replies:** 3\ **Last updated:** [August 21, 2020, 12:52pm UTC](https://discuss.elastic.co/t/can-not-parse-bash-output-as-json/245809 "2020-08-21T12:52:16Z")
I use shall script to generate dataset but logstash won't accept my message Thanks for feature help. --- ## [Getting Appdynamics Data into ELK](https://discuss.elastic.co/t/getting-appdynamics-data-into-elk/245905)
**Author:** [@sriram28](https://discuss.elastic.co/u/sriram28)\ **Replies:** 0\ **Last updated:** [August 21, 2020, 10:22am UTC](https://discuss.elastic.co/t/getting-appdynamics-data-into-elk/245905 "2020-08-21T10:22:47Z")
Hi All, Need your help. I am trying to get Appdynamics Data into Logstash . I am getting connect time out when trying to connect from logstash to appdynamics . +++++++++++++ t http\_request\_failure.error connect tim… --- ## [Logstash crash receiving data from filebeats](https://discuss.elastic.co/t/logstash-crash-receiving-data-from-filebeats/244425)
**Author:** [@M9k](https://discuss.elastic.co/u/M9k)\ **Replies:** 3\ **Last updated:** [August 21, 2020, 9:44am UTC](https://discuss.elastic.co/t/logstash-crash-receiving-data-from-filebeats/244425 "2020-08-21T09:44:05Z")
Hi, I have a problem with logstash, every time it receive a document from filebeat it crash. This is my filebeat configuration: filebeat.inputs: - paths: - c:\\windows\\system32\\dhcp\\DhcpSrvLog-\*.… --- ## [Logstash dosn't run file.conf in /etc/logstash/conf.d](https://discuss.elastic.co/t/logstash-dosnt-run-file-conf-in-etc-logstash-conf-d/244192)
**Author:** [@Amine\_Aloui](https://discuss.elastic.co/u/Amine_Aloui)\ **Replies:** 1\ **Last updated:** [August 21, 2020, 8:30am UTC](https://discuss.elastic.co/t/logstash-dosnt-run-file-conf-in-etc-logstash-conf-d/244192 "2020-08-21T08:30:52Z")
in /etc/logstash.yml i put this config path.config: /etc/logstash/conf.d/\*.conf config.reload.automatic: true config.reload.interval: 3s in /etc/logstash/conf.d i put my index.conf tail -f /var/log/logstash/logst… --- ## [Logstash how to parse different format message in one logfile](https://discuss.elastic.co/t/logstash-how-to-parse-different-format-message-in-one-logfile/245863)
**Author:** [@Champion\_Xie](https://discuss.elastic.co/u/Champion_Xie)\ **Replies:** 2\ **Last updated:** [August 21, 2020, 7:58am UTC](https://discuss.elastic.co/t/logstash-how-to-parse-different-format-message-in-one-logfile/245863 "2020-08-21T07:58:55Z")
The original log 07/20/20 09:23:14 INFO login 2493 us.(channel/Channel.cpp:575) 07/20/20 09:23:14 INFO deletecity:cn(city/CityDeleteBusiness.cpp:55) 07/20/20 09:23:14 INFO ---p\_dbSession-\>deleteCity() 14066… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=311) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=313)