# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=302 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 303 --- ## [Parsing whole file as a single event](https://discuss.elastic.co/t/parsing-whole-file-as-a-single-event/249317)
**Author:** [@thdesy](https://discuss.elastic.co/u/thdesy)\ **Replies:** 1\ **Last updated:** [September 21, 2020, 1:59pm UTC](https://discuss.elastic.co/t/parsing-whole-file-as-a-single-event/249317 "2020-09-21T13:59:57Z")
Hi all, I would like to monitor a directory, where per event a file is written; each event-file consists of several key=value pairs. \> /var/log/dir/history.\* \> head /var/log/dir/history.1234 key1 = value1 key… --- ## [Logstash - Symantec Pipeline](https://discuss.elastic.co/t/logstash-symantec-pipeline/249334)
**Author:** [@amolmegheELK](https://discuss.elastic.co/u/amolmegheELK)\ **Replies:** 0\ **Last updated:** [September 21, 2020, 9:55am UTC](https://discuss.elastic.co/t/logstash-symantec-pipeline/249334 "2020-09-21T09:55:36Z")
Hello, I am using ELK 6.8.0, and i am performing the logstash Pipeline for symantec. earlier indices created as per pipeline and our custom requirements . however newly created indices is different from old one. Older i… --- ## [Grok Multiline file](https://discuss.elastic.co/t/grok-multiline-file/248302)
**Author:** [@LuigiDelavega](https://discuss.elastic.co/u/LuigiDelavega)\ **Replies:** 4\ **Last updated:** [September 11, 2020, 2:57pm UTC](https://discuss.elastic.co/t/grok-multiline-file/248302 "2020-09-11T14:57:28Z")
Hi there, I'm looking to grok mails which were not sent. So i would like the entire file to be read as one log with 5 fields : From, To, Date, Subject and the rest as Message. Here is what it looks like: From: TEXT To… --- ## [Logstash PQ Java 14 and Closed queue](https://discuss.elastic.co/t/logstash-pq-java-14-and-closed-queue/249242)
**Author:** [@Jugsofbeer](https://discuss.elastic.co/u/Jugsofbeer)\ **Replies:** 0\ **Last updated:** [September 20, 2020, 8:32am UTC](https://discuss.elastic.co/t/logstash-pq-java-14-and-closed-queue/249242 "2020-09-20T08:32:13Z")
Hi, Just wondering if anyone else has struck the issue I have recently with Persistent Queue enabled and the pipeline sometimes getting into a unrecoverable state where it throws the message "cannot write to a closed qu… --- ## [Dynamic input :Http\_poller url](https://discuss.elastic.co/t/dynamic-input-http-poller-url/249277)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 1\ **Last updated:** [September 20, 2020, 7:42pm UTC](https://discuss.elastic.co/t/dynamic-input-http-poller-url/249277 "2020-09-20T19:42:22Z")
Hi all, in my logstash file i use a http\_poller a input. input { http\_poller { urls =\> { url2020 =\> "http://xxxxx/XXXx/Xxxx/XXXXXXXstartDate=01/01/2020&endDate=30/09/2020" } request\_timeout =\> 60 schedule =\> { "… --- ## [How to Json log format output for my system log](https://discuss.elastic.co/t/how-to-json-log-format-output-for-my-system-log/249238)
**Author:** [@poddraj](https://discuss.elastic.co/u/poddraj)\ **Replies:** 0\ **Last updated:** [September 20, 2020, 7:29am UTC](https://discuss.elastic.co/t/how-to-json-log-format-output-for-my-system-log/249238 "2020-09-20T07:29:00Z")
Hi, I have my log in below format, it actually has all fields delimited by ^. However, in between we can have few other lines which are nothing but the field headers. I am unable to get this log converted into json form… --- ## [Input file plugin - file is not read when modified](https://discuss.elastic.co/t/input-file-plugin-file-is-not-read-when-modified/249228)
**Author:** [@xyz2](https://discuss.elastic.co/u/xyz2)\ **Replies:** 3\ **Last updated:** [September 20, 2020, 2:14am UTC](https://discuss.elastic.co/t/input-file-plugin-file-is-not-read-when-modified/249228 "2020-09-20T02:14:52Z")
HI, I am using file input plugin in logstash. My file gets replaced in regular time interval . I need the file to be read from the beginning each time it is replaced. I have below setting . It reads from beginning onl… --- ## [Logstash error](https://discuss.elastic.co/t/logstash-error/249194)
**Author:** [@grazer](https://discuss.elastic.co/u/grazer)\ **Replies:** 4\ **Last updated:** [September 20, 2020, 1:25am UTC](https://discuss.elastic.co/t/logstash-error/249194 "2020-09-20T01:25:43Z")
Running version 7.8.0 everthing seems to be running fine however, I'm getting error... not sure what it is.. I have the correct java version, all the mutations seem to work, the paths are correct etc.. This extract is fr… --- ## [Coordination Node Where I put them?](https://discuss.elastic.co/t/coordination-node-where-i-put-them/249156)
**Author:** [@Samuel\_Amico](https://discuss.elastic.co/u/Samuel_Amico)\ **Replies:** 7\ **Last updated:** [September 19, 2020, 10:42pm UTC](https://discuss.elastic.co/t/coordination-node-where-i-put-them/249156 "2020-09-19T22:42:17Z")
Guys, I have a doubt. The coordination node, I put at the entrance of the cluster (logstash sends it to the coordination node) or I put it behind the kibana (kibana points to the coordination node). I have 2 coordinatio… --- ## [How to remove\_field for json data by http\_poller at all once](https://discuss.elastic.co/t/how-to-remove-field-for-json-data-by-http-poller-at-all-once/248545)
**Author:** [@unisuke](https://discuss.elastic.co/u/unisuke)\ **Replies:** 1\ **Last updated:** [September 19, 2020, 2:47pm UTC](https://discuss.elastic.co/t/how-to-remove-field-for-json-data-by-http-poller-at-all-once/248545 "2020-09-19T14:47:38Z")
Hi. I want to filter for jason getting by http\_poller plugin. For example, it's below. "content" =\> \[ \[0\] { "AList" =\> \[ \[0\] { "name" =\> "X", }, \[1\] { "name" =\> "Y, } … --- ## [Can logstash add array field to elasticsearch?](https://discuss.elastic.co/t/can-logstash-add-array-field-to-elasticsearch/249202)
**Author:** [@recordable542](https://discuss.elastic.co/u/recordable542)\ **Replies:** 0\ **Last updated:** [September 19, 2020, 7:09am UTC](https://discuss.elastic.co/t/can-logstash-add-array-field-to-elasticsearch/249202 "2020-09-19T07:09:58Z")
My Environment mysql logstash (with jdbc plugin) 7.8.0 elasticsearch 7.6.2 pipeline.yaml ... statement =\> "SELECT id, name, location1, location2 FROM users;" ... I have location1 and location2 column in mysql users ta… --- ## [S3 input plugin is not reading AWS-KMS (CMK) encrypted bucket](https://discuss.elastic.co/t/s3-input-plugin-is-not-reading-aws-kms-cmk-encrypted-bucket/249199)
**Author:** [@deepanshumarwah](https://discuss.elastic.co/u/deepanshumarwah)\ **Replies:** 0\ **Last updated:** [September 19, 2020, 6:38am UTC](https://discuss.elastic.co/t/s3-input-plugin-is-not-reading-aws-kms-cmk-encrypted-bucket/249199 "2020-09-19T06:38:18Z")
Hi All I am running a logstash container with s3 pipeline as per below configuration: input { s3 { id =\> "pipeline\_s3\_example\_bucket\_input" bucket =\> "example-bucket" region =\> "ap-southeast-1" access… --- ## [I have problem with the logstash configuration](https://discuss.elastic.co/t/i-have-problem-with-the-logstash-configuration/249189)
**Author:** [@Pablo\_Rivadeneira](https://discuss.elastic.co/u/Pablo_Rivadeneira)\ **Replies:** 0\ **Last updated:** [September 18, 2020, 11:43pm UTC](https://discuss.elastic.co/t/i-have-problem-with-the-logstash-configuration/249189 "2020-09-18T23:43:46Z")
this is my configuration file input { file { path =\> \["C:/Users/user03/Documents/sistema\_TPG/88.-Elasticsearch/testlog.log"\] start\_position =\> "beginning" sincedb\_path =\> "C:/logstash/logstash/sincedb/logdbpath.txt" … --- ## [Processing data from logstash to elastic](https://discuss.elastic.co/t/processing-data-from-logstash-to-elastic/249183)
**Author:** [@gr0k](https://discuss.elastic.co/u/gr0k)\ **Replies:** 0\ **Last updated:** [September 18, 2020, 10:17pm UTC](https://discuss.elastic.co/t/processing-data-from-logstash-to-elastic/249183 "2020-09-18T22:17:52Z")
Worked through the RedELK install over the last week which is set to collect logs out of /root/cobaltstrike/logs. I made 1 edit to the teamserver filebeat config to collect logs from /opt/cobaltstrike/logs and it broke t… --- ## [Problem With Date Parse - Suddenly Started Last Night](https://discuss.elastic.co/t/problem-with-date-parse-suddenly-started-last-night/249172)
**Author:** [@dmb](https://discuss.elastic.co/u/dmb)\ **Replies:** 1\ **Last updated:** [September 18, 2020, 9:40pm UTC](https://discuss.elastic.co/t/problem-with-date-parse-suddenly-started-last-night/249172 "2020-09-18T21:40:28Z")
Hi All, Very new to the ELK stack and have set up a server to ingest all my syslogs and firewall logs. Up until last night at exactly 6:59 PM all my syslogs were being ingested and the date field was being parsed out, … --- ## [Using the date plugin to replace the value of @timestamp](https://discuss.elastic.co/t/using-the-date-plugin-to-replace-the-value-of-timestamp/249094)
**Author:** [@goncalobsantos](https://discuss.elastic.co/u/goncalobsantos)\ **Replies:** 1\ **Last updated:** [September 18, 2020, 7:36pm UTC](https://discuss.elastic.co/t/using-the-date-plugin-to-replace-the-value-of-timestamp/249094 "2020-09-18T19:36:04Z")
I am trying to write a .conf file to automatically build monthly indexes. I've read that Logstash uses @timestamp to identify the month of a document and that this field is in UTC by default. I'm in a timezone differe… --- ## [Containerized Logstash from it's tar ball](https://discuss.elastic.co/t/containerized-logstash-from-its-tar-ball/249158)
**Author:** [@aneeshansari](https://discuss.elastic.co/u/aneeshansari)\ **Replies:** 0\ **Last updated:** [September 18, 2020, 6:32pm UTC](https://discuss.elastic.co/t/containerized-logstash-from-its-tar-ball/249158 "2020-09-18T18:32:25Z")
Hello Everyone, can anybody help me to containerized logstash with filebeat 7.8.1. Regards, Aneesh --- ## [Run http\_poller only once and terminate logstash](https://discuss.elastic.co/t/run-http-poller-only-once-and-terminate-logstash/249029)
**Author:** [@hamsinihboomi](https://discuss.elastic.co/u/hamsinihboomi)\ **Replies:** 4\ **Last updated:** [September 18, 2020, 5:22pm UTC](https://discuss.elastic.co/t/run-http-poller-only-once-and-terminate-logstash/249029 "2020-09-18T17:22:05Z")
I have an API call to JIRA which should run only once every 4 weeks. I'm using logstash http\_poller to fetch the data from JIRA. The schedule with http\_poller does not allow a complex cron schedule so I was planning on … --- ## [How to create fields dynamically?](https://discuss.elastic.co/t/how-to-create-fields-dynamically/249115)
**Author:** [@11191](https://discuss.elastic.co/u/11191)\ **Replies:** 1\ **Last updated:** [September 18, 2020, 3:14pm UTC](https://discuss.elastic.co/t/how-to-create-fields-dynamically/249115 "2020-09-18T15:14:17Z")
I have a log looking like: \[2020-09-18 12:34 smth1=38 smth2=125 ... \] and I want "smth1" to be a field name and "38" its value, what would be best way to do it considering that "smth1" might be different ea… --- ## [How to implement task life cycle scenario](https://discuss.elastic.co/t/how-to-implement-task-life-cycle-scenario/248128)
**Author:** [@ram546](https://discuss.elastic.co/u/ram546)\ **Replies:** 9\ **Last updated:** [September 18, 2020, 3:09pm UTC](https://discuss.elastic.co/t/how-to-implement-task-life-cycle-scenario/248128 "2020-09-18T15:09:35Z")
hello, I am new to ELK stack and I want to create one scenario where I can track elapsed time. e.g I have a platform where user can CREATE ticket. and that ticket will be assigned to some department/person. then it will … --- ## [\[ERROR\]\[logstash.agent \] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: PipelineAction::Create\, action\_result: false", :backtrace=\>nil}](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-id-main-action-type-logstash-failedaction-message-could-not-execute-action-pipelineaction-create-main-action-result-false-backtrace-nil/249057)
**Author:** [@Sayontani\_Bose](https://discuss.elastic.co/u/Sayontani_Bose)\ **Replies:** 1\ **Last updated:** [September 18, 2020, 3:01pm UTC](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-id-main-action-type-logstash-failedaction-message-could-not-execute-action-pipelineaction-create-main-action-result-false-backtrace-nil/249057 "2020-09-18T15:01:23Z")
Hi I have Installed Logstash and whenever i m trying to run by "bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/01-wazuh.conf --config.reload.automatic I am getting this error - \[ERROR\]\[logstash.age… --- ## [Events Received Rate is climbing high after logstash restart](https://discuss.elastic.co/t/events-received-rate-is-climbing-high-after-logstash-restart/249063)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 1\ **Last updated:** [September 18, 2020, 10:09am UTC](https://discuss.elastic.co/t/events-received-rate-is-climbing-high-after-logstash-restart/249063 "2020-09-18T10:09:37Z")
Hi , I'm handling with strange behavior , after bouncing logstash Events Received Rate droped to 200 e/s after bounce again it climbed to 1000 e/s what can be the cause ? --- ## [Logstash, how to avoid output block when one elasticsearch node go down](https://discuss.elastic.co/t/logstash-how-to-avoid-output-block-when-one-elasticsearch-node-go-down/247419)
**Author:** [@daniele.lopez](https://discuss.elastic.co/u/daniele.lopez)\ **Replies:** 6\ **Last updated:** [September 18, 2020, 9:54am UTC](https://discuss.elastic.co/t/logstash-how-to-avoid-output-block-when-one-elasticsearch-node-go-down/247419 "2020-09-18T09:54:25Z")
Hi all, I have an elasticsearch cluster with 3 nodes. is a really basic cluster. On each node i have one logstash pointing to cluster nodes. If a node go down all logstash output fail, even if i have the master node u… --- ## [How does logstash read multiple gunzipped files](https://discuss.elastic.co/t/how-does-logstash-read-multiple-gunzipped-files/249074)
**Author:** [@Praneet\_Khandelwal](https://discuss.elastic.co/u/Praneet_Khandelwal)\ **Replies:** 0\ **Last updated:** [September 18, 2020, 9:07am UTC](https://discuss.elastic.co/t/how-does-logstash-read-multiple-gunzipped-files/249074 "2020-09-18T09:07:48Z")
I have set up a simple logstash pipeline with a single file input (mode= 'read'). I have given a regex pattern in the path attribute. I am trying to read a number of gunzipped files matching the regex in the path. The th… --- ## [Giop in logstash file](https://discuss.elastic.co/t/giop-in-logstash-file/248971)
**Author:** [@hiba](https://discuss.elastic.co/u/hiba)\ **Replies:** 6\ **Last updated:** [September 18, 2020, 8:52am UTC](https://discuss.elastic.co/t/giop-in-logstash-file/248971 "2020-09-18T08:52:26Z")
I create a giop filed in my logstash config file (location). In my data source the latitude and the longitude fields can be an empty string. so whene i load data into elasticserach i lose all the documents wich have th… --- ## [Logstash and JSON array split](https://discuss.elastic.co/t/logstash-and-json-array-split/247482)
**Author:** [@ranu\_shekhar](https://discuss.elastic.co/u/ranu_shekhar)\ **Replies:** 4\ **Last updated:** [September 18, 2020, 8:02am UTC](https://discuss.elastic.co/t/logstash-and-json-array-split/247482 "2020-09-18T08:02:29Z")
Hi, I have the following input in JSON file: \[{"id":"9906fce735f8282ef85ccd2e5d49cbd7d3aaef2c","short\_id":"9906fce7","created\_at":"2020-08-21T16:01:20.000+05:30","parent\_ids":"d4e1afec8e2069f514701697c7a6ffd4cda5fc74",… --- ## [Logstash](https://discuss.elastic.co/t/logstash/248924)
**Author:** [@sai92](https://discuss.elastic.co/u/sai92)\ **Replies:** 2\ **Last updated:** [September 17, 2020, 10:19pm UTC](https://discuss.elastic.co/t/logstash/248924 "2020-09-17T22:19:48Z")
Hello I am using elasticsearch and kibana for IoT platform to collect data from sensors I use seperate plugin of mine to calculate signal quality of each sensor and to collect data from sensor and push it to elasticsear… --- ## [JSON parse error - Suricata](https://discuss.elastic.co/t/json-parse-error-suricata/248997)
**Author:** [@Paulogbr](https://discuss.elastic.co/u/Paulogbr)\ **Replies:** 2\ **Last updated:** [September 17, 2020, 4:02pm UTC](https://discuss.elastic.co/t/json-parse-error-suricata/248997 "2020-09-17T16:02:30Z")
Hello Team, I mounted my lab with suricata and web servers. I tried send log suricata to ELK. But I received error when i did the parse My conf: input { udp { port =\> "514" type =\> "syslog" } } filter { json { … --- ## [Logstash file input mix up](https://discuss.elastic.co/t/logstash-file-input-mix-up/248942)
**Author:** [@logben](https://discuss.elastic.co/u/logben)\ **Replies:** 1\ **Last updated:** [September 17, 2020, 1:58pm UTC](https://discuss.elastic.co/t/logstash-file-input-mix-up/248942 "2020-09-17T13:58:49Z")
Hello! I read two Windows file shares with DNS logs with the following two seperate configuration files: DNS-Server 1: input { file { path =\> "/var/log/logstash/dns-share-pdc/dns.log" … --- ## [Auto Reload conf file + Pipelines.yml](https://discuss.elastic.co/t/auto-reload-conf-file-pipelines-yml/248865)
**Author:** [@Pankaj\_Jadhav](https://discuss.elastic.co/u/Pankaj_Jadhav)\ **Replies:** 3\ **Last updated:** [September 17, 2020, 1:56pm UTC](https://discuss.elastic.co/t/auto-reload-conf-file-pipelines-yml/248865 "2020-09-17T13:56:42Z")
I am running conf files via pipelines.yml. pipeline.id: pipeline19 config.reload.automatic: true config.reload.interval: 3s path.config: "/etc/logstash/conf.d/ds-recentsearch.conf" queue.type: persisted After sett… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=301) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=303)