# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=936 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 937 --- ## [Access Denied error with s3 plugin](https://discuss.elastic.co/t/access-denied-error-with-s3-plugin/95233)
**Author:** [@desai.amogh](https://discuss.elastic.co/u/desai.amogh)\ **Replies:** 0\ **Last updated:** [July 31, 2017, 7:26pm UTC](https://discuss.elastic.co/t/access-denied-error-with-s3-plugin/95233 "2017-07-31T19:26:57Z")
I am trying to use the s3 plugin and I get below error: \[\[main\]\"xxxxxx", access\_key… --- ## [LS loosing messages with persisted queue](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 4\ **Last updated:** [July 31, 2017, 7:11pm UTC](https://discuss.elastic.co/t/ls-loosing-messages-with-persisted-queue/93868 "2017-07-31T19:11:59Z")
Hello Logstash experts, I've encountered a very weird issue: I was indexing a database using the JDBC input, but realized, what the document count in ES was less than in the source table. It tuned out, that the reason … --- ## [How to merge two lines having different patterns in a log file?](https://discuss.elastic.co/t/how-to-merge-two-lines-having-different-patterns-in-a-log-file/94208)
**Author:** [@Amit\_Singh1](https://discuss.elastic.co/u/Amit_Singh1)\ **Replies:** 4\ **Last updated:** [July 31, 2017, 5:15pm UTC](https://discuss.elastic.co/t/how-to-merge-two-lines-having-different-patterns-in-a-log-file/94208 "2017-07-31T17:15:31Z")
SAMPLE LOG: 2017-07-14 17:47:11 +0000 \[INFO\] from application in application-akka.actor.default-dispatcher-9 - requestId: 8e5ccc9dbb4611cf9f363efbbe8e1c59,user: gaurav@sigmoid.com, organization: Vdopia, view: Auction, … --- ## [Logstash Using a lot of CPU](https://discuss.elastic.co/t/logstash-using-a-lot-of-cpu/95214)
**Author:** [@Warren\_Turner](https://discuss.elastic.co/u/Warren_Turner)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 5:08pm UTC](https://discuss.elastic.co/t/logstash-using-a-lot-of-cpu/95214 "2017-07-31T17:08:33Z")
Hey guys, I’m having an issue where logstash uses 5 CPU cores even when there is no configs. I’m running version 5.5.1 on Ubuntu 16.04. I tried increasing the JVM heap size (recommended in the performance troubleshooti… --- ## [Log Events/Records that Cause Exceptions](https://discuss.elastic.co/t/log-events-records-that-cause-exceptions/94501)
**Author:** [@swsw](https://discuss.elastic.co/u/swsw)\ **Replies:** 2\ **Last updated:** [July 31, 2017, 4:58pm UTC](https://discuss.elastic.co/t/log-events-records-that-cause-exceptions/94501 "2017-07-31T16:58:50Z")
Hello, How do I log the events/records that come into Logstash that cause exceptions (specifically illegal\_argument\_exception when putting data into Elasticsearch from Logstash). Situation: I am using Logstash and El… --- ## [Snmptrap plugin cannot decode Mac address correctly](https://discuss.elastic.co/t/snmptrap-plugin-cannot-decode-mac-address-correctly/94036)
**Author:** [@xianzhuzi](https://discuss.elastic.co/u/xianzhuzi)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 4:42pm UTC](https://discuss.elastic.co/t/snmptrap-plugin-cannot-decode-mac-address-correctly/94036 "2017-07-31T16:42:55Z")
Here is the log I get using snmptrap plugin. The Mac address in "message" is @value=\\"|\\x0E\\xCE\\xCD\\x8F0\\". And logstash automatically convert it into a strange string, like this: "|\\u000eÎ͏0". The Mac address in … --- ## ["mapper\_parsing\_exception" caused by new files input](https://discuss.elastic.co/t/mapper-parsing-exception-caused-by-new-files-input/94944)
**Author:** [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 2:22pm UTC](https://discuss.elastic.co/t/mapper-parsing-exception-caused-by-new-files-input/94944 "2017-07-31T14:22:40Z")
Hi, I noticed something on my elastic setup. Got elastic stack 5.4.0 on one machine for now I’m parsing log files from different servers, which I copy to the input folder every minute with batch scripts. The input of… --- ## [Grok regex with escaped "\[", "\]", "(", and ")" chars doesn't work](https://discuss.elastic.co/t/grok-regex-with-escaped-and-chars-doesnt-work/93842)
**Author:** [@timk519](https://discuss.elastic.co/u/timk519)\ **Replies:** 4\ **Last updated:** [July 31, 2017, 1:17pm UTC](https://discuss.elastic.co/t/grok-regex-with-escaped-and-chars-doesnt-work/93842 "2017-07-31T13:17:13Z")
Elastic newbie here - working with a new 5.5 install. I have a log line that looks like so: \[2015/10/01@19:48:22.785-0400\] P-4780 T-2208 I DBUTIL : (451) prostrct create session begin for timk519 on CON:. I… --- ## [Problem parsing Json from Beats](https://discuss.elastic.co/t/problem-parsing-json-from-beats/95177)
**Author:** [@dgoering](https://discuss.elastic.co/u/dgoering)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 1:02pm UTC](https://discuss.elastic.co/t/problem-parsing-json-from-beats/95177 "2017-07-31T13:02:07Z")
Hello I am new to the Elastic stack and am trying to get a fairly simple setup running: logback \> filebeats \> logstash \> elasticsearch \> Kibana I am using the logstash encoder in logback to create log entries as json f… --- ## [Split an event into multiple events](https://discuss.elastic.co/t/split-an-event-into-multiple-events/95174)
**Author:** [@Nikhil\_Utane](https://discuss.elastic.co/u/Nikhil_Utane)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 11:50am UTC](https://discuss.elastic.co/t/split-an-event-into-multiple-events/95174 "2017-07-31T11:50:09Z")
Hi, I have a following incoming event.I need to extract \[custombeat\]\[parameters\] fields (highlighted in bold) into multiple events. i.e. event1: type=lte, status=3 event2: type=cpu, status=0 event3: type=network, sta… --- ## [Config file structure for http output plugin (Getting 400)](https://discuss.elastic.co/t/config-file-structure-for-http-output-plugin-getting-400/94637)
**Author:** [@Vib1990](https://discuss.elastic.co/u/Vib1990)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 10:54am UTC](https://discuss.elastic.co/t/config-file-structure-for-http-output-plugin-getting-400/94637 "2017-07-31T10:54:19Z")
My events are json with about 4-5 top level fields. I am using the http output plugin to send the events to an http endpoint running on my Flask app. However, I am getting a 400 error code for each event sent. I am not… --- ## [Logstash - MariaDB to ElasticSearch - No Error, but doesnt load](https://discuss.elastic.co/t/logstash-mariadb-to-elasticsearch-no-error-but-doesnt-load/95127)
**Author:** [@pawanyv](https://discuss.elastic.co/u/pawanyv)\ **Replies:** 2\ **Last updated:** [July 31, 2017, 9:07am UTC](https://discuss.elastic.co/t/logstash-mariadb-to-elasticsearch-no-error-but-doesnt-load/95127 "2017-07-31T09:07:26Z")
Hi, I am trying to load data from MariaDB to elasticsearch using JDBC input plugin and elasticsearch output plugin. both logstash and elasticsearch versions are 5.5.0 My logstash.conf file is: input { jdbc { jdbc\_d… --- ## [It takes lots of time initializing the query when using logstash-input-jdbc](https://discuss.elastic.co/t/it-takes-lots-of-time-initializing-the-query-when-using-logstash-input-jdbc/95152)
**Author:** [@lunatic](https://discuss.elastic.co/u/lunatic)\ **Replies:** 0\ **Last updated:** [July 31, 2017, 9:06am UTC](https://discuss.elastic.co/t/it-takes-lots-of-time-initializing-the-query-when-using-logstash-input-jdbc/95152 "2017-07-31T09:06:35Z")
Hi, I’m using logstash-input-jdbc and I got some errors. There are about 5 million records in my db(MySQL) and I want to put them into es. And there is lots of ‘join’ operations like "select \* from table1 left join t… --- ## [Pipeline aborted due to error {:exception=\>#\ **Author:** [@ndarkness](https://discuss.elastic.co/u/ndarkness)\ **Replies:** 0\ **Last updated:** [July 31, 2017, 8:47am UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-exception-logstash-configurationerror-cipher/95148 "2017-07-31T08:47:00Z") HEllo, I am seeing the following errors on my logstash log \[2017-07-31T10:11:57,951\]\[INFO \]\[logstash.pipeline \] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch… --- ## [Remove duplicates](https://discuss.elastic.co/t/remove-duplicates/95135)
**Author:** [@raja\_gopal](https://discuss.elastic.co/u/raja_gopal)\ **Replies:** 0\ **Last updated:** [July 31, 2017, 7:43am UTC](https://discuss.elastic.co/t/remove-duplicates/95135 "2017-07-31T07:43:42Z")
I m using logstash to filter logs from jenkins server. I pull out the build.xml files and the problem is the same file is created as two events in elasticsearch. What happens is when the jenkins starts the build the buil… --- ## [Import old postgres data to elasticsearch](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090)
**Author:** [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\ **Replies:** 4\ **Last updated:** [July 31, 2017, 7:21am UTC](https://discuss.elastic.co/t/import-old-postgres-data-to-elasticsearch/95090 "2017-07-31T07:21:23Z")
I have a lot of data in my postgres database( on a remote). This is the data of the past 1 year, and I want to push it to elasticsearch now. The data has a time field in it in this format 2016-09-07 19:26:36.817039+00. … --- ## [\[WARN \]\[logstash.outputs.webhdfs \] Failed to flush outgoing items {:outgoing\_count=\>160, :exception=\>"WebHDFS::IOError"](https://discuss.elastic.co/t/warn-logstash-outputs-webhdfs-failed-to-flush-outgoing-items-outgoing-count-160-exception-webhdfs-ioerror/95128)
**Author:** [@regret](https://discuss.elastic.co/u/regret)\ **Replies:** 0\ **Last updated:** [July 31, 2017, 7:05am UTC](https://discuss.elastic.co/t/warn-logstash-outputs-webhdfs-failed-to-flush-outgoing-items-outgoing-count-160-exception-webhdfs-ioerror/95128 "2017-07-31T07:05:15Z")
Dear all: When I use logstash-output-webhdfs plugin to archive logs into hdfs, I met below error occasionly(not always). Does anyone know what happened and how to fix this ? \[WARN \]\[logstash.outputs.webhdfs \] Failed to… --- ## [Update existing index template](https://discuss.elastic.co/t/update-existing-index-template/95117)
**Author:** [@Shivram\_Subramanian](https://discuss.elastic.co/u/Shivram_Subramanian)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 5:03am UTC](https://discuss.elastic.co/t/update-existing-index-template/95117 "2017-07-31T05:03:45Z")
I have an existing template with specific properties that are applied when new indexes are created. For example, this exists { "final1-template": { "order": 0, "template": "final-\*", "setti… --- ## [Parse out Two Fields from Multiline data in Logstash coming from Filebetes](https://discuss.elastic.co/t/parse-out-two-fields-from-multiline-data-in-logstash-coming-from-filebetes/95055)
**Author:** [@vinaymishra](https://discuss.elastic.co/u/vinaymishra)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 4:05am UTC](https://discuss.elastic.co/t/parse-out-two-fields-from-multiline-data-in-logstash-coming-from-filebetes/95055 "2017-07-31T04:05:48Z")
I have huge message chunk’s coming out from filebetes . each message chunk has multiple below messages OUTPUT\> OUTPUT\>=============================================== OUTPUT\> HiveFilterPDO\_ExternalSource OUTPUT\> … --- ## [Split a string with delimiter and assign to new key - logstash](https://discuss.elastic.co/t/split-a-string-with-delimiter-and-assign-to-new-key-logstash/95056)
**Author:** [@Drift](https://discuss.elastic.co/u/Drift)\ **Replies:** 4\ **Last updated:** [July 31, 2017, 4:01am UTC](https://discuss.elastic.co/t/split-a-string-with-delimiter-and-assign-to-new-key-logstash/95056 "2017-07-31T04:01:47Z")
I am trying to import csv into elasticsearch using logstash. I have a field in the csv with is an IP:Port pair. I would like to split the IP and port, and assign them to two new keys though the logstash config file. I l… --- ## [Not able to check added field in index when drop is used](https://discuss.elastic.co/t/not-able-to-check-added-field-in-index-when-drop-is-used/95074)
**Author:** [@deepanshu\_mittal](https://discuss.elastic.co/u/deepanshu_mittal)\ **Replies:** 8\ **Last updated:** [July 31, 2017, 1:57am UTC](https://discuss.elastic.co/t/not-able-to-check-added-field-in-index-when-drop-is-used/95074 "2017-07-31T01:57:40Z")
I am trying to add the field if any condition matches and then drop . But it is not able to keep the field. Can you please help on same ? if “el5” in \[message\] { mutate { add\_field{ “linuxversion”, “5”} } drop{} } --- ## [Logstash on port 10514. need help for my assignment, thankyou](https://discuss.elastic.co/t/logstash-on-port-10514-need-help-for-my-assignment-thankyou/95101)
**Author:** [@Amity4](https://discuss.elastic.co/u/Amity4)\ **Replies:** 4\ **Last updated:** [July 31, 2017, 1:31am UTC](https://discuss.elastic.co/t/logstash-on-port-10514-need-help-for-my-assignment-thankyou/95101 "2017-07-31T01:31:14Z")
so, i need to take data from port 10514 on server from my virtual machine. i’ve tried but i didn’t success yet . can you help me to configure the logstash? my logstash-syslog.conf content is like this : and the outpu… --- ## [Logstash on port 10514. need help for my assignment](https://discuss.elastic.co/t/logstash-on-port-10514-need-help-for-my-assignment/95096)
**Author:** [@Amity4](https://discuss.elastic.co/u/Amity4)\ **Replies:** 1\ **Last updated:** [July 31, 2017, 1:05am UTC](https://discuss.elastic.co/t/logstash-on-port-10514-need-help-for-my-assignment/95096 "2017-07-31T01:05:53Z")
so, i need to take data from port 10514 on server from my virtual machine. i've tried but i didn't success yet . can you help me to configure the logstash? my logstash-syslog.conf content is like this : and the outpu… --- ## [Timestamp issue on storing logs](https://discuss.elastic.co/t/timestamp-issue-on-storing-logs/94972)
**Author:** [@gugansankar](https://discuss.elastic.co/u/gugansankar)\ **Replies:** 1\ **Last updated:** [July 30, 2017, 1:22pm UTC](https://discuss.elastic.co/t/timestamp-issue-on-storing-logs/94972 "2017-07-30T13:22:12Z")
I don't know why logstash creating timestamp with 1hr lag approx. Initially I thought , problem might be on Elasticsearch. but after I did some analysis, by just redirect logstash out to stdout file. then I saw "wrong t… --- ## [Logstash Started Service with Tow Error](https://discuss.elastic.co/t/logstash-started-service-with-tow-error/95058)
**Author:** [@omid](https://discuss.elastic.co/u/omid)\ **Replies:** 0\ **Last updated:** [July 30, 2017, 5:48am UTC](https://discuss.elastic.co/t/logstash-started-service-with-tow-error/95058 "2017-07-30T05:48:08Z")
im elk and filebeat Started Services but logstash Started (Running) Service with Tow Error ↓ \> logstash\[15340\]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to… --- ## [Logstash doesn't create new indexes](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047)
**Author:** [@Omoidashita](https://discuss.elastic.co/u/Omoidashita)\ **Replies:** 4\ **Last updated:** [July 29, 2017, 9:53pm UTC](https://discuss.elastic.co/t/logstash-doesnt-create-new-indexes/95047 "2017-07-29T21:53:45Z")
am a new user of ELK 5.5.1 (Elasticsearch, Logstash, Kibana). I am making a monitoring server using ELK with Ubuntu 16.4. I have for the moment two sources of data, Netflow from my rooter and Collectd from my server. P… --- ## [Multiple Output Reliability](https://discuss.elastic.co/t/multiple-output-reliability/94772)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 4\ **Last updated:** [July 29, 2017, 7:25pm UTC](https://discuss.elastic.co/t/multiple-output-reliability/94772 "2017-07-29T19:25:44Z")
How can I check if the data were successfully transferred to elasticsearch? For example I got this output. output { elasticsearch1 { hosts =\> \["xxx.xxx.xxx.250:9200"\] index =\> "logstash1-%{+YYYY.MM.dd}" } ela… --- ## [Where can I find rubydebug output](https://discuss.elastic.co/t/where-can-i-find-rubydebug-output/94905)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 3\ **Last updated:** [July 29, 2017, 7:25pm UTC](https://discuss.elastic.co/t/where-can-i-find-rubydebug-output/94905 "2017-07-29T19:25:07Z")
stdout { codec => rubydebug } TIA --- ## [Hot to send a data from logstash server to a remote elasticserach?](https://discuss.elastic.co/t/hot-to-send-a-data-from-logstash-server-to-a-remote-elasticserach/94668)
**Author:** [@kamalbeg](https://discuss.elastic.co/u/kamalbeg)\ **Replies:** 10\ **Last updated:** [July 29, 2017, 8:38am UTC](https://discuss.elastic.co/t/hot-to-send-a-data-from-logstash-server-to-a-remote-elasticserach/94668 "2017-07-29T08:38:05Z")
I have a server where I have installed logstash. I am trying to forward a log file from my Linux server to a remote elastic search server. What config do I need to do to achieve this? I have install ELK only 2 days ago. … --- ## [How to Parse App Log from Apache nifi](https://discuss.elastic.co/t/how-to-parse-app-log-from-apache-nifi/95015)
**Author:** [@john5634](https://discuss.elastic.co/u/john5634)\ **Replies:** 1\ **Last updated:** [July 28, 2017, 10:54pm UTC](https://discuss.elastic.co/t/how-to-parse-app-log-from-apache-nifi/95015 "2017-07-28T22:54:05Z")
We are trying to get the first 3 fields from a text app log file 2017-07-28 11:40:00,256 WARN \[Timer-Driven Process Thread-7\] o.a.h.c.protocol.ResponseProcessCookies Invalid cookie header: "set-cookie: guest\_id=v1%3A150… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=935) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=937)