# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=904 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 905 --- ## [Need to config codec-cef as output](https://discuss.elastic.co/t/need-to-config-codec-cef-as-output/101596)
**Author:** [@GabayAvi](https://discuss.elastic.co/u/GabayAvi)\ **Replies:** 0\ **Last updated:** [September 24, 2017, 10:04am UTC](https://discuss.elastic.co/t/need-to-config-codec-cef-as-output/101596 "2017-09-24T10:04:48Z")
i need to take data from elk and put in arcsight syslog elasticsearch \>\> logstash \>\> archsight syslog did any one have an example of logstash configuration for this kind of request? thanks Avinoam --- ## [Best way to put pcap traffic into logstash/Elasticsearch](https://discuss.elastic.co/t/best-way-to-put-pcap-traffic-into-logstash-elasticsearch/101237)
**Author:** [@nuggetinu](https://discuss.elastic.co/u/nuggetinu)\ **Replies:** 2\ **Last updated:** [September 24, 2017, 9:35am UTC](https://discuss.elastic.co/t/best-way-to-put-pcap-traffic-into-logstash-elasticsearch/101237 "2017-09-24T09:35:31Z")
Hi, I am interested in creating a dashboard which will visualize network traffic collected from access points. I currently have tcpdump running on an access point and outputting to a pcap file. My intention is to pcap d… --- ## [Jdbc\_page\_size doesn't dump all my data to elastic search](https://discuss.elastic.co/t/jdbc-page-size-doesnt-dump-all-my-data-to-elastic-search/101590)
**Author:** [@woshitom](https://discuss.elastic.co/u/woshitom)\ **Replies:** 0\ **Last updated:** [September 24, 2017, 4:44am UTC](https://discuss.elastic.co/t/jdbc-page-size-doesnt-dump-all-my-data-to-elastic-search/101590 "2017-09-24T04:44:31Z")
I'd like to export tom\_test2 postgresql table to elastic search. The table has 176805 rows: =\> select count(\*) from tom\_test2; count -------- 176805 (1 row) The following logstach conf file import correctly my data… --- ## [Logstash-plugins: 'logstash-input-beats'](https://discuss.elastic.co/t/logstash-plugins-logstash-input-beats/101170)
**Author:** [@Invictus](https://discuss.elastic.co/u/Invictus)\ **Replies:** 1\ **Last updated:** [September 23, 2017, 6:12pm UTC](https://discuss.elastic.co/t/logstash-plugins-logstash-input-beats/101170 "2017-09-23T18:12:24Z")
Hello Elastic-Forum, i have a question concerning plugins on logstash. Is the plugin 'logstash-input-beats' already preinstalled after setting up logstash? Because it is mentioned in the list: D:\\ELK-Stack\\logstash\\bi… --- ## [Extract source filename in logstash received from Filebeat](https://discuss.elastic.co/t/extract-source-filename-in-logstash-received-from-filebeat/101108)
**Author:** [@hasan\_sharekhan](https://discuss.elastic.co/u/hasan_sharekhan)\ **Replies:** 1\ **Last updated:** [September 23, 2017, 6:08pm UTC](https://discuss.elastic.co/t/extract-source-filename-in-logstash-received-from-filebeat/101108 "2017-09-23T18:08:58Z")
I am using ELK v5.5 How do i extract only log file name from the source log file name send by filebeat to logstash I am processing dozens of logs files and will be having different paths and log file names, I just want… --- ## [The existance of logstash](https://discuss.elastic.co/t/the-existance-of-logstash/101569)
**Author:** [@hsafe](https://discuss.elastic.co/u/hsafe)\ **Replies:** 1\ **Last updated:** [September 23, 2017, 4:30pm UTC](https://discuss.elastic.co/t/the-existance-of-logstash/101569 "2017-09-23T16:30:40Z")
Hi everyone, I have been trying to figure out if logstash needs to be present anymore after the beats ability injecting directly to elasticsearch in post 5.4 versions in particular. Now we see beats of all sorts capable… --- ## [Create new event in ruby filter](https://discuss.elastic.co/t/create-new-event-in-ruby-filter/101528)
**Author:** [@vl.zemtsov](https://discuss.elastic.co/u/vl.zemtsov)\ **Replies:** 4\ **Last updated:** [September 23, 2017, 3:28pm UTC](https://discuss.elastic.co/t/create-new-event-in-ruby-filter/101528 "2017-09-23T15:28:59Z")
Hi. I cant create new event in fuby filter. (Use logstash 5.6.0) In filter: filter { ruby { code =\> " yield LogStash::Event.new( 'timestamp' =\> event.get('timestamp'),… --- ## [Logstash Conf File for Apache Log won't interpret "message"](https://discuss.elastic.co/t/logstash-conf-file-for-apache-log-wont-interpret-message/101378)
**Author:** [@dahansi](https://discuss.elastic.co/u/dahansi)\ **Replies:** 14\ **Last updated:** [September 23, 2017, 7:53am UTC](https://discuss.elastic.co/t/logstash-conf-file-for-apache-log-wont-interpret-message/101378 "2017-09-23T07:53:41Z")
Hey there, I am pretty new to Elasticsearch. My ELK-setup on ubuntu is working so far and I receive the logfiles from the client server with filebeat. Now I want to interpret the accesslog file. It looks like this: … --- ## [Logstash 5.6.0 deserialization errors when persistent queues enabled](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952)
**Author:** [@Zt\_Zeng](https://discuss.elastic.co/u/Zt_Zeng)\ **Replies:** 4\ **Last updated:** [September 23, 2017, 1:27am UTC](https://discuss.elastic.co/t/logstash-5-6-0-deserialization-errors-when-persistent-queues-enabled/100952 "2017-09-23T01:27:16Z")
The same problem in a closed post Version: 5.5.0 & 5.6.0 OS: Linux ubuntu-100 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86\_64 x86\_64 x86\_64 GNU/Linux Config file: logstash.yml with x… --- ## [Gelf input message being truncated after 2KB](https://discuss.elastic.co/t/gelf-input-message-being-truncated-after-2kb/101547)
**Author:** [@ksrb](https://discuss.elastic.co/u/ksrb)\ **Replies:** 0\ **Last updated:** [September 22, 2017, 11:56pm UTC](https://discuss.elastic.co/t/gelf-input-message-being-truncated-after-2kb/101547 "2017-09-22T23:56:17Z")
Hi I'm using the gelf driver with docker some of our logs are being truncated after exactly 16384 characters (2 KBs). Is this expected and if it is there some configuration to increase the message buffer size? Related … --- ## [The fliter plugin mutate-convert method only convert the first element of an array](https://discuss.elastic.co/t/the-fliter-plugin-mutate-convert-method-only-convert-the-first-element-of-an-array/47100)
**Author:** [@Morriaty](https://discuss.elastic.co/u/Morriaty)\ **Replies:** 4\ **Last updated:** [September 22, 2017, 8:32pm UTC](https://discuss.elastic.co/t/the-fliter-plugin-mutate-convert-method-only-convert-the-first-element-of-an-array/47100 "2017-09-22T20:32:55Z")
I used jdbc input plugin to select a column named available\_cities from mysql. It was a string field which formatted like this "12, 13, 143, 123...." And the filter was filter { mutate { split =\> { "availa… --- ## [S3 Output filenames to long on v4.x](https://discuss.elastic.co/t/s3-output-filenames-to-long-on-v4-x/101529)
**Author:** [@js77usf](https://discuss.elastic.co/u/js77usf)\ **Replies:** 0\ **Last updated:** [September 22, 2017, 6:36pm UTC](https://discuss.elastic.co/t/s3-output-filenames-to-long-on-v4-x/101529 "2017-09-22T18:36:30Z")
When using the S3 Output plugin with Windows, the UUID has so many characters in the output file name that often the output file name (with tags applied) is too long for Windows OS and the process fails. This happens wh… --- ## [Data won't appear in elasticsearch/kibana](https://discuss.elastic.co/t/data-wont-appear-in-elasticsearch-kibana/101525)
**Author:** [@marleybobby](https://discuss.elastic.co/u/marleybobby)\ **Replies:** 0\ **Last updated:** [September 22, 2017, 6:12pm UTC](https://discuss.elastic.co/t/data-wont-appear-in-elasticsearch-kibana/101525 "2017-09-22T18:12:30Z")
I need to parse the log files of nginx. But no data appear in elastichsearch/kibana. Elasticsearch is up and running and receives data from beats like metricbeat, heartbeat. nginx-pipeline.conf: input { beats… --- ## [Logstash output not working when using logstash-s3-plugin](https://discuss.elastic.co/t/logstash-output-not-working-when-using-logstash-s3-plugin/101404)
**Author:** [@dwdw](https://discuss.elastic.co/u/dwdw)\ **Replies:** 3\ **Last updated:** [September 22, 2017, 5:31pm UTC](https://discuss.elastic.co/t/logstash-output-not-working-when-using-logstash-s3-plugin/101404 "2017-09-22T17:31:05Z")
Configuration with input file from local file system writes to output file and output elasticsearch as expected. When using logtash-input-s3 input{s3{ etc }} the output to elasticsearch and file only write one of the fil… --- ## [Logstash-output-zabbix (question about multi\_value)](https://discuss.elastic.co/t/logstash-output-zabbix-question-about-multi-value/101515)
**Author:** [@vl.zemtsov](https://discuss.elastic.co/u/vl.zemtsov)\ **Replies:** 3\ **Last updated:** [September 22, 2017, 4:21pm UTC](https://discuss.elastic.co/t/logstash-output-zabbix-question-about-multi-value/101515 "2017-09-22T16:21:30Z")
Hello. I dont understand how use multi\_value in logstash (zabbix output). (Use logstash 5.5.2) For example: in filter: filter { mutate { add\_field =\> { "zabbix\_key" =\> \[ "zabbix.key.1", "zabb… --- ## [RabbitMQ output not sending any event](https://discuss.elastic.co/t/rabbitmq-output-not-sending-any-event/101510)
**Author:** [@Flavio\_Campana](https://discuss.elastic.co/u/Flavio_Campana)\ **Replies:** 0\ **Last updated:** [September 22, 2017, 3:09pm UTC](https://discuss.elastic.co/t/rabbitmq-output-not-sending-any-event/101510 "2017-09-22T15:09:17Z")
Hi, we are trying to ship some IIS log to Elasticsearch using logstash, with a RabbitMQ instance to act as a buffer, something like (IIS Server \[IIS W3C logs\] -\> \[logstash\]) -\> \[RabbitMQ\] -\> \[Logstash\] -\> \[ElasticSearch… --- ## [S3 output plugin](https://discuss.elastic.co/t/s3-output-plugin/101321)
**Author:** [@mlrgoh](https://discuss.elastic.co/u/mlrgoh)\ **Replies:** 1\ **Last updated:** [September 22, 2017, 2:21pm UTC](https://discuss.elastic.co/t/s3-output-plugin/101321 "2017-09-22T14:21:31Z")
I've a question about S3 output plugin which I can't find any answer in the Logstash documentation site. Does the S3 output plugin upload logstash events over secure communication protocol? As the events I'm uploading … --- ## [Transaction/Acknowledgement for JMS plugin](https://discuss.elastic.co/t/transaction-acknowledgement-for-jms-plugin/101481)
**Author:** [@roblopes](https://discuss.elastic.co/u/roblopes)\ **Replies:** 0\ **Last updated:** [September 22, 2017, 12:24pm UTC](https://discuss.elastic.co/t/transaction-acknowledgement-for-jms-plugin/101481 "2017-09-22T12:24:48Z")
Hi, Is there any way to rollback the message back to JMS (I'm using TIBCO EMS' implementation of JMS)? I'm not comfortable having the auto-acknowledgement, considering that something might go wrong. I can't put the mess… --- ## [Persistent sequence number and sorting the data in Kibana](https://discuss.elastic.co/t/persistent-sequence-number-and-sorting-the-data-in-kibana/101475)
**Author:** [@mberger1](https://discuss.elastic.co/u/mberger1)\ **Replies:** 0\ **Last updated:** [September 22, 2017, 11:58am UTC](https://discuss.elastic.co/t/persistent-sequence-number-and-sorting-the-data-in-kibana/101475 "2017-09-22T11:58:27Z")
Hey! We are using the Elastic Stack with version 5.5. Until now, I managed to solve every issue, but this one got me good. When our devices produce more than one event per millisecond, Kibana wont show the events in t… --- ## [Rabbitmq input metadata](https://discuss.elastic.co/t/rabbitmq-input-metadata/101348)
**Author:** [@Charlotte\_EGM](https://discuss.elastic.co/u/Charlotte_EGM)\ **Replies:** 3\ **Last updated:** [September 22, 2017, 11:19am UTC](https://discuss.elastic.co/t/rabbitmq-input-metadata/101348 "2017-09-22T11:19:26Z")
Hi, I would like to see the metadata of my rabbitmq input. I have set metadata\_enabled to true and then i added this : add\_field =\> { "topic" =\> "%{\[@metadata\]\[rabbitmq\_properties\]\[routing-key\]}"} Unfortunately I en… --- ## [How to split and parse message properly with different plugins?](https://discuss.elastic.co/t/how-to-split-and-parse-message-properly-with-different-plugins/101453)
**Author:** [@sipelg](https://discuss.elastic.co/u/sipelg)\ **Replies:** 1\ **Last updated:** [September 22, 2017, 11:10am UTC](https://discuss.elastic.co/t/how-to-split-and-parse-message-properly-with-different-plugins/101453 "2017-09-22T11:10:04Z")
Hi, I've been cracking my head for some time now but i just can't get it right. Logstash version 5.5.2 The goal is simple: take first part of message, treat it as 'timestamp' \[ optional: drop/omit the first part, \], t… --- ## [Help with date parsing](https://discuss.elastic.co/t/help-with-date-parsing/101472)
**Author:** [@111126](https://discuss.elastic.co/u/111126)\ **Replies:** 0\ **Last updated:** [September 22, 2017, 10:54am UTC](https://discuss.elastic.co/t/help-with-date-parsing/101472 "2017-09-22T10:54:51Z")
Hello How I do parse date like this September 22nd 2017, 12:53:17.000 I try that: match =\> \["time\_local", "MMMM dd yyyy, HH:mm:ss.SSS"\] But I have tag: \_dateparsefailure --- ## [Logstash process keeps dying](https://discuss.elastic.co/t/logstash-process-keeps-dying/96069)
**Author:** [@cuttlefishjones](https://discuss.elastic.co/u/cuttlefishjones)\ **Replies:** 14\ **Last updated:** [September 22, 2017, 9:26am UTC](https://discuss.elastic.co/t/logstash-process-keeps-dying/96069 "2017-09-22T09:26:58Z")
Hi, I have a couple of test instances of ELK running and I am running into a weird issue with Logstash. The process keeps hanging and refuses to restart with this error: ERROR logstash.agent - Cannot create pipeline {:… --- ## [GROK filters timeout and 'value too large' message](https://discuss.elastic.co/t/grok-filters-timeout-and-value-too-large-message/99825)
**Author:** [@cuttlefishjones](https://discuss.elastic.co/u/cuttlefishjones)\ **Replies:** 1\ **Last updated:** [September 22, 2017, 9:25am UTC](https://discuss.elastic.co/t/grok-filters-timeout-and-value-too-large-message/99825 "2017-09-22T09:25:33Z")
Hi, I'm having problems parsing custom Apache logs. I've built custom GROKS using the debugger website. On there they match instantly but when I put them into logstash they just constantly timeout or throw the 'value too… --- ## [Logstash nested array JSON parsing](https://discuss.elastic.co/t/logstash-nested-array-json-parsing/101100)
**Author:** [@aditya\_soni](https://discuss.elastic.co/u/aditya_soni)\ **Replies:** 14\ **Last updated:** [September 22, 2017, 8:49am UTC](https://discuss.elastic.co/t/logstash-nested-array-json-parsing/101100 "2017-09-22T08:49:17Z")
I have a JSON log of my application which contains elements in a nested array form, here is the sample of it:- { "msgs": \[ { "ts": "2017-09-04T07:07:46.37098Z", "tid": 25, "lvl": "Informat… --- ## [Kafka with elk stack](https://discuss.elastic.co/t/kafka-with-elk-stack/101430)
**Author:** [@Jay\_Dabhi](https://discuss.elastic.co/u/Jay_Dabhi)\ **Replies:** 5\ **Last updated:** [September 22, 2017, 8:48am UTC](https://discuss.elastic.co/t/kafka-with-elk-stack/101430 "2017-09-22T08:48:47Z")
Hi, When should we use kafka in ELK stack? Where should we put kafka module in ELK stack provided filebeat is being used for log shipping? --- ## [Not able to install Translate Plugin](https://discuss.elastic.co/t/not-able-to-install-translate-plugin/101447)
**Author:** [@sreejiths](https://discuss.elastic.co/u/sreejiths)\ **Replies:** 1\ **Last updated:** [September 22, 2017, 8:48am UTC](https://discuss.elastic.co/t/not-able-to-install-translate-plugin/101447 "2017-09-22T08:48:07Z")
Hi ..I am trying to install Translate plugin on Logstash but hav issues in Installing ..My logstash is running on CENTOS 7 (UAT) & RHEL 7 (PROD) and running logtash version 5.2 ..I need to perform offline installation i… --- ## [Match a IP and add a Name](https://discuss.elastic.co/t/match-a-ip-and-add-a-name/101436)
**Author:** [@lueneburger](https://discuss.elastic.co/u/lueneburger)\ **Replies:** 2\ **Last updated:** [September 22, 2017, 8:03am UTC](https://discuss.elastic.co/t/match-a-ip-and-add-a-name/101436 "2017-09-22T08:03:33Z")
Hi everyone, short question: is it possible to match the client\_Ip field against a set of ips -\> names? right now i collect my haproxy logs into ES and can see the client\_IP without any problem, all IPs are whiteliste… --- ## [Scaling ELK stack](https://discuss.elastic.co/t/scaling-elk-stack/101416)
**Author:** [@manthan\_patel](https://discuss.elastic.co/u/manthan_patel)\ **Replies:** 4\ **Last updated:** [September 22, 2017, 6:33am UTC](https://discuss.elastic.co/t/scaling-elk-stack/101416 "2017-09-22T06:33:12Z")
Hi, I have ELK stack on AWS. It has capacity of around 5000 UDP msg/sec. I want to scale it up to capacity of 20,000 msg/sec. What would be the recommended architecture for ELK stack. Will it help If I keep N number o… --- ## [How to define metric's @timestamp from Data collection side(which is beat side)?](https://discuss.elastic.co/t/how-to-define-metrics-timestamp-from-data-collection-side-which-is-beat-side/100500)
**Author:** [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\ **Replies:** 3\ **Last updated:** [September 22, 2017, 6:26am UTC](https://discuss.elastic.co/t/how-to-define-metrics-timestamp-from-data-collection-side-which-is-beat-side/100500 "2017-09-22T06:26:11Z")
Dear folks, Recently I've been working on customizing beat with bash to generate the json data that logstash needed instead of to write own beat with go. Cuz I'm not the go expert. I'm good to implement customizing be… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=903) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=905)