# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=871 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 872 --- ## [How use translate in json object?](https://discuss.elastic.co/t/how-use-translate-in-json-object/108421)
**Author:** [@kar](https://discuss.elastic.co/u/kar)\ **Replies:** 5\ **Last updated:** [November 21, 2017, 7:45am UTC](https://discuss.elastic.co/t/how-use-translate-in-json-object/108421 "2017-11-21T07:45:09Z")
Hello This is my json: {"serial\_number":"36307","time":"2017-11-20 08:10:40.340 +03:00","transaciton\_id":"","card\_hash":"","component":"READER","level":"DEBUG","message":"Request": SrvOpen: SrvOpenRequest: {"TgComName… --- ## [Validate logstash conf file merge](https://discuss.elastic.co/t/validate-logstash-conf-file-merge/108489)
**Author:** [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Replies:** 1\ **Last updated:** [November 21, 2017, 6:30am UTC](https://discuss.elastic.co/t/validate-logstash-conf-file-merge/108489 "2017-11-21T06:30:39Z")
Greetings folks, I am trying to wrap my head around how logstash merges files, and would like clarification on my setup if possible. in my /etc/logstash/conf.d dir i have the 4 files. -rw-r--r-- 1 root root 186 Nov 2… --- ## [How to create a date and hour field into a timestamp](https://discuss.elastic.co/t/how-to-create-a-date-and-hour-field-into-a-timestamp/108213)
**Author:** [@pgervais](https://discuss.elastic.co/u/pgervais)\ **Replies:** 1\ **Last updated:** [November 21, 2017, 6:25am UTC](https://discuss.elastic.co/t/how-to-create-a-date-and-hour-field-into-a-timestamp/108213 "2017-11-21T06:25:37Z")
I need to combine two fields , one of which contains a date field and the other an hour field into a timestamp . For example, the Date field : "01OCT17" i.e. 1'st of OCT 2017. The Hour field is a numeric value from 0-2… --- ## [Excluding AWS ALB Health check from nginx logs](https://discuss.elastic.co/t/excluding-aws-alb-health-check-from-nginx-logs/108149)
**Author:** [@kersmacko](https://discuss.elastic.co/u/kersmacko)\ **Replies:** 2\ **Last updated:** [November 21, 2017, 6:21am UTC](https://discuss.elastic.co/t/excluding-aws-alb-health-check-from-nginx-logs/108149 "2017-11-21T06:21:29Z")
Hello, I've got a couple of NGINX instances running behind an ALB in AWS. I have Logstash forwarding the instances' access logs to elastic search instances. I've been trying to filter out the ELB health checks from be… --- ## [Kinesis plugin not sending data to s3](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370)
**Author:** [@hemasalma](https://discuss.elastic.co/u/hemasalma)\ **Replies:** 4\ **Last updated:** [November 21, 2017, 6:11am UTC](https://discuss.elastic.co/t/kinesis-plugin-not-sending-data-to-s3/108370 "2017-11-21T06:11:42Z")
Hello, We have created kinesis Stream and sent data to it... logstash version is 5.4 we are trying to send data in Stream to s3 using logstash input plugin (kinesis) and output plugin as (s3) Below is the configurati… --- ## [How to add/create clusters in logstash?](https://discuss.elastic.co/t/how-to-add-create-clusters-in-logstash/108318)
**Author:** [@dsakpal](https://discuss.elastic.co/u/dsakpal)\ **Replies:** 4\ **Last updated:** [November 21, 2017, 5:04am UTC](https://discuss.elastic.co/t/how-to-add-create-clusters-in-logstash/108318 "2017-11-21T05:04:05Z")
I want to add clustering mechanism into the logstash. Can anybody help me out in this? Thanks. --- ## [My Grok filter is not working](https://discuss.elastic.co/t/my-grok-filter-is-not-working/108446)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 5\ **Last updated:** [November 21, 2017, 4:24am UTC](https://discuss.elastic.co/t/my-grok-filter-is-not-working/108446 "2017-11-21T04:24:17Z")
Hi Guys, I am struggling with basic grok filter and seeking your help please Here is my file and would like to parse as geotype, attack\_type "103.40.130.84": "bad reputation" "103.40.132.18": "known attacker" "103.4… --- ## [\[Working example\]Fortigate config for Logstash](https://discuss.elastic.co/t/working-example-fortigate-config-for-logstash/108493)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [November 21, 2017, 4:02am UTC](https://discuss.elastic.co/t/working-example-fortigate-config-for-logstash/108493 "2017-11-21T04:02:00Z")
Hi, I've been working on a Logstash filter for Fortigate syslogs and I finally have it working. I noticed a lot of people on this board and other places asking for a Fortigate config so I decided to upload mine here. Ho… --- ## [Extract data from { }](https://discuss.elastic.co/t/extract-data-from/107862)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 5\ **Last updated:** [November 21, 2017, 2:48am UTC](https://discuss.elastic.co/t/extract-data-from/107862 "2017-11-21T02:48:35Z")
Hi, I'm struggling with parsing Fortigate logs. Right now I got up to the point where I got all log data within a field marked with { logdata }. I'm using a grok filter and a KV filter after that. The problem with the … --- ## [Multiple pipelines using Logstash 6.0.0 docker image on kubernetes](https://discuss.elastic.co/t/multiple-pipelines-using-logstash-6-0-0-docker-image-on-kubernetes/108479)
**Author:** [@troy](https://discuss.elastic.co/u/troy)\ **Replies:** 0\ **Last updated:** [November 21, 2017, 12:45am UTC](https://discuss.elastic.co/t/multiple-pipelines-using-logstash-6-0-0-docker-image-on-kubernetes/108479 "2017-11-21T00:45:00Z")
In order to get multiple pipelines working w/ the official Logstash 6.0.0 docker image, I set the PATH\_CONFIG environment variable to "", which sets path.config: null in /usr/share/logstash/config/logstash.yml. Since pat… --- ## [Create New Index in Kibana without restarting logstash (5.6.2)](https://discuss.elastic.co/t/create-new-index-in-kibana-without-restarting-logstash-5-6-2/108475)
**Author:** [@shashwat](https://discuss.elastic.co/u/shashwat)\ **Replies:** 1\ **Last updated:** [November 20, 2017, 11:44pm UTC](https://discuss.elastic.co/t/create-new-index-in-kibana-without-restarting-logstash-5-6-2/108475 "2017-11-20T23:44:41Z")
Hi, I need to ask a very basic question. Whenever I update my logsatash configuration file, I restart it to get the changes reflected in Kibana. For example, when I add new index in logstash.conf file to read some new l… --- ## [Logstash 5.6 Plugin Installation Conflicts with JMX Remote Port](https://discuss.elastic.co/t/logstash-5-6-plugin-installation-conflicts-with-jmx-remote-port/108474)
**Author:** [@Jorn](https://discuss.elastic.co/u/Jorn)\ **Replies:** 0\ **Last updated:** [November 20, 2017, 11:15pm UTC](https://discuss.elastic.co/t/logstash-5-6-plugin-installation-conflicts-with-jmx-remote-port/108474 "2017-11-20T23:15:27Z")
Good afternoon all, I'm currently chasing a scenario where our puppet config management client runs fail due to an issue installing newer versions of the logstash-output-s3 plugin. The error cited in the logstash-plugi… --- ## [Logstash service stuck in paused state](https://discuss.elastic.co/t/logstash-service-stuck-in-paused-state/108454)
**Author:** [@Ronald\_Gene\_Humphrey](https://discuss.elastic.co/u/Ronald_Gene_Humphrey)\ **Replies:** 0\ **Last updated:** [November 20, 2017, 8:17pm UTC](https://discuss.elastic.co/t/logstash-service-stuck-in-paused-state/108454 "2017-11-20T20:17:03Z")
Installed service and it will not start need a little bit of help used NSSM for install and set paths as follows Path: c:\\ELK-Stack\\logstash\\bin\\logstash.bat Startup directory: c:\\ELK-Stack\\logstash\\bin Arguments: -f … --- ## [Issues with logstash opening ports](https://discuss.elastic.co/t/issues-with-logstash-opening-ports/108250)
**Author:** [@1ntgr](https://discuss.elastic.co/u/1ntgr)\ **Replies:** 3\ **Last updated:** [November 20, 2017, 7:08pm UTC](https://discuss.elastic.co/t/issues-with-logstash-opening-ports/108250 "2017-11-20T19:08:54Z")
Hi everyone, I've done a bit of digging, but have not had much success. I am using v5.6.4 of ELK 1ntgr@elk:~$ /usr/share/logstash/bin/logstash --version logstash 5.6.4 1ntgr@elk:~$ /usr/share/kibana/bin/kibana --ver… --- ## [Logstash aggregate query](https://discuss.elastic.co/t/logstash-aggregate-query/107827)
**Author:** [@Pratiksha\_Tewary](https://discuss.elastic.co/u/Pratiksha_Tewary)\ **Replies:** 1\ **Last updated:** [November 20, 2017, 6:48pm UTC](https://discuss.elastic.co/t/logstash-aggregate-query/107827 "2017-11-20T18:48:49Z")
Hi, We are trying to implement the aggregate query in Logstash using the Elasticsearch input plugin and have the following in our input portion of the Logstash configuration: input { elasticsearch { hosts =\> "\ **Author:** [@outcoldman](https://discuss.elastic.co/u/outcoldman)\ **Replies:** 1\ **Last updated:** [November 20, 2017, 4:39pm UTC](https://discuss.elastic.co/t/logstash-x-pack-is-using-deprecated-feature/108304 "2017-11-20T16:39:43Z") Logstash is reporting \[2017-11-20T01:04:57,970\]\[WARN \]\[logstash.outputs.elasticsearch\] You are using a deprecated config setting "document\_type" set in elasticsearch. Deprecated settings will continue to work, but are s… --- ## [Logstash S3 filename parsing](https://discuss.elastic.co/t/logstash-s3-filename-parsing/108415)
**Author:** [@erikkn](https://discuss.elastic.co/u/erikkn)\ **Replies:** 0\ **Last updated:** [November 20, 2017, 3:26pm UTC](https://discuss.elastic.co/t/logstash-s3-filename-parsing/108415 "2017-11-20T15:26:09Z")
For the input I am using the Logstash S3 plugin. We are putting quite some information in the name of our log files, an example log filename: \-\-\-\.log Could someone help me how to parse … --- ## [Logstash-input-dynamodb plugin is not installing on logstash-5.4.0](https://discuss.elastic.co/t/logstash-input-dynamodb-plugin-is-not-installing-on-logstash-5-4-0/108403)
**Author:** [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Replies:** 0\ **Last updated:** [November 20, 2017, 2:18pm UTC](https://discuss.elastic.co/t/logstash-input-dynamodb-plugin-is-not-installing-on-logstash-5-4-0/108403 "2017-11-20T14:18:15Z")
Hi All... I am trying to install logstash-input-dynamodb plugin in logstash-5.4.0 by following command ./logstash-plugin install logstash-input-dynamodb but i am getting following error ./logstash-plugin install logs… --- ## [Need Help in Logstash Performance Issue](https://discuss.elastic.co/t/need-help-in-logstash-performance-issue/107888)
**Author:** [@Arun\_Raj](https://discuss.elastic.co/u/Arun_Raj)\ **Replies:** 8\ **Last updated:** [November 20, 2017, 1:06pm UTC](https://discuss.elastic.co/t/need-help-in-logstash-performance-issue/107888 "2017-11-20T13:06:16Z")
Hi All, I am having a requirement to filter some 50 GB data thru logstash to view the real time data in kibana currently there is a slight lag in logstash processing could you pl let know what all the possible ways to … --- ## [Logstash Parsing - kv filter](https://discuss.elastic.co/t/logstash-parsing-kv-filter/107714)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 2\ **Last updated:** [November 20, 2017, 12:06pm UTC](https://discuss.elastic.co/t/logstash-parsing-kv-filter/107714 "2017-11-20T12:06:50Z")
Hi All, Iam trying to parse the events using kv filter , my kibana looks likes this \[image\] How to make just from : ebay@ebay.com but iam getting like 78AE89C45C: from : ebay@ebay.com moreover iam getting duplic… --- ## [Docker logging driver gelf doesn't work](https://discuss.elastic.co/t/docker-logging-driver-gelf-doesnt-work/108365)
**Author:** [@dzx912](https://discuss.elastic.co/u/dzx912)\ **Replies:** 1\ **Last updated:** [November 20, 2017, 11:34am UTC](https://discuss.elastic.co/t/docker-logging-driver-gelf-doesnt-work/108365 "2017-11-20T11:34:16Z")
Hello I'm trying run ELK for Docker logging driver I've cloned this project and put a little code: add container generator logs and edit logstash.conf https://github.com/dzx912/docker-elk/commit/b79663fd5f2d1e036d… --- ## [Collecting Logs from IBM SAN controller](https://discuss.elastic.co/t/collecting-logs-from-ibm-san-controller/108367)
**Author:** [@eamonn](https://discuss.elastic.co/u/eamonn)\ **Replies:** 0\ **Last updated:** [November 20, 2017, 10:59am UTC](https://discuss.elastic.co/t/collecting-logs-from-ibm-san-controller/108367 "2017-11-20T10:59:23Z")
As part of the Log collection system I need to design I will need to pick up various infrastructure logs. Specifically i would like to know how do i collect logs from IBM SAN controller, SAN Directors, San storage array… --- ## [JMS input plugin perfromance](https://discuss.elastic.co/t/jms-input-plugin-perfromance/108142)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 3\ **Last updated:** [November 20, 2017, 10:23am UTC](https://discuss.elastic.co/t/jms-input-plugin-perfromance/108142 "2017-11-20T10:23:48Z")
Any help appreciated! I am used JMS input plugin, to read from ActiveMQ topic. The JMS input plugin works, but it seems to only get a fraction of messages, I do not get all the messages from the topic. It seems I only g… --- ## [Occasional mapping failure for numeric field](https://discuss.elastic.co/t/occasional-mapping-failure-for-numeric-field/108358)
**Author:** [@Matthew\_Field](https://discuss.elastic.co/u/Matthew_Field)\ **Replies:** 0\ **Last updated:** [November 20, 2017, 10:19am UTC](https://discuss.elastic.co/t/occasional-mapping-failure-for-numeric-field/108358 "2017-11-20T10:19:43Z")
I am getting occasional mapping errors like this: "type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse", "caused\_by"=\>{"type"=\>"number\_format\_exception", "reason"=\>"For input string: "%{\[doc\]\[geoip\]\[latitude\]}… --- ## [Logstah error \[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will restart this plugin](https://discuss.elastic.co/t/logstah-error-error-logstash-pipeline-a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/108348)
**Author:** [@sailq21](https://discuss.elastic.co/u/sailq21)\ **Replies:** 0\ **Last updated:** [November 20, 2017, 9:18am UTC](https://discuss.elastic.co/t/logstah-error-error-logstash-pipeline-a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/108348 "2017-11-20T09:18:18Z")
\[2017-11-20T17:11:02,435\]\[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will restart this plugin. Plugin: \5044, id=\>"823584548602b7a239986c4d01ff915c41b4c954-1", e… --- ## [Parsing java map to json in logstash](https://discuss.elastic.co/t/parsing-java-map-to-json-in-logstash/108333)
**Author:** [@abhilash3011](https://discuss.elastic.co/u/abhilash3011)\ **Replies:** 2\ **Last updated:** [November 20, 2017, 9:09am UTC](https://discuss.elastic.co/t/parsing-java-map-to-json-in-logstash/108333 "2017-11-20T09:09:04Z")
Hi All, I am storing a java map to my database which is in this form {id=sampleid,name=samplename} actual data is larger but I want to extract individual fields(like id and name) from above log. I tried using json filt… --- ## [Accessing the parameters of ActiveMQ with Logstash](https://discuss.elastic.co/t/accessing-the-parameters-of-activemq-with-logstash/107404)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 2\ **Last updated:** [November 20, 2017, 9:08am UTC](https://discuss.elastic.co/t/accessing-the-parameters-of-activemq-with-logstash/107404 "2017-11-20T09:08:51Z")
I am using stomp as an input for Logstash, which handles ActiveMQ as my source for messages. I would like to access the properties of the message, which has some valuable information for me and I would like to pass it i… --- ## [Logstash warning](https://discuss.elastic.co/t/logstash-warning/107368)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 9\ **Last updated:** [November 20, 2017, 8:54am UTC](https://discuss.elastic.co/t/logstash-warning/107368 "2017-11-20T08:54:10Z")
i get this warning in logstash log \[2017-11-13T10:41:22,803\]\[WARN \]\[logstash.filters.json \] Error parsing json {:source=\>"LogMsg", :raw=\>"UID:420: Out /getData/{city}/{speciality}/{doctorName} all/all/all", :exception=\>#… --- ## [Split field data](https://discuss.elastic.co/t/split-field-data/108327)
**Author:** [@hsiuming](https://discuss.elastic.co/u/hsiuming)\ **Replies:** 1\ **Last updated:** [November 20, 2017, 8:45am UTC](https://discuss.elastic.co/t/split-field-data/108327 "2017-11-20T08:45:18Z")
Hello We now have a ELK system with logstash,elasticsearch and kibana. The logstash configure is: input { beats { port =\> 5044 } } filter { grok { match =\> \[ "message", "%{COMMONAPACHELOG}" \] } } output { i… --- ## [JDBC Input | Statement Help](https://discuss.elastic.co/t/jdbc-input-statement-help/108245)
**Author:** [@obthearchitect](https://discuss.elastic.co/u/obthearchitect)\ **Replies:** 2\ **Last updated:** [November 20, 2017, 6:54am UTC](https://discuss.elastic.co/t/jdbc-input-statement-help/108245 "2017-11-20T06:54:34Z")
Greetings, I'm trying to create a test pipeline that takes data from MySQL and dumps it into Elasticsearch and so far so good, I'm finally able to get Data into the DB. Now what I would like to do is have logstash run e… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=870) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=872)