# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1123 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1124 --- ## [Use case logic](https://discuss.elastic.co/t/use-case-logic/56310)
**Author:** [@Kishore](https://discuss.elastic.co/u/Kishore)\ **Replies:** 1\ **Last updated:** [July 27, 2016, 1:28pm UTC](https://discuss.elastic.co/t/use-case-logic/56310 "2016-07-27T13:28:26Z")
We are working on the Use case that 'User access from different locations', whenever there is change in country name should be captured. Below is the Scenario: kishore: India kishore: USA For the above scenario i wa… --- ## [Converting Splunk Field Extraction in Logstash Equivalent](https://discuss.elastic.co/t/converting-splunk-field-extraction-in-logstash-equivalent/56504)
**Author:** [@darioc](https://discuss.elastic.co/u/darioc)\ **Replies:** 1\ **Last updated:** [July 27, 2016, 1:16pm UTC](https://discuss.elastic.co/t/converting-splunk-field-extraction-in-logstash-equivalent/56504 "2016-07-27T13:16:59Z")
Hello, I have the following field extraction in Splunk (?i)- site\\:(?P\[^;:\]+) How can I achieve a similar result with logstash? Sample log message: INFO 2016-07-21 13:17:48,139 \[http-bio-8080-exec-5\] com.vendo… --- ## [Logstash partial updates](https://discuss.elastic.co/t/logstash-partial-updates/56378)
**Author:** [@lorello](https://discuss.elastic.co/u/lorello)\ **Replies:** 2\ **Last updated:** [July 27, 2016, 7:22am UTC](https://discuss.elastic.co/t/logstash-partial-updates/56378 "2016-07-27T07:22:24Z")
Can logstash run partial updates reading from a logfile? Example: index contains fields ID (used as document id), NAME, VALUE I read data from a logfile made of json lines and containing updates on the same records … --- ## [Parsing a JSON array of objects](https://discuss.elastic.co/t/parsing-a-json-array-of-objects/55441)
**Author:** [@gigavinyl](https://discuss.elastic.co/u/gigavinyl)\ **Replies:** 5\ **Last updated:** [July 27, 2016, 9:23am UTC](https://discuss.elastic.co/t/parsing-a-json-array-of-objects/55441 "2016-07-27T09:23:49Z")
I have a log file that's an array of objects that looks something like this: \[ { "cate1": "data1a", "cate2": "data2a" }, { "cate1": "data1b", "cate2": "data2b" } \] and I need each object … --- ## [Logstash Stuck at Pipeline Main Started](https://discuss.elastic.co/t/logstash-stuck-at-pipeline-main-started/56471)
**Author:** [@Deepak\_Patnaik](https://discuss.elastic.co/u/Deepak_Patnaik)\ **Replies:** 7\ **Last updated:** [July 27, 2016, 6:17am UTC](https://discuss.elastic.co/t/logstash-stuck-at-pipeline-main-started/56471 "2016-07-27T06:17:11Z")
Hi, I ran logstash to index an apache log file. It worked. After that I used an application log file. It also worked. Then all of a sudden when I try to re-execute the apache log file, logstash is stuck at 'Pipeline main… --- ## [Logstash not populating .raw values](https://discuss.elastic.co/t/logstash-not-populating-raw-values/56333)
**Author:** [@john.miles](https://discuss.elastic.co/u/john.miles)\ **Replies:** 6\ **Last updated:** [July 27, 2016, 2:27am UTC](https://discuss.elastic.co/t/logstash-not-populating-raw-values/56333 "2016-07-27T02:27:43Z")
I have filebeat set up to ship my nginx logs over to logstash and logstash is updating my indices with the log data. I am using the logstash-\* template (though my index is called logstash-appname-prod-web-access-logs-YY… --- ## [Assigning value to add\_field new field](https://discuss.elastic.co/t/assigning-value-to-add-field-new-field/56430)
**Author:** [@zoplex](https://discuss.elastic.co/u/zoplex)\ **Replies:** 3\ **Last updated:** [July 27, 2016, 12:16am UTC](https://discuss.elastic.co/t/assigning-value-to-add-field-new-field/56430 "2016-07-27T00:16:09Z")
Hi, I get the field defined in filebeat.yml file as: - paths: - /var/logs/mylog.log document\_type: LOG1 fields: mytype: FORMAT1 ,defining different format spec for each o… --- ## [SNMPTrap not reading snmp traffic through logstash](https://discuss.elastic.co/t/snmptrap-not-reading-snmp-traffic-through-logstash/56199)
**Author:** [@apomerenk](https://discuss.elastic.co/u/apomerenk)\ **Replies:** 11\ **Last updated:** [July 26, 2016, 7:40pm UTC](https://discuss.elastic.co/t/snmptrap-not-reading-snmp-traffic-through-logstash/56199 "2016-07-26T19:40:48Z")
I am trying to send SNMP traffic through Logstash using the SNMPtrap filter. I can see traffic coming into the machine that Logstash is on using TCPdump, and it is directed at the IP and port 162. I have run Logstash as … --- ## [Sending out a filed aggregation count](https://discuss.elastic.co/t/sending-out-a-filed-aggregation-count/55802)
**Author:** [@yossiv](https://discuss.elastic.co/u/yossiv)\ **Replies:** 2\ **Last updated:** [July 26, 2016, 6:18pm UTC](https://discuss.elastic.co/t/sending-out-a-filed-aggregation-count/55802 "2016-07-26T18:18:28Z")
Dear community i want to create a filter that will send me out every minute the aggregated number of on a specific filed for example there is aa query called: SOAPAction:Deposit i want to send every minute the number o… --- ## [Logstash-netflow-codec](https://discuss.elastic.co/t/logstash-netflow-codec/55539)
**Author:** [@abdalians](https://discuss.elastic.co/u/abdalians)\ **Replies:** 2\ **Last updated:** [July 26, 2016, 4:56pm UTC](https://discuss.elastic.co/t/logstash-netflow-codec/55539 "2016-07-26T16:56:31Z")
Hi We are testing out the netflow input plugin for logstash but cant seem to get logstash to output anything. input { udp { port =\> 1963 codec =\> netflow { ipfix\_definitions =\> "/opt/logstash/vendor/bundle/jru… --- ## [Cannot recover from a SERVICE\_UNAVAILABLE/2/no master](https://discuss.elastic.co/t/cannot-recover-from-a-service-unavailable-2-no-master/56165)
**Author:** [@MikeC](https://discuss.elastic.co/u/MikeC)\ **Replies:** 1\ **Last updated:** [July 26, 2016, 2:37pm UTC](https://discuss.elastic.co/t/cannot-recover-from-a-service-unavailable-2-no-master/56165 "2016-07-26T14:37:21Z")
I have a bunch of hosts that im monitoring with logstash and they all send data to my elastic cluster. When the clusters health turn red for reasons, some of the logstash instances seem to die and stop sending data ba… --- ## [JDBC input logstash (JTDS support)](https://discuss.elastic.co/t/jdbc-input-logstash-jtds-support/26367)
**Author:** [@Alvaro\_Gareppe](https://discuss.elastic.co/u/Alvaro_Gareppe)\ **Replies:** 4\ **Last updated:** [July 26, 2016, 1:10pm UTC](https://discuss.elastic.co/t/jdbc-input-logstash-jtds-support/26367 "2016-07-26T13:10:59Z")
Any experience using jdbc input with jtds connecting to a SQL server DB ? I'm having an issue on start that says "are you sure you've included the correct jdbc driver in: jdbc\_driver\_library? " jar is in the config… --- ## [Run Daemon Logstash](https://discuss.elastic.co/t/run-daemon-logstash/56384)
**Author:** [@hatienkma](https://discuss.elastic.co/u/hatienkma)\ **Replies:** 3\ **Last updated:** [July 26, 2016, 11:04am UTC](https://discuss.elastic.co/t/run-daemon-logstash/56384 "2016-07-26T11:04:13Z")
The same thread name. I have problem via Logstash when starting. I want to insert logfile (eve.json) from Suricata to Etaslicsearch. I have use Logstash to import. When i enter command: "/opt/logstash/bin/logstash -f… --- ## [Processing log files while using logrotate](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174)
**Author:** [@rspangle1](https://discuss.elastic.co/u/rspangle1)\ **Replies:** 8\ **Last updated:** [July 26, 2016, 8:52am UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174 "2016-07-26T08:52:33Z")
I was wondering what everybody's opinion is on processing log files read by logstash but need to be rotated using logrotate. It appears that the two don't work very well together. I'm trying to make sure we don't loose… --- ## [Should I expect 2.3 kafka plugins to work with kafka 0.10?](https://discuss.elastic.co/t/should-i-expect-2-3-kafka-plugins-to-work-with-kafka-0-10/56213)
**Author:** [@memelet](https://discuss.elastic.co/u/memelet)\ **Replies:** 4\ **Last updated:** [July 26, 2016, 7:53am UTC](https://discuss.elastic.co/t/should-i-expect-2-3-kafka-plugins-to-work-with-kafka-0-10/56213 "2016-07-26T07:53:31Z")
All our other kafka 0.9 clients work fine with 0.10 brokers. But whenever I start a kafka-output logstash hangs, and can only be recovered with kill -9. Nothing in the logs. (Have not run with -debug yet.) kafka-input … --- ## [Configure log4j for logstash using dockerfile](https://discuss.elastic.co/t/configure-log4j-for-logstash-using-dockerfile/55656)
**Author:** [@ssable](https://discuss.elastic.co/u/ssable)\ **Replies:** 2\ **Last updated:** [July 26, 2016, 7:41am UTC](https://discuss.elastic.co/t/configure-log4j-for-logstash-using-dockerfile/55656 "2016-07-26T07:41:32Z")
I recently started working on logstash. I am running it in docker and I want to read its logs using 'docker logs ' command. But I am getting error log4j:WARN No appenders could be found for logger (org.apache.http.clie… --- ## [How to use --pluginpath in logstash 2.3.4](https://discuss.elastic.co/t/how-to-use-pluginpath-in-logstash-2-3-4/56087)
**Author:** [@Lyne\_Lin](https://discuss.elastic.co/u/Lyne_Lin)\ **Replies:** 1\ **Last updated:** [July 26, 2016, 7:01am UTC](https://discuss.elastic.co/t/how-to-use-pluginpath-in-logstash-2-3-4/56087 "2016-07-26T07:01:37Z")
I read official tutorial, searched google and this forum, but still failed to get my local aaa.rb filter work correctly. Please help to see whether I missed something. In logstash 2.3.4 (/opt/logstash), 1. put my aaa.r… --- ## [Logs are not collecting](https://discuss.elastic.co/t/logs-are-not-collecting/56348)
**Author:** [@disalassalyjose](https://discuss.elastic.co/u/disalassalyjose)\ **Replies:** 3\ **Last updated:** [July 26, 2016, 4:43am UTC](https://discuss.elastic.co/t/logs-are-not-collecting/56348 "2016-07-26T04:43:46Z")
Hi, I am able to get the logs from one client server which is windows 12 r2 but not able to get the logs from another client server, below are the configuration that are done in both server side. Server :Ubuntu 14.04 cli… --- ## [Running multiple independent logstash config files with input,filter and output](https://discuss.elastic.co/t/running-multiple-independent-logstash-config-files-with-input-filter-and-output/29757)
**Author:** [@shovik](https://discuss.elastic.co/u/shovik)\ **Replies:** 6\ **Last updated:** [July 25, 2016, 4:41pm UTC](https://discuss.elastic.co/t/running-multiple-independent-logstash-config-files-with-input-filter-and-output/29757 "2016-07-25T16:41:27Z")
Hi, We have multiple applications running on a single server and we are collecting the logs from these application and sending it to a intermediate queues using logstash forwarders. As, the log formats, processing a… --- ## [Error "Ignoring ffi-1.9.13 because its extensions are not built"](https://discuss.elastic.co/t/error-ignoring-ffi-1-9-13-because-its-extensions-are-not-built/56166)
**Author:** [@Saurabh\_Jambhule](https://discuss.elastic.co/u/Saurabh_Jambhule)\ **Replies:** 2\ **Last updated:** [July 25, 2016, 4:02pm UTC](https://discuss.elastic.co/t/error-ignoring-ffi-1-9-13-because-its-extensions-are-not-built/56166 "2016-07-25T16:02:35Z")
I am getting error while installing/ updating plugin logstash-input-beats as: "Ignoring ffi-1.9.13 because its extensions are not built. Try: gem pristine ffi --version 1.9.13" But when do that (gem pristine ffi --v… --- ## [Kafka input/output swapped?](https://discuss.elastic.co/t/kafka-input-output-swapped/56285)
**Author:** [@memelet](https://discuss.elastic.co/u/memelet)\ **Replies:** 1\ **Last updated:** [July 25, 2016, 12:10pm UTC](https://discuss.elastic.co/t/kafka-input-output-swapped/56285 "2016-07-25T12:10:58Z")
I seem to getting my input and output swapped: \[ops-elk-1\]$ sudo ./logstash-plugin install --version='3.0.2' logstash-input-kafka Validating logstash-input-kafka-3.0.2 Installing logstash-input-kafka Installation succe… --- ## [Logstash file plugin](https://discuss.elastic.co/t/logstash-file-plugin/56267)
**Author:** [@Kishore](https://discuss.elastic.co/u/Kishore)\ **Replies:** 12\ **Last updated:** [July 25, 2016, 9:58am UTC](https://discuss.elastic.co/t/logstash-file-plugin/56267 "2016-07-25T09:58:29Z")
I am using file Output plugin to store parsed events, is there any option to clear that file for every 24hr? Below is the Example: output { if ( \[status\] =~ "Invalid credentials" ) and "throttled" not in \[tags\] { f… --- ## [Transport error : Cannot get new connection from pool](https://discuss.elastic.co/t/transport-error-cannot-get-new-connection-from-pool/53319)
**Author:** [@rupal](https://discuss.elastic.co/u/rupal)\ **Replies:** 1\ **Last updated:** [July 25, 2016, 9:57am UTC](https://discuss.elastic.co/t/transport-error-cannot-get-new-connection-from-pool/53319 "2016-07-25T09:57:22Z")
Transport error : Cannot get new connection from pool. How to solve connection error. --- ## [Config works if I run logstash on it, but not when run as logstash service](https://discuss.elastic.co/t/config-works-if-i-run-logstash-on-it-but-not-when-run-as-logstash-service/53469)
**Author:** [@timogoosen](https://discuss.elastic.co/u/timogoosen)\ **Replies:** 2\ **Last updated:** [July 25, 2016, 9:46am UTC](https://discuss.elastic.co/t/config-works-if-i-run-logstash-on-it-but-not-when-run-as-logstash-service/53469 "2016-07-25T09:46:55Z")
Hi guys. I have a config that works if I run logstash on the config individually like this: $ /opt/logstash/bin/logstash -f someconfig.conf Or if I run it as root for that matter, but if I run the logstash as a serv… --- ## [Http input plugin - multiple log entries group into one in Kibana](https://discuss.elastic.co/t/http-input-plugin-multiple-log-entries-group-into-one-in-kibana/56101)
**Author:** [@hawaiipersson](https://discuss.elastic.co/u/hawaiipersson)\ **Replies:** 4\ **Last updated:** [July 25, 2016, 7:55am UTC](https://discuss.elastic.co/t/http-input-plugin-multiple-log-entries-group-into-one-in-kibana/56101 "2016-07-25T07:55:48Z")
Hi I am fairly new to Logstash so excuse me in advanced. Using the http input plugin to send logs from a application to logstash. They are sent on a schedule, 5-20 logs at a time. Each log entry is sent with its own … --- ## [Bad File descriptor error with Large dictionary file (\>10MB) in Logstash](https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225)
**Author:** [@nishant\_goel](https://discuss.elastic.co/u/nishant_goel)\ **Replies:** 2\ **Last updated:** [July 24, 2016, 6:31pm UTC](https://discuss.elastic.co/t/bad-file-descriptor-error-with-large-dictionary-file-10mb-in-logstash/56225 "2016-07-24T18:31:12Z")
I have multiple log messages in a file which I am processing using logstash filter plugins. Then, the filtered logs are getting sent to elasticsearch. There is one field called addID in a log message. I want to drop al… --- ## [Check if port is open with Logstash/Elasticsearch](https://discuss.elastic.co/t/check-if-port-is-open-with-logstash-elasticsearch/56177)
**Author:** [@beelastic](https://discuss.elastic.co/u/beelastic)\ **Replies:** 2\ **Last updated:** [July 24, 2016, 11:24am UTC](https://discuss.elastic.co/t/check-if-port-is-open-with-logstash-elasticsearch/56177 "2016-07-24T11:24:56Z")
I want to be alerted (with wather) if a specific port is closed. To achiev this i want to create a field with logstash (e.g. port\_status), and fill it with open or close, whether the port is open or closed. How can i… --- ## [Accessing field of JSON Input](https://discuss.elastic.co/t/accessing-field-of-json-input/56068)
**Author:** [@Daniel\_Tebernum](https://discuss.elastic.co/u/Daniel_Tebernum)\ **Replies:** 2\ **Last updated:** [July 24, 2016, 10:36am UTC](https://discuss.elastic.co/t/accessing-field-of-json-input/56068 "2016-07-24T10:36:17Z")
Hello! I have a problem in accessing a json field. I've tried multiple solutions but had no success. What I've done: I have this config file: input { http { port =\> 5001 codec =\> "json" } } filter { … --- ## [Gem dependency version conflict among logstash plugins](https://discuss.elastic.co/t/gem-dependency-version-conflict-among-logstash-plugins/55590)
**Author:** [@kaung](https://discuss.elastic.co/u/kaung)\ **Replies:** 4\ **Last updated:** [July 24, 2016, 5:28am UTC](https://discuss.elastic.co/t/gem-dependency-version-conflict-among-logstash-plugins/55590 "2016-07-24T05:28:39Z")
I am pretty new to ruby ecosystem; so apologies first if I am missing something basic :slight\_smile: I understand that plugins are maintained and released separately from logstash-core. Furthermore, plugins are mainta… --- ## [Using KV filter](https://discuss.elastic.co/t/using-kv-filter/56221)
**Author:** [@soodlikesjava](https://discuss.elastic.co/u/soodlikesjava)\ **Replies:** 3\ **Last updated:** [July 24, 2016, 2:11am UTC](https://discuss.elastic.co/t/using-kv-filter/56221 "2016-07-24T02:11:49Z")
Hi team I have a requirement to create fields by matching key\>value in the event . but KV filter should consider only those KV filters which are enclosed under brackets like message : logging data (key1\>value1,key2\>val… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1122) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1124)