# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1128 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1129 --- ## [Log4j in integration with logstash](https://discuss.elastic.co/t/log4j-in-integration-with-logstash/55333)
**Author:** [@pavana](https://discuss.elastic.co/u/pavana)\ **Replies:** 4\ **Last updated:** [July 13, 2016, 1:39pm UTC](https://discuss.elastic.co/t/log4j-in-integration-with-logstash/55333 "2016-07-13T13:39:16Z")
Is it possible to configure the data in message field in logged data. For example , I want to give data in message field while logging, whose logfield is INFO as shown below. 2016-06-16 INFO main - Airtel,500060,1000… --- ## [Logstash Pipeline Main Started, yet no output](https://discuss.elastic.co/t/logstash-pipeline-main-started-yet-no-output/55375)
**Author:** [@Rohit\_Shukla](https://discuss.elastic.co/u/Rohit_Shukla)\ **Replies:** 4\ **Last updated:** [July 13, 2016, 9:09am UTC](https://discuss.elastic.co/t/logstash-pipeline-main-started-yet-no-output/55375 "2016-07-13T09:09:33Z")
Hi, Yesterday I was able to read the file sent it to Elasticsearch and view it in Kibana using below config: input { file { path =\> "C:/ELK/logstash/bin/data/TestSysData" start\_position =\> beginni… --- ## [Prevent Logstash from stopping](https://discuss.elastic.co/t/prevent-logstash-from-stopping/55295)
**Author:** [@NicoYUE](https://discuss.elastic.co/u/NicoYUE)\ **Replies:** 2\ **Last updated:** [July 13, 2016, 8:24am UTC](https://discuss.elastic.co/t/prevent-logstash-from-stopping/55295 "2016-07-13T08:24:55Z")
Hello, I currently use Logstash to retrieve data from a SQL Database and send it my Elasticsearch cluster. I run Logstash with a command line (service won't work but it's another issue): /opt/logstash/bin/logstash … --- ## [Pushing data to kafka topic using logstash](https://discuss.elastic.co/t/pushing-data-to-kafka-topic-using-logstash/46766)
**Author:** [@rohit\_prusty](https://discuss.elastic.co/u/rohit_prusty)\ **Replies:** 6\ **Last updated:** [July 13, 2016, 7:53am UTC](https://discuss.elastic.co/t/pushing-data-to-kafka-topic-using-logstash/46766 "2016-07-13T07:53:40Z")
We are using kafka-0.8.2.1 and integrating it with logstash-2.3.0 and elasticsearch-2.3.0. We are able to read from the kafka topic using "logstash-input-kafka (2.0.6)" input plugin and store data in elasticsearch. B… --- ## [Insert to one field "One Of" two fields ot tokens](https://discuss.elastic.co/t/insert-to-one-field-one-of-two-fields-ot-tokens/55371)
**Author:** [@shaigb](https://discuss.elastic.co/u/shaigb)\ **Replies:** 2\ **Last updated:** [July 13, 2016, 7:27am UTC](https://discuss.elastic.co/t/insert-to-one-field-one-of-two-fields-ot-tokens/55371 "2016-07-13T07:27:11Z")
Hi I'm trying to populate a certain field (lets call it ioc) with one of 2 values: in 1 log it will be an IP Address and in another it will be a Domain. Is there a way to tell logstash to take the field that exists? when… --- ## [Grok and date filter](https://discuss.elastic.co/t/grok-and-date-filter/55376)
**Author:** [@Chanawit\_Onchoo](https://discuss.elastic.co/u/Chanawit_Onchoo)\ **Replies:** 5\ **Last updated:** [July 13, 2016, 7:22am UTC](https://discuss.elastic.co/t/grok-and-date-filter/55376 "2016-07-13T07:22:20Z")
Hello I have some question to ask about date filter how can I try to change type that get from grok to date (input) : notBefore=Nov 10 00:00:00 2015 GMT notAfter=Nov 28 23:59:59 2016 GMT issuer= /C=US/O=SSS/OU=STT/CN=… --- ## [Pattern Cisco-ASA syslog message : Grokparsefailure ASA410001 ASA313005](https://discuss.elastic.co/t/pattern-cisco-asa-syslog-message-grokparsefailure-asa410001-asa313005/55197)
**Author:** [@OncleThorgal](https://discuss.elastic.co/u/OncleThorgal)\ **Replies:** 3\ **Last updated:** [July 13, 2016, 7:14am UTC](https://discuss.elastic.co/t/pattern-cisco-asa-syslog-message-grokparsefailure-asa410001-asa313005/55197 "2016-07-13T07:14:09Z")
Hi everyone ! I used a cisco-asa pattern to filter my syslog message but i don't know why, i've tags:\_grokparsefailure. My patterns are : #== Cisco ASA == CISCO\_TAGGED\_SYSLOG \<%{POSINT:syslog\_pri}\>:%%{CISCOTAG:cis… --- ## [How to feed entire logfile to elasticsearch as a message](https://discuss.elastic.co/t/how-to-feed-entire-logfile-to-elasticsearch-as-a-message/54518)
**Author:** [@sharath3185](https://discuss.elastic.co/u/sharath3185)\ **Replies:** 3\ **Last updated:** [July 13, 2016, 6:53am UTC](https://discuss.elastic.co/t/how-to-feed-entire-logfile-to-elasticsearch-as-a-message/54518 "2016-07-13T06:53:57Z")
'm trying to use logstash to feed a whole text file as a message into ElasticSearch. I have used multiline filter now as below. multiline { pattern =\> "/.\*./gm" negate =\> true what =\> … --- ## [How to index the logstash filter and filter not showing up in Kibana](https://discuss.elastic.co/t/how-to-index-the-logstash-filter-and-filter-not-showing-up-in-kibana/55277)
**Author:** [@Preeti\_Sharma](https://discuss.elastic.co/u/Preeti_Sharma)\ **Replies:** 2\ **Last updated:** [July 13, 2016, 5:50am UTC](https://discuss.elastic.co/t/how-to-index-the-logstash-filter-and-filter-not-showing-up-in-kibana/55277 "2016-07-13T05:50:04Z")
Here is my logstash conf input { file { path =\> '/home/logs/corporate\_access.log' } } filter { grok { match =\> { "message" =\> "User id: %{NUMBER:UserId}, Client id:%{NUMBER:ClientId}" } … --- ## [Custom logstash grok filter's pattern](https://discuss.elastic.co/t/custom-logstash-grok-filters-pattern/55327)
**Author:** [@asoong-94](https://discuss.elastic.co/u/asoong-94)\ **Replies:** 1\ **Last updated:** [July 13, 2016, 5:38am UTC](https://discuss.elastic.co/t/custom-logstash-grok-filters-pattern/55327 "2016-07-13T05:38:41Z")
I am trying to parse through log files that are separated by a row of dashes before and after each log file condensed into one document. I was thinking of creating a custom grok pattern and putting in patterns\_dir. And t… --- ## [Where can i get the old version of logstash?](https://discuss.elastic.co/t/where-can-i-get-the-old-version-of-logstash/55369)
**Author:** [@kk\_op](https://discuss.elastic.co/u/kk_op)\ **Replies:** 1\ **Last updated:** [July 13, 2016, 5:30am UTC](https://discuss.elastic.co/t/where-can-i-get-the-old-version-of-logstash/55369 "2016-07-13T05:30:18Z")
where can i get the old version of logstash, because my redhat has only java1.6 installed. --- ## [Is args reserved keyword, failed to parse \[args\]](https://discuss.elastic.co/t/is-args-reserved-keyword-failed-to-parse-args/55187)
**Author:** [@Fan\_Ming\_Chen](https://discuss.elastic.co/u/Fan_Ming_Chen)\ **Replies:** 1\ **Last updated:** [July 13, 2016, 4:45am UTC](https://discuss.elastic.co/t/is-args-reserved-keyword-failed-to-parse-args/55187 "2016-07-13T04:45:30Z")
Pipeline main started { "args": { "type": 0 } } e\[33mFailed action. {:status=\>400, :action=\>\["index", {:id=\>nil, :index=\>"log stash-2016.07.11", :type=\>"logs", :routing=\>nil}, #@metadata\_accessors=#@store={}, @lut={}\>,… --- ## [JSON object aggregation](https://discuss.elastic.co/t/json-object-aggregation/55243)
**Author:** [@asoong-94](https://discuss.elastic.co/u/asoong-94)\ **Replies:** 1\ **Last updated:** [July 13, 2016, 4:10am UTC](https://discuss.elastic.co/t/json-object-aggregation/55243 "2016-07-13T04:10:50Z")
my output into JSON looks like this { JSON Object } { JSON Object } { JSON Object } is there anyway I could make it appear all together as one JSON object instead of three separate ones? --- ## [Logstash 5 error to start with configuration](https://discuss.elastic.co/t/logstash-5-error-to-start-with-configuration/55210)
**Author:** [@dhy3ives](https://discuss.elastic.co/u/dhy3ives)\ **Replies:** 1\ **Last updated:** [July 13, 2016, 4:06am UTC](https://discuss.elastic.co/t/logstash-5-error-to-start-with-configuration/55210 "2016-07-13T04:06:16Z")
Hi I'm using logstash 5 - zip file version, and trying run a conf, but receive this erros bellow. My start config \[elk@elk bin\]$ ./logstash -f beats-syslog.conf -r -l /opt/elk/logs/logstash-syslog-apache.log --log.leve… --- ## [Logstash Kafka output storing topic offset option](https://discuss.elastic.co/t/logstash-kafka-output-storing-topic-offset-option/47696)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 13\ **Last updated:** [July 12, 2016, 7:17pm UTC](https://discuss.elastic.co/t/logstash-kafka-output-storing-topic-offset-option/47696 "2016-07-12T19:17:26Z")
Hello, Due to some shared architecture that I have to deal with, I am asked to find out if logstash kafka output when writing to kafka topic, can save the offset in kafka as opose to the zookeper. Is there a way to do … --- ## [Logstash errors while executing Ruby Code](https://discuss.elastic.co/t/logstash-errors-while-executing-ruby-code/55352)
**Author:** [@Ramdev\_Wudali](https://discuss.elastic.co/u/Ramdev_Wudali)\ **Replies:** 0\ **Last updated:** [July 12, 2016, 8:47pm UTC](https://discuss.elastic.co/t/logstash-errors-while-executing-ruby-code/55352 "2016-07-12T20:47:39Z")
Hi : I am trying to calculate the difference between two timestamp fields to see if we have an issue in consuming the data. (the @timestamp field automagically created to indicate index time and date field generated … --- ## [Filter help](https://discuss.elastic.co/t/filter-help/55237)
**Author:** [@Diggy](https://discuss.elastic.co/u/Diggy)\ **Replies:** 3\ **Last updated:** [July 12, 2016, 6:40pm UTC](https://discuss.elastic.co/t/filter-help/55237 "2016-07-12T18:40:45Z")
I've added the following filter to only capture Windows event types "WARNING" AND "ERROR": if \[type\] == "eventlog" and "INFO" in \[EventType\] { drop { } } Now, I'd like to exclude some hosts from this fil… --- ## [\[HELP?\] Can only import last entry using JDBC with Logstash and ES](https://discuss.elastic.co/t/help-can-only-import-last-entry-using-jdbc-with-logstash-and-es/55283)
**Author:** [@Nindy876](https://discuss.elastic.co/u/Nindy876)\ **Replies:** 2\ **Last updated:** [July 12, 2016, 12:19pm UTC](https://discuss.elastic.co/t/help-can-only-import-last-entry-using-jdbc-with-logstash-and-es/55283 "2016-07-12T12:19:32Z")
So I'm new to ELK and I'm trying to import data from a postgres DB into ES for searching within Kibana. I have the JDBC plugin all set up and I have a config file to import data. It runs fine but I only seem to get the… --- ## [Getting errors after upgrading logstash version from 1.5.3 to 2.3.3](https://discuss.elastic.co/t/getting-errors-after-upgrading-logstash-version-from-1-5-3-to-2-3-3/55274)
**Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 1\ **Last updated:** [July 12, 2016, 11:09am UTC](https://discuss.elastic.co/t/getting-errors-after-upgrading-logstash-version-from-1-5-3-to-2-3-3/55274 "2016-07-12T11:09:13Z")
Hi Team, I have upgraded Logstash version from 1.5.3 to 2.3.3, On logstash debug getting below errors: Unknown setting 'cluster' for elasticsearch {:level=\>:error, :file=\>"logstash/config/mixin.rb", :line=\>"329", :m… --- ## [Backfilling logs and timestamp](https://discuss.elastic.co/t/backfilling-logs-and-timestamp/55286)
**Author:** [@swagce](https://discuss.elastic.co/u/swagce)\ **Replies:** 1\ **Last updated:** [July 12, 2016, 11:05am UTC](https://discuss.elastic.co/t/backfilling-logs-and-timestamp/55286 "2016-07-12T11:05:56Z")
Hi everyone, I have some issue with the "@timestamp" field when i'm importing old logs. The thing I want to do is to replace the value of the "@timestamp" with the value of when the log was created (here i'm looking at… --- ## [Logstash elastichsearch output in case of ES cluster](https://discuss.elastic.co/t/logstash-elastichsearch-output-in-case-of-es-cluster/34782)
**Author:** [@gildor7](https://discuss.elastic.co/u/gildor7)\ **Replies:** 7\ **Last updated:** [July 12, 2016, 9:35am UTC](https://discuss.elastic.co/t/logstash-elastichsearch-output-in-case-of-es-cluster/34782 "2016-07-12T09:35:03Z")
Hello, I have the following situation: - 1 node with Logstash indexer - 3 nodes with Elastichsearch How should I set the output logstash? output { elasticsearch { host =\> "?" cluster =\> "ES-cluster" } } Hav… --- ## [Extract information from text and add as value to new field using grok and regular expression](https://discuss.elastic.co/t/extract-information-from-text-and-add-as-value-to-new-field-using-grok-and-regular-expression/55269)
**Author:** [@Tyngd\_Punkt](https://discuss.elastic.co/u/Tyngd_Punkt)\ **Replies:** 2\ **Last updated:** [July 12, 2016, 8:45am UTC](https://discuss.elastic.co/t/extract-information-from-text-and-add-as-value-to-new-field-using-grok-and-regular-expression/55269 "2016-07-12T08:45:18Z")
Hello, I'm a total newbie. I monitor a twitter feed using ELK. I want to be able to visualize in Kibana based on CVE numbers. I have been trying to use grok to add a new filed containing only a CVE number if it can be … --- ## [Removing ansi color codes with mutate](https://discuss.elastic.co/t/removing-ansi-color-codes-with-mutate/41958)
**Author:** [@encee](https://discuss.elastic.co/u/encee)\ **Replies:** 1\ **Last updated:** [July 12, 2016, 8:10am UTC](https://discuss.elastic.co/t/removing-ansi-color-codes-with-mutate/41958 "2016-07-12T08:10:07Z")
Hello, I have been trying to strip the color codes out of my logs so that I can more easily use logstash but I have been unable to actually get it to work. I found a comment and a small article that had a mutate stri… --- ## [Configtest passes, but still get error on load](https://discuss.elastic.co/t/configtest-passes-but-still-get-error-on-load/54891)
**Author:** [@memelet](https://discuss.elastic.co/u/memelet)\ **Replies:** 4\ **Last updated:** [July 12, 2016, 7:42am UTC](https://discuss.elastic.co/t/configtest-passes-but-still-get-error-on-load/54891 "2016-07-12T07:42:05Z")
I have a config with translate filter: translate { field =\> "level" dictionary =\> \[ "DEBUG", "DEBUG" "INFO", "INFO" "WARN", "WARN" "ERR", "ERROR" \] fallback =\> "INFO" exact =\> true } The … --- ## [Multiple replacement in single mutate in Logstash](https://discuss.elastic.co/t/multiple-replacement-in-single-mutate-in-logstash/55262)
**Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 1\ **Last updated:** [July 12, 2016, 6:35am UTC](https://discuss.elastic.co/t/multiple-replacement-in-single-mutate-in-logstash/55262 "2016-07-12T06:35:12Z")
Hi Team, I want to change the countryCode in transaction Log to the respective country name Now iam using multiple if loops like: if\[countryCD\]== "978"{ mutate { replace =\> \["countryCD"… --- ## [Logstash can't add fields?](https://discuss.elastic.co/t/logstash-cant-add-fields/55258)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 3\ **Last updated:** [July 12, 2016, 6:34am UTC](https://discuss.elastic.co/t/logstash-cant-add-fields/55258 "2016-07-12T06:34:27Z")
I have been using logstash to read some DB restore logs. Here is some lines of sample records. 07/08/2016 6:33:22.50: START restore database SQL2540W Restore is successful, however a warning "2539" was encounte… --- ## [How to parsing multiline messages with conditionals issues in Logstash?](https://discuss.elastic.co/t/how-to-parsing-multiline-messages-with-conditionals-issues-in-logstash/55257)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 2\ **Last updated:** [July 12, 2016, 6:21am UTC](https://discuss.elastic.co/t/how-to-parsing-multiline-messages-with-conditionals-issues-in-logstash/55257 "2016-07-12T06:21:03Z")
I have a log file reading through the ftp process and I am using logstash to ingest the data. A sample record is like that Enter an FTP subcommand. \> PUT REPE533 AMTD\_Statusfile.txt 227 Entering Passive Mode (10,133,… --- ## [Redis Vs Kafka Vs RabbitMQ in ELK](https://discuss.elastic.co/t/redis-vs-kafka-vs-rabbitmq-in-elk/55212)
**Author:** [@kamal\_bisht](https://discuss.elastic.co/u/kamal_bisht)\ **Replies:** 6\ **Last updated:** [July 12, 2016, 5:35am UTC](https://discuss.elastic.co/t/redis-vs-kafka-vs-rabbitmq-in-elk/55212 "2016-07-12T05:35:49Z")
Hi I am going to use ELK for my log analysis/monitoring. I need to use broker between logstash shipper and logstash indexer. Which tool Redis or Kafka or RabbitMQ I should use for this ? Please help me on this. --- ## [Logstash stopps processing at midnight UTC](https://discuss.elastic.co/t/logstash-stopps-processing-at-midnight-utc/55145)
**Author:** [@Govind\_Raj](https://discuss.elastic.co/u/Govind_Raj)\ **Replies:** 1\ **Last updated:** [July 12, 2016, 2:36am UTC](https://discuss.elastic.co/t/logstash-stopps-processing-at-midnight-utc/55145 "2016-07-12T02:36:57Z")
Hi Team we have a logstash configuration which process and push's the logs to our ES and kibana. At midnight UTC it stops processing the data . Here is our configuration input { file { path =\> "/var/log/orglog… --- ## [Question regarding new logstash / netflow config](https://discuss.elastic.co/t/question-regarding-new-logstash-netflow-config/54992)
**Author:** [@Gary\_Wright](https://discuss.elastic.co/u/Gary_Wright)\ **Replies:** 5\ **Last updated:** [July 11, 2016, 11:33pm UTC](https://discuss.elastic.co/t/question-regarding-new-logstash-netflow-config/54992 "2016-07-11T23:33:14Z")
Hello All, I have a new ubuntu 16.04 instance in which I have just loaded a fresh install of the ELK stack on. I currently have logstash 2.3.4-1 installed and have also installed the netflow codec using the following … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1127) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1129)