# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=836 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 837 --- ## [Error sending logs to Elastic After porting a pipeline configuration to docker](https://discuss.elastic.co/t/error-sending-logs-to-elastic-after-porting-a-pipeline-configuration-to-docker/116757)
**Author:** [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Replies:** 4\ **Last updated:** [January 23, 2018, 10:20pm UTC](https://discuss.elastic.co/t/error-sending-logs-to-elastic-after-porting-a-pipeline-configuration-to-docker/116757 "2018-01-23T22:20:26Z")
We have a working pipeline configuration file which we are porting to Docker. We are seeing the following warnings in the logs. We do not have any security configured on our Elastic Search. What is very puzzling is t… --- ## [Multiline: flush downstream error -\> NPE](https://discuss.elastic.co/t/multiline-flush-downstream-error-npe/116761)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 10:06pm UTC](https://discuss.elastic.co/t/multiline-flush-downstream-error-npe/116761 "2018-01-23T22:06:52Z")
I have a setup of filebeats pushing data to LS which in turn pushes data to ES. I was harvesting on a few files to pump data. To test potential speedup, I started 2 filebeat processes each processing a half of the set of… --- ## [If/and/or clause](https://discuss.elastic.co/t/if-and-or-clause/116736)
**Author:** [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Replies:** 1\ **Last updated:** [January 23, 2018, 9:25pm UTC](https://discuss.elastic.co/t/if-and-or-clause/116736 "2018-01-23T21:25:01Z")
Hello, Will this IF clause work? if \[field\_A\] == "aaa" and ( \[field\_B\] == "bbb" or \[field\_C\] == "ccc") { ... } Documentation tells me I can run a complex IF clause, but looking for examples I found none. I don't h… --- ## [Date filter failer](https://discuss.elastic.co/t/date-filter-failer/115334)
**Author:** [@avih](https://discuss.elastic.co/u/avih)\ **Replies:** 1\ **Last updated:** [January 23, 2018, 9:14pm UTC](https://discuss.elastic.co/t/date-filter-failer/115334 "2018-01-23T21:14:09Z")
Hi, My Date filter fails although the string I'm trying to convert seems to match the matched pattern: Some of the timestamp string have the following format : 2018-01-12 10:54:20,246 and some have the following : 201… --- ## [Need help with Logstash Configuration Error - \_grokparsefailure](https://discuss.elastic.co/t/need-help-with-logstash-configuration-error-grokparsefailure/115316)
**Author:** [@taweisse](https://discuss.elastic.co/u/taweisse)\ **Replies:** 3\ **Last updated:** [January 23, 2018, 9:11pm UTC](https://discuss.elastic.co/t/need-help-with-logstash-configuration-error-grokparsefailure/115316 "2018-01-23T21:11:31Z")
Hi all, I have been struggling with my Logstash configuration file for 2 days now and cannot figure out what the issue is. I am trying to parse regularly structured data using the grok filter, and no matter what I try, I… --- ## [Logstash process](https://discuss.elastic.co/t/logstash-process/115081)
**Author:** [@Sumit\_Gupta1](https://discuss.elastic.co/u/Sumit_Gupta1)\ **Replies:** 1\ **Last updated:** [January 23, 2018, 9:03pm UTC](https://discuss.elastic.co/t/logstash-process/115081 "2018-01-23T21:03:49Z")
Hello, I have some query: 1.) When schedule job runs, then what will be the process? Whole database data push in the elastic search or only updated data? 2.) When schedule job runs, will be any acknowledgement from l… --- ## [Best practices for sending Logstash output duplicates off-site](https://discuss.elastic.co/t/best-practices-for-sending-logstash-output-duplicates-off-site/116289)
**Author:** [@Aceman87](https://discuss.elastic.co/u/Aceman87)\ **Replies:** 1\ **Last updated:** [January 23, 2018, 8:47pm UTC](https://discuss.elastic.co/t/best-practices-for-sending-logstash-output-duplicates-off-site/116289 "2018-01-23T20:47:44Z")
Hi, We are getting Beats events and Luberjack input to our logstash. We then do some filtering and store these into our Elasticsearch. However, we also need to send this output off-site through a HTTP proxy. We have … --- ## [Index is not creating](https://discuss.elastic.co/t/index-is-not-creating/115911)
**Author:** [@Sumit\_Gupta1](https://discuss.elastic.co/u/Sumit_Gupta1)\ **Replies:** 5\ **Last updated:** [January 23, 2018, 8:39pm UTC](https://discuss.elastic.co/t/index-is-not-creating/115911 "2018-01-23T20:39:24Z")
I' getting following error: status=\>400, : action=\>\[ "index", { : \_id=\>nil, : \_index=\>"mongodata", : \_type=\>"mytype1", : \_routing=\>nil }, # \], : response=\>{ "index"=\>{ "\_index"=\>"mongodata", "\_type"=\>"myty… --- ## [Setting Queue Quota for Outputs](https://discuss.elastic.co/t/setting-queue-quota-for-outputs/116738)
**Author:** [@Jorn](https://discuss.elastic.co/u/Jorn)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 7:30pm UTC](https://discuss.elastic.co/t/setting-queue-quota-for-outputs/116738 "2018-01-23T19:30:35Z")
We have a shared aggregation cluster running Logstash 5.6 that has persistent queues enabled, but as far as we can tell, this resource is shared between all configured outputs. Is there a way for us to configure our out… --- ## [Maximum number of bytes for tcp input plugin](https://discuss.elastic.co/t/maximum-number-of-bytes-for-tcp-input-plugin/116733)
**Author:** [@cappy](https://discuss.elastic.co/u/cappy)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 7:10pm UTC](https://discuss.elastic.co/t/maximum-number-of-bytes-for-tcp-input-plugin/116733 "2018-01-23T19:10:41Z")
I'm using the tcp input plugin with the JSON codec to pull in JSON-formatted syslog, and when processing events, sometimes I will get some larger than 8192 bytes. When this happens, the event is split in two records (an… --- ## [Jdbc input plugin, sql\_last\_value is always 0 (or 1 January 1970)](https://discuss.elastic.co/t/jdbc-input-plugin-sql-last-value-is-always-0-or-1-january-1970/116707)
**Author:** [@Mikhail](https://discuss.elastic.co/u/Mikhail)\ **Replies:** 1\ **Last updated:** [January 23, 2018, 6:39pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-sql-last-value-is-always-0-or-1-january-1970/116707 "2018-01-23T18:39:05Z")
Many hours are spent, many neurons are lost, a number of gray hairs are earned... All this in the battle with sql\_last\_value refreshment in .logstash\_jdbc\_last\_run file. My configuration: MS SQL Server database table -… --- ## [Converting an Ubuntu based Dockerfile to CentOS fails](https://discuss.elastic.co/t/converting-an-ubuntu-based-dockerfile-to-centos-fails/116727)
**Author:** [@tezarin](https://discuss.elastic.co/u/tezarin)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 6:29pm UTC](https://discuss.elastic.co/t/converting-an-ubuntu-based-dockerfile-to-centos-fails/116727 "2018-01-23T18:29:23Z")
Hi all, I have successfully building the sebp/elk image which is based on an Ubuntu image: https://hub.docker.com/r/sebp/elk/~/dockerfile/ and I can run the container by running this command: sudo docker run -p 5601:56… --- ## [Logstash creating an array when I add a string field that not exist yet](https://discuss.elastic.co/t/logstash-creating-an-array-when-i-add-a-string-field-that-not-exist-yet/116723)
**Author:** [@rgheleri](https://discuss.elastic.co/u/rgheleri)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 5:35pm UTC](https://discuss.elastic.co/t/logstash-creating-an-array-when-i-add-a-string-field-that-not-exist-yet/116723 "2018-01-23T17:35:16Z")
Hello everyone I have a logstash conf file that fetch two different queries from different tables in mysql and both are merged in the same index of elasticsearch. One specific string field (code\_serial) exists in one ta… --- ## [Mongodb input plugin for logstash does not get any document](https://discuss.elastic.co/t/mongodb-input-plugin-for-logstash-does-not-get-any-document/109638)
**Author:** [@rgheleri](https://discuss.elastic.co/u/rgheleri)\ **Replies:** 2\ **Last updated:** [January 23, 2018, 5:39pm UTC](https://discuss.elastic.co/t/mongodb-input-plugin-for-logstash-does-not-get-any-document/109638 "2018-01-23T17:39:40Z")
I'm trying to use Logstash to get data from Mongodb and insert into Elasticsearch, using the logstash-input-mongodb plugin. Aparently I can connect in mongo with success but no document is stored in elasticsearch, not ev… --- ## [Logstash does not listen for beats](https://discuss.elastic.co/t/logstash-does-not-listen-for-beats/115181)
**Author:** [@wweeder](https://discuss.elastic.co/u/wweeder)\ **Replies:** 3\ **Last updated:** [January 23, 2018, 4:50pm UTC](https://discuss.elastic.co/t/logstash-does-not-listen-for-beats/115181 "2018-01-23T16:50:32Z")
Hello, it appears that logstash fails to listen on 5044. My config: # egrep -v '#|^$' logstash.yml path.data: /var/lib/logstash path.config: /etc/logstash/conf.d/\*.conf path.logs: /var/log/logstash # cat conf.d/bea… --- ## [Aggregation of document](https://discuss.elastic.co/t/aggregation-of-document/116706)
**Author:** [@AnaisH](https://discuss.elastic.co/u/AnaisH)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 3:44pm UTC](https://discuss.elastic.co/t/aggregation-of-document/116706 "2018-01-23T15:44:57Z")
Hello, I would like to concat different events in logstash/elasticsearch. I receive this table \[51\] And I want to have this result in Kibana \[58\] But in Kibana I have this result \[26\] I think that it's necessary … --- ## [Logstash create index in elastic search failure](https://discuss.elastic.co/t/logstash-create-index-in-elastic-search-failure/116698)
**Author:** [@yinzhi7](https://discuss.elastic.co/u/yinzhi7)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 3:12pm UTC](https://discuss.elastic.co/t/logstash-create-index-in-elastic-search-failure/116698 "2018-01-23T15:12:00Z")
Sorry about my English. I will try to describe what's going on. And also still quite newbie on the ELK process.. Here is the exception i found in logstash's log: \[2018-01-23T09:07:20,699\]\[INFO \]\[logstash.outputs.elas… --- ## [Parsing Array Object in Logstash](https://discuss.elastic.co/t/parsing-array-object-in-logstash/116637)
**Author:** [@Lorenzo\_Gargiullo](https://discuss.elastic.co/u/Lorenzo_Gargiullo)\ **Replies:** 7\ **Last updated:** [January 23, 2018, 11:41am UTC](https://discuss.elastic.co/t/parsing-array-object-in-logstash/116637 "2018-01-23T11:41:24Z")
Hi, i want to pars an array of object in Logstash. This is mi conf: input { stdin { } } filter { grok { match =\> { "message" =\> "%{DATESTAMP:timestamp}%{SPACE}%{NOTSPACE:num\_column}%{SPACE}%{NOTSPACE:sensor001}%{… --- ## [Composite ID](https://discuss.elastic.co/t/composite-id/115284)
**Author:** [@devil\_srj7](https://discuss.elastic.co/u/devil_srj7)\ **Replies:** 6\ **Last updated:** [January 23, 2018, 12:27pm UTC](https://discuss.elastic.co/t/composite-id/115284 "2018-01-23T12:27:46Z")
I require a way to create a composite key for the document in elastic search through logstash corresponding to the composite key in my MySQL database. --- ## [How to create custom document \_id in logstash?](https://discuss.elastic.co/t/how-to-create-custom-document-id-in-logstash/116629)
**Author:** [@kasliwalroshni](https://discuss.elastic.co/u/kasliwalroshni)\ **Replies:** 6\ **Last updated:** [January 23, 2018, 12:23pm UTC](https://discuss.elastic.co/t/how-to-create-custom-document-id-in-logstash/116629 "2018-01-23T12:23:48Z")
This is sample logstash output filter. output { elasticsearch { hosts =\> \["localhost"\] sniffing =\> true manage\_template =\> false index =\> "mqtt-index-%{+YYYY.MM.dd}" documen… --- ## [Error while running logstash on ubtuntu server](https://discuss.elastic.co/t/error-while-running-logstash-on-ubtuntu-server/116660)
**Author:** [@krishnaSH](https://discuss.elastic.co/u/krishnaSH)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 11:48am UTC](https://discuss.elastic.co/t/error-while-running-logstash-on-ubtuntu-server/116660 "2018-01-23T11:48:14Z")
I am trying to connect my ms sql server to elastic search through logstash. I have used the steps given here: https://www.elastic.co/guide/en/logstash/6.1/installing-logstash.html , And when I execute the command : sudo… --- ## [How to convert String "date+time+tz" to timestamp](https://discuss.elastic.co/t/how-to-convert-string-date-time-tz-to-timestamp/115577)
**Author:** [@marouane](https://discuss.elastic.co/u/marouane)\ **Replies:** 3\ **Last updated:** [January 23, 2018, 11:32am UTC](https://discuss.elastic.co/t/how-to-convert-string-date-time-tz-to-timestamp/115577 "2018-01-23T11:32:34Z")
Hello every one, I want to replace 3 fields that contains string(date) , string(time) and string(tz) by a new field with date type that contains (date+time+tz) converted. { "query":{ "bool":{ "must\_not":{ … --- ## [How to parse JSON data in Logstash](https://discuss.elastic.co/t/how-to-parse-json-data-in-logstash/116550)
**Author:** [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)\ **Replies:** 1\ **Last updated:** [January 23, 2018, 10:10am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-logstash/116550 "2018-01-23T10:10:32Z")
I am parsing a .txt file which have json format values . When I try to parse the file using JSON filter the data is not getting parsed. Logstash.config input { beats{ port =\> 7360 codec =\> "json" } } … --- ## [Logstash-filter-aggregate](https://discuss.elastic.co/t/logstash-filter-aggregate/116600)
**Author:** [@piingluo](https://discuss.elastic.co/u/piingluo)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 6:34am UTC](https://discuss.elastic.co/t/logstash-filter-aggregate/116600 "2018-01-23T06:34:41Z")
Hey, I was merging three event into one event, I was wondering if someone could tell me what I'm doing wrong. input example: type=SYSCALL msg=audit(1516094800.649:1198): arch=c000003e syscall=59 success=yes exit=0 a0=2… --- ## [Grok Parsing Timestamp Error](https://discuss.elastic.co/t/grok-parsing-timestamp-error/116576)
**Author:** [@rijinmp](https://discuss.elastic.co/u/rijinmp)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 3:35am UTC](https://discuss.elastic.co/t/grok-parsing-timestamp-error/116576 "2018-01-23T03:35:46Z")
Log : in24.inetnebr.com - - \[01/Aug/1995:00:00:01 -0400\] "GET /shuttle/missions/sts-68/news/sts-68-mcc-05.txt HTTP/1.0" 200 1839 Pattern : %{HOSTNAME:vhost} - - %{DATA:Raw\_timestamp} "%{DATA:Method} /%{DATA:File} %{DA… --- ## [Logstash Drop Filter for UFW BLOCK in Syslog](https://discuss.elastic.co/t/logstash-drop-filter-for-ufw-block-in-syslog/116573)
**Author:** [@cailen](https://discuss.elastic.co/u/cailen)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 3:15am UTC](https://discuss.elastic.co/t/logstash-drop-filter-for-ufw-block-in-syslog/116573 "2018-01-23T03:15:57Z")
Greetings, I have been fighting something for 2 days now that severely alludes me at this point: I cannot get the \[UFW BLOCK\] entries in syslog to either get blocked by the filebeat system module via regex or by filteri… --- ## [How do i make sure that logstash received data from aws cloudwatch logs?](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-aws-cloudwatch-logs/116572)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 0\ **Last updated:** [January 23, 2018, 3:14am UTC](https://discuss.elastic.co/t/how-do-i-make-sure-that-logstash-received-data-from-aws-cloudwatch-logs/116572 "2018-01-23T03:14:48Z")
Hello there How do i make sure that logstash received data from aws cloudwatch logs? this is my input config: input { cloudwatch\_logs { log\_group =\> \[ "LOGGRP001" \] access\_key\_id =\> "ABCDDEEDDDD" secret\_access… --- ## [Limit input data to recent data only, like Last 24 hours](https://discuss.elastic.co/t/limit-input-data-to-recent-data-only-like-last-24-hours/116422)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 2\ **Last updated:** [January 22, 2018, 9:40pm UTC](https://discuss.elastic.co/t/limit-input-data-to-recent-data-only-like-last-24-hours/116422 "2018-01-22T21:40:40Z")
Hello, I'm using Elasticsearch input plugin with Csv output plugin as part of my pipeline and even though everything works as expected, my pipeline would write all data from elasticsearch into csv file, yet I would like… --- ## [IF conditional not dropping field](https://discuss.elastic.co/t/if-conditional-not-dropping-field/116188)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [January 22, 2018, 9:24pm UTC](https://discuss.elastic.co/t/if-conditional-not-dropping-field/116188 "2018-01-22T21:24:56Z")
Hi, I'm trying to filter out some data but for some reason the conditional I'm using isn't working. I'm using a similar config for another log and there it's working fine. I also tried putting the conditional after ren… --- ## [Split not working in filter](https://discuss.elastic.co/t/split-not-working-in-filter/116178)
**Author:** [@mahajan344](https://discuss.elastic.co/u/mahajan344)\ **Replies:** 1\ **Last updated:** [January 22, 2018, 9:24pm UTC](https://discuss.elastic.co/t/split-not-working-in-filter/116178 "2018-01-22T21:24:11Z")
Hi, I have array of data which i want to split to make data inside array as main data. E.g. my json is as below, its output from logstash pipeline "\_embedded" =\> \[ \[0\] { "jobtitle" =\> "NDT SUPERVISOR", … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=835) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=837)