# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=784 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 785 --- ## [Location to set max\_files\_open settings](https://discuss.elastic.co/t/location-to-set-max-files-open-settings/127384)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 0\ **Last updated:** [April 9, 2018, 8:23pm UTC](https://discuss.elastic.co/t/location-to-set-max-files-open-settings/127384 "2018-04-09T20:23:52Z")
I've spent some time searching through the forums (and Google - but many sources are 2+ years old and the paths have changed) to find the locations to set max\_files\_open for logstash since I'm hitting the 4095 maximum l… --- ## [Logstash Indexing Error - Aggregate plugin: For task\_id pattern '%{id}', there are more than one filter](https://discuss.elastic.co/t/logstash-indexing-error-aggregate-plugin-for-task-id-pattern-id-there-are-more-than-one-filter/127376)
**Author:** [@halogeek](https://discuss.elastic.co/u/halogeek)\ **Replies:** 1\ **Last updated:** [April 9, 2018, 8:23pm UTC](https://discuss.elastic.co/t/logstash-indexing-error-aggregate-plugin-for-task-id-pattern-id-there-are-more-than-one-filter/127376 "2018-04-09T20:23:31Z")
Am using Elasticsearch 5.5.0 and logstash 5.5.0 on Linux - AWS ec2-instance. Note: Have already posted the same exact question on Stack Overflow here. Have a logstash\_etl.conf file which resides in /etc/logstash/conf.d… --- ## [Log between {}](https://discuss.elastic.co/t/log-between/127325)
**Author:** [@Sebastian\_Herrera](https://discuss.elastic.co/u/Sebastian_Herrera)\ **Replies:** 5\ **Last updated:** [April 9, 2018, 8:20pm UTC](https://discuss.elastic.co/t/log-between/127325 "2018-04-09T20:20:25Z")
Hello, i am having my log between brackets and i am not able to parse that part, can anyone help me?? Regards my log is \[Priority: 2\] {TCP} 192.168.16.247:51650 -\> 52.184.227.73:443 i was able to parse the priority, b… --- ## [Need to pull cassandra db latest values](https://discuss.elastic.co/t/need-to-pull-cassandra-db-latest-values/126900)
**Author:** [@aarthinim](https://discuss.elastic.co/u/aarthinim)\ **Replies:** 1\ **Last updated:** [April 9, 2018, 8:16pm UTC](https://discuss.elastic.co/t/need-to-pull-cassandra-db-latest-values/126900 "2018-04-09T20:16:05Z")
My sql table structure is "create table emp (empid int primary key, emp\_first varchar, emp\_last varchar, emp\_dept varchar);" and my logstash.conf file input { jdbc{ jdbc\_connection\_string =\> "jdbc:cassandra://localho… --- ## [Help with logstash config using datadog output plugin](https://discuss.elastic.co/t/help-with-logstash-config-using-datadog-output-plugin/127141)
**Author:** [@caljess599](https://discuss.elastic.co/u/caljess599)\ **Replies:** 3\ **Last updated:** [April 9, 2018, 8:15pm UTC](https://discuss.elastic.co/t/help-with-logstash-config-using-datadog-output-plugin/127141 "2018-04-09T20:15:36Z")
I'm using the old datadog output plugin for logstash, which I'm running inside a container. For the life of me, I cannot get my JSON input (coming in via the http input filter) to show up in DataDog. Basically, I don't… --- ## [Immense term and maximum length exception in elasticsearch output](https://discuss.elastic.co/t/immense-term-and-maximum-length-exception-in-elasticsearch-output/78536)
**Author:** [@fritzhardy](https://discuss.elastic.co/u/fritzhardy)\ **Replies:** 1\ **Last updated:** [April 9, 2018, 8:10pm UTC](https://discuss.elastic.co/t/immense-term-and-maximum-length-exception-in-elasticsearch-output/78536 "2018-04-09T20:10:16Z")
I recently began running into this: "max\_bytes\_length\_exceeded\_exception: bytes can be at most 32766 in length; got 338076" Full error: {"type"=\>"illegal\_argument\_exception", "reason"=\>"Document contains at least o… --- ## [Best practice to track data loss when sending data to ElasticSeach](https://discuss.elastic.co/t/best-practice-to-track-data-loss-when-sending-data-to-elasticseach/126471)
**Author:** [@raychen11](https://discuss.elastic.co/u/raychen11)\ **Replies:** 4\ **Last updated:** [April 9, 2018, 5:14pm UTC](https://discuss.elastic.co/t/best-practice-to-track-data-loss-when-sending-data-to-elasticseach/126471 "2018-04-09T17:14:36Z")
Wondering if we have any suitable ways to check the data loss when sending data via Logstash to ElasticSearch endpoint. Ways I know check logstash.log file to see if any errors are there 1.1 Works but seems to need a… --- ## [How to filter events based on certain string?](https://discuss.elastic.co/t/how-to-filter-events-based-on-certain-string/127190)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 8\ **Last updated:** [April 9, 2018, 3:29pm UTC](https://discuss.elastic.co/t/how-to-filter-events-based-on-certain-string/127190 "2018-04-09T15:29:03Z")
Hi, I am trying to filter out events if it contains either source\_type=\\”APP/PROC/WEB\\” or source\_type=APP in the event. Can someone please help me on this? Thanks --- ## [Aggregate plugin - problem with micro-batching](https://discuss.elastic.co/t/aggregate-plugin-problem-with-micro-batching/124979)
**Author:** [@rxxg](https://discuss.elastic.co/u/rxxg)\ **Replies:** 6\ **Last updated:** [April 9, 2018, 3:10pm UTC](https://discuss.elastic.co/t/aggregate-plugin-problem-with-micro-batching/124979 "2018-04-09T15:10:56Z")
Hi, I've been wrestling with my Aggregate plugin configuration and it seems like it interacts poorly with the Logstash micro-batching framework in my case. I have log lines similar to this (simplified): 2018-03-21 10:… --- ## [Mutate multiple fields from multiple tables](https://discuss.elastic.co/t/mutate-multiple-fields-from-multiple-tables/122775)
**Author:** [@jurquiza](https://discuss.elastic.co/u/jurquiza)\ **Replies:** 4\ **Last updated:** [April 9, 2018, 1:59pm UTC](https://discuss.elastic.co/t/mutate-multiple-fields-from-multiple-tables/122775 "2018-04-09T13:59:32Z")
Hi everyone! After some time of fighting with logstash i managed to insert my sql server database to elastic. Im having some trouble when im emitting a field that looks like a phone number(a field with a "-" character) … --- ## [Merge multiple input into one document](https://discuss.elastic.co/t/merge-multiple-input-into-one-document/127084)
**Author:** [@Tiago\_Pinto](https://discuss.elastic.co/u/Tiago_Pinto)\ **Replies:** 5\ **Last updated:** [April 9, 2018, 1:32pm UTC](https://discuss.elastic.co/t/merge-multiple-input-into-one-document/127084 "2018-04-09T13:32:08Z")
How can I run two sql queries and merge the info into one single document? I'm trying to use the jdbc input plugin to make a data merge into one document, the data comes from two diferent databases. In one database I h… --- ## [Error when loading data into Logstash](https://discuss.elastic.co/t/error-when-loading-data-into-logstash/127297)
**Author:** [@Bode](https://discuss.elastic.co/u/Bode)\ **Replies:** 4\ **Last updated:** [April 9, 2018, 11:47am UTC](https://discuss.elastic.co/t/error-when-loading-data-into-logstash/127297 "2018-04-09T11:47:13Z")
Hello All, I am new to ELK and I was testing a sample file with the config file below. On running the log stash application, I got the error "No Configurationfound in the configures sources". I am using running it on Wi… --- ## [Error Importing csv files using logstash](https://discuss.elastic.co/t/error-importing-csv-files-using-logstash/127113)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 2\ **Last updated:** [April 9, 2018, 9:43am UTC](https://discuss.elastic.co/t/error-importing-csv-files-using-logstash/127113 "2018-04-09T09:43:13Z")
Hey Guys, I have been struggling to get through this issue from past one week. Trying to upload a sample csv file using logstash and keep getting error, digged up in lot many documents and places and still didnt get an… --- ## [How to Filter log base on log line](https://discuss.elastic.co/t/how-to-filter-log-base-on-log-line/126918)
**Author:** [@Bhargav\_Patel](https://discuss.elastic.co/u/Bhargav_Patel)\ **Replies:** 5\ **Last updated:** [April 9, 2018, 11:13am UTC](https://discuss.elastic.co/t/how-to-filter-log-base-on-log-line/126918 "2018-04-09T11:13:40Z")
Hi I am newbee in logstash. can someone let me know how to filter below log. I want ot break this line base on timestamp java class and my custom text like "Updated By" and "Audit Log" and log level etc. 2018-04-05 17… --- ## [File input not updating](https://discuss.elastic.co/t/file-input-not-updating/126979)
**Author:** [@Callahan](https://discuss.elastic.co/u/Callahan)\ **Replies:** 5\ **Last updated:** [April 9, 2018, 10:39am UTC](https://discuss.elastic.co/t/file-input-not-updating/126979 "2018-04-09T10:39:34Z")
Hi, I've got a small issue with the updating of an index from a csv. The PowerShell script I've written to convert a xml file into a csv file works by copying the xml file to the local storage, converting it and writing… --- ## [Why it tell me that \[0\] "\_dateparsefailure" when i change the @timestamp](https://discuss.elastic.co/t/why-it-tell-me-that-0-dateparsefailure-when-i-change-the-timestamp/127272)
**Author:** [@chenzilong444](https://discuss.elastic.co/u/chenzilong444)\ **Replies:** 2\ **Last updated:** [April 9, 2018, 8:55am UTC](https://discuss.elastic.co/t/why-it-tell-me-that-0-dateparsefailure-when-i-change-the-timestamp/127272 "2018-04-09T08:55:31Z")
{ "timestamp" =\> "2018-04-09 15:33:45.294", "@version" =\> "1", "@timestamp" =\> 2018-04-09T08:19:32.987Z, "tags" =\> \[ \[0\] "\_dateparsefailure" \], "message" =\> "\[2018-04-09 15:33:45.294\]\\r", "host" =\> "BIH-D-6331" … --- ## [How can I use if condition in filter for xml](https://discuss.elastic.co/t/how-can-i-use-if-condition-in-filter-for-xml/127055)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 18\ **Last updated:** [April 9, 2018, 7:38am UTC](https://discuss.elastic.co/t/how-can-i-use-if-condition-in-filter-for-xml/127055 "2018-04-09T07:38:21Z")
Is there any solution about using if condition in filter? I want if there is some tags in xml source, do filter about xml, mutae, ruby , ect... First, my xml source is like this : \ **Author:** [@tatdat](https://discuss.elastic.co/u/tatdat)\ **Replies:** 3\ **Last updated:** [April 9, 2018, 7:35am UTC](https://discuss.elastic.co/t/office-365-help-parse-json-with-key-value-in-data/125850 "2018-04-09T07:35:18Z") Im working on log of O365 via O365 API. I have sample log look like "o365": { "ExtendedProperties": \[ { "Name": "UserAgent", "Value": "Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outloo… --- ## [LogStash & Netflow : Wrong values?](https://discuss.elastic.co/t/logstash-netflow-wrong-values/126890)
**Author:** [@Julzor](https://discuss.elastic.co/u/Julzor)\ **Replies:** 2\ **Last updated:** [April 9, 2018, 6:07am UTC](https://discuss.elastic.co/t/logstash-netflow-wrong-values/126890 "2018-04-09T06:07:27Z")
Hi, I deployed a LogStash server dedicated to collecting Netflow flows (16 CPU). The Netflow codec is working, i can see all my datas in the predifined dashboards in Kibana, and i also can graph' it on Grafana. Everyt… --- ## [Logstash-output-elasticsearch for netflow](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505)
**Author:** [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)\ **Replies:** 12\ **Last updated:** [April 9, 2018, 12:02am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505 "2018-04-09T00:02:45Z")
I've been having this issue fora while now, I've posted on github but the problem is not the netflow codec jorritfolmer has explained why and given me some insight but I thought I ask here for a bit more help Blockq… --- ## [Input file plugin for Rotating Logs on NFS](https://discuss.elastic.co/t/input-file-plugin-for-rotating-logs-on-nfs/127228)
**Author:** [@servo](https://discuss.elastic.co/u/servo)\ **Replies:** 0\ **Last updated:** [April 8, 2018, 8:52pm UTC](https://discuss.elastic.co/t/input-file-plugin-for-rotating-logs-on-nfs/127228 "2018-04-08T20:52:23Z")
What is ES recommended solution for sending Rotating Logs in NFS mounted directory to HTTP using Logstash? I've read that input file plugin is not tested/well supported for log files on NFS. Has this been resolved/im… --- ## [How can i put event json in an array](https://discuss.elastic.co/t/how-can-i-put-event-json-in-an-array/127204)
**Author:** [@sky](https://discuss.elastic.co/u/sky)\ **Replies:** 1\ **Last updated:** [April 8, 2018, 4:00pm UTC](https://discuss.elastic.co/t/how-can-i-put-event-json-in-an-array/127204 "2018-04-08T16:00:38Z")
Hi,all！ I need logstash to output the event json in array, example, {"field":"value"} to \[{"field":"value"}\] , how can i do? thanks! --- ## [Logstash Netflow Docs Out of Date?](https://discuss.elastic.co/t/logstash-netflow-docs-out-of-date/127207)
**Author:** [@Jarrad](https://discuss.elastic.co/u/Jarrad)\ **Replies:** 2\ **Last updated:** [April 8, 2018, 1:13pm UTC](https://discuss.elastic.co/t/logstash-netflow-docs-out-of-date/127207 "2018-04-08T13:13:44Z")
Hi. I've been following the docs here: https://www.elastic.co/guide/en/logstash/6.x/netflow-module.html for an ELK stack running ELK 6.2.3 I've built today running on Ubuntu 16.04 server. I'd like to get Netflow integra… --- ## [Setting protocol type in Unix Input Plugin](https://discuss.elastic.co/t/setting-protocol-type-in-unix-input-plugin/127205)
**Author:** [@mibu](https://discuss.elastic.co/u/mibu)\ **Replies:** 0\ **Last updated:** [April 8, 2018, 10:39am UTC](https://discuss.elastic.co/t/setting-protocol-type-in-unix-input-plugin/127205 "2018-04-08T10:39:47Z")
Hello everybody, I am trying to receive journald messages by forwarding them to syslog. My Linux is a systemd-system (OpenSUSE Leap 42.2) with no syslog daemon installed and I would like to avoid installing one. The Sy… --- ## [Logstash on many files](https://discuss.elastic.co/t/logstash-on-many-files/126458)
**Author:** [@pcouas](https://discuss.elastic.co/u/pcouas)\ **Replies:** 1\ **Last updated:** [April 8, 2018, 8:18am UTC](https://discuss.elastic.co/t/logstash-on-many-files/126458 "2018-04-08T08:18:36Z")
Hi, My Kibana5.6.8 logstash configuration seems only reading one log file ? then logdate field is viewing as string and not as date ? My logstash.conf on /home/elastichsearch/confLogs is input { file { type =\> "sta… --- ## [While running my conf file it is throwing this error?](https://discuss.elastic.co/t/while-running-my-conf-file-it-is-throwing-this-error/127174)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 1\ **Last updated:** [April 7, 2018, 8:03pm UTC](https://discuss.elastic.co/t/while-running-my-conf-file-it-is-throwing-this-error/127174 "2018-04-07T20:03:15Z")
Actually I'm trying to import sql server data to elasticsearch and this is the initial phase if this works I need to export search functionality to elasticsearch.! \[img\] Please help with this error --- ## [How would I mutate a header hash object that has headers that change from request to request?](https://discuss.elastic.co/t/how-would-i-mutate-a-header-hash-object-that-has-headers-that-change-from-request-to-request/127155)
**Author:** [@alilland](https://discuss.elastic.co/u/alilland)\ **Replies:** 1\ **Last updated:** [April 7, 2018, 12:58pm UTC](https://discuss.elastic.co/t/how-would-i-mutate-a-header-hash-object-that-has-headers-that-change-from-request-to-request/127155 "2018-04-07T12:58:32Z")
I am parsing my filebeat logs, The original string looks like this: \[2018-04-04 13:18:37 -0700\] INFO -- 200 -- db=0.0 total=51.66 view=51.66 -- GET /trades headers={"Version"=\>"HTTP/1.1", "Host"=\>"localhost:3000", "Con… --- ## [Add Remote IP address in winlogbeat data](https://discuss.elastic.co/t/add-remote-ip-address-in-winlogbeat-data/126912)
**Author:** [@suchand](https://discuss.elastic.co/u/suchand)\ **Replies:** 3\ **Last updated:** [April 7, 2018, 10:55am UTC](https://discuss.elastic.co/t/add-remote-ip-address-in-winlogbeat-data/126912 "2018-04-07T10:55:34Z")
Hi, I have added the below mutate filter for adding IP address in winlogbeat index. mutate{ add\_field =\> { "remote\_ip" =\> "%{\[@metadata\]\[ip\_address\]}" } } But it's not getting the output as expected. It's showing li… --- ## [Limit of total fields \[1000\] in index has been exceeded"](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-has-been-exceeded/126929)
**Author:** [@cowensel](https://discuss.elastic.co/u/cowensel)\ **Replies:** 2\ **Last updated:** [April 7, 2018, 1:16am UTC](https://discuss.elastic.co/t/limit-of-total-fields-1000-in-index-has-been-exceeded/126929 "2018-04-07T01:16:57Z")
I know this query keeps cropping up. Is there a way we can identify the event that triggered this error? and if we increase the limit on the template are there any overheads added. --- ## [Timestamp for logstash syslog output when specifying RFC5424](https://discuss.elastic.co/t/timestamp-for-logstash-syslog-output-when-specifying-rfc5424/126839)
**Author:** [@cplvic](https://discuss.elastic.co/u/cplvic)\ **Replies:** 1\ **Last updated:** [April 6, 2018, 9:36pm UTC](https://discuss.elastic.co/t/timestamp-for-logstash-syslog-output-when-specifying-rfc5424/126839 "2018-04-06T21:36:50Z")
Hi I just started using the syslog output plugin. I am processing past logs (2017). I specified rfc =\> "rfc5424" I am also using the date filter to map the log date to @timestamp. On my receiving server, the timestamp … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=783) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=785)