# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=803 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 804 --- ## [Manage dynamic fields type](https://discuss.elastic.co/t/manage-dynamic-fields-type/123599)
**Author:** [@OptiFine](https://discuss.elastic.co/u/OptiFine)\ **Replies:** 1\ **Last updated:** [March 13, 2018, 2:11pm UTC](https://discuss.elastic.co/t/manage-dynamic-fields-type/123599 "2018-03-13T14:11:46Z")
Hello, I have in my input something like this : \[ hibernate.transaction.count=\[10\], hibernate.flush.count=\[9\], hibernate.entity0=\[foo.foo.foo.Class\], hibernate.entity0.fetchcount=\[2\], hibernate.entity1=\[foo.foo.foo.Cla… --- ## [Add Multiple input plugins in Logstash](https://discuss.elastic.co/t/add-multiple-input-plugins-in-logstash/123585)
**Author:** [@zakkhan](https://discuss.elastic.co/u/zakkhan)\ **Replies:** 2\ **Last updated:** [March 13, 2018, 1:48pm UTC](https://discuss.elastic.co/t/add-multiple-input-plugins-in-logstash/123585 "2018-03-13T13:48:25Z")
I need to add two different input plugins in logstash i.e. Filebeat and jdbc but only jdbc is working. I am not sure is this the right way i am adding both of them? Please let me know what to do in this kind of scenario: … --- ## [Logstash to run in background services in Windows](https://discuss.elastic.co/t/logstash-to-run-in-background-services-in-windows/123604)
**Author:** [@vaishali1](https://discuss.elastic.co/u/vaishali1)\ **Replies:** 2\ **Last updated:** [March 13, 2018, 1:46pm UTC](https://discuss.elastic.co/t/logstash-to-run-in-background-services-in-windows/123604 "2018-03-13T13:46:50Z")
How to run the logstash automatically "Services" in Windows, once when the system restarted? --- ## [Elasticsearch Input Plugin - Faraday::ConnectionFailed](https://discuss.elastic.co/t/elasticsearch-input-plugin-faraday-connectionfailed/123703)
**Author:** [@f.hiekel](https://discuss.elastic.co/u/f.hiekel)\ **Replies:** 1\ **Last updated:** [March 13, 2018, 1:21pm UTC](https://discuss.elastic.co/t/elasticsearch-input-plugin-faraday-connectionfailed/123703 "2018-03-13T13:21:04Z")
Hey, currently i try to execute an aggregation function and wirte the resul back to an index. Therefore I thought it would be a good idea to do it with the elasticserach input plugin to execute the query. My configurat… --- ## [How to escape square bracket in Logstash Grok filter?](https://discuss.elastic.co/t/how-to-escape-square-bracket-in-logstash-grok-filter/123721)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 6\ **Last updated:** [March 13, 2018, 1:08pm UTC](https://discuss.elastic.co/t/how-to-escape-square-bracket-in-logstash-grok-filter/123721 "2018-03-13T13:08:46Z")
Hi, I am using ELK GA 5.0.0. I am consuming from Kafka topic using Logstash. Messages are JSON encoded. In the message, I have date field like \\t\[05/Feb/2018:10:39:47 +0000\]\\t. In the grok filter, I am parsing it like; … --- ## [Grok Filter Pattern](https://discuss.elastic.co/t/grok-filter-pattern/123694)
**Author:** [@muthu\_kumar1](https://discuss.elastic.co/u/muthu_kumar1)\ **Replies:** 3\ **Last updated:** [March 13, 2018, 12:54pm UTC](https://discuss.elastic.co/t/grok-filter-pattern/123694 "2018-03-13T12:54:37Z")
Hi please help me to get pattern for the below log string in grok \[22/11/2017#09:06:37:126\] 1511341597077#O##servername#3322132#SCREEN#Controller#getURL#STEP#W1W8#RN19191199999111#231672998#Success#48 --- ## [Logstash date conversion from string to date not working](https://discuss.elastic.co/t/logstash-date-conversion-from-string-to-date-not-working/123699)
**Author:** [@ahir](https://discuss.elastic.co/u/ahir)\ **Replies:** 3\ **Last updated:** [March 13, 2018, 12:42pm UTC](https://discuss.elastic.co/t/logstash-date-conversion-from-string-to-date-not-working/123699 "2018-03-13T12:42:12Z")
Hi, I am a beginner in the elastic stack. I have a CSV file, contents of which I want to load into elastic search. In the CSV there is a "date" column which has dates in the following format: 2/27/2018 M/d/yyyy I am … --- ## [Kv filter - bracket removal in the field](https://discuss.elastic.co/t/kv-filter-bracket-removal-in-the-field/122247)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 2\ **Last updated:** [March 13, 2018, 11:45am UTC](https://discuss.elastic.co/t/kv-filter-bracket-removal-in-the-field/122247 "2018-03-13T11:45:01Z")
Hi All, Iam trying to remove the brackets in the value \[image\] but am not sure how to do it is it like this or something else kv { remove\_char\_value =\> "\[\]" include\_brackets =\> true } } Thanks, Ra… --- ## [Beats-pipeline.conf Spikes CPU 100% w/2+ outputs](https://discuss.elastic.co/t/beats-pipeline-conf-spikes-cpu-100-w-2-outputs/122909)
**Author:** [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Replies:** 2\ **Last updated:** [March 13, 2018, 11:30am UTC](https://discuss.elastic.co/t/beats-pipeline-conf-spikes-cpu-100-w-2-outputs/122909 "2018-03-13T11:30:15Z")
Hi, Outside of the individual performance of the targets I haven't seen any examples like this. Doesn't seem to matter which output I use as long as I only use one so I don't feel target performance is the issue and I m… --- ## [Parsing dockers files with logstash](https://discuss.elastic.co/t/parsing-dockers-files-with-logstash/123717)
**Author:** [@Asier\_Saluena](https://discuss.elastic.co/u/Asier_Saluena)\ **Replies:** 1\ **Last updated:** [March 13, 2018, 11:01am UTC](https://discuss.elastic.co/t/parsing-dockers-files-with-logstash/123717 "2018-03-13T11:01:37Z")
Hi, I'm including my dockers json logs to the ELK stack, but I can't send them to ES and I don't know why not My filebeat shipper look like this: - type: log paths: json.message\_key: log json.keys\_under\_root: true fie… --- ## [A plugin had an unrecoverable error1 5.6.8](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error1-5-6-8/123702)
**Author:** [@liyongbin0419](https://discuss.elastic.co/u/liyongbin0419)\ **Replies:** 2\ **Last updated:** [March 13, 2018, 9:24am UTC](https://discuss.elastic.co/t/a-plugin-had-an-unrecoverable-error1-5-6-8/123702 "2018-03-13T09:24:58Z")
hi i ha ve a problem when i run logstash5.6.8version ,i cant slove. this is my configure error: Error: Permission denied \[2018-03-13T16:52:47,053\]\[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable err… --- ## [SOLVED - Long & Lat Help - I've been reading many posts & still no answers](https://discuss.elastic.co/t/solved-long-lat-help-ive-been-reading-many-posts-still-no-answers/122137)
**Author:** [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Replies:** 12\ **Last updated:** [March 13, 2018, 8:03am UTC](https://discuss.elastic.co/t/solved-long-lat-help-ive-been-reading-many-posts-still-no-answers/122137 "2018-03-13T08:03:06Z")
Team, My data source has Longtitude and Latitude in it. I have time series data of aircraft and want to plot the map progress. I am using Logstash 5.3.0 from an event stream. Below is my configuration. Currently when t… --- ## [Logstash xml parsing - Fields extraction](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201)
**Author:** [@satkumvnr](https://discuss.elastic.co/u/satkumvnr)\ **Replies:** 8\ **Last updated:** [March 13, 2018, 7:04am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201 "2018-03-13T07:04:04Z")
HI, I am new to Elastic Search I am streaming the xml files from Apache Kafka to elastic search, where I need to assign the fields for xml attributes. I am struggling to get pass through it. I need to get the below fie… --- ## [High Availibility for Logstash Input Processing](https://discuss.elastic.co/t/high-availibility-for-logstash-input-processing/123653)
**Author:** [@ninjada](https://discuss.elastic.co/u/ninjada)\ **Replies:** 1\ **Last updated:** [March 13, 2018, 6:55am UTC](https://discuss.elastic.co/t/high-availibility-for-logstash-input-processing/123653 "2018-03-13T06:55:21Z")
hello, i'm currently using a logstash server to process logs in an AWS s3 bucket via the s3 input, and push them to an AWS elasticsearch service cluster. however, i need it to have production level redundancy ie. high a… --- ## [S3 input issues with images](https://discuss.elastic.co/t/s3-input-issues-with-images/122999)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 15\ **Last updated:** [March 13, 2018, 6:49am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999 "2018-03-13T06:49:36Z")
I've been trying to figure out how to solve this: I have an Amazon S3 bucket with a lot of files and filetypes. I only care about the xml files. I can't figure out how to have it only process xml files. It keeps trying t… --- ## [Persistent Queues delay output](https://discuss.elastic.co/t/persistent-queues-delay-output/123255)
**Author:** [@Hai](https://discuss.elastic.co/u/Hai)\ **Replies:** 3\ **Last updated:** [March 13, 2018, 3:47am UTC](https://discuss.elastic.co/t/persistent-queues-delay-output/123255 "2018-03-13T03:47:12Z")
when logstash enable queue.type: persisted, other parameters is default, i found logstash output very slow, but queue has buffered hundreds of megabytes of data. from visualize i got follow pic, normally data jitter fr… --- ## [Permission Denied - permission settings?](https://discuss.elastic.co/t/permission-denied-permission-settings/123359)
**Author:** [@DevT](https://discuss.elastic.co/u/DevT)\ **Replies:** 4\ **Last updated:** [March 13, 2018, 1:27am UTC](https://discuss.elastic.co/t/permission-denied-permission-settings/123359 "2018-03-13T01:27:46Z")
I get these errors when trying to start Logstash. I tried changing permissions. What do your permission settings look like on Ubuntu? logstash --path.settings /etc/logstash \`ERROR Unable to create file /var/log/logstas… --- ## [Filter for multi-lines PostgreSQL log](https://discuss.elastic.co/t/filter-for-multi-lines-postgresql-log/123657)
**Author:** [@Luiz\_Siqueira](https://discuss.elastic.co/u/Luiz_Siqueira)\ **Replies:** 0\ **Last updated:** [March 13, 2018, 12:40am UTC](https://discuss.elastic.co/t/filter-for-multi-lines-postgresql-log/123657 "2018-03-13T00:40:01Z")
Hello, great! I really need your help. I need to merge the lines of the same PostgreSQL event. My "/var/log/postgresql/postgresql-9.5-main.log" events are as follows: 2018-03-12 20:14:54 AMT \[11143\]: \[1-1\] LOG: conne… --- ## [How to send logs from logstash to a remote Elasticsearch Server](https://discuss.elastic.co/t/how-to-send-logs-from-logstash-to-a-remote-elasticsearch-server/123517)
**Author:** [@Hanen](https://discuss.elastic.co/u/Hanen)\ **Replies:** 3\ **Last updated:** [March 12, 2018, 9:20pm UTC](https://discuss.elastic.co/t/how-to-send-logs-from-logstash-to-a-remote-elasticsearch-server/123517 "2018-03-12T21:20:06Z")
Hi, I have a server where I have installed logstash. I am trying to forward a log windows using winlogbeat to a remote Elasticsearch server. i specified the elasticserver's host address and ports in the output configur… --- ## [Trimming of whitespace from kv filter fields unexpected](https://discuss.elastic.co/t/trimming-of-whitespace-from-kv-filter-fields-unexpected/123643)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 0\ **Last updated:** [March 12, 2018, 9:18pm UTC](https://discuss.elastic.co/t/trimming-of-whitespace-from-kv-filter-fields-unexpected/123643 "2018-03-12T21:18:58Z")
Is there anything in the documentation for the kv filter that suggests the following config input { generator { message =\> 'TLSID: d8658382-91d6-45f4-9b36-ab9893b5e185|Content-Length: 395|Content-Type: application/json… --- ## [Problem with geoip.location](https://discuss.elastic.co/t/problem-with-geoip-location/123564)
**Author:** [@franco.federico](https://discuss.elastic.co/u/franco.federico)\ **Replies:** 8\ **Last updated:** [March 12, 2018, 9:16pm UTC](https://discuss.elastic.co/t/problem-with-geoip-location/123564 "2018-03-12T21:16:15Z")
Hi all I'm trying to load in ES a SSH.cef file. I'm used this configuration file for logstash input { tcp { # The delimiter config used is for TCP interpretation codec =\> cef { delimiter =\> "\\r\\n"} port =\> 5000 t… --- ## [Update field based on values in multiple documents](https://discuss.elastic.co/t/update-field-based-on-values-in-multiple-documents/123601)
**Author:** [@Makra](https://discuss.elastic.co/u/Makra)\ **Replies:** 0\ **Last updated:** [March 12, 2018, 5:26pm UTC](https://discuss.elastic.co/t/update-field-based-on-values-in-multiple-documents/123601 "2018-03-12T17:26:45Z")
Hi I am sending logs/events to ES after parsing the log files though Logstash. Now what i want is to add a field after evaluating fields in multiple documents. e.g following are the documents from the ES index. pid=1,… --- ## [How to send JSON directly to elasticsearch w/o any parsing in logstash](https://discuss.elastic.co/t/how-to-send-json-directly-to-elasticsearch-w-o-any-parsing-in-logstash/123331)
**Author:** [@raychen11](https://discuss.elastic.co/u/raychen11)\ **Replies:** 2\ **Last updated:** [March 12, 2018, 5:20pm UTC](https://discuss.elastic.co/t/how-to-send-json-directly-to-elasticsearch-w-o-any-parsing-in-logstash/123331 "2018-03-12T17:20:42Z")
hey folks, My log files are already in JSON format and I have full control of how they look. When I use logstash, looks like I have to specify the input as JSON either in File plugin (use codec) or Filter (use json). I … --- ## [Logstash data pipeline from Elastic to SQL Server](https://discuss.elastic.co/t/logstash-data-pipeline-from-elastic-to-sql-server/123589)
**Author:** [@jruzicka](https://discuss.elastic.co/u/jruzicka)\ **Replies:** 1\ **Last updated:** [March 12, 2018, 4:37pm UTC](https://discuss.elastic.co/t/logstash-data-pipeline-from-elastic-to-sql-server/123589 "2018-03-12T16:37:30Z")
We have a series of reports running in SSRS. There is data we are only sending to Elastic that I'd like to reference in SSRS. What I'd like to do is create a pipeline that can query elastic and output specific fields t… --- ## [Help Setting Up JSON Filter](https://discuss.elastic.co/t/help-setting-up-json-filter/123363)
**Author:** [@Kurorion](https://discuss.elastic.co/u/Kurorion)\ **Replies:** 2\ **Last updated:** [March 12, 2018, 3:57pm UTC](https://discuss.elastic.co/t/help-setting-up-json-filter/123363 "2018-03-12T15:57:31Z")
Hello, I have a JSON array in the following format that I need to create a set of filters for to get the desired output. The JSON is using the SenML format if anyone is curious. \[ {"bname":"urn:dev:ow:10e2073a01080… --- ## [Doc + meta =\> csv + email](https://discuss.elastic.co/t/doc-meta-csv-email/123586)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 0\ **Last updated:** [March 12, 2018, 3:50pm UTC](https://discuss.elastic.co/t/doc-meta-csv-email/123586 "2018-03-12T15:50:23Z")
My goal: I have a several docs spread across of multiple indices that I'd like to export as CSV with addition meta data set and email as attachment on daily basis using Logstash. Logstash plugin: input: Elasticsear… --- ## [Logstash to Elasticsearch JDBC Current Version Conflict Issue](https://discuss.elastic.co/t/logstash-to-elasticsearch-jdbc-current-version-conflict-issue/123314)
**Author:** [@ayeager](https://discuss.elastic.co/u/ayeager)\ **Replies:** 2\ **Last updated:** [March 12, 2018, 3:26pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-jdbc-current-version-conflict-issue/123314 "2018-03-12T15:26:00Z")
I'm trying to import data from a mysql database with the jdbc input plugin and based on the way I'm getting the data out of the database, I'm running into concurrency errors similar to the one below \[2018-03-09T17:40:02… --- ## [GROK filter for multi-line Postgresql log](https://discuss.elastic.co/t/grok-filter-for-multi-line-postgresql-log/39854)
**Author:** [@EugeneBorshch](https://discuss.elastic.co/u/EugeneBorshch)\ **Replies:** 7\ **Last updated:** [March 12, 2018, 2:52pm UTC](https://discuss.elastic.co/t/grok-filter-for-multi-line-postgresql-log/39854 "2018-03-12T14:52:47Z")
Hi All, I am having hard times trying to merge multi-line Postgresql logs with GROK into one Logstash event. My log file look like this: Jan 22 09:45:01 myhost postgres\[19015\]: \[2-1\] postgres postgres 19015 ::1 ERROR… --- ## [Load old data to ElasticSearch](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557)
**Author:** [@Codrin\_Costea](https://discuss.elastic.co/u/Codrin_Costea)\ **Replies:** 6\ **Last updated:** [March 12, 2018, 2:47pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557 "2018-03-12T14:47:29Z")
Hello ! I am new to Elastic Search and I would need some help in one problem. I try to load multiple log files from my proxy to Elastic Search, through logstast. The steps that I am following are: Create the .conf fil… --- ## [The best way to automatically delete data from certain index by list of IDs](https://discuss.elastic.co/t/the-best-way-to-automatically-delete-data-from-certain-index-by-list-of-ids/122701)
**Author:** [@Mikhail](https://discuss.elastic.co/u/Mikhail)\ **Replies:** 1\ **Last updated:** [March 12, 2018, 2:29pm UTC](https://discuss.elastic.co/t/the-best-way-to-automatically-delete-data-from-certain-index-by-list-of-ids/122701 "2018-03-12T14:29:34Z")
Hi, Could you please help me with the following. I have some SQL environment that consists of: Table1. It is main table with data. Here I have unique identifier SKID of each entry. Table2. Accumulates SKIDs of entrie… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=802) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=804)