# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=411 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 412 --- ## [4096 characters input limitation! missing part of logs in message field](https://discuss.elastic.co/t/4096-characters-input-limitation-missing-part-of-logs-in-message-field/217522)
**Author:** [@kasra](https://discuss.elastic.co/u/kasra)\ **Replies:** 2\ **Last updated:** [February 2, 2020, 2:03pm UTC](https://discuss.elastic.co/t/4096-characters-input-limitation-missing-part-of-logs-in-message-field/217522 "2020-02-02T14:03:17Z")
I have issue with sending logs which have more than 4096 characters! the thing is some of my logs contain more than 4096 characters and the result is jsonparsefailure . I have also used multiline codec for my input but… --- ## [Mcfee Epo XMl part contains unwanted \\".How to remove this](https://discuss.elastic.co/t/mcfee-epo-xml-part-contains-unwanted-how-to-remove-this/217519)
**Author:** [@Ozil](https://discuss.elastic.co/u/Ozil)\ **Replies:** 1\ **Last updated:** [February 2, 2020, 11:25am UTC](https://discuss.elastic.co/t/mcfee-epo-xml-part-contains-unwanted-how-to-remove-this/217519 "2020-02-02T11:25:27Z")
Mcafee EPO XML part contains extra " " " .Because of this xml become invalid. i would like to replace " with ". is that possible.i tried with Gsub mutate { gsub =\> \[ "syslog5424\_msg","\[\\"\]", """\] } its not working p… --- ## [Logstash Avro Kafka Codec issue](https://discuss.elastic.co/t/logstash-avro-kafka-codec-issue/217510)
**Author:** [@Nick11](https://discuss.elastic.co/u/Nick11)\ **Replies:** 0\ **Last updated:** [February 2, 2020, 5:05am UTC](https://discuss.elastic.co/t/logstash-avro-kafka-codec-issue/217510 "2020-02-02T05:05:13Z")
Hi all, I am running a simple logstash pipeline to consume a message from a Kafka topic, which is in AVRO format and display it i on the stdout. I get all empty elements in the logstash output. AVRO Schema (bestMatchin… --- ## [Creating multiple indexes with multiple s3 inputs](https://discuss.elastic.co/t/creating-multiple-indexes-with-multiple-s3-inputs/217005)
**Author:** [@Sulayman](https://discuss.elastic.co/u/Sulayman)\ **Replies:** 1\ **Last updated:** [February 1, 2020, 9:30pm UTC](https://discuss.elastic.co/t/creating-multiple-indexes-with-multiple-s3-inputs/217005 "2020-02-01T21:30:42Z")
Hi there, I have a problem where I have two S3 buckets. Bucket A monitors access logs and filters with grok. Bucket B has csv files. Every time i run logstash, the csv filter incorrectly tries to filter the access-lo… --- ## [MongoDB Atlas Integration with Logstash through logstash-output-mongodb plugin](https://discuss.elastic.co/t/mongodb-atlas-integration-with-logstash-through-logstash-output-mongodb-plugin/217500)
**Author:** [@rodher](https://discuss.elastic.co/u/rodher)\ **Replies:** 0\ **Last updated:** [February 1, 2020, 4:15pm UTC](https://discuss.elastic.co/t/mongodb-atlas-integration-with-logstash-through-logstash-output-mongodb-plugin/217500 "2020-02-01T16:15:33Z")
Hi I am trying to connect Logstash with MongoDB Atlas using logstash-output-mongodb plugin. However, some issues happen. Currently, I am using logstash 6.7.2, and logtsash-output-plugin version 3.1.6 The integration I… --- ## [Testing Logstash Custom Conf file with Custom File Input](https://discuss.elastic.co/t/testing-logstash-custom-conf-file-with-custom-file-input/217285)
**Author:** [@ellje](https://discuss.elastic.co/u/ellje)\ **Replies:** 3\ **Last updated:** [February 1, 2020, 1:16am UTC](https://discuss.elastic.co/t/testing-logstash-custom-conf-file-with-custom-file-input/217285 "2020-02-01T01:16:36Z")
Would like to have Logstash use my custom .conf file and custom input file input and generate any errors (if any) or outputs. Is there a way to do this? --- ## [Best Way To Match Hundreds of Value Possibilities in Logstash](https://discuss.elastic.co/t/best-way-to-match-hundreds-of-value-possibilities-in-logstash/217471)
**Author:** [@joshuasmith](https://discuss.elastic.co/u/joshuasmith)\ **Replies:** 1\ **Last updated:** [February 1, 2020, 12:03am UTC](https://discuss.elastic.co/t/best-way-to-match-hundreds-of-value-possibilities-in-logstash/217471 "2020-02-01T00:03:18Z")
Good day, I am trying to find the most efficient way in Logstash filters to say "if field X is one of these dozens/hundreds/thousands of values do something". For instance, I have multiple log sources (Windows logs, fi… --- ## [How to convert nummeric values 201901020 into date 02.01.2019](https://discuss.elastic.co/t/how-to-convert-nummeric-values-201901020-into-date-02-01-2019/217084)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 6\ **Last updated:** [January 31, 2020, 11:52pm UTC](https://discuss.elastic.co/t/how-to-convert-nummeric-values-201901020-into-date-02-01-2019/217084 "2020-01-31T23:52:37Z")
Hi all, I need your help with this case: How can I convert fields containing values like 201901020 into date 02.01.2019 (the last zero of the value is useless but there) I tried different mappings using follwoing sim… --- ## [Service isnt on in Logstash Docker Container?](https://discuss.elastic.co/t/service-isnt-on-in-logstash-docker-container/217464)
**Author:** [@fallenreaper](https://discuss.elastic.co/u/fallenreaper)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 9:07pm UTC](https://discuss.elastic.co/t/service-isnt-on-in-logstash-docker-container/217464 "2020-01-31T21:07:15Z")
Ideally, I want a logstash container to process all files under a mounted directory to the container. Before I started trying to create a folder in which will be looked at through the logstash file, I figured to try it … --- ## [Filebeat to Logstash resending last message](https://discuss.elastic.co/t/filebeat-to-logstash-resending-last-message/216813)
**Author:** [@d3mak](https://discuss.elastic.co/u/d3mak)\ **Replies:** 1\ **Last updated:** [January 31, 2020, 8:12pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-resending-last-message/216813 "2020-01-31T20:12:41Z")
I have a simple test-setup where I send data to Logstash via Filebeat. Both are in Version 7.5.2. All processes run on the localhost. On Logstash I pipe the output to stdout. I am sending contents of log-files via Fileb… --- ## [Ship FortiGate logs to ELK 6.8.6 with TLS enabled](https://discuss.elastic.co/t/ship-fortigate-logs-to-elk-6-8-6-with-tls-enabled/217430)
**Author:** [@claud81](https://discuss.elastic.co/u/claud81)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 4:56pm UTC](https://discuss.elastic.co/t/ship-fortigate-logs-to-elk-6-8-6-with-tls-enabled/217430 "2020-01-31T16:56:56Z")
Hi, I'm trying to ship FortiGate 6.0 logs to ELK 6.8.6. With udp input plugin all works fine on another ELK 6.8.6 stack without TLS, I can't see all the logs into Kibana. If I try to enable the tcp input plugin with c… --- ## [Extract multiple field/value pairs from single value in JSON string](https://discuss.elastic.co/t/extract-multiple-field-value-pairs-from-single-value-in-json-string/217367)
**Author:** [@Adi\_Buta](https://discuss.elastic.co/u/Adi_Buta)\ **Replies:** 2\ **Last updated:** [January 31, 2020, 3:11pm UTC](https://discuss.elastic.co/t/extract-multiple-field-value-pairs-from-single-value-in-json-string/217367 "2020-01-31T15:11:22Z")
Hello, I'm trying to extract some field/value pairs from a single field in a json string. My sample message: { ... "comment":"Ticket number : INC:I16393898, status : Work In Progress, group : Ent.Support.SOC.", ... … --- ## [Logstash maintain order of updates](https://discuss.elastic.co/t/logstash-maintain-order-of-updates/217309)
**Author:** [@noelzubinvictor](https://discuss.elastic.co/u/noelzubinvictor)\ **Replies:** 1\ **Last updated:** [January 31, 2020, 3:00pm UTC](https://discuss.elastic.co/t/logstash-maintain-order-of-updates/217309 "2020-01-31T15:00:25Z")
I have a logstash jdbc pipeline that syncs data from postgres to elastic search. If I set the number of logstash workers to say 16. Does the pipeline still use only 1 thread. If it does use more than 1 thread then does l… --- ## [Environment Variable in logstash export](https://discuss.elastic.co/t/environment-variable-in-logstash-export/217392)
**Author:** [@saravana\_hariharan](https://discuss.elastic.co/u/saravana_hariharan)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 2:08pm UTC](https://discuss.elastic.co/t/environment-variable-in-logstash-export/217392 "2020-01-31T14:08:03Z")
How to set environment variable in logstash input { tcp { port =\> "${TCP\_PORT}" } } Now let’s set the value of TCP\_PORT: export TCP\_PORT=12345(i dont know, where to do this export please help) --- ## [Translate filter with +2.5 million dictionary entries](https://discuss.elastic.co/t/translate-filter-with-2-5-million-dictionary-entries/217390)
**Author:** [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 2:01pm UTC](https://discuss.elastic.co/t/translate-filter-with-2-5-million-dictionary-entries/217390 "2020-01-31T14:01:52Z")
Hi, I have a case when I need to use a translate filter with a very large YAML dictionary (around 2.5 million of entries). I was reading this post where it recommends to use memcached in order to speed up the transla… --- ## [Duplicate records in elastic search during load of records with Logstash](https://discuss.elastic.co/t/duplicate-records-in-elastic-search-during-load-of-records-with-logstash/217372)
**Author:** [@ramypala](https://discuss.elastic.co/u/ramypala)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 12:41pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elastic-search-during-load-of-records-with-logstash/217372 "2020-01-31T12:41:09Z")
Hi Team, I am loading a ServiceNow table data into Elastic search using Logstash config with HTTP Poller Plugin.For example if the table has 40 records,after the load of 40 records in Elastic search, data load is not ge… --- ## [Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 127.0.0.1:5044: connect: connection refused](https://discuss.elastic.co/t/failed-to-connect-to-backoff-async-tcp-localhost-5044-dial-tcp-127-0-0-1-connect-connection-refused/213713)
**Author:** [@Robin020](https://discuss.elastic.co/u/Robin020)\ **Replies:** 5\ **Last updated:** [January 31, 2020, 12:11pm UTC](https://discuss.elastic.co/t/failed-to-connect-to-backoff-async-tcp-localhost-5044-dial-tcp-127-0-0-1-connect-connection-refused/213713 "2020-01-31T12:11:04Z")
My setup = filebeat --\> logstash --\> elasticsearch The first part (filebeat --\> logstash ) doens't work. I got this message in the filebeat log: INFO pipeline/output.go:95 Connecting to backoff(async(tcp://localh… --- ## [Logstash jdbc plugin how to ingest data from mysql table](https://discuss.elastic.co/t/logstash-jdbc-plugin-how-to-ingest-data-from-mysql-table/217335)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 3\ **Last updated:** [January 31, 2020, 11:49am UTC](https://discuss.elastic.co/t/logstash-jdbc-plugin-how-to-ingest-data-from-mysql-table/217335 "2020-01-31T11:49:19Z")
Hi everyone! I'm trying to transport data from MySQL table INFORMATION\_SCHEMA.PROCESSLIST via Logstash jdbc plugin. I never worked with jdbc before so I installed jdbc with sudo apt-get install libmysql-java command an… --- ## [Extract data from SQL Server](https://discuss.elastic.co/t/extract-data-from-sql-server/217360)
**Author:** [@Mbt\_Mbt](https://discuss.elastic.co/u/Mbt_Mbt)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 11:35am UTC](https://discuss.elastic.co/t/extract-data-from-sql-server/217360 "2020-01-31T11:35:34Z")
Good Morning, I am having trouble accessing the data in the SQLExpress database.I don't receive answers in elasticsearch Downloaded de JDBC driver and put it in te route: C:\\Users\\admin\\Desktop\\sqljdbc\_4.2\\esn\\jre8\\sql… --- ## [Dissect with a backslash at the end of the log line](https://discuss.elastic.co/t/dissect-with-a-backslash-at-the-end-of-the-log-line/216630)
**Author:** [@Vincent\_Maury](https://discuss.elastic.co/u/Vincent_Maury)\ **Replies:** 4\ **Last updated:** [January 31, 2020, 10:47am UTC](https://discuss.elastic.co/t/dissect-with-a-backslash-at-the-end-of-the-log-line/216630 "2020-01-31T10:47:55Z")
Hi, I'm having a log (from EMC Celerra) that ends with a backslash like this: \:\:\ daemon \\ I'm trying this dissect: dissect { mapping =\> { "message" =\> "%{event\_source}:%{severity}:%{id} d… --- ## [Config file that checks conditions before naming fields](https://discuss.elastic.co/t/config-file-that-checks-conditions-before-naming-fields/217253)
**Author:** [@teezy](https://discuss.elastic.co/u/teezy)\ **Replies:** 4\ **Last updated:** [January 31, 2020, 10:01am UTC](https://discuss.elastic.co/t/config-file-that-checks-conditions-before-naming-fields/217253 "2020-01-31T10:01:06Z")
Hi, I have a secure log file with the following content: Jan 12 08:28:24 name sshd\[11483\]: Did not receive identification string from xx.xxx.xxx.xx port xxxxx Jan 12 08:38:54 name proftpd\[12559\]: xx.xxx.xxx.xx (xx.xxx.x… --- ## [Correct date format for data ingestion](https://discuss.elastic.co/t/correct-date-format-for-data-ingestion/217331)
**Author:** [@T\_Martin\_T](https://discuss.elastic.co/u/T_Martin_T)\ **Replies:** 1\ **Last updated:** [January 31, 2020, 9:58am UTC](https://discuss.elastic.co/t/correct-date-format-for-data-ingestion/217331 "2020-01-31T09:58:06Z")
Hi guys, Im trying to ingest some data and I encountered a problem with the correct date assignment. The following sample of data looks like this: "price\_eur" =\> 4922.28, "engine\_power" =\> 122, "maker" =\> "mazda", … --- ## [Logstash JSON codec vs JSON filter performance](https://discuss.elastic.co/t/logstash-json-codec-vs-json-filter-performance/217334)
**Author:** [@TWhitehouse](https://discuss.elastic.co/u/TWhitehouse)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 9:24am UTC](https://discuss.elastic.co/t/logstash-json-codec-vs-json-filter-performance/217334 "2020-01-31T09:24:50Z")
Hey If I have input coming into a pipeline in JSON format, which is faster to use? I have multiple pipeline workers, so multiple threads executing filter+output. I was thinking input would be essentially 1 thread as i… --- ## [Logstash as syslog server - TCP is not working](https://discuss.elastic.co/t/logstash-as-syslog-server-tcp-is-not-working/217044)
**Author:** [@ptselios](https://discuss.elastic.co/u/ptselios)\ **Replies:** 1\ **Last updated:** [January 31, 2020, 8:35am UTC](https://discuss.elastic.co/t/logstash-as-syslog-server-tcp-is-not-working/217044 "2020-01-31T08:35:34Z")
Hello, I am completely new to the ELK stack and I have just installed an ELK all-in-one machine for testing purposes. In our organization we want to send all syslog messages to logstash and then to ES. I have the follo… --- ## [ElasticSearch Input and data manipulation](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252)
**Author:** [@Stefano\_Bossi](https://discuss.elastic.co/u/Stefano_Bossi)\ **Replies:** 3\ **Last updated:** [January 31, 2020, 8:04am UTC](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252 "2020-01-31T08:04:01Z")
I am trying to use the ES input plugin for logstash to further manipulate some data and reinsert the data into ES. I have a logstash filter like this: input { elasticsearch { hosts =\> \[ "ms-master… --- ## [MongoDB Input threw an exception, restarting {:exception=\>#\ **Author:** [@Raghala\_Purushotham](https://discuss.elastic.co/u/Raghala_Purushotham)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 6:33am UTC](https://discuss.elastic.co/t/mongodb-input-threw-an-exception-restarting-exception-bson/217302 "2020-01-31T06:33:22Z") Hi, I am trying to import the data from MongoDB to Elastic. As part of that, I am using the "logstash-input-mongodb" as an input plugin. But I am facing the below error while importing it. Conf file: input { mongodb { … --- ## [How To remove \\ from the existing log formate](https://discuss.elastic.co/t/how-to-remove-from-the-existing-log-formate/217295)
**Author:** [@Ozil](https://discuss.elastic.co/u/Ozil)\ **Replies:** 3\ **Last updated:** [January 31, 2020, 5:19am UTC](https://discuss.elastic.co/t/how-to-remove-from-the-existing-log-formate/217295 "2020-01-31T05:19:11Z")
Hi all I am trying to parse McFee EPO logs using logstash. I am using the combination of grok and XML filter to parse the log. In the input data set there are few \\ within the XML part so I am not able to parse the log … --- ## [Logstash Mysql](https://discuss.elastic.co/t/logstash-mysql/217301)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 6:24am UTC](https://discuss.elastic.co/t/logstash-mysql/217301 "2020-01-31T06:24:32Z")
I make connectivity of elk stack with mysql. I parse the data from mysql it works perfectly but when i parse the one table with select \* command it tells me that unique id \>0 but when i make a view of that table it parse… --- ## [Time mismatch @timestamp and message](https://discuss.elastic.co/t/time-mismatch-timestamp-and-message/216999)
**Author:** [@faridmmv](https://discuss.elastic.co/u/faridmmv)\ **Replies:** 11\ **Last updated:** [January 31, 2020, 5:37am UTC](https://discuss.elastic.co/t/time-mismatch-timestamp-and-message/216999 "2020-01-31T05:37:45Z")
Hi. How can i configure logstash to have @timestamp and message time equal. My Config is: input { beats { port =\> 5044 } } filter { if \[type\] == "syslog" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:… --- ## [Scaling :logstash with reading MARIADB as input and elasticsearch is the output](https://discuss.elastic.co/t/scaling-logstash-with-reading-mariadb-as-input-and-elasticsearch-is-the-output/217292)
**Author:** [@pranab155](https://discuss.elastic.co/u/pranab155)\ **Replies:** 0\ **Last updated:** [January 31, 2020, 4:06am UTC](https://discuss.elastic.co/t/scaling-logstash-with-reading-mariadb-as-input-and-elasticsearch-is-the-output/217292 "2020-01-31T04:06:18Z")
Hi , i have below queries , 1.can we read by two instance of logstash from same table ensuring there is no duplicate record read by either of instance and table has pk column name as "seq\_No" 2. if elastic search is … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=410) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=412)