# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=496 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 497 --- ## [Indent any json file nested fields and make it flat](https://discuss.elastic.co/t/indent-any-json-file-nested-fields-and-make-it-flat/194447)
**Author:** [@baselai](https://discuss.elastic.co/u/baselai)\ **Replies:** 5\ **Last updated:** [August 8, 2019, 7:33pm UTC](https://discuss.elastic.co/t/indent-any-json-file-nested-fields-and-make-it-flat/194447 "2019-08-08T19:33:06Z")
I have the following json code, it has a nested fields and I'm trying to make it flat but I couldn't! Here is the source json file { "body": { "BlockName": "SA", "input": { "type": "arra… --- ## [Logstash: File Input in Read Mode does not work - Docker image and Java Version](https://discuss.elastic.co/t/logstash-file-input-in-read-mode-does-not-work-docker-image-and-java-version/194488)
**Author:** [@MJochim](https://discuss.elastic.co/u/MJochim)\ **Replies:** 1\ **Last updated:** [August 8, 2019, 7:29pm UTC](https://discuss.elastic.co/t/logstash-file-input-in-read-mode-does-not-work-docker-image-and-java-version/194488 "2019-08-08T19:29:01Z")
Hi, I’ve started testing what Logstash (7.3.0) can do and I’m having a problem with the File input plugin‘s “read mode.” It fails on gzip'd files. The problem has been reported before, e.g. here: Apparently, the read… --- ## [Trying to follow Logstash tutorial](https://discuss.elastic.co/t/trying-to-follow-logstash-tutorial/194487)
**Author:** [@bschooly](https://discuss.elastic.co/u/bschooly)\ **Replies:** 1\ **Last updated:** [August 8, 2019, 7:15pm UTC](https://discuss.elastic.co/t/trying-to-follow-logstash-tutorial/194487 "2019-08-08T19:15:47Z")
On centos 7, I've installed logstash, and beats via the repo mentioned in the docs. The very first test at "https://www.elastic.co/guide/en/logstash/current/first-event.html" works. Now I'm trying "https://www.elastic.c… --- ## [Error: com.mysql.cj.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc\_driver\_library?](https://discuss.elastic.co/t/error-com-mysql-cj-jdbc-driver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/194133)
**Author:** [@manideep](https://discuss.elastic.co/u/manideep)\ **Replies:** 5\ **Last updated:** [August 8, 2019, 3:46pm UTC](https://discuss.elastic.co/t/error-com-mysql-cj-jdbc-driver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/194133 "2019-08-08T15:46:03Z")
Hi All, I don't understand what's wrong in below conf file. input { jdbc { jdbc\_driver\_library =\> "/usr/share/logstash/myjars/mysql-connector-java-8.0.11.jar" jdbc\_driver\_class =\> "com.mysql.cj.jdbc.Driver" jdbc\_co… --- ## [How to list plugins on a logstash docker container?](https://discuss.elastic.co/t/how-to-list-plugins-on-a-logstash-docker-container/194460)
**Author:** [@kmiklas](https://discuss.elastic.co/u/kmiklas)\ **Replies:** 1\ **Last updated:** [August 8, 2019, 3:17pm UTC](https://discuss.elastic.co/t/how-to-list-plugins-on-a-logstash-docker-container/194460 "2019-08-08T15:17:08Z")
Logstash is running in a docker container, and I'd like to list its plugins. For a VM or metal installation, Here is the command \[Ref 1\] sudo bin/plugin list How do I run this command in a Docker container? I've tried… --- ## [Problem with add\_tag](https://discuss.elastic.co/t/problem-with-add-tag/194426)
**Author:** [@Arnodl](https://discuss.elastic.co/u/Arnodl)\ **Replies:** 8\ **Last updated:** [August 8, 2019, 2:45pm UTC](https://discuss.elastic.co/t/problem-with-add-tag/194426 "2019-08-08T14:45:10Z")
Hello, I would like to change the tags field to a lowercase field. My original message like this: { "@timestamp": "2019-08-08T13:33:38.86", "appid": "bla", "tags": \[ "SUCC:AUTHEN"\] } My filter: mutate { rename =\> { … --- ## [Failed to parse date field error](https://discuss.elastic.co/t/failed-to-parse-date-field-error/194445)
**Author:** [@ssi](https://discuss.elastic.co/u/ssi)\ **Replies:** 1\ **Last updated:** [August 8, 2019, 2:38pm UTC](https://discuss.elastic.co/t/failed-to-parse-date-field-error/194445 "2019-08-08T14:38:35Z")
hi i get these errors with one of the indexes (microsoft defender ATP data) \[2019-08-08T15:57:01,663\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_i… --- ## [Logstash in-mem queues and input plugins](https://discuss.elastic.co/t/logstash-in-mem-queues-and-input-plugins/194427)
**Author:** [@ep4sh](https://discuss.elastic.co/u/ep4sh)\ **Replies:** 1\ **Last updated:** [August 8, 2019, 1:04pm UTC](https://discuss.elastic.co/t/logstash-in-mem-queues-and-input-plugins/194427 "2019-08-08T13:04:07Z")
Good day, dear forum users! My logs transfer scheme is as follows: App servers -\> AWS SQS (queue) -\> Logstash (input =\> sqs, output =\> elasticsearch) -\> AWS Elasticsearch Sometimes Elasticsearch is blocked for writing… --- ## [Multiple pipelines - distributor pattern](https://discuss.elastic.co/t/multiple-pipelines-distributor-pattern/194058)
**Author:** [@penguinairlines](https://discuss.elastic.co/u/penguinairlines)\ **Replies:** 21\ **Last updated:** [August 8, 2019, 12:53pm UTC](https://discuss.elastic.co/t/multiple-pipelines-distributor-pattern/194058 "2019-08-08T12:53:25Z")
Looks like I'm having a similar problem to another thread that was closed without being resolved. I am trying to use the distributor pattern. The linked doc makes some sense, but I have a few remaining questions. Rega… --- ## [Fail to log data from server database to elasticsearch](https://discuss.elastic.co/t/fail-to-log-data-from-server-database-to-elasticsearch/193801)
**Author:** [@samer](https://discuss.elastic.co/u/samer)\ **Replies:** 2\ **Last updated:** [August 8, 2019, 12:46pm UTC](https://discuss.elastic.co/t/fail-to-log-data-from-server-database-to-elasticsearch/193801 "2019-08-08T12:46:15Z")
I had installed the jdbc plugin with the command logstash-plugin install ..... and I had set the JAVA\_HOME with the command "setx JAVA\_HOME "path of java file" this the pipeline: input { jdbc { jdbc\_connection\_strin… --- ## [Logstash-webhdfs-Kerberos : No Kerberos credentials available (default cache: FILE:/tmp/krb5cc\_220590)](https://discuss.elastic.co/t/logstash-webhdfs-kerberos-no-kerberos-credentials-available-default-cache-file-tmp-krb5cc-220590/194429)
**Author:** [@shilpaawati](https://discuss.elastic.co/u/shilpaawati)\ **Replies:** 0\ **Last updated:** [August 8, 2019, 12:09pm UTC](https://discuss.elastic.co/t/logstash-webhdfs-kerberos-no-kerberos-credentials-available-default-cache-file-tmp-krb5cc-220590/194429 "2019-08-08T12:09:58Z")
Hi, I am using the webhdfs plugin of Logstash to send files to HDFS host system which uses the Kerberos authentication. After the config I see errors while the Logstash startup. Any suggestions about this issue? Webhd… --- ## [Can't get text on a START\_OBJECT at 1:1043](https://discuss.elastic.co/t/cant-get-text-on-a-start-object-at-1-1043/194387)
**Author:** [@LEK-LD](https://discuss.elastic.co/u/LEK-LD)\ **Replies:** 4\ **Last updated:** [August 8, 2019, 10:22am UTC](https://discuss.elastic.co/t/cant-get-text-on-a-start-object-at-1-1043/194387 "2019-08-08T10:22:28Z")
Hey guys, I'm facing a problem with my elasticsearch-output in Logstash. What I'm trying to archieve is indexing my logs, sent by filebeat, via logstash in elasticsearch. My logs are in json-codec and I'm using version … --- ## [Elasticsearch to csv using logstash](https://discuss.elastic.co/t/elasticsearch-to-csv-using-logstash/194371)
**Author:** [@Karnish\_Master](https://discuss.elastic.co/u/Karnish_Master)\ **Replies:** 0\ **Last updated:** [August 8, 2019, 8:24am UTC](https://discuss.elastic.co/t/elasticsearch-to-csv-using-logstash/194371 "2019-08-08T08:24:30Z")
Hi All , I am trying to export elasticsearch index data to CSV using logstash , Below is my configuration :- input { elasticsearch { hosts =\> "localhost:9200" index =\> "index\_name" query =\> ' { "query": { "matc… --- ## [Logstash failing with "ERROR::EMFILE- Too many open files"](https://discuss.elastic.co/t/logstash-failing-with-error-emfile-too-many-open-files/190715)
**Author:** [@Laddu\_ps](https://discuss.elastic.co/u/Laddu_ps)\ **Replies:** 15\ **Last updated:** [August 8, 2019, 5:39am UTC](https://discuss.elastic.co/t/logstash-failing-with-error-emfile-too-many-open-files/190715 "2019-08-08T05:39:31Z")
Hi. I'm having issues running logstash withi large sets of data. I'm trying to push data to Elasticsearch via Logstash Where both the applications are hosted on CENTOS servers. I'm trying to send 1000 files per minute (e… --- ## [Grok an IP address and username](https://discuss.elastic.co/t/grok-an-ip-address-and-username/194325)
**Author:** [@uklipse](https://discuss.elastic.co/u/uklipse)\ **Replies:** 2\ **Last updated:** [August 8, 2019, 2:39am UTC](https://discuss.elastic.co/t/grok-an-ip-address-and-username/194325 "2019-08-08T02:39:01Z")
Hi, I have the following text below that I'm trying to parse just the IP and username. I've tried adding all the text prior to the IP but still nothing. Even if I just put 1.1.1.1 into the Grok Debugger and %{IP} as the … --- ## [Logstash plugin offline pack not persisted](https://discuss.elastic.co/t/logstash-plugin-offline-pack-not-persisted/194333)
**Author:** [@jfreeman](https://discuss.elastic.co/u/jfreeman)\ **Replies:** 0\ **Last updated:** [August 7, 2019, 9:48pm UTC](https://discuss.elastic.co/t/logstash-plugin-offline-pack-not-persisted/194333 "2019-08-07T21:48:26Z")
I have created logstash-offline-plugins zip with all the plugins I want. I am able to install this pack using the command bin/logstash-plugin install file:///var/logstash\_offline/logstash-offline-plugins-6.7.0.zip Then… --- ## [RegEx in Logstash Conf File: loop through/retrieve all matches](https://discuss.elastic.co/t/regex-in-logstash-conf-file-loop-through-retrieve-all-matches/194020)
**Author:** [@dundundundun](https://discuss.elastic.co/u/dundundundun)\ **Replies:** 8\ **Last updated:** [August 7, 2019, 8:48pm UTC](https://discuss.elastic.co/t/regex-in-logstash-conf-file-loop-through-retrieve-all-matches/194020 "2019-08-07T20:48:34Z")
Let's suppose my log messages are as following: GroupName::Cars: mileage=19274 ~ year=2000 GroupName::school: students = 10000 ~ location = USA ~ classes = 50 ~ staff = 75 How can I pattern match both of these l… --- ## [Docker tls problem](https://discuss.elastic.co/t/docker-tls-problem/194321)
**Author:** [@sweb](https://discuss.elastic.co/u/sweb)\ **Replies:** 0\ **Last updated:** [August 7, 2019, 8:24pm UTC](https://discuss.elastic.co/t/docker-tls-problem/194321 "2019-08-07T20:24:08Z")
I have tls that root ca sign by me, With normal http input everything work fine: input { http { port =\> 3389 # ssl =\> true # ssl\_certificate\_authorities =\> \["/opt/certs/ca.pem"\] # ssl\_certificate =\> "… --- ## [Grok Stops Reading at a Backslash?](https://discuss.elastic.co/t/grok-stops-reading-at-a-backslash/194085)
**Author:** [@jakecramer17](https://discuss.elastic.co/u/jakecramer17)\ **Replies:** 4\ **Last updated:** [August 7, 2019, 6:28pm UTC](https://discuss.elastic.co/t/grok-stops-reading-at-a-backslash/194085 "2019-08-07T18:28:04Z")
This is an extension to a previously solved topic. I have this log here (as it appears in the console) 2019-08-05 08:55:15 - jcramer(Software Programmer) - Successfully updated profile that I've successfully been able… --- ## [Data loss using Json filter, messages dropped without any errors or warnings](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255)
**Author:** [@sherifabdlnaby](https://discuss.elastic.co/u/sherifabdlnaby)\ **Replies:** 3\ **Last updated:** [August 7, 2019, 3:48pm UTC](https://discuss.elastic.co/t/data-loss-using-json-filter-messages-dropped-without-any-errors-or-warnings/194255 "2019-08-07T15:48:50Z")
During setting up our ELK stack for application logs, some logs come structured and some aren't, I am running the JSON filter on every message to tag logs if they're structured or not. After some errors happened and we … --- ## [Logstash and Filebeat netflow events per second - losing data?](https://discuss.elastic.co/t/logstash-and-filebeat-netflow-events-per-second-losing-data/194277)
**Author:** [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Replies:** 0\ **Last updated:** [August 7, 2019, 3:48pm UTC](https://discuss.elastic.co/t/logstash-and-filebeat-netflow-events-per-second-losing-data/194277 "2019-08-07T15:48:15Z")
Hi there, we are currently trying to use Logstash and Filebeat to process a large netflow stream. We are running filebeat and logstash in docker containers using the official elastic search container images. The server… --- ## [Pattern matching filebeat to logstash](https://discuss.elastic.co/t/pattern-matching-filebeat-to-logstash/194268)
**Author:** [@Rovaldy\_Applyrs](https://discuss.elastic.co/u/Rovaldy_Applyrs)\ **Replies:** 1\ **Last updated:** [August 7, 2019, 3:21pm UTC](https://discuss.elastic.co/t/pattern-matching-filebeat-to-logstash/194268 "2019-08-07T15:21:41Z")
I wanted to know if it is possible to pattern match the path of the files coming in from file beat in log stash? I am trying to extract the path but it did not work. I tried something like this input { beats { port =\>… --- ## [CSV filter plugin and autodetect column names](https://discuss.elastic.co/t/csv-filter-plugin-and-autodetect-column-names/194244)
**Author:** [@Zedd](https://discuss.elastic.co/u/Zedd)\ **Replies:** 2\ **Last updated:** [August 7, 2019, 3:18pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-and-autodetect-column-names/194244 "2019-08-07T15:18:17Z")
Hi, I'm wondering if there is a way to parse one single CSV log using two distinct logstash conf files assuming each conf file would have a CSV filter plugin with autodetect\_column\_names set to true. As I understand it… --- ## [Dockerized Logstash and netflow module install issue](https://discuss.elastic.co/t/dockerized-logstash-and-netflow-module-install-issue/193651)
**Author:** [@iukea](https://discuss.elastic.co/u/iukea)\ **Replies:** 11\ **Last updated:** [August 7, 2019, 2:54pm UTC](https://discuss.elastic.co/t/dockerized-logstash-and-netflow-module-install-issue/193651 "2019-08-07T14:54:40Z")
Hello, I am currently having issues with installing the Netflow module inside of dockerized logstash Docker file FROM docker.elastic.co/logstash/logstash:7.3.0 ADD modules.yml /usr/share/logstash/config/ RUN bin/logs… --- ## [Reading over multiple lines with grok](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786)
**Author:** [@Kapi](https://discuss.elastic.co/u/Kapi)\ **Replies:** 4\ **Last updated:** [August 7, 2019, 2:47pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786 "2019-08-07T14:47:47Z")
Hi I have this Log where I want to match the start of a thread of device with the corresponding spoolid. 30 Jul 2019 09:56:28 \ DEVICE0 START\_THREAD THREAD\_ID(B01234) 30 Jul 2019 09:56:28 \ DEVICE1 START\_THREAD… --- ## [Having multiple logstash instances (more than one server) for reading from same s3 bucket using s3 input plugin](https://discuss.elastic.co/t/having-multiple-logstash-instances-more-than-one-server-for-reading-from-same-s3-bucket-using-s3-input-plugin/194251)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [August 7, 2019, 2:01pm UTC](https://discuss.elastic.co/t/having-multiple-logstash-instances-more-than-one-server-for-reading-from-same-s3-bucket-using-s3-input-plugin/194251 "2019-08-07T14:01:57Z")
Would like to run logstash instances on two servers for High availability. Would like to know how to configure logstash to read from the same S3 bucket so that if one logstash server goes down other logstash instance can… --- ## [Message in json but not full](https://discuss.elastic.co/t/message-in-json-but-not-full/194013)
**Author:** [@Arnodl](https://discuss.elastic.co/u/Arnodl)\ **Replies:** 3\ **Last updated:** [August 7, 2019, 1:39pm UTC](https://discuss.elastic.co/t/message-in-json-but-not-full/194013 "2019-08-07T13:39:25Z")
Hello, I have problem with parsing the message: fun.host.pl: /storage/test / applications / central / logger\_data\_20190801: prm: { "@timestamp": "2019-08-01T10:24:07.32", "appid": "prm", "tags": \["SUCC:AUTHEN"\], "c… --- ## [Help with installing 3rd party plugin](https://discuss.elastic.co/t/help-with-installing-3rd-party-plugin/194246)
**Author:** [@bigdamhero](https://discuss.elastic.co/u/bigdamhero)\ **Replies:** 0\ **Last updated:** [August 7, 2019, 1:38pm UTC](https://discuss.elastic.co/t/help-with-installing-3rd-party-plugin/194246 "2019-08-07T13:38:11Z")
I am trying to install the logtash-okta-input plugin. I believe I have built the jruby gem correctly. I build it on a separate machine which matched the Java ver as the server where this will run. I am trying to insta… --- ## [\[HELP!\]prepare logs for ML](https://discuss.elastic.co/t/help-prepare-logs-for-ml/194232)
**Author:** [@c00ld](https://discuss.elastic.co/u/c00ld)\ **Replies:** 2\ **Last updated:** [August 7, 2019, 1:31pm UTC](https://discuss.elastic.co/t/help-prepare-logs-for-ml/194232 "2019-08-07T13:31:13Z")
Hello! I have logs from dns, i need prepare it to Machine Learning in url https://www.elastic.co/products/stack/machine-learning/recipes/dns-data-exfiltration-tunneling Have a field: "my.support.base.net" How to modi… --- ## [How to keep syslog header when saving to file?](https://discuss.elastic.co/t/how-to-keep-syslog-header-when-saving-to-file/194130)
**Author:** [@dickens88](https://discuss.elastic.co/u/dickens88)\ **Replies:** 4\ **Last updated:** [August 7, 2019, 11:52am UTC](https://discuss.elastic.co/t/how-to-keep-syslog-header-when-saving-to-file/194130 "2019-08-07T11:52:13Z")
Guys, when I use logstash to input syslog and keep the message to file, I found it seems logstash won't keep syslog header. my configuration looks like this: input{ syslog { port =\> 1522 } } output{ … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=495) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=497)