# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=611 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 612 --- ## [Logstash Sending Old Data to ES](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974)
**Author:** [@Aralex](https://discuss.elastic.co/u/Aralex)\ **Replies:** 11\ **Last updated:** [January 28, 2019, 2:26pm UTC](https://discuss.elastic.co/t/logstash-sending-old-data-to-es/165974 "2019-01-28T14:26:38Z")
Hi, my pipeline consists of: filebeat -\> logstash -\> ES -\> kibana. I use it to read CSV files in a folder and index/mirror any changes or updates in real time on kibana. Problem 1: My test CSV file has 16 entries bus … --- ## [How to convert lat/lon data coming from MongoDB to geo\_point type?](https://discuss.elastic.co/t/how-to-convert-lat-lon-data-coming-from-mongodb-to-geo-point-type/165980)
**Author:** [@topgun743](https://discuss.elastic.co/u/topgun743)\ **Replies:** 1\ **Last updated:** [January 28, 2019, 2:02pm UTC](https://discuss.elastic.co/t/how-to-convert-lat-lon-data-coming-from-mongodb-to-geo-point-type/165980 "2019-01-28T14:02:26Z")
I am trying to index a collection to Elasticsearch from MongoDB. In MongoDB a document contains a Position field which has in turn lat and lon (float) fields that describe the location of any vehicle. I want these poin… --- ## [Accessing files inside a folder](https://discuss.elastic.co/t/accessing-files-inside-a-folder/165976)
**Author:** [@vinu89](https://discuss.elastic.co/u/vinu89)\ **Replies:** 1\ **Last updated:** [January 28, 2019, 1:57pm UTC](https://discuss.elastic.co/t/accessing-files-inside-a-folder/165976 "2019-01-28T13:57:41Z")
Hi, I have to access the files inside a folder . The folder structure is like this MainFolder -\>folder1 -\> 1.txt -\>folder2 -\>2.txt -\>1.txt -\>2.txt when i write the logstash parser like this file { path =\> "/root… --- ## [Drupal watchdog logs are combining with phperror logs and local apache error logs in elasticsearch api](https://discuss.elastic.co/t/drupal-watchdog-logs-are-combining-with-phperror-logs-and-local-apache-error-logs-in-elasticsearch-api/166002)
**Author:** [@Bdr](https://discuss.elastic.co/u/Bdr)\ **Replies:** 0\ **Last updated:** [January 28, 2019, 1:36pm UTC](https://discuss.elastic.co/t/drupal-watchdog-logs-are-combining-with-phperror-logs-and-local-apache-error-logs-in-elasticsearch-api/166002 "2019-01-28T13:36:59Z")
Starting while importing Drupal watchdog log file, phperrorlog file and local apache log into logstash it's generating elasticsearch api separately but after some hours when i called phperrorlog api then drupal watchdog… --- ## [RuntimeError 500 error with Logstash netflow module](https://discuss.elastic.co/t/runtimeerror-500-error-with-logstash-netflow-module/165972)
**Author:** [@Ayushi](https://discuss.elastic.co/u/Ayushi)\ **Replies:** 0\ **Last updated:** [January 28, 2019, 9:46am UTC](https://discuss.elastic.co/t/runtimeerror-500-error-with-logstash-netflow-module/165972 "2019-01-28T09:46:50Z")
Hi folks, I am new to ELK stack and trying to setup netflow module in logstash: command line command : bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055 -M netflow.var.elasticsearch.hosts="local… --- ## [Can't import .csv through logstash](https://discuss.elastic.co/t/cant-import-csv-through-logstash/165768)
**Author:** [@Reena\_Roseline](https://discuss.elastic.co/u/Reena_Roseline)\ **Replies:** 1\ **Last updated:** [January 28, 2019, 7:59am UTC](https://discuss.elastic.co/t/cant-import-csv-through-logstash/165768 "2019-01-28T07:59:10Z")
I am using windows 10, when I run logstash -f , I could see logstash agent started message but cant find any data in kibana? Any help would be appreciated. input { file { path =\> "D:/ES/SalesJan2009.csv" start\_posit… --- ## [How hide password in /etc/sysconfig/logstash?](https://discuss.elastic.co/t/how-hide-password-in-etc-sysconfig-logstash/165539)
**Author:** [@pr0b3l](https://discuss.elastic.co/u/pr0b3l)\ **Replies:** 0\ **Last updated:** [January 24, 2019, 7:26am UTC](https://discuss.elastic.co/t/how-hide-password-in-etc-sysconfig-logstash/165539 "2019-01-24T07:26:45Z")
I use logstash-keystore. How can i hide password in /etc/sysconfig/logstash? Why logstash-keystore is different on Kibana-keystore or Elasticsearch-keystore? --- ## [Append metadata to every event in XML file being ingested to Elasticsearch](https://discuss.elastic.co/t/append-metadata-to-every-event-in-xml-file-being-ingested-to-elasticsearch/165914)
**Author:** [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\ **Replies:** 2\ **Last updated:** [January 28, 2019, 7:00am UTC](https://discuss.elastic.co/t/append-metadata-to-every-event-in-xml-file-being-ingested-to-elasticsearch/165914 "2019-01-28T07:00:23Z")
Hey, I have a xml file as follows and I want to append the metadata which are the first few lines of the document to every event in the xml file in subsequent lines. The xml file is attached here. The expected output fo… --- ## [Is it possible to send data more then 4Mb of data through udp to logstash?](https://discuss.elastic.co/t/is-it-possible-to-send-data-more-then-4mb-of-data-through-udp-to-logstash/165932)
**Author:** [@naga\_kunchala](https://discuss.elastic.co/u/naga_kunchala)\ **Replies:** 0\ **Last updated:** [January 28, 2019, 4:49am UTC](https://discuss.elastic.co/t/is-it-possible-to-send-data-more-then-4mb-of-data-through-udp-to-logstash/165932 "2019-01-28T04:49:14Z")
is it possible to send data more then 4Mb of data through udp to logstash? --- ## [How to define a field when pushing the same document to multiple indexes?](https://discuss.elastic.co/t/how-to-define-a-field-when-pushing-the-same-document-to-multiple-indexes/165706)
**Author:** [@jmkim](https://discuss.elastic.co/u/jmkim)\ **Replies:** 2\ **Last updated:** [January 28, 2019, 2:41am UTC](https://discuss.elastic.co/t/how-to-define-a-field-when-pushing-the-same-document-to-multiple-indexes/165706 "2019-01-28T02:41:20Z")
Hello, I am collecting logs and trying to output the logs to multiple indexes. Can I define a field for each index? My Logstash output configuration is as follows. input: ip, name, url, msg output { if "agg"… --- ## [Stuck at Successfully started Logstash API endpoint {:port=\>9600}](https://discuss.elastic.co/t/stuck-at-successfully-started-logstash-api-endpoint-port-9600/165855)
**Author:** [@Be1](https://discuss.elastic.co/u/Be1)\ **Replies:** 3\ **Last updated:** [January 28, 2019, 2:08am UTC](https://discuss.elastic.co/t/stuck-at-successfully-started-logstash-api-endpoint-port-9600/165855 "2019-01-28T02:08:32Z")
Please help me with this error.this is my conf file input { file { path =\> "C:/Users/Administrator/Documents/roi1.csv" start\_position =\> "beginning" sincedb\_path =\> "NUL" } } filter { csv { separator =\> "," co… --- ## [Json parse failure on curl post data](https://discuss.elastic.co/t/json-parse-failure-on-curl-post-data/165884)
**Author:** [@hoff](https://discuss.elastic.co/u/hoff)\ **Replies:** 3\ **Last updated:** [January 27, 2019, 9:03pm UTC](https://discuss.elastic.co/t/json-parse-failure-on-curl-post-data/165884 "2019-01-27T21:03:53Z")
I have a script that sends json via curl. My question is am I able to display data that is not parsed by json codec? \[2019-01-25T11:10:01,403\]\[WARN \]\[logstash.filters.json \] Parsed JSON object/hash requires a tar… --- ## [Logstash-filter-elasticsearch URI parsing error when ssl set to true](https://discuss.elastic.co/t/logstash-filter-elasticsearch-uri-parsing-error-when-ssl-set-to-true/163814)
**Author:** [@zhaider](https://discuss.elastic.co/u/zhaider)\ **Replies:** 4\ **Last updated:** [January 27, 2019, 6:07pm UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-uri-parsing-error-when-ssl-set-to-true/163814 "2019-01-27T18:07:17Z")
This is my plugin config: elasticsearch { hosts =\> \[ "${LS\_ES\_HOST1}" \] user =\> "${LS\_ES\_USER}" password =\> "${LS\_ES\_PASSWORD}" ssl =\> true ca\_file =\> "${LS\_ES\_CACER… --- ## [Logstash File Input Plugin Read Mode Not Loading Data](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869)
**Author:** [@Kenneth\_M\_Kolano](https://discuss.elastic.co/u/Kenneth_M_Kolano)\ **Replies:** 3\ **Last updated:** [January 27, 2019, 3:14am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-read-mode-not-loading-data/165869 "2019-01-27T03:14:10Z")
I'm having trouble getting Logstash to load data to Elasticsearch when using read mode. input { file { mode =\> "read" path =\> \["/var/log/PASS/analyses/\*"\] sincedb\_path =\> \["/var/lib/logstash/file-analysis\_sincedb… --- ## [\[Solved\] Can't get logstash to catch cowrie filebeat](https://discuss.elastic.co/t/solved-cant-get-logstash-to-catch-cowrie-filebeat/165866)
**Author:** [@jonsag](https://discuss.elastic.co/u/jonsag)\ **Replies:** 3\ **Last updated:** [January 26, 2019, 10:07pm UTC](https://discuss.elastic.co/t/solved-cant-get-logstash-to-catch-cowrie-filebeat/165866 "2019-01-26T22:07:07Z")
Can't get it working with cowrie-\>filebeat-\>logstash-\>elasticsearch-\>kibana My cowrie logs to file, which is read by filebeat. Filebeat version $ /usr/share/filebeat/bin/filebeat version filebeat version 6.1.1 (arm), … --- ## [After grok filter definition in logstash, index cannot be created](https://discuss.elastic.co/t/after-grok-filter-definition-in-logstash-index-cannot-be-created/165846)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [January 26, 2019, 1:18pm UTC](https://discuss.elastic.co/t/after-grok-filter-definition-in-logstash-index-cannot-be-created/165846 "2019-01-26T13:18:56Z")
hi all, I am using logstash to ship log files into elasticsearch. first my configuration file of logstash is as following that successfully ship the files int elasticsearch and I can see the logs in kibana: input { fil… --- ## [Messing with "Umlaute" and how to manage Ö = ö Ü = ü Ä = ä á & ó = á é = é](https://discuss.elastic.co/t/messing-with-umlaute-and-how-to-manage-o-a-u-a-a-a-a-o-a-e-a/165790)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 2\ **Last updated:** [January 26, 2019, 1:14pm UTC](https://discuss.elastic.co/t/messing-with-umlaute-and-how-to-manage-o-a-u-a-a-a-a-o-a-e-a/165790 "2019-01-26T13:14:24Z")
Hi there, I have some issues with Ö Ü Ä ö ü ä á ó é etc. Codec plain is installed! log = csv by log4j2 standard charset ISO-8859-1 My Config: Filebeat: - type: log encoding: ISO-8859-1 Logstash: input { beats… --- ## [Guidance for parsing log file](https://discuss.elastic.co/t/guidance-for-parsing-log-file/165604)
**Author:** [@Hamed\_khosravi](https://discuss.elastic.co/u/Hamed_khosravi)\ **Replies:** 2\ **Last updated:** [January 26, 2019, 8:19am UTC](https://discuss.elastic.co/t/guidance-for-parsing-log-file/165604 "2019-01-26T08:19:44Z")
hi dears I trying to pars some log files for WebSphere MQ and I'm not familiar with this logs, some lines of those logs different with others lines. for example : """""""""""""""""""""""""""""""""""""""""""""""""""""""… --- ## [Logstash filebeat pipeline to ingest json logs - parse array](https://discuss.elastic.co/t/logstash-filebeat-pipeline-to-ingest-json-logs-parse-array/165669)
**Author:** [@fefo69](https://discuss.elastic.co/u/fefo69)\ **Replies:** 2\ **Last updated:** [January 25, 2019, 8:53pm UTC](https://discuss.elastic.co/t/logstash-filebeat-pipeline-to-ingest-json-logs-parse-array/165669 "2019-01-25T20:53:56Z")
Hi, I'm trying to ingest json log files, running filebeat and logstash, in log file I have transactions with the following format. I need to parse the logs and send to elastic a few set of fields of each transaction, b… --- ## [How to get Username from GUID with Winlogbeats sending to Logstash](https://discuss.elastic.co/t/how-to-get-username-from-guid-with-winlogbeats-sending-to-logstash/165813)
**Author:** [@snovak](https://discuss.elastic.co/u/snovak)\ **Replies:** 4\ **Last updated:** [January 25, 2019, 8:52pm UTC](https://discuss.elastic.co/t/how-to-get-username-from-guid-with-winlogbeats-sending-to-logstash/165813 "2019-01-25T20:52:44Z")
We recently setup Winlogbeats to send events to Logstash which then goes into Elastic Search and can be queried and visualized with Kibana. All is good there except many of the security related events that we receive ha… --- ## [How to create a custom named time based indexing using logstash?](https://discuss.elastic.co/t/how-to-create-a-custom-named-time-based-indexing-using-logstash/165751)
**Author:** [@Arpit\_Khanna](https://discuss.elastic.co/u/Arpit_Khanna)\ **Replies:** 1\ **Last updated:** [January 25, 2019, 5:25pm UTC](https://discuss.elastic.co/t/how-to-create-a-custom-named-time-based-indexing-using-logstash/165751 "2019-01-25T17:25:59Z")
Currently i am trying to create a time based indexing in an Elastic search using Logstash. I am trying to create a time based indexing. I have come across and successfully tried the following output mechanism in Logstas… --- ## [Unexpected problem when running logstash with flag](https://discuss.elastic.co/t/unexpected-problem-when-running-logstash-with-flag/165792)
**Author:** [@tszyro](https://discuss.elastic.co/u/tszyro)\ **Replies:** 1\ **Last updated:** [January 25, 2019, 3:24pm UTC](https://discuss.elastic.co/t/unexpected-problem-when-running-logstash-with-flag/165792 "2019-01-25T15:24:29Z")
I am trying to implement new filter in Logstash. To do so I try to run production logstash with local configuration. I created all necessary files in the folder from which I run. When specify flag -f to make sure the cor… --- ## [Date not working](https://discuss.elastic.co/t/date-not-working/165566)
**Author:** [@huangliusong](https://discuss.elastic.co/u/huangliusong)\ **Replies:** 3\ **Last updated:** [January 25, 2019, 2:12am UTC](https://discuss.elastic.co/t/date-not-working/165566 "2019-01-25T02:12:29Z")
my conf input { beats { port =\> "5044" } } filter { dissect { mapping =\> { "message" =\> "%{uat\_log\_date} %{uat\_log\_time} %{uat\_log\_ip} %{uat\_log\_traid} %{uat\_log\_action} %{uat\_log\_flag… --- ## [Logstash's Elasticsearch output script context is getting an Integer field as a String](https://discuss.elastic.co/t/logstashs-elasticsearch-output-script-context-is-getting-an-integer-field-as-a-string/165779)
**Author:** [@ffknob](https://discuss.elastic.co/u/ffknob)\ **Replies:** 1\ **Last updated:** [January 25, 2019, 1:54pm UTC](https://discuss.elastic.co/t/logstashs-elasticsearch-output-script-context-is-getting-an-integer-field-as-a-string/165779 "2019-01-25T13:54:26Z")
Hello, I'm confused by why the Elasticsearch output is getting an Integer field as String. The effect I'm getting is that when I tryh to increment this field, it appends the incremental value (1) to the end of the field… --- ## [Best way to set a field value to be a (json) object?](https://discuss.elastic.co/t/best-way-to-set-a-field-value-to-be-a-json-object/165654)
**Author:** [@jdmcalee](https://discuss.elastic.co/u/jdmcalee)\ **Replies:** 2\ **Last updated:** [January 25, 2019, 1:49pm UTC](https://discuss.elastic.co/t/best-way-to-set-a-field-value-to-be-a-json-object/165654 "2019-01-25T13:49:05Z")
I'm trying to configure a logstash filter to add a field that contains one of more objects. In other words, I want the document json for the field to end up looking something like this: {"links":\[{"label":"google","url"… --- ## [Parse out #012 from data](https://discuss.elastic.co/t/parse-out-012-from-data/165782)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 1\ **Last updated:** [January 25, 2019, 1:31pm UTC](https://discuss.elastic.co/t/parse-out-012-from-data/165782 "2019-01-25T13:31:19Z")
I have some data I'm sending via syslog to LS. It is json when it is collected: { "updated\_date": null, "status": null, "name": null, "dnssec": null, "city": null, "expiration\_date": null, "… --- ## [Forgett UNC for Filebeat - Admin pls delete this post -](https://discuss.elastic.co/t/forgett-unc-for-filebeat-admin-pls-delete-this-post/165372)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 0\ **Last updated:** [January 23, 2019, 10:04am UTC](https://discuss.elastic.co/t/forgett-unc-for-filebeat-admin-pls-delete-this-post/165372 "2019-01-23T10:04:37Z")
pls delete ! --- ## [Input painfully slow while fetching messages from activemq topic](https://discuss.elastic.co/t/input-painfully-slow-while-fetching-messages-from-activemq-topic/165765)
**Author:** [@BlueSpirit](https://discuss.elastic.co/u/BlueSpirit)\ **Replies:** 0\ **Last updated:** [January 25, 2019, 12:00pm UTC](https://discuss.elastic.co/t/input-painfully-slow-while-fetching-messages-from-activemq-topic/165765 "2019-01-25T12:00:08Z")
I have configured JMS input in logstash to subscribe to JMS topic messages and push messages to elastic search. Current config: input { jms { id =\> "my\_first\_jms" yaml\_file =\> "D:\\softwares\\logstash-6.4.0\\conf… --- ## [Logstash error, jar-dependencies gemfile](https://discuss.elastic.co/t/logstash-error-jar-dependencies-gemfile/165762)
**Author:** [@mortueta](https://discuss.elastic.co/u/mortueta)\ **Replies:** 0\ **Last updated:** [January 25, 2019, 11:49am UTC](https://discuss.elastic.co/t/logstash-error-jar-dependencies-gemfile/165762 "2019-01-25T11:49:24Z")
Hi, After ELK stack upgrade (from 6.5.3 to 6.5.4) I have encountered with some problems with logstash. I have followed the documentation: Upgrading the Elastic Stack | Installation and Upgrade Guide \[6.5\] | Elastic The… --- ## [Postfix parsing -](https://discuss.elastic.co/t/postfix-parsing/165193)
**Author:** [@fribse](https://discuss.elastic.co/u/fribse)\ **Replies:** 7\ **Last updated:** [January 25, 2019, 10:47am UTC](https://discuss.elastic.co/t/postfix-parsing/165193 "2019-01-25T10:47:56Z")
I found Tin Le's setup on Linked In, and I think it looked really nice, so I wanted to add those logfiles to our ELK as well. I've created the patterns file, and modified my logstash.con, I'll be getting the logfiles wi… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=610) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=612)