# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=628 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 629 --- ## [Reindex + enrich data](https://discuss.elastic.co/t/reindex-enrich-data/161762)
**Author:** [@andre22](https://discuss.elastic.co/u/andre22)\ **Replies:** 2\ **Last updated:** [December 20, 2018, 11:35pm UTC](https://discuss.elastic.co/t/reindex-enrich-data/161762 "2018-12-20T23:35:20Z")
Hi, I am trying to reindex sensor data in order to enrich it via lookup. My goal is to add multiple fields, as shown in the csv plugin config below. While the reindexing itself starts, I see no difference in the new ind… --- ## [Array for condition](https://discuss.elastic.co/t/array-for-condition/161054)
**Author:** [@andre22](https://discuss.elastic.co/u/andre22)\ **Replies:** 2\ **Last updated:** [December 20, 2018, 8:21pm UTC](https://discuss.elastic.co/t/array-for-condition/161054 "2018-12-20T20:21:05Z")
Hi, I want to drop a message if the "method" field has one of several contents. I've been searching the docs and forums for a while now but wont find the right syntax. I have this, which doesn't work: if \[ "validation"… --- ## [Seperate multiple events in input into seperate documents in elasticsearch index](https://discuss.elastic.co/t/seperate-multiple-events-in-input-into-seperate-documents-in-elasticsearch-index/161753)
**Author:** [@dshetty](https://discuss.elastic.co/u/dshetty)\ **Replies:** 0\ **Last updated:** [December 20, 2018, 7:47pm UTC](https://discuss.elastic.co/t/seperate-multiple-events-in-input-into-seperate-documents-in-elasticsearch-index/161753 "2018-12-20T19:47:54Z")
INPUT in logstash : { "Teacher": { "Name": "Mary", "age": 20, }, "Student": \[ { "Name": "Tim", "age"12 }, { "Name": "Eric", "age":13 } \] } Need to filter this input using logstash to send three seperate … --- ## [Wrap logstash information into another object](https://discuss.elastic.co/t/wrap-logstash-information-into-another-object/161749)
**Author:** [@luksi1](https://discuss.elastic.co/u/luksi1)\ **Replies:** 0\ **Last updated:** [December 20, 2018, 7:22pm UTC](https://discuss.elastic.co/t/wrap-logstash-information-into-another-object/161749 "2018-12-20T19:22:44Z")
We're performing Elasticsearch updates with scripts and upserts and we want to wrap everything in the Logstash filter into another object. Take for instance: grok { patterns\_dir =\> "/etc/logstash/grok\_patterns" bre… --- ## [Unable to get the new field in Kibana, defined in logstash!](https://discuss.elastic.co/t/unable-to-get-the-new-field-in-kibana-defined-in-logstash/161518)
**Author:** [@soumith](https://discuss.elastic.co/u/soumith)\ **Replies:** 0\ **Last updated:** [December 19, 2018, 1:00pm UTC](https://discuss.elastic.co/t/unable-to-get-the-new-field-in-kibana-defined-in-logstash/161518 "2018-12-19T13:00:52Z")
Hello, I am new to ELK stack and working on basics like getting a log through filebeat and view that in the kibana dashboard. I want to get an extra field in kibana from the part of existing message field. To do so, I ma… --- ## [How to create unique id in logstash/elastic search for apache logs in a distributed server environment](https://discuss.elastic.co/t/how-to-create-unique-id-in-logstash-elastic-search-for-apache-logs-in-a-distributed-server-environment/161241)
**Author:** [@elk\_dni\_1](https://discuss.elastic.co/u/elk_dni_1)\ **Replies:** 9\ **Last updated:** [December 20, 2018, 5:56pm UTC](https://discuss.elastic.co/t/how-to-create-unique-id-in-logstash-elastic-search-for-apache-logs-in-a-distributed-server-environment/161241 "2018-12-20T17:56:29Z")
How to create unique id in logstash/elastic search for apache logs in a distributed server environment so that when you reupload apache logs, logstash/es will update them instead of creating duplicate records. --- ## [Update CSV file changes to ELK](https://discuss.elastic.co/t/update-csv-file-changes-to-elk/161219)
**Author:** [@kb2295](https://discuss.elastic.co/u/kb2295)\ **Replies:** 8\ **Last updated:** [December 20, 2018, 5:46pm UTC](https://discuss.elastic.co/t/update-csv-file-changes-to-elk/161219 "2018-12-20T17:46:33Z")
Hello, If I have a CSV file that adds additional rows every 24 hours, is there a way that I can have ELK upload only the additional rows? I need a system in which I can keep changing the CSV file and not have ELK upload… --- ## [How to get logstash pipeline UI within Kibana](https://discuss.elastic.co/t/how-to-get-logstash-pipeline-ui-within-kibana/161569)
**Author:** [@ciphee](https://discuss.elastic.co/u/ciphee)\ **Replies:** 8\ **Last updated:** [December 20, 2018, 5:40pm UTC](https://discuss.elastic.co/t/how-to-get-logstash-pipeline-ui-within-kibana/161569 "2018-12-20T17:40:59Z")
Hello, I recently installed logstash and connected it to my elasticsearch. I have not been able to get the logstash and pipeline UI option to show up from within Kibana, is there a setting I need to enable in order to g… --- ## [Best approach to parse this line into tokens](https://discuss.elastic.co/t/best-approach-to-parse-this-line-into-tokens/161715)
**Author:** [@swright-eti](https://discuss.elastic.co/u/swright-eti)\ **Replies:** 1\ **Last updated:** [December 20, 2018, 4:39pm UTC](https://discuss.elastic.co/t/best-approach-to-parse-this-line-into-tokens/161715 "2018-12-20T16:39:19Z")
I am trying to figure out the best approach to parse this line. I want a couple of pieces of information parsed into tokens, and other stripped away. This was captured from the Logstash log when it was sent by Filebeat… --- ## [Setup Logstash so that default config file is read when starting as a service](https://discuss.elastic.co/t/setup-logstash-so-that-default-config-file-is-read-when-starting-as-a-service/161728)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 0\ **Last updated:** [December 20, 2018, 4:27pm UTC](https://discuss.elastic.co/t/setup-logstash-so-that-default-config-file-is-read-when-starting-as-a-service/161728 "2018-12-20T16:27:53Z")
I was having trouble getting Logstash to run correctly when I started it as a service. The logstash user was created correctly but when Logstash was setup as a service no data was reaching our ES cluster and thus not sh… --- ## [Logstash to parse only part of apache log file](https://discuss.elastic.co/t/logstash-to-parse-only-part-of-apache-log-file/161682)
**Author:** [@elk\_dni\_1](https://discuss.elastic.co/u/elk_dni_1)\ **Replies:** 3\ **Last updated:** [December 20, 2018, 1:38pm UTC](https://discuss.elastic.co/t/logstash-to-parse-only-part-of-apache-log-file/161682 "2018-12-20T13:38:33Z")
Hi Guys, I am re-indexing few ones. They are monthly. I deleted the a month index say Oct 2018 one. Oct Index contains data : Oct 1st UTC - Oct 31 UTC But my apache logs are in PST and are rotated daily. So I need … --- ## [How to Use dynamic URL in http\_poller](https://discuss.elastic.co/t/how-to-use-dynamic-url-in-http-poller/161658)
**Author:** [@kausik](https://discuss.elastic.co/u/kausik)\ **Replies:** 0\ **Last updated:** [December 20, 2018, 10:37am UTC](https://discuss.elastic.co/t/how-to-use-dynamic-url-in-http-poller/161658 "2018-12-20T10:37:26Z")
I am using http\_poller input plugin of Logstash to read logs stored in an URL endpoint. But the URL is not constant. It will change every time. I can restart the Logstash agent every time whenever new URL comes but it no… --- ## [Different files are entered into logstash . kibana fields show the same](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243)
**Author:** [@CCH0124](https://discuss.elastic.co/u/CCH0124)\ **Replies:** 9\ **Last updated:** [December 20, 2018, 10:13am UTC](https://discuss.elastic.co/t/different-files-are-entered-into-logstash-kibana-fields-show-the-same/161243 "2018-12-20T10:13:40Z")
Use filebeat to pass the json file to logstash. Read the CSV file using logstash. Both are different indexes. But the two indexes on kibana are reading the same csv file. How to solve it ? json filebeat filebeat.conf… --- ## [Grokparsefailure IIS Logs](https://discuss.elastic.co/t/grokparsefailure-iis-logs/161312)
**Author:** [@nu11ahnung](https://discuss.elastic.co/u/nu11ahnung)\ **Replies:** 4\ **Last updated:** [December 20, 2018, 9:52am UTC](https://discuss.elastic.co/t/grokparsefailure-iis-logs/161312 "2018-12-20T09:52:58Z")
I want to filter IIS Logs. I chose the default logging options + bytes sent. I configured my grok filter like this filter { if "iis" in \[tags\] { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:log\_timestam… --- ## [Logstash doesn't index all files in a folder](https://discuss.elastic.co/t/logstash-doesnt-index-all-files-in-a-folder/161643)
**Author:** [@Satyabrata\_Dash](https://discuss.elastic.co/u/Satyabrata_Dash)\ **Replies:** 0\ **Last updated:** [December 20, 2018, 9:01am UTC](https://discuss.elastic.co/t/logstash-doesnt-index-all-files-in-a-folder/161643 "2018-12-20T09:01:04Z")
Hello All, I am new to Elastic Stack and have been working on it for last 2 weeks. I am facing an issue with Logstash where I am reading log files from a specific folder. But when I tried to find some data on Kibana, I o… --- ## [Aggregation does not work](https://discuss.elastic.co/t/aggregation-does-not-work/161613)
**Author:** [@jmkim](https://discuss.elastic.co/u/jmkim)\ **Replies:** 1\ **Last updated:** [December 20, 2018, 6:35am UTC](https://discuss.elastic.co/t/aggregation-does-not-work/161613 "2018-12-20T06:35:12Z")
Hi, everyone! I am trying to figure out the log count by device. There is an error No data accumulates in the Elastic Search. aggregate { task\_id =\> "%{device}" code =\> " map\['log\_cnt'\] ||= 0 map\['log\_cnt… --- ## [About logstash.pipeline ERROR](https://discuss.elastic.co/t/about-logstash-pipeline-error/161453)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 0\ **Last updated:** [December 19, 2018, 6:32am UTC](https://discuss.elastic.co/t/about-logstash-pipeline-error/161453 "2018-12-19T06:32:23Z")
Has anyone seen this error? \[2018-12-19T15:25:46,073\]\[ERROR\]\[logstash.pipeline \] ash.vendor.bundle.jruby.$2\_dot\_3\_dot\_0.gems.sequel\_minus\_5\_dot\_5\_dot\_0.lib.sequel.dataset.actions.invokeOther2:each(usr/share/logst… --- ## [Mask the IP address using custom grok pattern](https://discuss.elastic.co/t/mask-the-ip-address-using-custom-grok-pattern/161293)
**Author:** [@Manjushree\_Gokhale](https://discuss.elastic.co/u/Manjushree_Gokhale)\ **Replies:** 2\ **Last updated:** [December 20, 2018, 5:53am UTC](https://discuss.elastic.co/t/mask-the-ip-address-using-custom-grok-pattern/161293 "2018-12-20T05:53:10Z")
Hi, I am new to Logstash and grok and want to mask/ change first octets of ip addresses in my logs for security purpose. but unable to create a grok pattern which will work for the situation. I am getting IP from logs… --- ## [How to remove duplicate message from json](https://discuss.elastic.co/t/how-to-remove-duplicate-message-from-json/161402)
**Author:** [@Fabio\_Luiz\_Portioli](https://discuss.elastic.co/u/Fabio_Luiz_Portioli)\ **Replies:** 1\ **Last updated:** [December 20, 2018, 4:55am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-message-from-json/161402 "2018-12-20T04:55:09Z")
Hi, I'm new working with grok and would like to know how could I remove the duplicate value? Ex: I'm sending a JSON with value: \[ { "source": "ELK Dev", "type": "Software", "node": "da5361e4-7e06-4e9f-a5ba-1e30e6… --- ## [Convert Epoch time to "yyyy/MM/dd HH:mm:ss.SSS"](https://discuss.elastic.co/t/convert-epoch-time-to-yyyy-mm-dd-hhss-sss/160912)
**Author:** [@emailshak](https://discuss.elastic.co/u/emailshak)\ **Replies:** 2\ **Last updated:** [December 20, 2018, 3:25am UTC](https://discuss.elastic.co/t/convert-epoch-time-to-yyyy-mm-dd-hhss-sss/160912 "2018-12-20T03:25:03Z")
I am receiving epoch time as input and would like to persist this in the following format "yyyy/MM/dd HH:mm:ss.SSS" in Elasticsearch. I am performing the following at present in logstash date { match =\> \["mydate", "UN… --- ## [Including new Gems to work with logstash-filter-ruby](https://discuss.elastic.co/t/including-new-gems-to-work-with-logstash-filter-ruby/160447)
**Author:** [@Smitto](https://discuss.elastic.co/u/Smitto)\ **Replies:** 1\ **Last updated:** [December 19, 2018, 10:04pm UTC](https://discuss.elastic.co/t/including-new-gems-to-work-with-logstash-filter-ruby/160447 "2018-12-19T22:04:04Z")
I'm working on applying some custom filters using the logstash-filter-ruby plugin that make use of the Timezone gem at https://github.com/panthomakos/timezone. When simply adding gem 'timezone', '~\> 1.0' to my logstash… --- ## [Full Scan in BD with jdbc](https://discuss.elastic.co/t/full-scan-in-bd-with-jdbc/161594)
**Author:** [@ISABEL\_CRISTINA\_L](https://discuss.elastic.co/u/ISABEL_CRISTINA_L)\ **Replies:** 0\ **Last updated:** [December 19, 2018, 9:00pm UTC](https://discuss.elastic.co/t/full-scan-in-bd-with-jdbc/161594 "2018-12-19T21:00:41Z")
Hello. I require your support with the following: Through a database oracle from logstash, however it has not been possible to define a filter so that I do not make a complete scan to the database, you can indicate in … --- ## [Unable to get the new field in Kibana!](https://discuss.elastic.co/t/unable-to-get-the-new-field-in-kibana/161531)
**Author:** [@soumith](https://discuss.elastic.co/u/soumith)\ **Replies:** 0\ **Last updated:** [December 19, 2018, 1:35pm UTC](https://discuss.elastic.co/t/unable-to-get-the-new-field-in-kibana/161531 "2018-12-19T13:35:24Z")
Hello, I am new to ELK stack and working on basics like getting a log through filebeat and view that in the kibana dashboard. I want to get an extra field in kibana from the part of existing message field. To do so, I ma… --- ## [Unable to input files in multiple directory levels](https://discuss.elastic.co/t/unable-to-input-files-in-multiple-directory-levels/161590)
**Author:** [@BrianZimmel](https://discuss.elastic.co/u/BrianZimmel)\ **Replies:** 0\ **Last updated:** [December 19, 2018, 8:31pm UTC](https://discuss.elastic.co/t/unable-to-input-files-in-multiple-directory-levels/161590 "2018-12-19T20:31:51Z")
Hello. I'm having trouble configuring the logstash file input plugin with a path pattern to perform a recursive search across multiple levels in a particular directory. Every time I run logstash it never picks up any of … --- ## [Run query and recreate index every run](https://discuss.elastic.co/t/run-query-and-recreate-index-every-run/154679)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 20\ **Last updated:** [December 19, 2018, 8:06pm UTC](https://discuss.elastic.co/t/run-query-and-recreate-index-every-run/154679 "2018-12-19T20:06:46Z")
I have logstash config file and would like to run this everytime and recreate index everytime. that way it mimic live data basically when running job's status is completed it should drop from query and I don't want to s… --- ## [Azure & Office 365, Oh My](https://discuss.elastic.co/t/azure-office-365-oh-my/161574)
**Author:** [@randomuserid](https://discuss.elastic.co/u/randomuserid)\ **Replies:** 1\ **Last updated:** [December 19, 2018, 7:45pm UTC](https://discuss.elastic.co/t/azure-office-365-oh-my/161574 "2018-12-19T19:45:55Z")
Is there a "good" way to ingest logs from Azure, Azure AD and / or Office 365 via logstash? I have just been handed these requirements.. --- ## [Headers in CSV File](https://discuss.elastic.co/t/headers-in-csv-file/161420)
**Author:** [@kb2295](https://discuss.elastic.co/u/kb2295)\ **Replies:** 6\ **Last updated:** [December 19, 2018, 7:04pm UTC](https://discuss.elastic.co/t/headers-in-csv-file/161420 "2018-12-19T19:04:26Z")
Hello. How can I get ELK to ignore the first line in my CSV file (the headers). I keep getting the error: \[2018-12-18T16:43:18,640\]\[WARN \]\[logstash.outputs.elasticsearch\] Failed action. {:status=\>400, :action=\>\["index… --- ## [What is the difference between a logstash pipeline and a logstash config file in /config.d?](https://discuss.elastic.co/t/what-is-the-difference-between-a-logstash-pipeline-and-a-logstash-config-file-in-config-d/161568)
**Author:** [@ciphee](https://discuss.elastic.co/u/ciphee)\ **Replies:** 0\ **Last updated:** [December 19, 2018, 5:33pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-a-logstash-pipeline-and-a-logstash-config-file-in-config-d/161568 "2018-12-19T17:33:47Z")
Hi there, I was wondering what the difference was between the logstash conf files that you put in /conf.d and pipelines that you can configure within the kibana UI. Are they the same thing that can be configured in 2 pl… --- ## [Allocation of cpu cores for logstash instances in a 64 vCPU cores server](https://discuss.elastic.co/t/allocation-of-cpu-cores-for-logstash-instances-in-a-64-vcpu-cores-server/161041)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 3\ **Last updated:** [December 19, 2018, 3:55pm UTC](https://discuss.elastic.co/t/allocation-of-cpu-cores-for-logstash-instances-in-a-64-vcpu-cores-server/161041 "2018-12-19T15:55:14Z")
Would like to know the proper worker thread configuration for a logstash server having 64 vCPU cores and 256GB ram. There are 21 logstash instances running with different configuration files. In the documentation I see t… --- ## [Change regexp filter to cidr filter](https://discuss.elastic.co/t/change-regexp-filter-to-cidr-filter/161538)
**Author:** [@dfesenko](https://discuss.elastic.co/u/dfesenko)\ **Replies:** 0\ **Last updated:** [December 19, 2018, 2:11pm UTC](https://discuss.elastic.co/t/change-regexp-filter-to-cidr-filter/161538 "2018-12-19T14:11:28Z")
I have a configuration file for filtering documents from the specific IP-addresses. Also, it adds or changes some fields in the documents. This file uses regular expressions and mutate plugin. For input it uses netflow p… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=627) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=629)