# Can't pull from kafka topic that has date stamp **URL:** \ **Category:** Logstash\ **Created:** [April 8, 2016, 12:46am UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757 "2016-04-08T00:46:50Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [April 8, 2016, 12:46am UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/1 "2016-04-08T00:46:50Z")
Hello, I have this issue, when I am able to generate a kafka topic that has date stamp in the name, but when I try to read from it, it fails to. verified that topic created, and i can read from it. Snipped bellow kafkacat -C -b "broker-0..consul:7000" -t f5-logs-wc1-2016.04.07 -p 0 -e | more \<189\>Apr 6 17:57:01 blah-LB1 notice syslog-ng: Duplicate stats counter; counter='udp((null):514)' \<45\>Apr 6 17:57:01 Blah-LB1 notice syslog-ng[23098]: syslog-ng starting up; version='2.1.4' \<189\>Apr 6 17:57:01Blah-LB1 notice syslog-ng: syslog-ng startup succeeded This is my kafka output ``` kafka { bootstrap_servers => "broker-0.consul:7000" topic_id => "f5-logs-wc1%{+YYYY.MM.dd}" #topic_id => "f5-logs-wc1" codec => plain { format => "%{message}" charset => "CP1252" } ``` Here is my kafka input kafka { ``` zk_connect => "blah.com:2181" topic_id => "f5-logs-wc1%{+YYYY.MM.dd}" #topic_id => "f5-logs-wc1" codec => plain auto_offset_reset => "smallest" reset_beginning => true } ``` But if I switch topic\_id name in both input and output configuration to a simple string name (please see commented) then it can read. Anything I am doing wrong in my kafka input? Maybe that's not how I need to append the date stamp to prefix string? ---
**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\ **Post date:** [April 13, 2016, 3:58pm UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/2 "2016-04-13T15:58:56Z")
On the output, topic gets sprintf(@topic\_id) but the inputs does not. I wouldn't recommend that anyway for the input. For example what if the Kafka topic is behind, right at midnight, the input would cease reading from that topic that was behind. Luckily you can use white\_list and black\_list instead of topic\_id which allows the strings that are java compatible regular expression. So try white\_list =\> "f5-logs-wc1.\*". That should pick up new topics as they appear. [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-white\_list](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-white_list) ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [April 13, 2016, 8:37pm UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/3 "2016-04-13T20:37:39Z")
So correct me if I am wrong. Will the white\_list =\> "f5-logs-wc1.\*". allow me to read from multiple topics? In other words, the case scenario that you mentioned, where topic is behind, and there is a new topic generation. Will i be reading from two topic simultaneously? ---
**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\ **Post date:** [April 13, 2016, 8:59pm UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/4 "2016-04-13T20:59:40Z")
Yes, in fact if you create a topic every day you will eventually be reading a ton of topics. You should consider just having a logging topic with a [delete.retention.ms](http://delete.retention.ms) set to a value you are comfortable with. The default is 24 hours. 7 days is nice. ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [April 13, 2016, 9:13pm UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/5 "2016-04-13T21:13:34Z")
Got it. Also, at the moment I have one partition for the topic, but two consumers. Will both consumers be reading from the same partition and generating duplicate data, or one will be on idle? Also, what if I have more partitions than consumers, how will that work? ---
**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\ **Post date:** [April 13, 2016, 11:54pm UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/6 "2016-04-13T23:54:28Z")
When the number of threads exceeds the number of consumable partitions amongst a consumer group, excessive threads will idle. When # threads \< # partitions, all partitions in a consumer group will be distributed amongst threads. ---
**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\ **Post date:** [April 18, 2016, 5:38pm UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/7 "2016-04-18T17:38:35Z")
Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:01am UTC](https://discuss.elastic.co/t/cant-pull-from-kafka-topic-that-has-date-stamp/46757/8 "2017-07-06T05:01:51Z")