I am new to elasticsearch and want to index my website logs which are stored on HDFS for fast querying. I have a well structured pipeline which runs a script every 20 minutes to ingest the data into HDFS. I want to integrate elasticsearch with it, so that it also indexes these logs based on particular field(s) and thereby giving faster query results using spark SQL. So, my question is, can I index my data based on particular field(s) only? Also, my logs are saved in avro file format. Does es provides a way to directly index avro serialized data or do I need to convert it into some other format?
Thank you in advance.