# Bash to import data - what could I be doing better

**URL:** <https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872>\
**Category:** Elasticsearch\
**Created:** [June 17, 2015, 7:54pm UTC](https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872 "2015-06-17T19:54:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bdunbar](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@bdunbar](https://discuss.elastic.co/u/bdunbar)\
**Post date:** [June 17, 2015, 7:54pm UTC](https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872/1 "2015-06-17T19:54:21Z")

</div>

I have log files from 8 apache servers from 2014 that I need imported into elasticsearch.

After much blood, sweat, and tears, this appears to be working. But I'd like to know if there is, perhaps, a better way of doing things.

Log files are in /var/log/muo-2014/$servername/_yyyymmdd_

The basic idea is the extend the do-while loop to add host app02, app03, and so forth, then call the function with the new variable.

in day\_process function if-then is to avoid processing a non-existent file (some days, some servers were not online).

logger lines to write the file size to syslog so I can track what's going on.

```
function day_process() {
    if [-f /var/log/muo-2014/${HOST}/*${PROC_DATE}*]
    then
        logger "Log Import start ${HOST}/${PROC_DATE}"
        logger "Log Import start ${HOST}/${PROC_DATE} File Size follows"
        find /var/log/muo-2014/${HOST}/*${PROC_DATE}* -size +1000000c -print0 | du -c --files0-from=- | awk 'END{print $1}'
        find /var/log/muo-2014/${HOST}/*${PROC_DATE}* -size +1000000c -print0 | du -ch --files0-from=- | awk 'END{print $1}'
        cat /var/log/muo-2014/${HOST}/*${PROC_DATE}* | /opt/logstash/bin/logstash -f /root/muo/fileimport.conf
        logger "Log Import End ${HOST}/${PROC_DATE}"
        sleep 1
    else
        echo "/var/log/muo-2014/${HOST}/*${PROC_DATE}* does not exist moving along"
    fi

}

echo "Starting APP01"
HOST=app01

DATE=2014-06-01
for i in {0..30}
do
    NEXT_DATE=$(date +%m-%d-%Y -d "$DATE + $i day")
    PROC_DATE=$(date +%Y%m%d -d "$DATE + $i day")
    day_process
    echo "echo next_date is $NEXT_DATE"
    echo "echo proc-date is $PROC_DATE"
done
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 18, 2015, 2:39am UTC](https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872/2 "2015-06-18T02:39:29Z")

</div>

Yes, use Logstash! It's just what is was made for.

---

<div class="post-metadata">

**Author:** ![bdunbar](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@bdunbar](https://discuss.elastic.co/u/bdunbar)\
**Post date:** [June 18, 2015, 1:11pm UTC](https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872/3 "2015-06-18T13:11:49Z")

</div>

This is using logstash?

```
cat /var/log/muo-2014/logfile | /opt/logstash/bin/logstash

```

Or did you mean instead something like this?

import.conf

```
file {
    path => "/var/log/muo-2014/logfile"
}

```

I did try that but ran into the issue where it did not close (because it mimics 'tail -f', which left me confused about when to exit the process.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 18, 2015, 9:57pm UTC](https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872/4 "2015-06-18T21:57:20Z")

</div>

Use the cat method, that's fine.

(I initially misread your post sorry, was operating with reduced sleep yesterday)

---

<div class="post-metadata">

**Author:** ![bdunbar](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@bdunbar](https://discuss.elastic.co/u/bdunbar)\
**Post date:** [June 18, 2015, 10:12pm UTC](https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872/5 "2015-06-18T22:12:26Z")

</div>

> [@warkolm](#):
>
> (I initially misread your post sorry, was operating with reduced sleep yesterday)

No worries: Been there, done that, got the t-shirt.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:06am UTC](https://discuss.elastic.co/t/bash-to-import-data-what-could-i-be-doing-better/23872/6 "2017-07-06T00:06:37Z")

</div>