# Reindexing packetbeats

**URL:** <https://discuss.elastic.co/t/reindexing-packetbeats/245511>\
**Category:** Elasticsearch\
**Created:** [August 19, 2020, 12:34am UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511 "2020-08-19T00:34:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zead](https://avatars.discourse-cdn.com/v4/letter/z/977dab/32.png) [@Zead](https://discuss.elastic.co/u/Zead)\
**Post date:** [August 19, 2020, 12:34am UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511/1 "2020-08-19T00:34:33Z")

</div>

Ok i have been following this tutorial and i have almost completed it to predict.

> **[Machine learning in cybersecurity: Detecting DGA activity in network data...](https://www.elastic.co/blog/machine-learning-in-cybersecurity-detecting-dga-activity-in-network-data)**
>
> Bad actors (and their malware) use domain generated algorithms (DGA) to avoid detection, but with Elastic Stack machine learning, you can easily build models to help you see right through their tricks. Learn how to detect their activity in your...

but when i am trying to reindex the packetbeat i am only reindexing the previous data not actually live.

so how can i capture the packetsbeat and captrue it live.

thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 19, 2020, 12:36am UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511/2 "2020-08-19T00:36:47Z")

</div>

Welcome to our community! 😃

It's not clear what you are asking here. Are you talking about reindexing the live data? Or capturing the live data?

---

<div class="post-metadata">

**Author:** ![Zead](https://avatars.discourse-cdn.com/v4/letter/z/977dab/32.png) [@Zead](https://discuss.elastic.co/u/Zead)\
**Post date:** [August 19, 2020, 9:11am UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511/3 "2020-08-19T09:11:38Z")

</div>

thank you for your welcoming.  
i am capturing my network data through packetsbeats. after that i am preprocessing the domain name to detect wether it's malicious packet or not. i have already done all that.  
but because i am reindexing the i am only getting previous data.  
like if i am indexing at 3:01 pm i am only data from 3:01 pm and previous.  
i want to index live data and continuous. how can i do that

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 19, 2020, 9:37pm UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511/4 "2020-08-19T21:37:24Z")

</div>

Ah ok. How are you doing the processing exactly?

---

<div class="post-metadata">

**Author:** ![Zead](https://avatars.discourse-cdn.com/v4/letter/z/977dab/32.png) [@Zead](https://discuss.elastic.co/u/Zead)\
**Post date:** [August 19, 2020, 9:55pm UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511/5 "2020-08-19T21:55:05Z")

</div>

I am taking the packetsbeats data (specifically domain name) and take n-gram of the name through painless script then i am putting the script through the pipeline so i can do inference the model.

this is the code :

```auto

    PUT _ingest/pipeline/dga_ngram_expansion_inference
    {
        "description": "Expands a domain into unigrams, bigrams and trigrams and makes a prediction of maliciousness",
        "processors": [
          {
            "script": {
              "id": "ngram-extractor-packetbeat",
              "params":{
                "ngram_count":1
              }
            }
          },
           {
            "script": {
              "id": "ngram-extractor-packetbeat",
              "params":{
                "ngram_count":2
              }
            }
          },
           {
            "script": {
              "id": "ngram-extractor-packetbeat",
              "params": {
                "ngram_count":3
              }
            }
          },
                  {
      "inference": {
        "model_id": "tes22-1597491011440",
        "target_field": "predicted_label",
        "field_map": {},
        "inference_config": { "classification": {"num_top_classes": 2} }
      }
    },
      {
               "script": {
              "id": "ngram-remover-packetbeat",
              "params":{
                "ngram_count":1
              }
            }
          },
           {
            "script": {
              "id": "ngram-remover-packetbeat",
              "params":{
                "ngram_count":2
              }
            }
          },
           {
            "script": {
              "id": "ngram-remover-packetbeat",
              "params": {
                "ngram_count":3
              }
            }
      }
        ]
    }

    PUT _ingest/pipeline/dns_classification_pipeline
    {
      "description": "A pipeline of pipelines for performing DGA detection",
      "version": 1,
      "processors": [
        {
          "pipeline": {
            "if": "ctx.containsKey('dns') && ctx['dns'].containsKey('question') && ctx['dns']['question'].containsKey('registered_domain') && !ctx['dns']['question']['registered_domain'].empty",
            "name": "dga_ngram_expansion_inference"
          }
        }
      ]
    }

    GET _ingest/pipeline/

    POST _reindex?wait_for_completion=false&refresh=true
    { 
      "source": {
        "index": "packetbeat-7.8.1-*",
        "query": {
          "bool": {
            "must": [
              {
                "range": {
            "@timestamp": {
              "gte": "now-1h/h",
              "lt": "now"
            }
          }
          },
          {
              "match": {
            "method": "Query"
          }
          }
              
            ]
          }
        }
      },
      "dest": {
        "index": "dga-detection_with_live_data17",
        "pipeline": "dns_classification_pipeline",
        "op_type": "index"
      }
    } 

```

i have made some work around i have created a python script that runs every 5 min to index the last 5 min.  
but i know this isn't really a solution as in production it will have a big impact on the system.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 19, 2020, 10:58pm UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511/6 "2020-08-19T22:58:43Z")

</div>

Ok, this is outside the scope of what Packetbeat does at the moment.

However it's a great idea, and I would encourage you to raise a feature request in GitHub to see if there might be something that can be built in in future.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 16, 2020, 10:58pm UTC](https://discuss.elastic.co/t/reindexing-packetbeats/245511/7 "2020-09-16T22:58:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.