# How to create a separate index which has only required fields

**URL:** <https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125>\
**Category:** Logstash\
**Created:** [October 23, 2020, 12:04pm UTC](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125 "2020-10-23T12:04:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ronnie16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronnie16/32/66399_2.png) [@Ronnie16](https://discuss.elastic.co/u/Ronnie16)\
**Post date:** [October 23, 2020, 12:04pm UTC](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125/1 "2020-10-23T12:04:03Z")

</div>

I have ELK cluster setup and i delete indices older than 10 days but i want to create a separate index which only has 1 required field so that it is always present .  
The main index has huge data and multiple fields, i only want below info in new index

```auto
 message:
      {"log_model": "request", "written_at": "2020-10-23T02:20:55.289Z", "written_ts": 1603419655289617000, "remote_user": "-", "request_url": "/login", "referer": "https://mytestingapp.com/testapp/", "x_forwarded_for": "192.168.1.1", "protocol": "HTTP/1.1", "method": "POST", "remote_ip": "10.234.2.1", "request_size_b": 49, "remote_host": "10.234.2.1", "remote_port": "58070", "request_received_at": "2020-10-23T02:20:55.289Z", "log_type": "user-service", "correlation_id": null}

```

i only want the below data

````auto
"request_url": "/login", "referer": "https://mytestingapp.com/testapp/",```

````

For this i had created another logstash.conf and added below

```auto
input {
  beats {
    port => 5044
    add_field => { "log_level" => "-" }
    tags => ["filebeat-log"]
    type => "beats"
    client_inactivity_timeout => 300
  }
}
filter {
  if [type] == "beats" {
   json {
      source => "message"
    }

   prune {
       interpolate => true
       whitelist_names => ["request_url"]
   }
  }
}
output {
  if [type] == "beats" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "login-index"
      action => "index"
    }
  }
}

```

but i dont se the index "login-index" in indices so i'm unable to create index pattern.  
Also i have stopped receiving any data in other index since i added the above

---

<div class="post-metadata">

**Author:** ![Ronnie16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronnie16/32/66399_2.png) [@Ronnie16](https://discuss.elastic.co/u/Ronnie16)\
**Post date:** [November 3, 2020, 5:59am UTC](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125/2 "2020-11-03T05:59:34Z")

</div>

Hi,  
any input will be appreciated.  
I have create a separate pipeline but getting below error in logstash.

```auto
warning: thread "[login-pipeline]>worker2" terminated with exception (report_on_exception is true):
Nov 3 05:29:06 ip-122-13-4-31 logstash[29473]: java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[^`

```

The logstash syntax test does not show any syntax errors though

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 3, 2020, 3:49pm UTC](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125/3 "2020-11-03T15:49:38Z")

</div>

I suspect that is a problem with interpolation. Why have you set

```
interpolate => true
```

---

<div class="post-metadata">

**Author:** ![Ronnie16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ronnie16/32/66399_2.png) [@Ronnie16](https://discuss.elastic.co/u/Ronnie16)\
**Post date:** [November 4, 2020, 5:26am UTC](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125/4 "2020-11-04T05:26:54Z")

</div>

Thanks @Badger, I saw it somewhere while checking for prune filter.

After removing that, i dont see any errors but i don't see the new index in index paterns

cat pipeline.yml

```auto
- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: login-pipeline
  path.config: "/home/ronnie/login_index.conf"

```

login-index.conf

```auto
input {
  beats {
    port => 5044
    client_inactivity_timeout => 300
  }
}
filter {
   json {
      source => "message"
    }

   prune {
       whitelist_names => ["^_source"]
   }
}
output {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "login-index"
    }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 2, 2020, 5:26am UTC](https://discuss.elastic.co/t/how-to-create-a-separate-index-which-has-only-required-fields/253125/5 "2020-12-02T05:26:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.