# How to capture text from a path or s3 prefix

**URL:** <https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998>\
**Category:** Logstash\
**Created:** [August 6, 2017, 1:58am UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998 "2017-08-06T01:58:16Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 6, 2017, 1:58am UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/1 "2017-08-06T01:58:16Z")

</div>

I have a slightly different requirement where i have a directory structure something like this

/data/aws-test1  
/data/aws-test2 .. and so on

Is there a way when using the file input template to capture aws-test1 and aws-test2 and use that name as the name of the index when it tries to ingest files from that directory.

I am using something like this as of now.

```
input {
file {
       path => "/data/**/*.gz"
       codec => "cloudtrail"
       start_position => "beginning"
       sincedb_path => "/usr/share/logstash/.sincedb"
        sincedb_path => "/var/lib/logstash/.sincedb"
       type => "cloudtrail"
        max_open_files => "1024"
  }
}

filter {
   grok {
       match => {"path" => "(/data/?<tstmp>\S+)/.*"}
   }
}

output {
  stdout { codec => rubydebug }

  elasticsearch {
        hosts => ["xx.xx.xx.xx:9200"]
        index => "%{[tstmp]}-%{+YYYY-MM}"
   }
}

```

Is this even possible to do.?

--  
Niraj

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 6, 2017, 8:23am UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/2 "2017-08-06T08:23:45Z")

</div>

Change

```
"(/data/?<tstmp>\S+)/.*"

```

to

```
"^/data/(?<tstmp>[^/]+)/"

```

but otherwise it should be fine.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 6, 2017, 8:08pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/3 "2017-08-06T20:08:53Z")

</div>

Hi @magnusbaeck,

```
yellow open %{[tstmp]}-2017-07 yAWsiuiLQzGGADBNJWP3hw 5 1 7 0 128.5kb 128.5kb
yellow open %{[tstmp]}-2017-08 5XJUW7ejTgmeQSkdFll_Xg 5 1 23 0 254.7kb 254.7kb

```

This is what i get when the index gets created.

--  
Niraj

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 6, 2017, 8:12pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/4 "2017-08-06T20:12:36Z")

</div>

@magnusbaeck Seems like there was a type in my copy paste. It works now.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 6, 2017, 9:34pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/5 "2017-08-06T21:34:30Z")

</div>

@magnusbaeck  
One more problem i have now is ingesting the \*.gz files of aws cloudtrails. As these gzip files have json inside it which does not have a new line , Logstash thinks that it is expecting more input as the file has no end to it.

logstash.inputs.file - each: file grew --\> This is what i see and it goes on and on forever.

Is there a way to overcome this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 7, 2017, 5:25am UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/6 "2017-08-07T05:25:18Z")

</div>

The file input expects log entries to end with a newline character. I don't think there's a workaround for this.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 7, 2017, 5:50am UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/7 "2017-08-07T05:50:40Z")

</div>

Thanks @magnusbaeck for the feedback. So what i have right now is like pre-process the s3 files , untar it, add a new line to every file with a script and let remain the existing files ( \*.gz ) so i can revert back if required to gz types. The ingestion is working fine now with few of the fields showing in kibana as not recognized but i believe those are due to mapping issues.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 23, 2017, 4:42pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/8 "2017-08-23T16:42:41Z")

</div>

@magnusbaeck I think i am stuck again. I am flipping back and forth between input file and input s3 to compare performance and the hassle of processing data. Here is my config with the pattern to grab for index creation. But somehow it is not able to grab the output i want.

```
input {
  s3 {
    type => "cloudtrail"
    bucket => "xxxxxxxxxxxxxxxx"
    prefix => "AWSLogs/xxxxxxxxxxxxx/CloudTrail/us-east-1/2017/02/21/"
    backup_to_dir => "/etc/s3backup/"
    add_field => { source => gzfiles }
    codec => cloudtrail {}
    region => "us-east-1"
    access_key_id => "xxxxxxxxxxxxxxxxxxxx"
    secret_access_key => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    sincedb_path => "/etc/s3backup/sincedb"
  }
}

filter {
   grok {
       match => {"prefix" => "^AWSLogs/(?<tstmp>[^/]+)/"}
   }
 }

output {
  stdout { codec => rubydebug }
  elasticsearch {
       	index => "%{[tstmp]}-%{+YYYY-MM}"
        hosts => ["xxxxxxxxxxxxx:9200"]
   }
}

```

I want names after AWSLogs which is generally the account name for aws to be used as my index name but the index that are created is having

%{[tstmp]}-2017-02 as the index name. What am i doing wrong here?

--  
Niraj

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 24, 2017, 2:03pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/9 "2017-08-24T14:03:11Z")

</div>

Is there a `prefix` field? Show an example event produced by Logstash.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 24, 2017, 3:49pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/10 "2017-08-24T15:49:00Z")

</div>

@magnusbaeck i was in an assumption that i can extract from the "prefix" variable that is being used in the s3 input. Now i feel that it has to be an actual field. If this is the issue, is there a way i can grab the value what i see in "prefix" variable in input. The reason i want this is because there are no other fields in cloudtrail events that can give me this value.

Let me know your view or any suggestion.

Attached is the sample cloudtrail event

```
{
               "eventID" => "1fc18942-bf47-41e2-b20d-b09fbcaa7acb",
             "awsRegion" => "us-east-1",
          "eventVersion" => "1.05",
      "responseElements" => {
        "assumedRoleUser" => {
            "assumedRoleId" => "xxxxxxxxxxxxxxx",
                      "arn" => "xxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
            "credentials" => {
             "accessKeyId" => "xxxxxxxxxxxxxxxxxxx",
            "sessionToken" => "xxxxxxxxxxxxxxxxxxxx",
              "expiration" => "Feb 21, 2017 1:00:02 AM"
        }
    },
       "sourceIPAddress" => "xxxxxxxxxxxxxxxx",
           "eventSource" => "sts.amazonaws.com",
     "requestParameters" => {
                "roleArn" => "xxxxxxxxxxxxxxxxxx",
        "roleSessionName" => "xxxxxxxxxxxxxxx"
    },
             "resources" => [
        [0] {
            "accountId" => "xxxxxxxxxxxxx",
                 "type" => "AWS::IAM::Role",
                  "ARN" => "xxxxxxxxxxxxxxxxxxxx"
        }
    ],
             "userAgent" => "Boto/2.43.0 Python/2.7.3 Linux/3.2.0-119-generic",
          "userIdentity" => {
        "accessKeyId" => "xxxxxxxxxxxxxxxxx",
          "accountId" => "xxxxxxxxxxxxx",
        "principalId" => "xxxxxxxxxxxxxxxxx",
               "type" => "IAMUser",
                "arn" => "xxxxxxxxxxxxxxx",
           "userName" => "xxxxxxxxxxxxxx"
    },
             "eventType" => "AwsApiCall",
                "source" => "gzfiles",
                  "type" => "cloudtrail",
                  "tags" => [
        [0] "_grokparsefailure"
    ],
            "@timestamp" => 2017-02-21T00:00:02.000Z,
         "sharedEventID" => "2a9b9a58-6d9b-4d66-873c-ae3e08a6f4f6",
             "requestID" => "b1c387b7-f7c8-11e6-a1b7-a52c9bd268c6",
              "@version" => "1",
             "eventName" => "AssumeRole",
    "recipientAccountId" => "xxxxxxxxxxxxx"
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 24, 2017, 8:21pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/11 "2017-08-24T20:21:37Z")

</div>

> If this is the issue, is there a way i can grab the value what i see in “prefix” variable in input.

No, but you can use `add_field` in the input to add whatever field you like.

```plaintext
input {
  s3 {
    type => "cloudtrail"
    bucket => "xxxxxxxxxxxxxxxx"
    prefix => "AWSLogs/xxxxxxxxxxxxx/CloudTrail/us-east-1/2017/02/21/"
    ...
    add_field => {
      "prefix" => "AWSLogs/xxxxxxxxxxxxx/CloudTrail/us-east-1/2017/02/21/"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 24, 2017, 8:27pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/12 "2017-08-24T20:27:49Z")

</div>

@magnusbaeck My bad for not explaining this correctly. Apologies.

This was just for testing so i included the complete path to narrow down the ingestion speed.

Basically i have different AWS Accounts under AWSLogs directory.

Something like below

AWSLogs/aws-test/  
AWSLogs/aws-dev /

and so on. and each one of them should be treated as single index as they belong to different account. Will this trick still work?

--  
Niraj

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 21, 2017, 8:28pm UTC](https://discuss.elastic.co/t/how-to-capture-text-from-a-path-or-s3-prefix/95998/13 "2017-09-21T20:28:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.