Questions[filebeat](server)

I'm trying to exclude a specific type if lines from my logs but not sure what regex pattern I should be using. Please advise.

I want to exclude any lines that have "/ws/api/authenticate".

Example line:

255.255.255.255 [01/Mar/2022:13:16:10 -0500] "/ws/api/authenticate?licenseKey=XXXXXXXXX&product=PRODID&ipAddress=12.12.12.12&version=v20180104" - 200 0.071 [90033]

I have two servers. Server A is running Elasticsearch and Logstash. Server B is running filebeat and is also the server which contains all the logs I'm trying to analyse.

Server A is behind a firewall, it can reach out to the internet, but there's no possibility of allowing any inbound traffic.

Server B is in AWS and under my control for inbound and outbound traffic.

Is there any way I can get data from server B to server A's logstash? Filebeat in the usual way would have to be able to push data to logstash, but this isn't possible in my scenario.

We are running an Elastic Stack with ECK in EKS (7.8). We noticed that our filebeat daemonset and the AWS module were not processing logs from S3 and our SQS queues backing up. Looking at the logs on our FileBeat containers, we noticed the following repeating error in each of the filebeat pods:

ERROR [s3] s3/input.go:206 SQS ReceiveMessageRequest failed: 
EC2RoleRequestError: no EC2 instance role found caused by: 
EC2MetadataError: failed to make Client request

We have found:

1. Restarting the pods did not resolve the issue.
2. Removing and reapplying the daemonset did not resolve the issue.
3. Restarting the nodes did not resolve the issue.
4. Terminating the nodes and allowing them to be replaced DID resolve the issue. The filebeat-pods began to process data again.

This issue has repeated many times and at different intervals. The filebeat pods will successfully process data for weeks at a time, or sometimes only a few days before this error recurs.

Although we have a work-around (replacing the nodes) we still wanted to look into why this occurred. Working with AWS we finally tracked down what we feel is a hint as to the root cause.

When the pods successfully start up, it seems that we see a message like the following:

{"log":"2021-02-17T17:08:47.378Z\u0009INFO\u0009[add_cloud_metadata]
\u0009add_cloud_metadata/add_cloud_metadata.go:93\u0009add_cloud_metadata: hosting provider
type detected as aws, metadata={\"account\":{\"id\":\"##########\"},\"availability_zone
\":\"#########\",\"image\":{\"id\":\"#############\"},\"instance\":{\"id\":\"i-#########
\"},\"machine\":{\"type\":\"#######\"},\"provider\":\"aws\",\"region\":\"######
\"}\n","stream":"stderr","time":"2021-02-17T17:08:47.379170444Z"}

If we restart a pod and monitor it on a "failed" node, we see this log message:

{"log":"2021-02-17T17:08:47.439Z\u0009INFO\u0009[add_cloud_metadata]
\u0009add_cloud_metadata/add_cloud_metadata.go:89\u0009add_cloud_metadata: hosting provider
 type not detected.\n","stream":"stderr","time":"2021-02-17T17:08:47.439427267Z"}

We have also verified that the pods themselves CAN hit the EC2 metadata endpoint successfully via curl.

Any information on how to resolve this recurring issue would be appreciated.

Filebeat Config:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: filebeat-config
    data:
      filebeat.yml: |-
        filebeat.config:
          modules:
            path: ${path.config}/modules.d/*.yml
            reload.enabled: true

        processors:
          - add_cloud_metadata: ~
          - add_docker_metadata: ~

        output.elasticsearch:
          workers: 5
          bulk_max_size: 5000
          hosts: ["${ES_HOSTS}"]
          username: "${ES_USER}"
          password: "${ES_PASSWORD}"

I'm learning to use ELK and have a debian PC that runs as a test client. every 30 sec it logs a message :

021-01-18T08:29:59.656-0500#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":4096}}}},"cpu":{"system":{"ticks":171310,"time":{"ms":11}},"total":{"ticks":433770,"time":{"ms":24},"value":433770},"user":{"ticks":262460,"time":{"ms":13}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":13},"info":{"ephemeral_id":"e83bbdd5-2482-4ac4-85b9-5b50f2b64e7c","uptime":{"ms":444690082}},"memstats":{"gc_next":20543200,"memory_alloc":12804128,"memory_total":20951973208},"runtime":{"goroutines":54}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":2,"batches":2,"total":2},"read":{"bytes":681},"write":{"bytes":4714}},"pipeline":{"clients":3,"events":{"active":0,"published":2,"total":2},"queue":{"acked":2}}},"registrar":{"states":{"current":15,"update":2},"writes":{"success":2,"total":2}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}

I dont want to have this every 30 secconds, so i created a filebeat.yml filter it out in filebeat

under path's section i wrote

exclude_lines :['^INFO#011[monitoring]#011log/log.go:145#011Non-zero']

To no effect, how should the line be ?.

I have a few side questions,

• If it wasnt logged every 30sec it be fine to me, can this status be logged every 5 minutes ?
• Well i'm new to ELK, i would think this line shouldnt be in filebeat (as it should do logs not memmory stats of the system, or am i reading tis wrong ?) On the other hand it isnt bad to know those stats, but then why isnt it nicely shown in Gui

I have an Elasticsearch 7.1 and i have configure filebeat to collect all log. I want to check if i have password in log.

So anyone have an idea how can i find all password in log using filebeat.

Thank you