Home / server / Questions / 855231
Accepted
Andreas
Asked: 2017-06-13 03:05:59 +0800 CST

Logstash output to Azure blobstorage

  • 772

I am currently importing IIS-logs into Logstash using Filebeat, and the Logstash is configured to output the documents into Elasticsearch. Now I also want to output my IIS logs to Azure storage (blob) for longtime-backup purposes, but I cannot find a way to do it.

There is a working Logstash plugin for output to AWS. But I need to use Azure blob storage. I cannot find an output plugin for Azure Blobstorage in the elastic output list, or anywhere else..

Is there a way to output documents from Logstash to Azure blobstorage?

Seems like a valuable feature. I guess it would look like something below.

output {
    Azure_storage {
        account => "test"
        key => "SuperSecret"
        container => "Backup_Documents"
        blobName => "nameofblob"
    }
  }
azure

2 Answers

  1. Best Answer

    There isn't one in the official plugins, nor in the community-maintained list of plugins. Someone probably hasn't written one yet. Or if they have, they aren't sharing. This may be a case where you output to a local directory and use a scheduled task to sync the directory to blob-storage. We're using this method for one of our stranger workflows.

    • 1

  2. There is an open-source plugin: https://github.com/tuffk/Logstash-output-to-Azure-Blob

    Basic configuration:

      output {
       azure {
         storage_account_name => "my-azure-account"    # required
         storage_access_key => "my-super-secret-key"   # required
         container_name => "my-container"              # required
         size_file => 1024*1024*5                      # optional
         time_file => 10                               # optional
         restore => true                               # optional
         temporary_directory => "path/to/directory"    # optional
         prefix => "a_prefix"                          # optional
         upload_queue_size => 2                        # optional
         upload_workers_count => 1                     # optional
         rotation_strategy_val => "size_and_time"      # optional
         tags => []                                    # optional
         encoding => "none"                            # optional
       }
     }

    To make the plugin available in your Logstash environment, run the following command:

    bin/logstash-plugin install logstash-output-azure
    • 1