Questions[log-rotation](server)

We are running a stack of Docker Swarm services and, according to our Privacy Policy, we have to keep logs only for the last 6 months, but no older. I was trying to achieve it through different logging drivers, but all of them (AFAIK) work only with max file-size, but not with times/dates.

The one solution that I came up with: use json-file logging driver and then setup logrotate.d with rotate 180 and daily options.

The problem, however, is that json-file logging driver doesn't disable standard Docker logging (I mean docker logs -f ... or docker service logs -f ...) which uses some internal Docker storage for logs, as per documentation:

Only the json-file and journald drivers make the logs available directly from docker-compose up and docker-compose logs. Using any other driver does not print any logs.

In this case, the rotation will work for json-file logs, but the "internal" logs will be still kept longer.

So, I wonder, how can one achieve time-based Docker log rotation?

I have a production system on which there are multiple nginx apps hosted. those are nginx applications servers. Now for each of these nginx app servers a logrotate config file is configured in /etc/logrotate.d/ folder. one of the example of logrotate config file is given below.

/var/www/example/current/log/example.log {
  daily
  missingok
  rotate 1
  delaycompress
  compress
  notifempty
  copytruncate
}

I checked following Logrotate not working and checked my config and didn't find any duplicate entries for application servers or nginx as well. below is the definition of my nginx logrotate file.

/var/log/nginx/*.log {
    daily
    missingok
    rotate 3
  compress
  delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
    if [ -s /var/run/nginx.pid ]; then
      kill -USR1 `cat /var/run/nginx.pid`
    fi
    endscript
}

My logs are still getting written in *.log.1 and not in *.log Can anyone help with this? I am trying to understand what is wrong here. Below is the logrotate command output in debug mode.

logrotate --force -d /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/nginx

Handling 1 logs

rotating pattern: /var/log/nginx/*.log  forced from command line (3 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/nginx/access.log
  log does not need rotating
considering log /var/log/nginx/elasticsearch.access.log
  log does not need rotating
considering log /var/log/nginx/error.log
  log does not need rotating
considering log /var/log/nginx/grafana.access.log
  log does not need rotating
considering log /var/log/nginx/example-admin.access.log
  log needs rotating
considering log /var/log/nginx/example-admin.error.log
  log does not need rotating
considering log /var/log/nginx/example-administrators.access.log
  log needs rotating
considering log /var/log/nginx/example-administrators.error.log
  log does not need rotating
considering log /var/log/nginx/example-alot.access.log
  log does not need rotating
considering log /var/log/nginx/example-alot.error.log
  log does not need rotating
considering log /var/log/nginx/example-api_clients.access.log
  log does not need rotating
considering log /var/log/nginx/example-api_clients.error.log
  log does not need rotating
considering log /var/log/nginx/example-assets.access.log
  log does not need rotating
considering log /var/log/nginx/example-backgrounds.access.log
  log does not need rotating
considering log /var/log/nginx/example-backgrounds.error.log
  log does not need rotating
considering log /var/log/nginx/example-blog.access.log
  log does not need rotating
considering log /var/log/nginx/example-blog.error.log
  log does not need rotating
considering log /var/log/nginx/example-blog-service.access.log
  log does not need rotating
considering log /var/log/nginx/example-blog-service.error.log
  log does not need rotating
considering log /var/log/nginx/example-boards.access.log
  log does not need rotating
considering log /var/log/nginx/example-boards.error.log
  log does not need rotating
considering log /var/log/nginx/example-bolt.access.log
  log does not need rotating
considering log /var/log/nginx/example-bolt.error.log
  log does not need rotating
considering log /var/log/nginx/example-bonobo.access.log
  log does not need rotating
considering log /var/log/nginx/example-bonobo.error.log
  log does not need rotating
considering log /var/log/nginx/example-boss.access.log
  log does not need rotating
considering log /var/log/nginx/example-boss.error.log
  log does not need rotating
considering log /var/log/nginx/example-callbacks.access.log
  log does not need rotating
considering log /var/log/nginx/example-callbacks.error.log
  log does not need rotating
considering log /var/log/nginx/example-cashier.access.log
  log does not need rotating
considering log /var/log/nginx/example-cashier.error.log
  log does not need rotating
considering log /var/log/nginx/example-dashboard-old.access.log
  log does not need rotating
considering log /var/log/nginx/example-dcl_shipments.access.log
  log does not need rotating
considering log /var/log/nginx/example-dcl_shipments.error.log
  log does not need rotating
considering log /var/log/nginx/example-device-registry.access.log
  log does not need rotating
considering log /var/log/nginx/example-device-registry.error.log
  log does not need rotating
considering log /var/log/nginx/example-docs.access.log
  log does not need rotating
considering log /var/log/nginx/example-exports.access.log
  log does not need rotating
considering log /var/log/nginx/example-exports.error.log
  log does not need rotating
considering log /var/log/nginx/example-fake-api.access.log
  log does not need rotating
considering log /var/log/nginx/example-fake-api.error.log
  log does not need rotating
considering log /var/log/nginx/example-fashion-police.access.log
  log does not need rotating
considering log /var/log/nginx/example-fashion-police.error.log
  log does not need rotating
considering log /var/log/nginx/example-flight-recorder.access.log
  log does not need rotating
considering log /var/log/nginx/example-flight-recorder.error.log
  log does not need rotating
considering log /var/log/nginx/example-gone.access.log
  log does not need rotating
considering log /var/log/nginx/example-gone.error.log
  log does not need rotating
considering log /var/log/nginx/example-go-portal.access.log
  log does not need rotating
considering log /var/log/nginx/example-go-portal.error.log
  log does not need rotating
considering log /var/log/nginx/example-help-service.access.log
  log does not need rotating
considering log /var/log/nginx/example-help-service.error.log
  log does not need rotating
considering log /var/log/nginx/example-kfc.access.log
  log does not need rotating
considering log /var/log/nginx/example-kfc.error.log
  log does not need rotating
considering log /var/log/nginx/example-kredits.access.log
  log does not need rotating
considering log /var/log/nginx/example-kredits.error.log
  log does not need rotating
considering log /var/log/nginx/example-login.access.log
  log does not need rotating
considering log /var/log/nginx/example-login.error.log
  log does not need rotating
considering log /var/log/nginx/example-manual-activations.access.log
  log does not need rotating
considering log /var/log/nginx/example-manual-activations.error.log
  log does not need rotating
considering log /var/log/nginx/example-maps.access.log
  log does not need rotating
considering log /var/log/nginx/example-mobile-api.access.log
  log does not need rotating
considering log /var/log/nginx/example-mobile-api.error.log
  log does not need rotating
considering log /var/log/nginx/example-mqttproxy.access.log
  log does not need rotating
considering log /var/log/nginx/example-mqttproxy.error.log
  log does not need rotating
considering log /var/log/nginx/example-network-coverage.access.log
  log does not need rotating
considering log /var/log/nginx/example-network-coverage.error.log
  log does not need rotating
considering log /var/log/nginx/example-package_tracker.access.log
  log does not need rotating
considering log /var/log/nginx/example-package_tracker.error.log
  log does not need rotating
considering log /var/log/nginx/example-people.access.log
  log needs rotating
considering log /var/log/nginx/example-people.error.log
  log does not need rotating
considering log /var/log/nginx/example-postman-pat.access.log
  log does not need rotating
considering log /var/log/nginx/example-postman-pat.error.log
  log does not need rotating
considering log /var/log/nginx/example-premium.access.log
  log does not need rotating
considering log /var/log/nginx/example-premium.error.log
  log does not need rotating
considering log /var/log/nginx/example-push-notifications.access.log
  log does not need rotating
considering log /var/log/nginx/example-push-notifications.error.log
  log does not need rotating
considering log /var/log/nginx/example-referrific.access.log
  log does not need rotating
considering log /var/log/nginx/example-referrific.error.log
  log does not need rotating
considering log /var/log/nginx/example-services.access.log
  log does not need rotating
considering log /var/log/nginx/example-services.error.log
  log does not need rotating
considering log /var/log/nginx/example-store.access.log
  log does not need rotating
considering log /var/log/nginx/example-store.error.log
  log does not need rotating
considering log /var/log/nginx/example-subs.access.log
  log does not need rotating
considering log /var/log/nginx/example-subs.error.log
  log does not need rotating
considering log /var/log/nginx/example-updates.access.log
  log does not need rotating
considering log /var/log/nginx/example-updates.error.log
  log does not need rotating
considering log /var/log/nginx/example-website.access.log
  log does not need rotating
considering log /var/log/nginx/example-wildcard-redirect.access.log
  log does not need rotating
considering log /var/log/nginx/example-zendesk-proxy.access.log
  log does not need rotating
considering log /var/log/nginx/example-zendesk-proxy.error.log
  log does not need rotating
considering log /var/log/nginx/kibana.access.log
  log does not need rotating
considering log /var/log/nginx/prometheus.access.log
  log does not need rotating
considering log /var/log/nginx/prometheus_alertmanager.access.log
  log does not need rotating
considering log /var/log/nginx/prometheus_statsd_bridge.access.log
  log does not need rotating
rotating log /var/log/nginx/example-admin.access.log, log->rotateCount is 3
dateext suffix '-20170104'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/nginx/example-admin.access.log.3.gz to /var/log/nginx/example-admin.access.log.4.gz (rotatecount 3, logstart 1, i 3), 
renaming /var/log/nginx/example-admin.access.log.2.gz to /var/log/nginx/example-admin.access.log.3.gz (rotatecount 3, logstart 1, i 2), 
renaming /var/log/nginx/example-admin.access.log.1.gz to /var/log/nginx/example-admin.access.log.2.gz (rotatecount 3, logstart 1, i 1), 
renaming /var/log/nginx/example-admin.access.log.0.gz to /var/log/nginx/example-admin.access.log.1.gz (rotatecount 3, logstart 1, i 0), 
rotating log /var/log/nginx/example-administrators.access.log, log->rotateCount is 3
dateext suffix '-20170104'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/nginx/example-administrators.access.log.3.gz to /var/log/nginx/example-administrators.access.log.4.gz (rotatecount 3, logstart 1, i 3), 
renaming /var/log/nginx/example-administrators.access.log.2.gz to /var/log/nginx/example-administrators.access.log.3.gz (rotatecount 3, logstart 1, i 2), 
renaming /var/log/nginx/example-administrators.access.log.1.gz to /var/log/nginx/example-administrators.access.log.2.gz (rotatecount 3, logstart 1, i 1), 
renaming /var/log/nginx/example-administrators.access.log.0.gz to /var/log/nginx/example-administrators.access.log.1.gz (rotatecount 3, logstart 1, i 0), 
rotating log /var/log/nginx/example-people.access.log, log->rotateCount is 3
dateext suffix '-20170104'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/nginx/example-people.access.log.3.gz to /var/log/nginx/example-people.access.log.4.gz (rotatecount 3, logstart 1, i 3), 
renaming /var/log/nginx/example-people.access.log.2.gz to /var/log/nginx/example-people.access.log.3.gz (rotatecount 3, logstart 1, i 2), 
renaming /var/log/nginx/example-people.access.log.1.gz to /var/log/nginx/example-people.access.log.2.gz (rotatecount 3, logstart 1, i 1), 
renaming /var/log/nginx/example-people.access.log.0.gz to /var/log/nginx/example-people.access.log.1.gz (rotatecount 3, logstart 1, i 0), 
renaming /var/log/nginx/example-admin.access.log to /var/log/nginx/example-admin.access.log.1
creating new /var/log/nginx/example-admin.access.log mode = 0640 uid = 33 gid = 4
renaming /var/log/nginx/example-administrators.access.log to /var/log/nginx/example-administrators.access.log.1
creating new /var/log/nginx/example-administrators.access.log mode = 0640 uid = 33 gid = 4
renaming /var/log/nginx/example-people.access.log to /var/log/nginx/example-people.access.log.1
creating new /var/log/nginx/example-people.access.log mode = 0640 uid = 33 gid = 4
running postrotate script
running script with arg /var/log/nginx/*.log : "
    if [ -s /var/run/nginx.pid ]; then
      kill -USR1 `cat /var/run/nginx.pid`
    fi
"
removing old log /var/log/nginx/example-admin.access.log.4.gz
error: error opening /var/log/nginx/example-admin.access.log.4.gz: No such file or directory

And when i do force logrotate, I end up having errors below.

sudo logrotate -f /etc/logrotate.conf
logrotate_script: 2: logrotate_script: /etc/init.d/example-admin: not found
error: error running non-shared postrotate script for /var/www/admin/current/log/example-admin.log of '/var/www/admin/current/log/example-admin.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-administrators: not found
error: error running non-shared postrotate script for /var/www/administrators/current/log/example-administrators.log of '/var/www/administrators/current/log/example-administrators.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-alot: not found
error: error running non-shared postrotate script for /var/www/alot/current/log/example-alot.log of '/var/www/alot/current/log/example-alot.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-bonobo: not found
error: error running non-shared postrotate script for /var/www/bonobo/current/log/example-bonobo.log of '/var/www/bonobo/current/log/example-bonobo.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-cashier: not found
error: error running non-shared postrotate script for /var/www/cashier/current/log/example-cashier.log of '/var/www/cashier/current/log/example-cashier.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-dcl-shipments: not found
error: error running non-shared postrotate script for /var/www/dcl_shipments/current/log/example-dcl_shipments.log of '/var/www/dcl_shipments/current/log/example-dcl_shipments.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-people: not found
error: error running non-shared postrotate script for /var/www/people/current/log/example-people.log of '/var/www/people/current/log/example-people.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-store: not found
error: error running non-shared postrotate script for /var/www/store/current/log/example-store.log of '/var/www/store/current/log/example-store.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-storefront: not found
error: error running non-shared postrotate script for /var/www/storefront/current/log/example-storefront.log of '/var/www/storefront/current/log/example-storefront.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-subs: not found
error: error running non-shared postrotate script for /var/www/subs/current/log/example-subs.log of '/var/www/subs/current/log/example-subs.log '

Below is the logrotate command ouput in debug mode.

sudo logrotate -dv /etc/logrotate.d/example-alot
reading config file /etc/logrotate.d/example-alot

Handling 1 logs

rotating pattern: /var/www/alot/current/log/example-alot.log  after 1 days (1 rotations)
empty log files are not rotated, old logs are removed
considering log /var/www/alot/current/log/example-alot.log
  log needs rotating
rotating log /var/www/alot/current/log/example-alot.log, log->rotateCount is 1
dateext suffix '-20170105'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/www/alot/current/log/example-alot.log.1.gz to /var/www/alot/current/log/example-alot.log.2.gz (rotatecount 1, logstart 1, i 1),
renaming /var/www/alot/current/log/example-alot.log.0.gz to /var/www/alot/current/log/example-alot.log.1.gz (rotatecount 1, logstart 1, i 0),
copying /var/www/alot/current/log/example-alot.log to /var/www/alot/current/log/example-alot.log.1
truncating /var/www/alot/current/log/example-alot.log
running postrotate script
running script with arg /var/www/alot/current/log/example-alot.log: "
      echo "Appending to" example-alot > /var/www/alot/current/log/example-alot.log
      /etc/init.d/example-alot reload >> /dev/null
"
removing old log /var/www/alot/current/log/example-alot.log.2.gz


sudo logrotate -dv /etc/logrotate.d/example-subs
reading config file /etc/logrotate.d/example-subs

Handling 1 logs

rotating pattern: /var/www/subs/current/log/example-subs.log  after 1 days (1 rotations)
empty log files are not rotated, old logs are removed
considering log /var/www/subs/current/log/example-subs.log
  log does not need rotating

I'm using rotatelogs to create my daily apache logs in format host.<day>.<month>.<year>.access.log. Now I want to gzip and move log to different directory after it's been finished. How to do that?

Update: There was a little mistake. logrotate -> rotatelogs

I wonder is there a programme that'll do what I want. This is like an streaming log rotation programme. For the linux/unix command line.

Let's say I have some input that is streaming some data. I want to write it a file, but a different file per day (or hour or whatever), i.e. open a file with a specified datetime based pattern, and write the contents of stdin to that file. when the day changes (or the hour or the minute, or the week etc), close that file, then re-open a new one (which will have a new filename), and write lines to that file instead?

So something like:

my long | process | that's generating | input | datestampfilewriter --daily 'output.%Y-%m-%d.txt'

Does this tool exist?

My log files are in the format "2011-03-28.log.php". Log files are created every day. I want to keep 5 days log and rest I want to remove it, which means only 5 log files will be kept and rest everything will be deleted.

Is it possible using Linux logrotate ?