Rob Brandt's questions

Ubuntu 20.04.3. This is a server that I upgraded last winter from 16.04 to 20.04. I use Webmin/Virtualmin to do day-to-day management. As a renewal for a LetsEncrypt cert came to be due last week, I started getting an error: "cannot change profile for the next exec call: No such file or directory". The Webmin people have not seen it before and I get the error message when I try certbot from the commandline, so I don't think it's involved. The people at letsencypt gave a valiant effort in trying to find the problem, but have come up empty: https://community.letsencrypt.org/t/letsencrypt-error-cannot-change-profile-for-the-next-exec-call-no-such-file-or-directory/164500

But it seems that this is an AppArmor issue. I have checked the /var/log/kern.log logs and one frequently listed is: audit: type=1400 audit(1635877257.770:584): apparmor="DENIED" operation="change_onexec" info="label not found" error=-2 profile="unconfined" name="snap.certbot.certbot" pid=3272017 comm="snap-confine"

What does this mean? Anything else I should be looking at? There are no log files in /var/log/apparmor.

Thanks!

Well I don't know if it's blocking EVERYTHING else, but it's blocking a lot.

I have an AWS instance running Ubuntu 20.04. This is a web server that hosts dozens of php/mysql sites that are "archives" or old versions of a web site that gets completely revised annually. From time to time everything goes dead... apache, mysql, even the webmin system I used to manage it. Even much of the webmin stats collection stops.

It happened again last night, for about 2 minutes. I did find one stat still being collected, "services", and it shows packagekit spiking that same time other stat collection goes dead.

What gives?

I have a couple of Ubuntu 20.04 servers running on AWS, using MySQL 5.7 rather than 8.0.

When I set up the server I did some modifications to /etc/mysql/mysql.conf.d/mysqld.cnf and everything worked fine. This is what I did previously in 16.04.

But it seems that a subsequent update via apt overwrote my changes on both servers. What is the proper way to do this in 20.4?

I recently upgraded my web servers from Ubuntu 16.04 to 20.04. I notice that snapd is running and taking a lot of memory. I've read up a bit on the Snap Package Manager and it most of what I read seems to be in the context of a desktop environment. It seems that for a server, apt is still at the core. Do I really need this thing running?

I have an Ubuntu 16.04 server I want to upgrade to Ubuntu 20.04. That comes with MySQL 8, but I am not ready for that yet. I have found examples on how to remove MySQL 8 and install MySQL 5.7, and have done that successfully. But how to I migrate my data? I have dozens of databases.

Obviously when the Ubuntu 20.04 upgrade runs and includes MySQL 8, it upgrades the data to be compatible with it. Swapping in MySQL 5.7 obviously won't downgrade it. I tried copying the /var/lib/mysql contents, then moving them back in after the upgrade/downgrade, but MySQL won't connect. Probably I might need to be more selective? ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

The username and permissions match what was in my old server. It is the mysql user. Here is the error information from the log:

2021-01-27T22:55:18.309052Z 0 [ERROR] [FATAL] InnoDB: Table flags are 0 in the data dictionary but the flags in file ./ibdata1 are 0x4000! 2021-01-27 22:55:18 0x7f5e14b0d740 InnoDB: Assertion failure in thread 140042050787136 in file ut0ut.cc line 921 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 22:55:18 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=151 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68196 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x56457cddebeb] /usr/sbin/mysqld(handle_fatal_signal+0x377)[0x56457c6a1977] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f5e150663c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f5e14b5818b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f5e14b37859] /usr/sbin/mysqld(+0x6aacec)[0x56457c677cec] /usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0x66)[0x56457d120ed6] /usr/sbin/mysqld(+0x120025b)[0x56457d1cd25b] /usr/sbin/mysqld(+0x12007ed)[0x56457d1cd7ed] /usr/sbin/mysqld(Z6fil_ioRK9IORequestbRK9page_id_tRK11page_size_tmmPvS8+0x322)[0x56457d1d9962] /usr/sbin/mysqld(_Z13buf_read_pageRK9page_id_tRK11page_size_t+0xdd)[0x56457d18c6cd] /usr/sbin/mysqld(_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0x475)[0x56457d15e325] /usr/sbin/mysqld(_Z31trx_rseg_get_n_undo_tablespacesPm+0x14f)[0x56457d0ff42f] /usr/sbin/mysqld(+0x10f96b8)[0x56457d0c66b8] /usr/sbin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x2e3d)[0x56457d0caa3d] /usr/sbin/mysqld(+0xfc431d)[0x56457cf9131d] /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x56457c6f593f] /usr/sbin/mysqld(+0xbe68e5)[0x56457cbb38e5] /usr/sbin/mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x56457cbb6e05] /usr/sbin/mysqld(+0x6cccd0)[0x56457c699cd0] /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x786)[0x56457c69b3e6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f5e14b390b3] /usr/sbin/mysqld(_start+0x2a)[0x56457c69169a] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.

I have an newish Ubuntu 16.04 server that I am installing a Symfony based application on. It says date.timezone needs to be set. Fair enough, I've done that before. Php is configured as FPM, so it SHOULD be a local php.ini above doc root, right? Setting date.timezone there does no good.

phpinfo() says the ini files read are:

Configuration File (php.ini) Path /etc/php/5.6/fpm Loaded Configuration File /etc/php/5.6/fpm/php.ini Scan this dir for additional .ini files /etc/php/5.6/fpm/conf.d Additional .ini files parsed /etc/php/5.6/fpm/conf.d/10-mysqlnd.ini, /etc/php/5.6/fpm/conf.d/10-opcache.ini, /etc/php/5.6/fpm/conf.d/10-pdo.ini, /etc/php/5.6/fpm/conf.d/15-xml.ini, /etc/php/5.6/fpm/conf.d/20-calendar.ini, /etc/php/5.6/fpm/conf.d/20-ctype.ini, /etc/php/5.6/fpm/conf.d/20-curl.ini, /etc/php/5.6/fpm/conf.d/20-dom.ini, /etc/php/5.6/fpm/conf.d/20-exif.ini, /etc/php/5.6/fpm/conf.d/20-fileinfo.ini, /etc/php/5.6/fpm/conf.d/20-ftp.ini, /etc/php/5.6/fpm/conf.d/20-gd.ini, /etc/php/5.6/fpm/conf.d/20-geoip.ini, /etc/php/5.6/fpm/conf.d/20-gettext.ini, /etc/php/5.6/fpm/conf.d/20-iconv.ini, /etc/php/5.6/fpm/conf.d/20-imagick.ini, /etc/php/5.6/fpm/conf.d/20-json.ini, /etc/php/5.6/fpm/conf.d/20-mbstring.ini, /etc/php/5.6/fpm/conf.d/20-mcrypt.ini, /etc/php/5.6/fpm/conf.d/20-mysql.ini, /etc/php/5.6/fpm/conf.d/20-mysqli.ini, /etc/php/5.6/fpm/conf.d/20-pdo_mysql.ini, /etc/php/5.6/fpm/conf.d/20-phar.ini, /etc/php/5.6/fpm/conf.d/20-posix.ini, /etc/php/5.6/fpm/conf.d/20-readline.ini, /etc/php/5.6/fpm/conf.d/20-shmop.ini, /etc/php/5.6/fpm/conf.d/20-simplexml.ini, /etc/php/5.6/fpm/conf.d/20-sockets.ini, /etc/php/5.6/fpm/conf.d/20-sysvmsg.ini, /etc/php/5.6/fpm/conf.d/20-sysvsem.ini, /etc/php/5.6/fpm/conf.d/20-sysvshm.ini, /etc/php/5.6/fpm/conf.d/20-tokenizer.ini, /etc/php/5.6/fpm/conf.d/20-wddx.ini, /etc/php/5.6/fpm/conf.d/20-xmlreader.ini, /etc/php/5.6/fpm/conf.d/20-xmlwriter.ini, /etc/php/5.6/fpm/conf.d/20-xsl.ini, /etc/php/5.6/fpm/conf.d/20-zip.ini

I've tried setting date.timezone in /etc/php/5.6/fpm/php.ini but it also has no effect.

What's going on here?

I've been knocking my brains on this for a week...

Setup: Ubuntu 16.04 LAMP stack, php 5.6 (yes I know I need to upgrade this). The server runs many sites for one organization, one of which gets constant public usage. The others are mostly subdomains for narrow usages. Most of the sites are open source applications. On AWS T2.large (2 processors, 8GB RAM). Storage is SSD. Single server hosts both apache and MySQL.

This set up has run really well for a couple of years. In June there was a period of about 10 days where server load shot up super high (100+) and I did some tuning but ultimately solved the problem by creating a new instance of the server. Chalked up to degraded hardware (?).

It started happening again about 10 days ago. The pattern is interesting. It's not totally consistent but there tends to be a spike in load every 30 minutes. I see no cron jobs that correlate with this. The load surges up to around 16-18. Sometimes when this happens, the server becomes unresponsive (not surprising) but it doesn't look like the proverbial "wedged server". When it goes unresponsive, the load decreases until it's well under 0.5, and stays there without recovery. Basically, it's dead. An apache restart doesn't bring it back to life, but an apache stop/apache start DOES bring it back. When it's in it's "dead" state, top shows that mysql uses about 0.2% of CPU. When it's under heavy load, mysql uses about 30-60% of 2 CPU cores (60-120% of a single core's CPU).

I can't think of anything that's changed since this started, other than a continual addition of data. The "big" database is about 1gb in size.

I tried creating a new instance for the server as in June, this time it didn't help.

I have attempted to do tuning, but with all the restarts it's hard to accumulate good data.

MYSQL has been running for a day, here's the output of tuning-primer.sh:

 -- MYSQL PERFORMANCE TUNING PRIMER --
      - By: Matthew Montgomery -

MySQL Version 5.7.27-0ubuntu0.16.04.1-log x86_64

Uptime = 1 days 5 hrs 33 min 50 sec
Avg. qps = 338
Total Questions = 36005313
Threads Connected = 1

Warning: Server has not been running for at least 48hrs.
It may not be safe to use these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service

SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 10.000000 sec.
You have 210572 out of 36005334 that take longer than 10.000000 sec. to complete
Your long_query_time seems to be fine

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.7/en/point-in-time-recovery.html

WORKER THREADS
Current thread_cache_size = 16
Current threads_cached = 15
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 300
Current threads_connected = 1
Historic max_used_connections = 41
The number of used connections is 13% of the configured maximum.
Your max_connections variable seems to be fine.

INNODB STATUS
Current InnoDB index space = 70 M
Current InnoDB data space = 195 M
Current InnoDB buffer pool free = 25 %
Current innodb_buffer_pool_size = 384 M
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory

MEMORY USAGE
Max Memory Ever Allocated : 1.75 G
Configured Max Per-thread Buffers : 2.58 G
Configured Max Global Buffers : 1.40 G
Configured Max Memory Limit : 3.98 G
Physical Memory : 7.79 G
Max memory limit seem to be within acceptable norms

KEY BUFFER
Current MyISAM index space = 853 M
Current key_buffer_size = 1.00 G
Key cache miss rate is 1 : 16883
Key buffer free ratio = 78 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere

QUERY CACHE
Query cache is enabled
Current query_cache_size = 16 M
Current query_cache_used = 11 M
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 72.74 %
Current query_cache_min_res_unit = 4 K
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = 256 K
Current read_rnd_buffer_size = 256 K
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 8.00 M
You have had 6245 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.

OPEN FILES LIMIT
Current open_files_limit = 3910 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_open_cache = 1800 tables
Current table_definition_cache = 2200 tables
You have a total of 2189 tables
You have 1792 open tables.
Current table_cache hit rate is 2%
, while 99% of your table cache is in use
You should probably increase your table_cache

TEMP TABLES
Current max_heap_table_size = 384 M
Current tmp_table_size = 384 M
Of 238038 temp tables, 17% were created on disk
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = 128 K
Current table scan ratio = 125 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 52
You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=ALWAYS'.

Apache has been running for .5 hours. That won't yield useful recommendations but does show configuration data. Here's the output of apache2buddy.pl:

#####################################################################
apache2buddy.pl report for ip-xxx-xxx-xxx-xxx.ec2.internal (xxx.xxx.xxx.xxx)
#####################################################################
[ OK ] This script is being run as root.
[ OK ] The utility 'pmap' exists and is available for use: /usr/bin/pmap
[ OK ] The utility 'netstat' exists and is available for use: /bin/netstat
[ OK ] 'php' exists and is available for use: /usr/bin/php
[ OK ] The utility 'apachectl' exists and is available for use: /usr/sbin/apachectl
[ OK ] The 'python' binary exists and is available for use: /usr/bin/python
[ OK ] The port (port 80) is a valid port.
[ -- ] We are attempting to discover the operating system type and version number ...
[ -- ] Distro: Ubuntu
[ -- ] Version: 16.04
[ -- ] Codename: xenial
[ OK ] This distro is supported by apache2buddy.pl.
[ OK ] This distro version is supported by apache2buddy.pl.
[ -- ] Hostname: ip-xxx-xxx-xxx-xxx.ec2.internal
[ -- ] Primary IP: xxx.xxx.xxx.xxx
[ -- ] We are checking the service running on port 80...
[ -- ] The process listening on port 80 is /usr/sbin/apache2
[ -- ] The process running on port 80 is Apache/2.4.18 (Ubuntu).
[ -- ] The full path to the Apache config file is: /etc/apache2/apache2.conf
[ -- ] Apache is using prefork model.
[ -- ] pidfile setting is /var/run/apache2/apache2$SUFFIX.pid.
[ -- ] Actual pidfile is /var/run/apache2/apache2.pid.
[ -- ] Parent PID: 14833.
[ OK ] Memory usage of parent PID is less than 50MB: 9712 Kilobytes.
[ -- ] Apache has been running 0d 0h 08m 55s.
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.
[ -- ] Your server has 7981 MB of PHYSICAL memory.
[ >> ] ServerLimit directive not found, assuming default values.
[ -- ] Your ServerLimit setting is 256.
[ -- ] Your MaxRequestWorkers setting is 19.
[ OK ] Current Apache Process Count is 16, including the parent PID.
[ -- ] Number of vhosts detected: 30.
[ -- ]             |________ of which 23 are HTTP (specifically, port 80).
[ -- ]             |________ of which 7 are HTTPS (specifically, port 443).
[ @@ ] Current Apache vHost Count is greater than maxrequestworkers, which is unusual, but can be valid in some scenarios.
[ -- ] Your MaxRequestsPerChild setting is 5000.
[ -- ] This server is NOT running Plesk.
[ -- ] This server is NOT running cPanel.
[ -- ] Virtualmin Version:  6.07.gpl
[ -- ] Webmin Version:  1.930
Use of uninitialized value $real_config in concatenation (.) or string at -
    line 1212 (#1)
    (W uninitialized) An undefined value was used as if it were already
    defined.  It was interpreted as a "" or a 0, but maybe it was a mistake.
    To suppress this warning assign a defined value to your variables.

    To help you figure out what was undefined, perl will try to tell you
    the name of the variable (if any) that was undefined.  In some cases
    it cannot do this, so it also tells you what operation you used the
    undefined value in.  Note, however, that perl optimizes your program
    and the operation displayed in the warning may not necessarily appear
    literally in your program.  For example, "that $foo" is usually
    optimized into "that " . $foo, and the warning will refer to the
    concatenation (.) operator, even though there is no . in
    your program.

Use of uninitialized value $apache_proc_php in concatenation (.) or string at -
    line 2366 (#1)
[ -- ] Your PHP Memory Limit (Per-Process) is .
Use of uninitialized value $apache_proc_php in string eq at - line 2367 (#1)
[ -- ] MySQL Detected => Using 1554.09 MB of memory.

[ OK ] No large log files were found in /var/log/apache2.
[ >> ] MaxClients has been hit recently (maximum of 5 results shown), consider the dates and times below:
[Tue Sep 17 02:01:26.796315 2019] [mpm_prefork:error] [pid 3161] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Tue Sep 17 02:35:48.069254 2019] [mpm_prefork:error] [pid 6394] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Tue Sep 17 02:56:25.712864 2019] [mpm_prefork:error] [pid 8685] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Tue Sep 17 03:32:59.609792 2019] [mpm_prefork:error] [pid 12435] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Tue Sep 17 03:52:05.692572 2019] [mpm_prefork:error] [pid 14833] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[ OK ] No PHP Fatal Errors were found.
Use of uninitialized value $_ in pattern match (m//) at - line 2172 (#1)
[ -- ] apache2 is currently using 932.67 MB of memory.
[ -- ] The smallest apache process is using 3.35 MB of memory
[ -- ] The average apache process is using 22.50 MB of memory
[ -- ] The largest apache process is using 29.73 MB of memory
[ OK ] Going by the average Apache process, Apache can potentially use 427.50 MB RAM:
        Without considering services: 5.36 % of total installed RAM
        Considering extra services: 6.66 % of remaining RAM
[ OK ] Going by the largest Apache process, Apache can potentially use 564.88 MB RAM:
        Without considering services: 7.08 % of total installed RAM
        Considering extra services: 8.79 % of remaining RAM


--------------------------------------------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: ip-xxx-xxx-xxx-xxx.ec2.internal (xxx.xxx.xxx.xxx):

Settings considered for this report:
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.

    Your server's physical RAM:                                   7981 MB
    Remaining Memory after other services considered:             6426 MB
    Apache's MaxRequestWorkers directive:                         19       <--------- Current Setting    
    Apache MPM Model:                                             prefork
    Largest Apache process (by memory):                           29 MB
[ !! ]  Vhost count exceeds MaxRequestWorkers limits!
[ !! ]  Your MaxRequestWorkers setting is too low.
    Your recommended MaxRequestWorkers setting is between 194 and 216.     <------- Acceptable Range (10% of MAX)
    Max potential memory usage:                                   564 MB
    Percentage of TOTAL RAM allocated to Apache:                  7.08  %
    Percentage of REMAINING RAM allocated to Apache:              8.79  %
--------------------------------------------------------------------------------
A log file entry has been made in: /var/log/apache2buddy.log for future reference.

Last 5 entries:

2019/09/14 18:49:19 Uptime: "0d 01h 16m 17s" Model: "Prefork" Memory: "7981 MB" MaxRequestWorkers: "45" Recommended: "96" Smallest: "3.32 MB" Avg: "31.03 MB" Largest: "72.06 MB" Highest Pct Remaining RAM: "46.70%" (40.64% TOTAL RAM)
2019/09/15 01:53:51 Uptime: "0d 05h 02m 19s" Model: "Prefork" Memory: "7981 MB" MaxRequestWorkers: "45" Recommended: "71" Smallest: "3.32 MB" Avg: "53.25 MB" Largest: "96.15 MB" Highest Pct Remaining RAM: "63.15%" (54.22% TOTAL RAM)
2019/09/15 22:24:37 Uptime: "0d 0h 25m 22s" Model: "Prefork" Memory: "7981 MB" MaxRequestWorkers: "19" Recommended: "150" Smallest: "3.32 MB" Avg: "21.83 MB" Largest: "49.91 MB" Highest Pct Remaining RAM: "12.61%" (11.89% TOTAL RAM)
2019/09/16 05:47:19 Uptime: "0d 04h 24m 25s" Model: "Prefork" Memory: "7981 MB" MaxRequestWorkers: "19" Recommended: "96" Smallest: "8.30 MB" Avg: "43.31 MB" Largest: "69.24 MB" Highest Pct Remaining RAM: "19.68%" (16.49% TOTAL RAM)
2019/09/17 04:00:53 Uptime: "0d 0h 08m 55s" Model: "Prefork" Memory: "7981 MB" MaxRequestWorkers: "19" Recommended: "216" Smallest: "3.35 MB" Avg: "22.50 MB" Largest: "29.73 MB" Highest Pct Remaining RAM: "8.79%" (7.08% TOTAL RAM)

Ubuntu 14.04 on Amazon AWS EC2, has been in production for several years as a web server.

I was working on implementing a new backup process, and while doing so the server ran out of disk space. A 300gb magnetic drive, it's usage was at 290gb when I started this project. While backing up a rather large account, the drive ran out of space but was still mostly operational. OK, so I started deleting files. I deleted a lot of "temp" files local to a few particular web sites, I deleted some working backups (.gz files) that amounted to several gb, and I also checked the /tmp system folder and looked to see if there were any .Trashes folders. I believe in total I deleted at least 10gb of files. However my system still reported zero free space. And MySQL wouldn't start (which I know requires free space to start and operate).

I rebooted the server, to no avail. I shut down apache to make sure it wasn't holding open files. Eventually I expanded the disk drive to 350gb, and then everything started working. The server currently reports 298gb used. Should be much less than that, given all that I deleted.

1) Why would the server not recognize the space I was freeing up? 2) How can I get it to see it now?

Here is the output of df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            1.8G   12K  1.8G   1% /dev
tmpfs           358M  200K  358M   1% /run
/dev/xvda1      345G  283G   47G  86% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            1.8G     0  1.8G   0% /run/shm
none            100M     0  100M   0% /run/user
overflow        1.0M   76K  948K   8% /tmp

I am configuring a new web application server running Ubuntu 16.04. The only thing "unusual" I have done is install php 5.6 instead of php7. Php5.6 however does require a valid root certificate and I am getting errors in a variety of apps that care about this. Curl, cron, etc.

I have tried a variety of fixes to get this resolved with no improvement. Such as:

https://github.com/composer/composer/issues/3346#issuecomment-76593763

https://stackoverflow.com/questions/35821245/github-server-certificate-verification-failed/35824116#35824116

How do I install a root certificate?

Still no luck. The cron job I am running still generates:

curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option.

If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL).

If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.

On Ubuntu 12.04. This is a very old utility AWS server that has always run reliably. Today I notice that MySQL isn't running. I did a reboot just to be "clean". When I try and issue:

/etc/init.d/mysql start >/dev/null 2>&1 &

I get:

[1] 10836

Not sure what this means. I notice that there is a pending software update for w3m and so I use apt-get -y install w3m ..

and get

dpkg: dependency problems prevent configuration of mysql-server-5.5:
 mysql-server-5.5 depends on mysql-server-core-5.5 (= 5.5.46-0ubuntu0.12.04.2); however:
  Version of mysql-server-core-5.5 on system is 5.5.54-0ubuntu0.12.04.1.
dpkg: error processing mysql-server-5.5 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.5; however:
  Package mysql-server-5.5 is not configured yet.
dpkg: error processing mysql-server (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 mysql-server-5.5
 mysql-server
Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 mysql-server-5.5 : Depends: mysql-server-core-5.5 (= 5.5.46-0ubuntu0.12.04.2) but 5.5.54-0ubuntu0.12.04.1 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

I've tried googling this issue and these results but see no solutions. I've checked the MySQL error logs, and they are all zero bytes.

What's up?

I have an Ubuntu 12.04 server running on Amazon, it's a well seasoned instance that's been running for several years. This is not a shared hosting system, the whole thing is ours. We just started getting error messages relating to disk space, such as:

Warning: session_start(): open(/var/lib/php5/sess_6q18snb4mk50dlige4vb8ch2r3, O_RDWR) failed: No space left on device (28)

and

SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql_370_0.MYI' (Errcode: 28)

when I check the file system, I get:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      197G  165G   23G  89% /
udev            7.4G  8.0K  7.4G   1% /dev
tmpfs           1.5G  176K  1.5G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.4G     0  7.4G   0% /run/shm

Overall there's plenty of space left. I don't see any partitions in particular trouble. What's the problem?