jack's questions

I have a few servers with Dual E5-2680 v2 CPUs on Ubuntu 18.04. Each E5-2680 v2 has 10 physical cores or 20 threads. That being said, running "cat /proc/cpuinfo | grep processor | wc -l" should show 40.

However, some servers shows only 20. For these servers, output of command

cat /proc/cpuinfo | grep "physical id"

physical id : 0
physical id : 0
physical id : 0
physical id : 0
physical id : 0
physical id : 0
physical id : 0
physical id : 0
physical id : 0
physical id : 0
physical id : 1
physical id : 1
physical id : 1
physical id : 1
physical id : 1
physical id : 1
physical id : 1
physical id : 1
physical id : 1
physical id : 1

user@hostname:~# lscpu

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              20
On-line CPU(s) list: 0-19
Thread(s) per core:  1
Core(s) per socket:  10
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               62
Model name:          Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Stepping:            4
CPU MHz:             1200.814
CPU max MHz:         3600.0000
CPU min MHz:         1200.0000
BogoMIPS:            5600.17
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            25600K
NUMA node0 CPU(s):   0-9
NUMA node1 CPU(s):   10-19
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts flush_l1d

Do you have an idea what could cause this problem? Thanks.

My MySQL server run to crash due to (page file write error). I did a disk diagnosis but no errors found. I restarted MySQL server. It scanned the bin log like for 1-2 hours and recorded log as below:

InnoDB: Doing recovery: scanned up to log sequence number 51 2341175808
InnoDB: Doing recovery: scanned up to log sequence number 51 2346418688
InnoDB: Doing recovery: scanned up to log sequence number 51 2351661568
InnoDB: Doing recovery: scanned up to log sequence number 51 2356904448
InnoDB: Doing recovery: scanned up to log sequence number 51 2362147328

After that, I see a lot of errors as below:

InnoDB: Number of pending reads 128, pending pread calls 0
InnoDB: Error: InnoDB has waited for 50 seconds for pending
InnoDB: reads to the buffer pool to be finished.
InnoDB: Number of pending reads 128, pending pread calls 0
InnoDB: Error: InnoDB has waited for 50 seconds for pending
InnoDB: reads to the buffer pool to be finished.
InnoDB: Number of pending reads 128, pending pread calls 0
InnoDB: Error: InnoDB has waited for 50 seconds for pending
InnoDB: reads to the buffer pool to be finished.

I have wait a few hours but it just adds those lines to error log repeatly with no sign to stop.

What does above log messages mean? How can I make my MySQL server running again? I have a 80GB InnoDB database running on this server. Is there a way to force recover it without undo any uncommited transactions or try to recover the data from page file which caused the crash? I have no problem drop those data.

I tried "sudo -u mysql /usr/sbin/mysqld –innodb_force_recovery=6" to restart MySQL server but got new repeated errors as below:

InnoDB: stored checksum 2440779633, prior-to-4.0.14-form stored checksum 3425185587
InnoDB: Page lsn 51 2450779673, low 4 bytes of lsn at page end 2450779673
InnoDB: Page number (if stored to page already) 10824,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 4294967295 0
InnoDB: (index "CLUST_IND" of table "SYS_IBUF_TABLE_0")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 10824.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.

What should I do now?

It's an InnoDB database. The query cache has 1.2GB in size with 300k queries in the cache. Most of the cached queries are related to 3 major tables (260MB data length plus 600MB index length altogether). Everytime I tried to insert new entry or update existing entries in these 3 tables, MySQL slowed down a lot just like the server is dead. I got "Timeout" if I open my website in browser.

I check the Munin stats, it shows 95% of the query cache are cleared after the operation so I guess the sudden overload was caused by inserting entries to heavy-read tables. And during the overload, Munin just shows a white strip on the graph so you may get an idea on how bad the situation was.

How can I optimize this?

I killed a MySQL process which is running a LOAD DATA INFILE command to load a 9GB .csv file into a table in a InnoDB database.

Now, when I run mysql DATABASE, it just stopped and showed

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

What can I do now? I guess it's doing some repair and cleanning work caused by the interrupted LOAD DATA INFILE command.

Solution:

I tried service mysql restart but I decided to kill all mysqld processes forcibly after restarting took more than 1 hours and still no clue how many more hours ahead to complete.

Then I used service mysql start to run MySQL server again. Fortunately the InnoDB rebuild process is quite fast, took about 15 minutes to complete, no data loss.

How can I install python-psycopg (not python-psycopg2) on Ubuntu 9.10

"apt-get install python-psycopg" returns "Package python-psycopg has no installation candidate"

I also downloaded source code at psycopg-1.1.21.tar.gz but "make" command was no found in the archive.

I tried to change collation MySQL table to utf8_general_cs but got following error:

mysql> ALTER TABLE table_name CONVERT TO CHARACTER SET utf8 COLLATE 'utf8_general_cs';
ERROR 1273 (HY000): Unknown collation: 'utf8_general_cs'

I run "SHOW COLLATE" command and "utf8_general_cs" is not in the results.

What can I do now?

I have a munin-master and munin-node installed on same server (Ubuntu 9.10 server).

Most pre-installed plugins work well but the following plugins are with blank graph:

Memcached bytes used
Memcached connections
Memcached cache hits and misses
Memcached cached items
Memcached requests
Memcached network traffic
MySQL Queries Cache Size

I run the following 3 script in terminal and results were ok.

/etc/munin/plugins/memcached_bytes
/etc/munin/plugins/memcached_counters
/etc/munin/plugins/memcached_rates

But when I tried the command below after "telnet localhost 4949"

fetch memcached_bytes 
# Unknown service
etch memcached_bytes_
# timeout pid 28009 - killing...done

Does anyone know the reason?

Is that possible to bind all available public IP addresses on a server to one Postfix instance as its outgoing IP address pool and let it choose a random IP address or specified IP address from the pool each time it sends out an email?

If above is not possible, can it be configured to listen on one public IP address per instance and each time it delivers a message, it use the binded one as outgoing IP address?

I used rewrite directives below in nginx to rewrite urls of static files to external CDN server.

rewrite ^/static/(css|images|js)/([a-z_\-\.]+)$ http://cdn.domain.com/$1_$2 last;

It works but it redirect the url automatically in browsers.

How can I do the above rewrite without redirects?

I just added the following lines in /etc/mysql/my.cnf after I converted one database to use InnoDB engine.

innodb_buffer_pool_size = 2560M
innodb_log_file_size    = 256M
innodb_log_buffer_size  = 8M
innodb_flush_log_at_trx_commit  = 2
innodb_thread_concurrency   = 16
innodb_flush_method = O_DIRECT

But it raise "ERROR 2013 (HY000) at line 2: Lost connection to MySQL server during query" error restarting mysqld. And mysql error log shows the following

InnoDB: Error: log file ./ib_logfile0 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 268435456 bytes!
100118 20:52:52 [ERROR] Plugin 'InnoDB' init function returned error.
100118 20:52:52 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
100118 20:52:52 [ERROR] Unknown/unsupported table type: InnoDB
100118 20:52:52 [ERROR] Aborting

So I commented out this line

# innodb_log_file_size  = 256M

And it restarted mysql successfully.

I wonder what's the "5242880 bytes of log file" showed in mysql error? It's the first database on InnoDB engine on this server so when and where is that log file created? In this case, how can I enable innodb_log_file_size directive in my.cnf?

EDIT

I tried to delete /var/lib/mysql/ib_logfile0 and restart mysqld but it still failed. It now shows the following in error log.

100118 21:27:11  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 256 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100 200
InnoDB: Error: log file ./ib_logfile1 is of different size 0 5242880 bytes
InnoDB: than specified in the .cnf file 0 268435456 bytes!

Resolution

It works now after deleted both ib_logfile0 and ib_logfile1 in /var/lib/mysql

I found a problem using libicns's icontainer2icns command, it doesn't have an option to specify output folder but just put all extracted .icns files in current working directory.

For example, there is a file named "source.icontainer" under "/home/user/test"

user@localhost:/home/user$ icontainer2icns test/source.icontainer

Running above command will resulted in all .icns files be extracted to /home/user but not /home/user/test folder.

Is there a way to redirect all extracted files to whatever folder i specify?

I have installed ImageMagick 6.5.8 from source on Ubuntu 8.10 followed by the instructions listed here

> tar xvfz ImageMagick.tar.gz
> cd ImageMagick-6.5.8
> ./configure
> make
> sudo make install

Installation was successful but when I try to use 'liquid-rescale' operator it showed:

> convert source.png -liquid-rescale 500x500 output.png
convert: delegate library support not built-in `source.png' (LQR) @ resize.c/LiquidRescaleImage/1550.

Looks like it was compiled without liquid-rescale support.

I wonder how to turn on liquid-rescale when installing ImageMagick from source.

Thanks in advance.

I'm looking for a flexible bash script to do the following:

1. Convert .rar, .tar, .tar.gz, .bz2, .7z archives to .zip format
2. Keep all folder structures and filenames as source archive.
3. Convert it quietly, outputs "error" on failure and outputs "encrypted" on password protected archived.

Thanks in advance.

How to find out what charset encoding is used by current file system and how to change it to UTF-8?

EDIT:

Here is the output of mount:

/dev/sdb6 on / type ext3 (rw,relatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
/proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,nosuid,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
lrm on /lib/modules/2.6.27-11-generic/volatile type tmpfs (rw,mode=755)
securityfs on /sys/kernel/security type securityfs (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev)

Here is the output of "cat /etc/fstab"

# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# /dev/sdb7
UUID=50d660f1-1948-41e1-96af-3cb9bca338dd /               ext3    relatime,errors=remount-ro 0       0
# /dev/sdb8
UUID=efaee412-8e29-4f65-927d-f57252451088 none            swap    sw              0       0

I'm using Ubuntu. When I tried to create a new directory under a folder with more than 30,000 subfolders, it displayed following error message:

mkdir: cannot create directory `foldername': Too many links

How can I raise the limit of subfolders number in Ubuntu?

I consistently come up with "You have new mail in /var/mail/root" message in command line mode of my Ubuntu server. How to disable the new mail messages?

Currently MySQL server 5.4 is still in beta stage and not available in Ubuntu repository. Is it worth to build from source and upgrade to version 5.4? What's the pros and cons of version 5.4 compared to 5.1?

Edit: I read what's new document. It claims performance improvement on multi-core CPU, but mainly for InnoDB tables. I'm curious how much improvement it could make for basic SELECT/INSERT/UPDATE/DELETE queries, SELECT COUNT(*) queries and AVG(), SUM(), MAX(), MIN() aggregation queries on MyISAM tables.

It takes some time to install the new version and dump/load database to do a benchmark. So I would appreciate you share some of your experience if you have tried the new version.

From your experience, what's the upper limit of rows in a MyISAM table can MySQL handle efficiently on a server with Q9650 CPU (4-core, 3.0G) and 8G RAM.

I currently have a table with 15 million rows. It's pretty fast. If the scale increases to 1 billion rows, do I need to partition it into 10 tables with 100 million rows each?

Supposed local folder as below:

/test/subfolder
/test/subfolder/file1.txt
/test/subfolder/page1.htm
/test/subfolder/page2.htm
/test/.hiddenfolder
/test/./hidenfolder/file2

1. How to exclude .hiddenfolder to be transfer when executing "scp -r test user@ip:/"?

2. How to include *.htm files only and transfer them to corresponding subfolder on remote server?

3. Any other commands can do this job more easily?

How to install a Windows XP or Windows Server 2003 virtual machine on a remote Ubuntu server by command line?

Also, please advise what virtualization software is best for this kind of job?