Fabio's questions

I'm having an issue with a linux server. Once a week the running mysql instance hangs and there is no way to fully stop it. If I kill it, it remains in zombie status and init does not reap its pid.

The server is used for staging deployments and some internal tools, so it's not under heavy load. The only process constantly used id mysql and for this I think that it's the only process which suffer of this issue.

I've searched system logs for errors and the only thing I found is this error (repeated a couple of times) in dmesg output:

[706560.640085] INFO: task mysqld:31965 blocked for more than 120 seconds.
[706560.640198] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[706560.640312] mysqld          D ffff88032fd93f40     0 31965      1 0x00000000
[706560.640317]  ffff880242a27d18 0000000000000086 ffff88031a50dd00 ffff880242a27fd8
[706560.640321]  ffff880242a27fd8 ffff880242a27fd8 ffff88031e549740 ffff88031a50dd00
[706560.640325]  ffff88031a50dd00 ffff88032fd947f8 0000000000000002 ffffffff8112f250
[706560.640328] Call Trace:
[706560.640338]  [<ffffffff8112f250>] ? __lock_page+0x70/0x70
[706560.640344]  [<ffffffff816cb1b9>] schedule+0x29/0x70
[706560.640347]  [<ffffffff816cb28f>] io_schedule+0x8f/0xd0
[706560.640350]  [<ffffffff8112f25e>] sleep_on_page+0xe/0x20
[706560.640353]  [<ffffffff816c9900>] __wait_on_bit+0x60/0x90
[706560.640356]  [<ffffffff8112f390>] wait_on_page_bit+0x80/0x90
[706560.640360]  [<ffffffff8107dce0>] ? autoremove_wake_function+0x40/0x40
[706560.640363]  [<ffffffff8112f891>] filemap_fdatawait_range+0x101/0x190
[706560.640366]  [<ffffffff81130975>] filemap_write_and_wait_range+0x65/0x70
[706560.640371]  [<ffffffff8122e441>] ext4_sync_file+0x71/0x320
[706560.640376]  [<ffffffff811c3e6d>] do_fsync+0x5d/0x90
[706560.640379]  [<ffffffff811c40d0>] sys_fsync+0x10/0x20
[706560.640383]  [<ffffffff816d495d>] system_call_fastpath+0x1a/0x1f

When this happens the only way to make everything working again is a full reboot, but in order to do that I'm forced to use this command after I've manually stopped all running processes

echo b > /proc/sysrq-trigger

otherwise normal reboot process hangs forever. I've tracked reboots script and I've found out that also the reboot process hangs on a sync call, this one in /etc/init.d/sendsigs (I'm on ubuntu)

# Flush the kernel I/O buffer before we start to kill
# processes, to make sure the IO of already stopped services to
# not slow down the remaining processes to a point where they
# are accidentily killed with SIGKILL because they did not
# manage to shut down in time.
sync

I'm almost sure that the cause of this is an hardware issue (the RAID controller???) also because I've other two machines with the same hardware and software configuration and they don't suffer of this, but I can't find any hint in syslog or dmesg. I've also installed smartmontools and mcelog packages but none of them did report any issue.

What can I do to track the cause of this issue?

Today is happened again, here is the status of system after triggering a reboot

init─┬─console-kit-dae───64*[{console-kit-dae}]
     ├─dbus-daemon
     ├─mcelog
     ├─mysqld───{mysqld}
     ├─newrelic-daemon───newrelic-daemon───11*[{newrelic-daemon}]
     ├─ntpd
     ├─polkitd───{polkitd}
     ├─python3
     ├─rpc.idmapd
     ├─rpc.statd
     ├─rpcbind
     ├─sh───rc───S20sendsigs───sync
     ├─smartd
     ├─snmpd
     ├─sshd───sshd───zsh───sudo───zsh───pstree
     └─sshd───sshd───zsh───sudo───zsh

And here is the status of sync process

# ps aux | grep sync
root      3637  0.1  0.0   4352   372 ?        D    05:53   0:00 sync

i.e. Uninterruptible sleep...

Hardware specs as reported by lshw

I think the raid controller is a fake raid. I usually don't deal with hardware (and for the record I don't have physical access to it)

description: Computer
product: X7DBP ()
vendor: Supermicro
version: 0123456789
serial: 0123456789
width: 64 bits
capabilities: smbios-2.4 dmi-2.4 vsyscall32
configuration: administrator_password=disabled boot=normal frontpanel_password=unknown keyboard_password=unknown power-on_password=disabled uuid=53D19F64-D663-A017-8922-0030487C1FEE
*-core
   description: Motherboard
   product: X7DBP
   vendor: Supermicro
   physical id: 0
   version: PCB Version
   serial: 0123456789
 *-firmware
      description: BIOS
      vendor: Phoenix Technologies LTD
      physical id: 0
      version: 6.00
      date: 05/29/2007
      size: 106KiB
      capacity: 960KiB
      capabilities: pci pnp upgrade shadowing escd cdboot bootselect edd int13floppy2880 acpi usb ls120boot zipboot biosbootspecification

  *-storage
         description: RAID bus controller
         product: 631xESB/632xESB SATA RAID Controller
         vendor: Intel Corporation
         physical id: 1f.2
         bus info: pci@0000:00:1f.2
         version: 09
         width: 32 bits
         clock: 66MHz
         capabilities: storage pm bus_master cap_list
         configuration: driver=ahci latency=0
         resources: irq:19 ioport:18a0(size=8) ioport:1874(size=4) ioport:1878(size=8) ioport:1870(size=4) ioport:1880(size=32) memory:d8500400-d85007ff

We're moving dns servers (managed with bind) for our main domain (let's call it main.org) to new servers. Since we're having some issues to update glue records with current registrar we thought to change authoritative dns servers to another domain (that we directly manage)

Current setup is

dig -t ns main.org
;; ANSWER SECTION:
main.org.       140 IN  NS  ns1.main.org.
main.org.       140 IN  NS  ns2.main.org.

We don't directly manage glue records for ns1.main.org and ns2.main.org, so we'd like to change in this way

dig -t ns main.org
;; ANSWER SECTION:
main.org.       140 IN  NS  ns1.secondary.org.
main.org.       140 IN  NS  ns2.secondary.org.

in order to manage better A records for ns1.secondary.org and ns2.secondary.org.

However currently s1.secondary.org and ns2.secondary.org are not authoritative dns for secondary.org zone and my registrar doesn't allow me to do the change for this reason.

Is this a strict requirement for DNS delegation?

Or in other words must ns1.secondary.org and ns2.secondary.org be authoritative for secondary.org zone in order to be authoritative also for main.org?

More details

When I tried to insert ns1.secondary.org in the control panel for main.org (provided by registrar) I got an error saying that "ns1.secondary.org could not be added as nameserver for main.org". I opened a ticket with my reseller (it's an Enom reseller account) which in turn has forwarded the question to Enom and here is their answer:

Enom told me that the DNS servers "ns1.secondary.org" and "ns2.secondary.org" must be registered in your registrar as name servers. Once you do that, either you or us can update the DNS servers for the domain main.org

When running a vagrant instance the project folder is mounted on /vagrant automatically. However is mounted with the following permissions

# ll -d /vagrant
drwx------ 1 vagrant vagrant 612 Jun 13 14:41 /vagrant/

I need it to be mounted with (at least) 0770 but I can't find how. If I run the mount command I see this output

# mount
v-root on /vagrant type vboxsf (uid=1000,gid=100,rw)

I've tried both chmod and chown/chgrp, but they won't work on that mounted folder so my apache user can't access that folder. I read in Vagrant manual that I can change owner and group but it doesn't mention nothing about permission.

How can I do that?

Another option could be switch to NFS but in this way it won't work on Windows platforms and it need to edit local /etc/exports file and it would require root privileges and also it's pretty annoying, so I'd prefer to not make this change.

I currently have the following network configuration

• Apple Time Capsule (Gigabit Ethernet router)
• 2 Windows desktop wired with Gigabit ethernet
• 1 MacBook Pro in wireless N mode
• 1 iPhone (obviously wireless)

I bought a NAS and a Gigabit switch to extend my network and the nas asks me if I want to enable jumbo frames for the network connection. So my question, could I?

I think if I do that I will have problems for wireless clients and I need to reconfigure both the desktops and the time capsule to use jumbo frames.

Should I add a router instead of a switch to segment the network in two parts in order to use jumbo frames?

Does this have any sense? Will performance improve with jumbo frames?

I'm going crazy to solve this. I have a mysql installation (on machine db.example.org) which doesn't resolve a given hostname. I gave privileges using hostnames i.e.

GRANT USAGE ON *.* TO 'user'@'host1.example.org' IDENTIFIED BY PASSWORD 'secret'
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, INDEX ON `my_database`.* TO 'user'@'host1.example.org'

However when I try to connect using mysql -u user -p -h db.example.org I obtain

ERROR 1045 (28000): Access denied for user 'user'@'192.168.11.244' (using password: YES)

I already checked for correct name resolution in the dns system:

$ dig -x 192.168.11.244
;; ANSWER SECTION:
244.11.168.192.in-addr.arpa. 68900 IN   PTR host1.example.org.

I've also checked for skip-name-resolve option in mysql variables in fact if I can access from another machine on the same subnet using hostname privileges.

The only difference is that host1.example.org and db.example.org point the same ip on the same machine i.e. both db.example.org and host1.example.org have ip 192.168.11.244. In this way all the applications using that database can use the name db.example.org and we can move the data on other hosts (if needed) just by changing the dns record, leaving the application code unchanged.

What should I do to solve this or at least to understand what's happening?

I need to find all the instances of a given string in the whole filesystem, because I don't remember in which configuration files, script or any other programs I put it and I need to update that string with a new one.

I tried with the following command

`grep -nr 'needle' / --exclude-dir=.svn | mail [email protected] -s 'References on xxx'

If I run this command on a small directory it gives me the output I need in the form

/path1/:nn:line containing needle
/path2/:nn:line containing needle

where /path1 is the full path of the file, nn is the row containing the needle and last field is the content of the line.

However when I run the command on the root directory the grep process hang after a while. I run this script about 8 hours ago and even on a small filesystem (less than 5GB) it doesn't end and if I run top or ps the process seems sleeping

root 24909 0.0 0.1 3772 1520 pts/1 S+ Feb10 0:15 grep -nr needle / --exclude-dir=.svn

Why it doesn't end? Is there any better way to do this (it's a one time job, I don't need to execute this more than once)

Thanks.

Update: I found a working solution with find and xargs which seems to be working and uses less system resources than the find -exec solution. Here is my final command line:

find /{boot,etc,home,lib,lost+found,opt,root,sbin,usr,var} -type f -print0 | xargs -r0 grep -nr 'needle' | mail [email protected] -s 'References on xxx'

I used the /{dir1,dir2,...,dirn} syntax because I want the output lines with the fullpath including the leading slash, and in this way you can use the command without cd / i.e. from any directory.

I just upgraded my system from Lenny to Squeeze following the steps posted here. I followed all the steps using apt-get (in the upgrade from Etch to Lenny i used aptitude as suggested) and the upgrade went fine.

In the daily routine I use aptitude to maintain the system up to date and I really like the automatic handling of unused packages, so if I install package A that depends on B when I remove A, B is removed too keeping my system clean.

My question is the following: do apt-get and aptitude share the automatic handling of packages? I don't think so, because I found a lot of packages not marked as Auto in aptitude after the upgrade via apt-get.

Am I right? How can I fix this in automated way if the answer is yes?