Bruno Reis's questions

If I run fsfreeze on a mounted filesystem, it will not be possible to perform new "write" operations on that FS until it is unfrozen, and "ongoing" writes should "complete".

Now, will fsfreeze also flush the FS caches, after it starts blocking new writes, and after it completes those ongoing writes, and before it returns? Can you point me to any authoritative source, please?

I've spent time reading the docs, but couldn't find this explicitly mentioned anywhere. The closest I find is "creates a stable image on disk" -- does this mean that it flushes the FS caches?

Thanks.

I'm having some trouble connecting to a MySQL 5.1 server on an RDS instance on AWS from my laptop.

The detailed description of the problem is here: https://forums.aws.amazon.com/thread.jspa?messageID=323397

In short: I have 2 MySQL servers, both with the same db configuration and firewall (security group) configuration. One of them works fine: I can connect to it from my EC2 instances (ie, from inside the AWS cloud) and from my laptop. The other one doesn't: I can connect from my EC2 instances but not from my laptop.

The symptom: a connection attempt from my laptop just hangs, and then times out, as if there was a firewall blocking me (ie, silently dropping my SYN packets).

I must say that everything has been working fine for a very long time, and this problem began suddenly, 3 days ago, without any modifications to DB parameters or the security groups.

My current analysis of the situation:

• The firewall (ie, security group) cannot be the problem: both MySQL servers share the same firewall configuration -- I can connect to one of them but not to the other. Later on, I even added a rule to allow inbound connections from 0.0.0.0/0 (ie, I turned off the firewall), and nothing. Oh, I also created a new, fresh security group and changed this instance's SG to the new one (to which I first added my ip address, and then 0.0.0.0/0) but still nothing.
• The credentials cannot be the problem: I use the same from my laptop and from my EC2 instances -- and the user (which is what Amazon calls master user), in the database, has a host of '%'.
• MySQL is not blocking my IP due to, say, too many failed connection attemps: I've FLUSH HOSTS on the database, and also I tried to connect using many different source IP addresses, even from all around the world through a VPN proxy service.

What could I be missing?

I'm asking here because it's been about 36 hours since I've posted on AWS forums but got no answer at all over there... someone here might have a solution!

Any input is really appreciated, I'm out of ideas. Thanks!

EDIT: after many hours, I've found out that the problem has nothing to do with Eucalyptus. It looks like the image is buggy. Very, very buggy. More details in the end. I didn't manage to fix it, and I will file a bug.

EDIT 2: I managed to fix it, it apparently works.

I have a 4-machine cluster running Ubuntu Server Natty (11.04) x64. I've installed "Ubuntu Enterprise Cloud" from the installtion CD (then updated it) on each of these machines. The cloud seems to work fine, I have lots of virtual machines running Natty servers on them.

Now I'd like to run Oneiric in a virtual machine, but somehow I can't.

I downloaded Oneiric's (x64) image from http://cloud-images.ubuntu.com/oneiric/current/, published it (uec-publish-tarball oneiric-server-cloudimg-amd64.tar.gz oneiric-server-cloudimg-amd64) exactly as I did with Natty, then tried to launch an instance (euca-run-instances -n 1 -k my-key -t m1.small -z my-cloud emi-XXXXXXXX) using Oneiric's image, but the instance is not able to boot.

With euca-get-console-output I get the following:

[    0.461269] VFS: Cannot open root device "sda1" or unknown-block(0,0)
[    0.462388] Please append a correct "root=" boot option; here are the available partitions:
[    0.463855] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.465331] Pid: 1, comm: swapper Not tainted 3.0.0-13-generic #22-Ubuntu
[    0.466526] Call Trace:
[    0.466989]  [<ffffffff815d3ee5>] panic+0x91/0x194
[    0.467860]  [<ffffffff81ad1031>] mount_block_root+0xdc/0x18e
[    0.468891]  [<ffffffff81ad126a>] mount_root+0x54/0x59
[    0.469829]  [<ffffffff81ad13dc>] prepare_namespace+0x16d/0x1a7
[    0.470883]  [<ffffffff81ad0d76>] kernel_init+0x140/0x145
[    0.471837]  [<ffffffff815f38e4>] kernel_thread_helper+0x4/0x10
[    0.472889]  [<ffffffff81ad0c36>] ? start_kernel+0x3df/0x3df
[    0.473884]  [<ffffffff815f38e0>] ? gs_change+0x13/0x13

The filesystem is labeled "cloudimg-rootfs", inside the image both /etc/fstab and /boot/grub/grub.cfg always refer to the image by the label, everything seems to be correct, yet the kernel says it can't find the root file system.

I've spent many hours googling, but nothing came out. I've asked on #ubuntu-server, but nobody knew what to do. I've asked on #eucalyptus but got no answer at all.

Any ideas on why this is happening and how to solve it?

Thanks

EDIT: after many hours, I've found out that the problem has nothing to do with Eucalyptus. It looks like the image is buggy. Very, very buggy.

The first problem is that the Kernel in the image is a -generic kernel, while I suppose it should be a -virtual one. I chrooted into the image, removed the -generic packages, replaced it with the -virtual ones. Then I extracted the new kernel (and replaced the original one (-generic) that came with the tarball) because I need it when I publish and launch an image with Eucalyptus.

The problem described above was solved.

But then, the console started showing this:

mount: mount point ext4 does not exist

If you check the /etc/fstab file in the image, it says:

LABEL=cloudimg-rootfs  ext4 defaults    0   1

Damnt, where's my mount point? Note that it is missing /proc as well.

Well, when you think it is over, you will notice that your instance will have no network connectivity. Let's check /etc/network/interface:

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

Oh my! It is missing eth0... here I stopped. I can't take no more. I give up.

Looks like Canonical has just forgotten to properly set up this image. At first, I though: "have I downloaded a server image by mistake?", but no, I double checked. It is really the cloud image, it has even "cloud-init" installed (which is not, by default, on server images). They just forgot to prepare it.

I will file a bug (and reference it here once this is done), and hope they fix it soon!

EDIT 2: it looks like the network configuration was the last thing missing. I decided to test it with the fixes above, and it booted properly! However, I haven't got the slightest idea if the image is now good to go...

I'm having trouble with my cloud installation.

I've installed everything from the Ubuntu Server 11.04 CD, then upgraded the system to the latest version (apt-get update && apt-get dist-upgrade), installed NTP, correctly set it up, and booted everything. The cloud runs ok for sometime, but then the cloud controller / cluster controller will stop talking to one of the node controllers!

One of the symptoms is euca-describe-availability-zones verbose suddenly starts showing less resources than the appropriate (and initial) values.

If I look on cc.log, it shows:

[Mon Sep 19 19:09:07 2011][002531][EUCADEBUG ] DEBUG: requested URI http://10.20.200.10:8775/axis2/services/EucalyptusNC
[Mon Sep 19 19:09:07 2011][002531][EUCADEBUG ]  ncClientCall(ncDescribeResource): ppid=13403 client calling 'ncDescribeResource'
[Mon Sep 19 19:09:07 2011][002531][EUCAERROR ] ERROR: DescribeResource() could not be invoked (check NC host, port, and credentials)
[Mon Sep 19 19:09:07 2011][002531][EUCADEBUG ]  ncClientCall(ncDescribeResource): ppid=13403 done calling 'ncDescribeResource' with exit code '1'

Then, in the corresponding node controller, axis2c.log, I see this:

[Mon Sep 19 19:10:58 2011] [error] rampart_timestamp_token.c(179) [rampart]Timestamp not valid: Created time is not valid
[Mon Sep 19 19:10:58 2011] [error] rampart_sec_header_processor.c(612) [rampart]Timestamp is not valid
[Mon Sep 19 19:10:58 2011] [error] rampart_sec_header_processor.c(1911) [rampart]Timestamp processing failed
[Mon Sep 19 19:10:58 2011] [error] rampart_in_handler.c(143) [rampart][rampart_in_handler] Security Header processing failed.
[Mon Sep 19 19:10:58 2011] [error] phase.c(233) Handler RampartInHandler invoke failed within phase Security
[Mon Sep 19 19:10:58 2011] [error] engine.c(696) Invoking phase Security failed
[Mon Sep 19 19:10:58 2011] [error] engine.c(279) Invoking operation specific phases failed for operation ncDescribeResource
[Mon Sep 19 19:10:58 2011] [error] rampart_engine.c(159) [rampart][rampart_engine] Cannot get saved rampart_context
[Mon Sep 19 19:10:58 2011] [error] rampart_out_handler.c(136) [rampart][rampart_out_handler] ramaprt_context creation failed.
[Mon Sep 19 19:10:58 2011] [error] phase.c(233) Handler RampartOutHandler invoke failed within phase MessageOut
[Mon Sep 19 19:10:58 2011] [error] engine.c(696) Invoking phase MessageOut failed

So: there's a time synchronization problem.

However, NTP is installed, and correctly configured.

One thing I've noticed, by issuing lots of ntpq -np, is that the NC stops working once the time offset is positive. If the offset keeps negative, everything works fine. The offsets are very small (around 5ms, the absolute maximum I could see is 10ms).

Googling, I've found this rampart code: http://wso2.org/project/wsf/php/1.1.0/docs/code-coverage/rampartc/src/util/.libs/rampart_timestamp_token.c.gcov.html

/*Check whether created is less than current time or not*/
  current_val = rampart_generate_time(env, 0);
  validity = rampart_compare_date_time(env, current_val, created_val);
  if (validity == AXIS2_SUCCESS)
  {
      AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI, "[rampart][ts]Timestamp not valid: Created time is not valid");
              AXIS2_FREE(env->allocator, current_val);
              current_val = NULL;
      return AXIS2_FAILURE;
  }

As we can see, it apparently allows for deviation in time in one sense, but not in the other.

Am I missing something? Am I the only facing this issue? Isn't it stupid to verify timestamps with milissecond precision and allow only negative deviations?!

How do people deal with this problem? What can I do to keep my cloud alive?

I thought of some solutions:

1. Patch rampart, to simply remove the timestamp verification
2. Patch rampart, to allow for positive deviations as well
3. Find a way to make ntp or ntpdate adjust the time to some specific offset behind the server's reference time
4. Write my own time synchronization tool

What do you think?

EDIT: it looks like it is possible to disable Rampart in Axis2 configuration, but I can't figure out how to do that!

EDIT 2: The version of Rampart available on Ubuntu's repositories is 1.3.0, which is from 2007 or 2008... the latest released version is something like 1.6.0, from June 2011. Apparently this latest version allows packets "from the future". I'd really like to find this latest version from a PPA!

EDIT 3: I've found some parameters to change Rampart 1.3.0's behaviour: TimeToLive, ClockSkewBuffer and PrecisionInMilliseconds. I've added them (360, 60 and false, respectively) to EucalyptusNC.xml and EucalyptusCC.xml, and things are getting better. Occasionally I still see the timestamp error messages on the logs, but they are very rare now. I've also disabled NTP on the NCs and created a cron script (that runs every hour) to sync time (ntpdate -b) with the CC.

EDIT 4: Apparently this is a bug in Ubuntu's packaging of Eucalyptus. I've filed a bug on Launchpad, following recommendations from people in #eucalyptus on Freenode: https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/854946

Let R1 and R2 be two (wireless) routers. Both R1 and R2 have one wired interface eth0 and one wireless interface wlan0. On each router, both interfaces are bridged on br0. So:

• R1.br0 = R1.eth0 + R1.wlan0
• R2.br0 = R2.eth0 + R2.wlan0

Let C1 and C2 be wireless clients connected to R1 and R2 respectively.

What will happen exactly when C1 sends a broadcast ethernet frame B, that's supposed to be received by C2 (say, an ARP request for C2's IP address)?

My questions:

1. Upon receipt of B by R1, will it send the frame through R1.br0 (therefore sending it to both R1.eth0 and R1.wlan0) or will send the frame only through R1.eth0 (since it knows that the frame came through R1.wlan0)?

2. In any case, R2 will eventually receive B. Upon receipt of B by R2, a question mostly equivalent to the first one: will R2 send the frame through R2.br0 (...) or through R2.wlan0 (...)? I'm not sure the answer to both questions has necessarily to be the same, since the source interface is not really the same thing, one is wire, the other is wireless, that's why I ask both.

If the answer to question 2 is "R2 will send the frame through both R2.wlan0 and R2.eth0", won't there be some kind of loop (by applying the same reasoning of my question (2) to R1 when it receives B again, this time from R1.eth0)? How does ethernet deals with this? The most simple solution I see is to make bridges so that they send the broadcast frames to all the interfaces but from the one which it originated.

I don't know how wireless networks work, so, if it is true that a bridge won't send a packet to the interface from which it originated, how can we be sure that the other wireless clients will receive the broadcast frame? Do wireless clients on a wireless (802.11b/g/n, whatever) network communicate directly to each other?

Actually, is all of this defined anywhere, or is it implementation specific? If it is defined, where can I get the standards?

I've installed DD-WRT v24-SP2 micro on a Linksys WRT54G2 v1.0, and I'd appreciate some help on setting up a not-so-trivial network.

I've studied the documentation (notably http://www.dd-wrt.com/wiki/index.php/Default_Configuration_Overview, which describes the router's ports), but I can't figure out by myself how to achieve what I want. I have some knowledge on Linux networking and routing, but nothing too fancy.

Description of the networks:

I want to set up one wired LAN (maybe someday I will partition this into two and connect them with another router, maybe this information could change the way to configure the WRT54G2 now, I don't know).

I want to have 2 WLANs, one internal and another for guests.

The internal one should be completely connected to the LAN (that is, all boxes on the wired network and the internal wireless network must see each other).

The guest WLAN cannot see the LAN, nor the other WLAN (the internal one).

All the computers connected to any of these networks (LAN, internal WLAN and guest WLAN) should be able to obtain IP addresses and DNS server addresses from a DHCP server on the router.

Internet connectivity (dual WAN with failover):

Finally, I have 2 internet connections: a fast one through PPPoE and another, slower on, to which I must connect with DHCP (to obtain a public IP address from the ISP through the cable modem).

All the networks (LAN, internal WLAN and guest WLAN) must be able to access the Internet.

The two connections should work as follows: the faster (PPPoE) one is the "default", every computer should use that connection unless it is down (this may be detected by pinging the gateway, or by any other means if anyone has a better idea), in which case the router should automatically switch to the slower (DHCP) connection. If the PPPoE connection ever is available again, the router should switch back to it once it detects that the connection is available.

Can all this be done with a single DD-WRT micro running on a Linksys WRT54G2 v1.0? What are the VLANs needed, what bridges should I create?

I'm pretty sure I'd be able to setup these networks using a Linux box with 3 network cards (one for PPPoE, one for the other internet connection, and the third for the LAN) and two wireless routers... but after reading the description of all the hardware that can be controlled by DD-WRT, I have the impression that it can handle all of this!!

Thanks in advance!