Questions[uec](server)

I'm having trouble with my cloud installation.

I've installed everything from the Ubuntu Server 11.04 CD, then upgraded the system to the latest version (apt-get update && apt-get dist-upgrade), installed NTP, correctly set it up, and booted everything. The cloud runs ok for sometime, but then the cloud controller / cluster controller will stop talking to one of the node controllers!

One of the symptoms is euca-describe-availability-zones verbose suddenly starts showing less resources than the appropriate (and initial) values.

If I look on cc.log, it shows:

[Mon Sep 19 19:09:07 2011][002531][EUCADEBUG ] DEBUG: requested URI http://10.20.200.10:8775/axis2/services/EucalyptusNC
[Mon Sep 19 19:09:07 2011][002531][EUCADEBUG ]  ncClientCall(ncDescribeResource): ppid=13403 client calling 'ncDescribeResource'
[Mon Sep 19 19:09:07 2011][002531][EUCAERROR ] ERROR: DescribeResource() could not be invoked (check NC host, port, and credentials)
[Mon Sep 19 19:09:07 2011][002531][EUCADEBUG ]  ncClientCall(ncDescribeResource): ppid=13403 done calling 'ncDescribeResource' with exit code '1'

Then, in the corresponding node controller, axis2c.log, I see this:

[Mon Sep 19 19:10:58 2011] [error] rampart_timestamp_token.c(179) [rampart]Timestamp not valid: Created time is not valid
[Mon Sep 19 19:10:58 2011] [error] rampart_sec_header_processor.c(612) [rampart]Timestamp is not valid
[Mon Sep 19 19:10:58 2011] [error] rampart_sec_header_processor.c(1911) [rampart]Timestamp processing failed
[Mon Sep 19 19:10:58 2011] [error] rampart_in_handler.c(143) [rampart][rampart_in_handler] Security Header processing failed.
[Mon Sep 19 19:10:58 2011] [error] phase.c(233) Handler RampartInHandler invoke failed within phase Security
[Mon Sep 19 19:10:58 2011] [error] engine.c(696) Invoking phase Security failed
[Mon Sep 19 19:10:58 2011] [error] engine.c(279) Invoking operation specific phases failed for operation ncDescribeResource
[Mon Sep 19 19:10:58 2011] [error] rampart_engine.c(159) [rampart][rampart_engine] Cannot get saved rampart_context
[Mon Sep 19 19:10:58 2011] [error] rampart_out_handler.c(136) [rampart][rampart_out_handler] ramaprt_context creation failed.
[Mon Sep 19 19:10:58 2011] [error] phase.c(233) Handler RampartOutHandler invoke failed within phase MessageOut
[Mon Sep 19 19:10:58 2011] [error] engine.c(696) Invoking phase MessageOut failed

So: there's a time synchronization problem.

However, NTP is installed, and correctly configured.

One thing I've noticed, by issuing lots of ntpq -np, is that the NC stops working once the time offset is positive. If the offset keeps negative, everything works fine. The offsets are very small (around 5ms, the absolute maximum I could see is 10ms).

Googling, I've found this rampart code: http://wso2.org/project/wsf/php/1.1.0/docs/code-coverage/rampartc/src/util/.libs/rampart_timestamp_token.c.gcov.html

/*Check whether created is less than current time or not*/
  current_val = rampart_generate_time(env, 0);
  validity = rampart_compare_date_time(env, current_val, created_val);
  if (validity == AXIS2_SUCCESS)
  {
      AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI, "[rampart][ts]Timestamp not valid: Created time is not valid");
              AXIS2_FREE(env->allocator, current_val);
              current_val = NULL;
      return AXIS2_FAILURE;
  }

As we can see, it apparently allows for deviation in time in one sense, but not in the other.

Am I missing something? Am I the only facing this issue? Isn't it stupid to verify timestamps with milissecond precision and allow only negative deviations?!

How do people deal with this problem? What can I do to keep my cloud alive?

I thought of some solutions:

1. Patch rampart, to simply remove the timestamp verification
2. Patch rampart, to allow for positive deviations as well
3. Find a way to make ntp or ntpdate adjust the time to some specific offset behind the server's reference time
4. Write my own time synchronization tool

What do you think?

EDIT: it looks like it is possible to disable Rampart in Axis2 configuration, but I can't figure out how to do that!

EDIT 2: The version of Rampart available on Ubuntu's repositories is 1.3.0, which is from 2007 or 2008... the latest released version is something like 1.6.0, from June 2011. Apparently this latest version allows packets "from the future". I'd really like to find this latest version from a PPA!

EDIT 3: I've found some parameters to change Rampart 1.3.0's behaviour: TimeToLive, ClockSkewBuffer and PrecisionInMilliseconds. I've added them (360, 60 and false, respectively) to EucalyptusNC.xml and EucalyptusCC.xml, and things are getting better. Occasionally I still see the timestamp error messages on the logs, but they are very rare now. I've also disabled NTP on the NCs and created a cron script (that runs every hour) to sync time (ntpdate -b) with the CC.

EDIT 4: Apparently this is a bug in Ubuntu's packaging of Eucalyptus. I've filed a bug on Launchpad, following recommendations from people in #eucalyptus on Freenode: https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/854946

I just installed a private Ubuntu Enterprise Cloud with Ubuntu Server 11 (based on Eucalyptus 2) and managed to get everything working (in System Mode, so the instances get an IP from our DHCP. Eucalyptus recognizes this IP).

However, I have a really strange problem and could not find any info here or on Google.

The Problem is that I cannot connect via SSH to any instance. I tried different images from the store, 64 and 32 bit. Also from different source computers, including network PCs, the CC and the node itself.

Here is the output from the console:

cloud-controller:~$ ssh -i .euca/mykey.priv -v 192.168.2.141
OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 192.168.2.141 [192.168.2.141] port 22.
debug1: Connection established.
debug1: identity file .euca/mykey.priv type -1
debug1: identity file .euca/mykey.priv-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3ubuntu3
debug1: match: OpenSSH_5.3p1 Debian-3ubuntu3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
debug1: SSH2_MSG_KEXINIT sent
Read from socket failed: Connection reset by peer

So, the server is running but closes the connection immediately. What could be the cause for it? Unfortunately I have no other option than SSH to login into the instance, it's a fresh install.

Thanks for any solutions or hints! Michael

I have setup the Cloud-A with 1 - [CLC+CC] and 2 - [NC] computers. I have another Cloud-B with same configuration using the Ubuntu Enterprise Cloud

Both of them working fine individually, in the same LAN.

Now if I want to add the NC of Cloud-A to CC of Cloud-B, [in case the resources of Cloud-B are exhausted] how can I make it possible ? I guess this calls for the interoperability stuff...

Could you please explain what happens exactly when we ask for instance, the direct interaction happens between the client and NC or it goes through the CLC and CC ?

What I want to say is, say there are multiple cloud providers. A user is subscribed to any one of them, say Cloud-A for IaaS. As the requirements are dynamic, all the resources of Cloud-A may get exhausted. There may be another Cloud-B which can provide the services but that Cloud-A can't ask the client to go for Cloud-B.

So if it is possible to have some co-ordination between this two providers to share resources mutually, making client fully unaware of whats going on in the background....?

How does storage works with the ubuntu enterprise cloud?

If I have two disks (on separate machines) with free space 10gb and 20gb, can I have a VM running with 30gb of disk space?

I intend to do some virtualization with either of the two technologies: Ubuntu Enterprise Cloud and VMWare.

I would like to know the advantages (e.g. main features missing in the other one, long-run, financial advantages, support...) of using the Ubuntu Enterprise cloud compared to VMWare.

Thank you.