Dolan Antenucci's questions

After reading about some people's issue with Docker hosts' time going out of sync, I realized that my Docker host on Digital Ocean (created via docker-machine) may want ntp running on it, and that got me thinking about system updates to the Docker hosts in general.

There has already been a good discussion on applying updates to the actual docker services -- with general agreement that rebuilding images from updated base images is good solution -- but I haven't seen much focus on the Docker hosts themselves.

For those using Docker in a production environment, are you even bothering with docker-machine, or are you building and maintaining your Docker hosts with traditional tools like Chef/Puppet/etc?

AWS offers Elastic IP Addresses, which are great for switching website domains to a new instance/stack without having to wait for DNS changes to propagate.

Some cloud service providers (e.g., Rackspace) do not offer such reassigning to their public IP addresses. In Rackspace's case, one of their Load Balancers would accomplish what I want, but the cost (starts at $38/month if need SSL) becomes much greater than AWS's Elastic IP addresses (free if assigned to a box or ~$8/month if not used).

Is there a way to reproduce the convenience of AWS's Elastic IP addresses when hosting on Rackspace without using a load balancer (e.g., via some particular server setup or use of a 3rd party)?

I have an HBase cluster that is working, and I'm attempting to add some new servers to the cluster, but "SocketException: Invalid argument" and "FailedServerException: This server is in the failed servers list" errors keep getting generated in the logs.

2014-07-02 22:28:01,140 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.SocketException: Invalid argument
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:193)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupConnection(HBaseClient.java:392)
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:438)
    at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1141)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:988)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
    at com.sun.proxy.$Proxy10.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:141)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2040)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2086)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:748)
    at java.lang.Thread.run(Thread.java:701)
2014-07-02 22:28:31,764 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: <MY_MASTER_SERVER>/<MY_MASTER_NAME>:<MY_MASTER_PORT>
    at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:427)
    at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1141)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:988)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
    at com.sun.proxy.$Proxy10.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:141)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2040)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2086)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:748)
    at java.lang.Thread.run(Thread.java:701)

So far I can't find any differences between the old and new servers:

• both running Ubuntu 12.04 with all the latest updates and Cloudera's CDH4 for HBase
• neither have /etc/hosts have entries for master HBase (although I tried adding one on new servers, but still having same issue)
• firewalls should be configured the same without any local network restrictions (NOTE: on new servers, I can telnet to port 60000, my HBase Master's port, without any errors)

While debugging this, I saw a mention online about IPv6 configuration possibly causing issue, but as far as I know, both old and new servers have whatever default configurations Ubuntu uses for this.

Any ideas on how I can debug further and/or what the issue may be?

I have a 5 slave Hadoop cluster (using CDH4)---slaves are where DataNode and TaskNode run. Each slave has 4 partitions dedicated to HDFS storage. One of the slaves needed a reinstall and this caused one of the HDFS partitions to be lost. At this point, HDFS was complaining about 35K missing blocks.

A few days later, the reinstall was complete and I brought the node back online to Hadoop. HDFS remains in safe-mode and the new server is not registering anywhere near the amount of blocks that the other nodes are. For instance, under DFS Admin, the new node shows it has 6K blocks, while the other nodes have about 400K blocks.

Currently, the new node's DataNode logs show it is doing some verification (or copying?) on a variety of blocks, some of which fail as already existing. I believe this is HDFS just replicating existing data to the new node. Example of verification:

2013-08-09 17:05:02,113 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-143510735-141.212.113.141-1343417513962:blk_6568189110100209829_1733272

Example of failure:

2013-08-09 17:04:48,100 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: meez02.eecs.umich.edu:50010:DataXceiver error processing REPLACE_BLOCK operation  src: /141.212.113.141:52192 dest: /141.212.113.65:50010
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-143510735-141.212.113.141-1343417513962:blk_-4515068373845130948_756319 already exists in state FINALIZED and thus cannot be created.
    at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:813)
    at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:92)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(DataXceiver.java:846)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReplaceBlock(Receiver.java:137)
    at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:70)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
    at java.lang.Thread.run(Thread.java:679)

In DFS Admin, I can also see that this new node is at 61% capacity (matching other nodes' approximate usage), even though its number of blocks is about 2% of the other nodes. I'm guessing this is just the old data that HDFS is not recognizing.

I suspect one of a few things happened: (a) HDFS abandoned this node's data because of staleness; (b) the reinstall changed some system parameter so HDFS treats it as a brand new node (i.e. not an existing one with data); or (c) somehow the drive mapping got messed up, thus causing the partitions mapping to be changed and HDFS not able to find the old data (although the drives have labels, and I am 95% sure we got this right).

Main question: How can I get HDFS to re-recognize the data on this drive?

• answer: restart NameNode, and the nodes will re-report which blocks they have (see Update 1 below)

Sub-question 1: If my assumption of the new node's data usage is correct---that the 61% usage is ghost data---does it ever get cleaned up by HDFS, or do I need to manually remove this?

• less of an issue: since a large portion of drive seems to be recognized (see Update 1 below)

Sub-question 2: Currently, I cannot run listCorruptFileBlocks to find the missing blocks due to "replication queues have not been initialized." Any idea how to fix this? Do I have to wait for the new node to rebalance (i.e. this verification/copying phase to end)?

• answer: leaving Safe Mode let me run this (see Update 1 below)

Updates

Update 1: I thought I had fixed the issue by restarting my NameNode. This caused the new node's block count to jump up to approximately the same usage as the other nodes, and DFS changed its message to:

Safe mode is ON. The reported blocks 629047 needs additional 8172 blocks to reach the threshold 0.9990 of total blocks 637856. Safe mode will be turned off automatically.

I left it in this state for several hours, hoping that it would finally leave Safe Mode, but nothing has changed. I then manually turned off Safe Mode, and DFS's message changed to, "8800 blocks are missing". At this point, I was able to run hdfs fsk -list-corruptfileblocks, to see a large portion of files that are missing blocks.

Current remaining issue: how to get these missing blocks recovered... (should I spin this off in new question?)

I have 5 servers, all with similar hardware (i7, four 2tb 7200rpm drives, two 4tb 5400rpm drives, 430 watt power supply), and lately the machines have been freezing up. This has gotten worse in the last day or so, and I can't pinpoint any explanation. One recent change was adding the two 4tb hard drives. The crashes happen most often while running a large Hadoop job, so I was originally thinking the load was causing some issues, but last night one server just froze without any heavy load on the box (or so I think), other than HDFS (Hadoop's distributed file system) was probably rebalancing itself since two of the five nodes were offline.

If I plugin a monitor and keyboard to one of these frozen machines, I can't get any response or feedback on the screen.

Any ideas on possible points of failure and/or different logs I can look at to investigate? Thanks

Edit: The systems are running Ubuntu 10.04

Edit 2: More on hardware:

• intel core i7-930 bloomfield 2.8ghz processor (quad core)
• 12gb (6 x 2gb) kingston ddr3 1333 ram
• antec earthwatts green 430 power supply
• msi x58m lga 1366 motherboard

Edit 3: I pulled the two 4TB hard drives out temporarily to see if it helped with the crashing, and so far the servers are staying up, even under heavy Hadoop load. I will try the power meter soon to confirm if they are drawing too much power.

When I run a command for Hadoop on my Mac, Hadoop sends the logs to the command line. I could capture this on my own with > mylogfile.log, but I'm hoping there is simply a setting/option in Hadoop or Apache-logging that changes how this works.

Thanks!

Update: I found this https://stackoverflow.com/questions/2725858/can-apache-httpd-be-made-to-log-errors-to-console-instead-of-log-files-under-wind that has a suggestion on how to fix it for Apache. Since the two use the same logging system, maybe this will help someone suggest a way to set something similar?

This is sort of a follow-up question to an unanswered question I have regarding administration of Cloudera cluster, but I figure generalizing the question to all of Ubuntu may help me get an answer.

I want to be able to start/stop the same service on all my cluster's nodes. The only way I see how to do this is to create keys for the root user on my master node to each of the other nodes. Since I follow the Ubuntu recommendation to not use the root account, and instead use a sudo'ing user, I am hesitant to touch the root account (i.e. adding the keys).

Perhaps an alternative would be to add keys for my user to each of the boxes, then use some command to run sudo remotely?

I know there has to be other scenarios where people do something similar. What is the standard practice for this type of cluster management?

I've setup Hadoop to use Kerberos (following the Cloudera security guide), but it is unclear how I connect to hadoop with regular users (e.g. username=myuser).

Currently I have myself authenticated with Kerberos with my Keberos admin user (via kinit kerbadmin/admin), but that doesn't seem to help. Do I need to tell Hadoop that kerberos user "kerbadmin" is allowed to use Hadoop?

Running something like hadoop fs -ls / results in a permission issue:

11/09/13 11:13:03 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
Bad connection to FS. command aborted. exception: Call to <myservername>/127.0.1.1:54310 failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 

Also, sudo -u hdfs hadoop fs -ls / returns the same error message.

Looking at Cloudera's installation instructions, I don't see any mention of how to run jobs as regular users.

When I try to run a sample job, this is what I get:

hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar pi 2 100000
Number of Maps  = 2
Samples per Map = 100000
Wrote input for Map #0
Wrote input for Map #1
Starting Job
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=myuser, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

One solution would be to set perms of "/" to allow writing by all users.

Another solution I'm seeing online is to set a property mapreduce.jobtracker.staging.root.dir but I'm not sure where that is set at: http://getsatisfaction.com/cloudera/topics/unable_to_run_mapreduce_job_in_cdh3_cluster_permission_denied

I'm guessing there is a standard way this is handled (i.e. not all users running hadoop jobs have root access, nor is leaving permissions of HDFS root wide-open standard as well)

Edit: still stuck on this, but I reposted question to Cloudera's mailing list.. hopefully someone there or here will reply :) thanks!

I've got a cluster I'm setting up Cloudera 3 on, and it is unclear on whether I should be using these start/stop scripts like I used to with the standard Apache Hadoop setup (where I had a specific user account that ran all the Hadoop stuff). With CDH3, I got these services running as root.

What is the suggested secure and convenient way to issue global starts and stops with Cloudera 3? Do users who want to use these scripts setup keys so root can login to the other boxes? If so, how is this setup by a sudo'd user (just sudo su and setup keys?)?

Also, when I tried to run sudo /usr/lib/hadoop-0.20/bin/start-all.sh (with no keys setup for the root user to connect to the remove boxes), I get the following output:

starting namenode, logging to /usr/lib/hadoop-0.20/bin/../logs/hadoop-root-namenode-meez01.out
May not run daemons as root. Please specify HADOOP_NAMENODE_USER
<asks me for root@myserver's password>
....
<similar message for jobtracker>

update: I know someone out there has to be using Cloudera.. is this a conspiracy to get me to pay for their support?! j/k. if anything, let mek now how you are using it

I'm running 3 separate virtualhosts for my website (Django w/ wsgi for the main site, another Django w/ wsgi for the mobile version of the site, and a 3rd for Wordpress serving as the site's blog). After a few weeks, the swap memory climbs up to the point where my load & ping times become really slow. When I look at top, I see that there are several Apache processes each taking up a significant amount of memory, and have been running for at least an hour.

This is on an Ubuntu 10.04 server running on Rackspace cloud (medium instance).

I'm running the two django sites with wsgi in daemon mode (threads = 1, processes = 2).

My apache2.conf main settings look like this (with several "irrelevant" things stripped out to minimize the size of this post -- if you're expecting another setting, let me know, and I can check if I have it in there):

Timeout 120
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
<IfModule mpm_event_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

One thing I've seen suggested is to switch away from Apache to a "less bloated" web server. I'm open to this idea, but am guessing that I would be best moving Wordpress off the server so that I don't need PHP (or do these other web servers offer php and python solutions together?)

Let me know if you want any more information. Thanks!

I would like to fallback to a static webpage if my main web server is down (currently a Rackspace Cloud instance). This would be a sort of worse-case scenario that shouldn't happen, but has before (e.g. Backspace hardware failure). Redundancy of the servers would be an optimal solution, but budget is a concern.. so I'm looking to a low cost autoMatic fallback if something happens to the only server currently

On an Ubuntu 10.04 server w/ Apache2, I have auth_mysql turned on, and I am getting [debug] messages showing up. How can I turn these off? I'm guessing there is a way to turn this off for the module, as well as for all 'debug' messages. I am curious how to do both.

Update:
Originally I posted about auth-mysql only; however, I also want to control WSGI's output (currently showing [info] messages in error.log).