Questions[hadoop](server)

I set up a KDC Server and created a Realm EXAMPLE.COM. Here is my krb5.conf file:

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = EXAMPLE.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log

[realms]
  EXAMPLE.COM = {
    admin_server = my.linux-server.de
    kdc = my.linux-server.de
  }

I also added a user testuser with password abc via kadmin.local:

kadmin.local:  addprinc [email protected]

I can successfully log in on my Ubuntu VM:

[root@ubuntu-vm ~]# kinit testuser
Password for [email protected]:

Then klist shows:

[root@ubuntu-vm ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]

Valid starting       Expires              Service principal
01.12.2016 14:58:40  02.12.2016 14:58:40  krbtgt/[email protected]

and I can open my Kererized Hadoop UI.

========================================================================

Problem makes my Windows client. I set it up by copying the krb5.conf file from the KDC machine to the Windows client and renamed it to kdc5.ini.

I also set the computer's domain:

C:> Ksetup /setdomain EXAMPLE.COM

After a restart I tried to connect to my KDC Realm via

C:> kinit [email protected]
Password for [email protected]:
<empty row>

Everything looked fine so far, but when I call klist I get only the following result:

Aktuelle Anmelde-ID ist 0:0x7eca34

Zwischengespeicherte Tickets: (0)

In English something like ... cached tickets: (0)

I'm also unable to open my website on the Windows client, so I guess that there's an interoperability issue as I didn't have any problems connection via my Ubuntu client.

My browser (Firefox) should be correctly configured on both machines (Ubuntu & Windows), I set the network.negotiate-auth.trusted-uris property to http://my.linux-server.de (since I did this, the Ubuntu client can open the site). Curl also works for Ubuntu, but don't for Windows.

UPDATE: Also tried a second Windows client without any success...

I deployed the secondary namenode on the same machine is my main namenode:

This is wrong for performance and durability reasons (the secondary name node isn't a hot spare, but it does have a copy of needed metadata). I have found documentation on how to move the namenode, but nothing about moving the secondary namenode.

Does anyone with experience know how to do this safely?

I am still a newbie learner of Hadoop, and this time I was trying to process a 106GB file. I used -copyFromLocal to copy that big file to my Hadoop DFS, but since the file is big I have to wait for a long time without a clue about the current copying status.

Is there any way to show the current file copying status with this command?

Thank you guys in advance for your help!

I am working in a project and my job is to build a database system to manage about 60,000,000,000 data entries.

The project background is I have to do real-time storage for large number of messages that read from about 30,000 RFID readers every second. Suppose every RFID reader generates 6,000 messages per one day, I have to insert 180,000,000 entries into the database.

A possible data entry is like "time_stamp, Reader_ID, Tag_ID, other_msg_content"

The will be queries (SELECT) base on time range, Reader_ID, and Tag_ID. The queries won't be very complicated.

Now I am designing the database system, and I plan to use MySQL. My dump questions are:

1. Is it wise to use MySQL, or I should resort to Oracle (which is expensive), or HBase?

2. If I must use MySQL, any idea how I can build the cluster?

3. If I insert the messages into a table, soon the table will be very long. I would like to use Sharding techniques to split a long table to many short tables.

   3.a. I want to know the proper length for a MySQL InnoDB table, i.e., after how many data entries were inserted, I shall start to sharding?

   3.b. Is there any good sharding proxy solution out there? I know spock proxy and some others, need recommendations.

4. Do I have to use MySQL Cluster? OR I just use mysql master servers and sharding slaves, and use Replication to achieve high availability?

5. Assume I have to handle 20 TB data in MySQL (for 1 year), I plan to use 20 nodes (PC server, cheap), and to store 1 TB data per node, is it possible? Any comments are welcomed.

Many thanks.

I'm in a evaluation phase of some hw that could be used for setting up a hadoop cluster. This hw is refurbished (hp G6 servers w/ Smart Array 410/i controller) and probably we should/must use it... we haven't it yet. I've read that 410/i controller doesn't allow a JBOD configuration as required by hadoop, is there anybody that could confirm this statement ? Or not ? Seems that a workaround could be to configure raid0 on every single disk, again anybody confirms it or there is other ways of configuration ? Thank you Rob