How does the GCP/GKE VPC native network implement the alias IP assignment to each pod?

As in Google Kubernetes Engine – Networking, each Pod will have an IP from secondary IP CIDR range of the subnet (e..g 10.4.1.1).

Each pod in a Linux instance has its veth I/F. How does each veth(i) for pod(i) is linked to the network interface of the Linux box in which the pods run? Each pod (veth) has its own network interface attached to the Linux box, or all pods and veth interfaces share the single eth0?

According to Kubernetes networking on Google Cloud indicates the single eth0 of the Linux box is shared among pods. Then the single eth0 has multiple secondary alias IP addresses associated to it. Then how the Linux box can separate the packets to each alias IP and route them to the corresponding pod/veth? Is it using iptable to map an alias IP to a veth?

The GKE document Alias IP ranges defined in a VM network interface does not have the detail.

Why GCP Cloud NAT needs BGP/Cloud Router?

Cloud Router documentation

Cloud Router enables you to dynamically exchange routes between your Virtual Private Cloud (VPC) and on-premises networks by using Border Gateway Protocol (BGP)

Cloud NAT overview

You configure a NAT gateway on a Cloud Router, which provides the control plane for NAT, holding configuration parameters that you specify.

Each Cloud NAT gateway is associated with a single VPC network, region, and Cloud Router.

In my understanding, NAT maps between private address/port to public ip/port, and forward the mapped packet to the Internet Gateway (to get out to the Internet). Where and why does BGP (Cloud Router) come into play in this picture?

Question

Please help understand the cause and solution for the problem.

Problem

SageMaker Studio domain creation fails due to KMS permissions. The IAM Role specified to the SageMaker arn:aws:iam::316725000538:role/SageMaker has the permissions for KMS required as specified in https://docs.aws.amazon.com/sagemaker/latest/dg/api-permissions-reference.html.

Domain creation failed
Unable to create Amazon EFS for domain 'd-1dq5c9rpkswy' because you don't have permissions to use the KMS key 'arn:aws:kms:us-east-2:316725000538:key/1e2dbf9d-daa0-408d-a290-1633b615c54f'. See https://docs.aws.amazon.com/sagemaker/latest/dg/api-permissions-reference.html for required permissions for CreateDomain action.

tells the IAM permissions

IAM Permission for CreateDomain action

• Amazon SageMaker API Permissions: Actions, Permissions, and Resources Reference

The IAM permission required for the CreateDomain action have been attached to the IAM role.

Question

Is there a way to log the connections blocked by the AWS Network Firewall, or filter the logs of blocked connections?

Background

Currently having setup the rules, and would like to know which IP or domains have been blocked.

Looking at Logging network traffic from AWS Network Firewall but not clear if it is possible.

You can record flow logs and alert logs from your Network Firewall stateful engine.
Flow logs are standard network traffic flow logs. Each flow log record captures the network flow for a specific 5-tuple.

Alert logs report traffic that matches your stateful rules that have an action that sends an alert. A stateful rule sends alerts for the rule actions DROP and ALERT.

From the flow logs, it is not clear if it is passed or blocked.

{
    "firewall_name": "network-firewall-sagemaker-studio-anfw",
    "availability_zone": "us-east-1a",
    "event_timestamp": "1628236046",
    "event": {
        "timestamp": "2021-08-06T07:47:26.000068+0000",
        "flow_id": 1108238612337889,
        "event_type": "netflow",
        "src_ip": "51.222.5.114",
        "src_port": 57528,
        "dest_ip": "10.2.2.60",
        "dest_port": 8088,
        "proto": "TCP",
        "netflow": {
            "pkts": 1,
            "bytes": 40,
            "start": "2021-08-06T07:46:24.365793+0000",
            "end": "2021-08-06T07:46:24.365793+0000",
            "age": 0,
            "min_ttl": 239,
            "max_ttl": 239
        },
        "tcp": {
            "tcp_flags": "02",
            "syn": true
        }
    }
}

Question

It is possible to access the Internet from a private subnet only with NAT without an Internet Gateway associate with its VPC?

Background

VPC with public and private subnets (NAT) says below. Just with this sentence, it would be a fair conclusion that it only requires NAT to reach the Internet without an Internet Gateway attached its VPC.

A NAT gateway with its own Elastic IPv4 address. Instances in the private subnet can send requests to the internet through the NAT gateway over IPv4 (for example, for software updates).

I believe for any traffic from a VPC to reach the Internet, the VPC requires the Internet Gateway but the document does not clearly say so, hence being confused.

Objective

Clarify Computer, Not User, when integrating a Linux box with a Windows Domain.

Background

Having confusion because most SSSD AD information focuses on User Authentication, however apparently in a Windows Domain, the computer to which the users logon should be in the domain already (as a service principal?). It seems to me I need a help to get to the bottom.

Question - Computer to join a Windows domain.

I believe there are several ways. Please suggest if below are correct.

1. Run realm in the Linux box.
2. Run Samba net ads join in the Linux box (Creating Host Keytab with Samba).
3. Run setSPN and ktpass in the Domain Controller

Regarding using realm and samba, I suppose the command also will join the Linux box into the Domain and AD. It it correct? Also please point to resources to understand what is going on behind with regard to computer and domain if any.

These ways require an Administrator credential Or a user credentials who has the permission to add a computer to a domain, which would not always be available. Then I suppose need to ask a Windows domain administrator to do what realm and samba do. Which seem to me involving below but need corrections/confirmation if they are correct.

1. Create a A record and reverse lookup PTR record in the domain DNS.
2. Add a computer for the Linux box in the Computers management view. The UPN of the box will be <linux hostname>@<realm or domain>.
3. Create a SPN for the Linux box with setSPN. The SPN is like host/<name>@<realm or domain>.
4. Create a keytab with ktpass. The SPN is specified with -princ and the UPN is specified with -mapuser.
5. Copy the keytab to the linux box as /etc/krb5.keytab and change permissions. (Treat like pem file in ~/.ssh/)

Question - UPN and SPN?

Why having multiple identifiers for the same object (linux box)? Why SPN is required?

In Linux, what happens to the state of a process (curl, postman, mule, caml) when it has sent a HTTP request and waiting for a HTTP response? What is the way to know how long the process is waiting for the response?

Objective

Would like to understand the mechanism why using become to be the ansible user causes "Permission denied (publickey)".

User ansible to run ansible-playbook to checkout a Github repository. SSH keys have been copied with ssh-copy-id with the user.

Without become, the playbook runs.

[ansible@ip-172-31-39-108 playbooks]$ whoami
ansible

[ansible@ip-172-31-39-108 playbooks]$ ansible-playbook git.yml
PLAY [Git example] *************************************************************
TASK [setup] *******************************************************************
ok: [ub01]
TASK [check out the repository on the host] ************************************
changed: [ub01]
PLAY RECAP *********************************************************************
ub01                       : ok=2    changed=1    unreachable=0    failed=0

However, using "become: yes" causes the error.

[ansible@ip-172-31-39-108 playbooks]$ ansible-playbook git.yml
PLAY [Git example] *************************************************************
TASK [setup] *******************************************************************
ok: [ub01]
TASK [check out the repository on the host] ************************************
fatal: [ub01]: FAILED! => {"changed": false, "cmd": "/usr/bin/git clone --origin origin '' /home/ansible/project/mezzanine-example", "failed": true, "msg": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.", "rc": 128, "stderr": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n", "stdout": "", "stdout_lines": []}
        to retry, use: --limit @/home/ansible/playbooks/git.retry

PLAY RECAP *********************************************************************
ub01                       : ok=1    changed=0    unreachable=0    failed=1

ssh-agent is running and the private key has been added.

[ansible@ip-172-31-39-108 playbooks]$ eval $(ssh-agent -s)
Agent pid 1513

[ansible@ip-172-31-39-108 playbooks]$ ssh-add ~/.ssh/id_rsa
Identity added: /home/ansible/.ssh/id_rsa (/home/ansible/.ssh/id_rsa)

Question

Please provide explanations why this happen or point to the resource to look into.

Also I only started ssh-agent in the server that runs ansible-playbook, but not on the target server. How the Github SSH authentication happens on the target server?

Playbook

- name: Git example
  hosts: webservers
  become: no    # <----- Changing to yes cause the issue
  become_user: ansible
  become_method: sudo
  vars:
    repo_url: [email protected]:lorin/mezzanine-example.git
    proj_dirname: /home/ansible/project
    proj_name: mezzanine-example
    proj_path: "{{ proj_dirname }}/{{ proj_name }}"
  tasks:
    - name: check out the repository on the host
      git: repo={{ repo_url }} dest={{ proj_path }} accept_hostkey=yes

Configurations

hosts

[webservers]
ub01
#rh01

ansible.cfg

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes

Environment

Ansible playbook is run on RedHat.

NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"

The target host is Ubuntu.

DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

Related Issues

• Ansible with Github: Permission denied (Publickey)
• How to fix "Permission denied (publickey)." Ansible and "git clone" issue
• Accessing SSH_AUTH_SOCK from another non-root user

Console -vvvvv output

TASK [check out the repository on the host] ************************************
task path: /home/ansible/playbooks/git.yml:12
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/source_control/git.py
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r ub01 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1485919043.94-240537002849590 `" && echo ansible-tmp-1485919043.94-240537002849590="` echo /tmp/ansible-tmp-1485919043.94-240537002849590 `" ) && sleep 0'"'"''
<ub01> PUT /tmp/tmpAjaOMc TO /tmp/ansible-tmp-1485919043.94-240537002849590/git.py
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set sftp_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC sftp -b - -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r '[ub01]'
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r ub01 '/bin/sh -c '"'"'setfacl -m u:ansible:r-x /tmp/ansible-tmp-1485919043.94-240537002849590/ /tmp/ansible-tmp-1485919043.94-240537002849590/git.py && sleep 0'"'"''
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r -tt ub01 '/bin/sh -c '"'"'sudo -H -S -n -u ansible /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-cxuzmrsbxdvydelfnjrsmgvocgkeptxd; /usr/bin/python /tmp/ansible-tmp-1485919043.94-240537002849590/git.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<ub01> ESTABLISH SSH CONNECTION FOR USER: None
<ub01> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)(-o)(ForwardAgent=yes)
<ub01> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<ub01> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<ub01> SSH: PlayContext set ssh_common_args: ()
<ub01> SSH: PlayContext set ssh_extra_args: ()
<ub01> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r)
<ub01> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r ub01 '/bin/sh -c '"'"'rm -f -r /tmp/ansible-tmp-1485919043.94-240537002849590/ > /dev/null 2>&1 && sleep 0'"'"''
fatal: [ub01]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/git clone --origin origin '' /home/ansible/project/mezzanine-example",
    "failed": true,
    "invocation": {
        "module_args": {
            "accept_hostkey": true,
            "bare": false,
            "clone": true,
            "depth": null,
            "dest": "/home/ansible/project/mezzanine-example",
            "executable": null,
            "force": false,
            "key_file": null,
            "recursive": true,
            "reference": null,
            "refspec": null,
            "remote": "origin",
            "repo": "[email protected]:lorin/mezzanine-example.git",
            "ssh_opts": null,
            "track_submodules": false,
            "umask": null,
            "update": true,
            "verify_commit": false,
            "version": "HEAD"
        },
        "module_name": "git"
    },
    "msg": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.",
    "rc": 128,
    "stderr": "Cloning into '/home/ansible/project/mezzanine-example'...\nPermission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",
    "stdout": "",
    "stdout_lines": []
}
        to retry, use: --limit @/home/ansible/playbooks/git.retry

PLAY RECAP *********************************************************************
ub01                       : ok=1    changed=0    unreachable=0    failed=1

Update

Thanks to the answer from @Jakuje and other articles, understood that the SSH agent listens on a UNIX socket file. The filename is stored in SSH_AUTH_SOCK environment variable. However, the SUDO eliminates the environment variables, hence the sudo-ed SSH client does not know how to talk to SSH agent. Therefore, cannot pass the SSh authentication.

• Understanding ssh-agent and ssh-add
• SSH Definitive Guide 6.3. SSH Agents
• An Illustrated Guide to SSH Agent Forwarding

The solution is in the articled provided by @Jakuje.

Question

1. How is the number of partitions decided by Spark?
2. Do I need to specify the number of available CPU cores somewhere explicitly so that the number of partitions will be the same (such as numPartition arg of parallelize method, but then need to update program whenever the number of cores changed)?

Background

Installed a Spark cluster as in Environment with no changes to the spark-env.sh, spark-defaults.conf files nor SparkConf object in programs.

For a N Queen program, the number of partition was 2 and only one node was assigned tasks. For a word count program, the number of partition was 22 and tasks were allocated to all nodes. Used spark-submit for both programs.

Programs

N Queen

val sparkConf = new SparkConf().setAppName("NQueen").set("spark.files.overwrite", "true")
val sc = new SparkContext(sparkConf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
def isSafe(column: Int, placement: List[Int]): Boolean = { ... }
def placeQueensAt(row: Int, placements: Set[List[Int]]): Set[List[Int]] = { ... }

val initial = sc.parallelize(queensAtFirst)
//val initial = sc.parallelize(queensAtFirst, 12)
println("Partitions = %d".format(initial.partitions.size))

val result = initial.flatMap(x => placeQueensAt(1, Set(x))).collect()

Word Count

val sparkConf = new SparkConf().setAppName("WordCount").set("spark.files.overwrite", "true")
val sc = new SparkContext(sparkConf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)

val lines = sc.textFile("hdfs:/user/wynadmin/sfpd.csv")
println("Patitions = %d".format(lines.partitions.size))

val words = for (line <- lines; word <- line.split(",") if word.toLowerCase.matches("[a-z]+")) yield (word, 1)
val counts = words.reduceByKey(_ + _)

Environment

Spark 2.0.1 (3 nodes with 4 CPU each) on Ubuntu 14.04.
Standalone deployment (not YARN nor Mesos)

Question

Regarding the TeraSort demo in hadoop, please suggest if the symptom is as expected or the workload should be distributed.

Symptom

Started Hadoop (3 nodes in a cluster) and run the TeraSort benchmark as below in Executions.

I expected all 3 nodes would get busy and all CPU would be utilized (400% in top). However only the node on which the job started got busy and the CPU was not fully utilized. For example if it is started on sydspark02, top shows as below.

I wonder this is as expected or if there is a configuration issue by which the workload is not distributed among the nodes.

sydspark02

top - 13:37:12 up 5 days,  2:58,  2 users,  load average: 0.22, 0.06, 0.12
Tasks: 134 total,   1 running, 133 sleeping,   0 stopped,   0 zombie
%Cpu(s): 27.5 us,  2.7 sy,  0.0 ni, 69.8 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:   8175980 total,  1781888 used,  6394092 free,       68 buffers
KiB Swap:        0 total,        0 used,        0 free.   532116 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2602 hadoop    20   0 2191288 601352  22988 S 120.7  7.4   0:15.52 java
 1197 hadoop    20   0  105644    976      0 S   0.3  0.0   0:00.16 sshd
 2359 hadoop    20   0 2756336 270332  23280 S   0.3  3.3   0:08.87 java

sydspark01

top - 13:38:32 up 2 days, 19:28,  2 users,  load average: 0.15, 0.07, 0.11
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  1.2 sy,  0.0 ni, 96.6 id,  2.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  10240364 total, 10092352 used,   148012 free,      648 buffers
KiB Swap:        0 total,        0 used,        0 free.  8527904 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10635 hadoop    20   0 2766264 238540  23160 S   3.0  2.3   0:11.15 java
11353 hadoop    20   0 2770680 287504  22956 S   1.0  2.8   0:08.97 java
11057 hadoop    20   0 2888396 327260  23068 S   0.7  3.2   0:12.42 java

sydspark03

top - 13:44:21 up 5 days,  1:01,  1 user,  load average: 0.00, 0.01, 0.05
Tasks: 124 total,   1 running, 123 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.0 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8175980 total,  4552876 used,  3623104 free,     1156 buffers
KiB Swap:        0 total,        0 used,        0 free.  3818884 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
29374 hadoop    20   0 2729012 204180  22952 S   3.0  2.5   0:07.47 java

Executions

> sbin/start-dfs.sh
Starting namenodes on [sydspark01]
sydspark01: starting namenode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hadoop-namenode-sydspark01.out
sydspark03: starting datanode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hadoop-datanode-sydspark03.out
sydspark02: starting datanode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hadoop-datanode-sydspark02.out
sydspark01: starting datanode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hadoop-datanode-sydspark01.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-sydspark01.out

> sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-resourcemanager-sydspark01.out
sydspark01: starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-sydspark01.out
sydspark03: starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-sydspark03.out
sydspark02: starting nodemanager, logging to /usr/local/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-sydspark02.out

> hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar teragen 100000000 /user/hadoop/terasort-input
> hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar terasort /user/hadoop/terasort-input /user/hadoop/terasort-output

Configuration files

slaves

sydspark01
sydspark02
sydspark03

core-sites.xml

<configuration>
    <property>
      <name>hadoop.tmp.dir</name>
      <value>/home/hadoop/tmp</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://sydspark01:9000</value>
    </property>
</configuration>

Environment

Ubuntu 14.04.4 LTS 4CPU on VMWare
Hadoop 2.7.3
Java 8

Monitoring with JMC

When running the JMC on the data node in the node where the job is executed.

CPU

Only about 25% of CPU resource (1 CPU out of 4) is used.

Memory