We are using HDP cluster with 182 data node
machines:
HDP version - 2.6.4 Ambari version 2.6.1
We note the following behavior on the data nodes
machines (its happens on all data-node
machines and on all disks).
When we perform the command as above example:
ps -eo s,user,cmd | grep ^[RD]
D hdfs du -sk /grid/sdj/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
D hdfs du -sk /grid/sdm/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
R root ps -eo s,user,cmd
Note - each disk in the data node
is 5.4 T Bytes
.
We can see that HDFS
is running the "du -sk" on the data node
disks
We don't like this, because the meaning of that is consuming high load CPU Avrg and sometimes even bad performance.
We are understand that HDFS
need to run the "du -sk
" in order to verify the disks space, but on other hand its cost - high CPU load avrg and sometimes even poor performance.
Is it possible to tell HDFS
in some way to disable this verification?
0 Answers