Questions[big-data](server)

we have hadoop cluster ( Ambari platform with HDP version - 2.6.4 )

and we performed verification step in order to understand if we have under replica blocks

the first verification was with:

su hdfs
hdfs fsck / -   --> 

its gives the results:


 Total size:    17653549013347 B (Total open files size: 854433698229 B)
 Total dirs:    843714
 Total files:   11752836
 Total symlinks:                0 (Files currently being written: 16)
 Total blocks (validated):      11792203 (avg. block size 1497052 B) (Total open file blocks (not validated): 6381)
 Minimally replicated blocks:   11792203 (100.00001 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          6
 Number of racks:               1

so as we can see above Under-replicated blocks is 0

BUT

when we perform the next verification:

hdfs dfsadmin -report

then we get

Configured Capacity: 141275429535744 (128.49 TB)
Present Capacity: 140886991802565 (128.14 TB)
DFS Remaining: 84748655941292 (77.08 TB)
DFS Used: 56138335861273 (51.06 TB)
DFS Used%: 39.85%
Under replicated blocks: 4212067
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

so from above we can see that Under replicated blocks is --> 4212067

about to know what is the right under replica number:

why we get differences between hdfs fsck / and hdfs dfsadmin -report ?

BTW - from Ambari we get the ~ same results as from hdfs dfsadmin -report

What is going wrong with the Spark Cassandra Connector could you please help to solve this?

Scala File:

import com.datastax.spark.connector._
import org.apache.spark.sql.SparkSession
import org.apache.spark.{SparkConf, SparkContext}


object SparkTest {
  def main(args: Array[String]) {
    val spark = SparkSession.builder.appName("SparkTest").getOrCreate()
    println("Hello World2")

    val conf = new SparkConf().set("spark.cassandra.connection.host","localhost")
    val sc = new SparkContext(conf)
    val rdd1 = sc.cassandraTable("spark_parallalism_test", "test_table1")
    println(rdd1.first)
    sc.stop()
    spark.stop()
  }
}

SBT file looks like:

name := "SparkTest"

version := "1.0"

scalaVersion := "2.12.10"


libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.0.1"

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.12" % "3.0.0"

Commands to execute:

sbt package

spark-submit --class "SparkTest" --master local[4] target/scala-2.12/sparktest_2.12-1.0.jar

Error Log:

20/12/28 23:41:57 WARN Utils: Your hostname, subhrangshu-Lenovo-V110-15ISK resolves to a loopback address: 127.0.1.1; using 192.168.43.14 instead (on interface wlp2s0)
20/12/28 23:41:57 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/12/28 23:41:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/spark/connector/rdd/reader/RowReaderFactory
        at SparkTest.main(SparkTest.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.datastax.spark.connector.rdd.reader.RowReaderFactory
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 13 more
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

What is my mistake in this case?

How do megasites like Youtube perform backups? According to https://www.quora.com/Where-does-YouTube-store-so-many-videos, 2014 they stored 76 PB every year, a number that most certainly has increased a lot since then. Is it even possible to backup this or do they solve the problems backups usually are used for in some other way (such as so much redundancy that it simply can't fail)?

Presto can't handle many concurrent sessions.

what's the maximum number of concurrent sessions per presto and how to set parameter for this ?

and how to handle it's maximum JVM ?

For small files hashing is just ok, but with huge ones you can easily find md5sum is CPU bound. Is there any hashing algorithm able to scale out on multiple cores? Any workarounds? Ideas? Anything? :)