Ali Majed HA's questions

I have a spark streaming program, that reads data from a socket I have craeted using:

nc -lk 9999

The program reads the data from socket and exclude the "Error" messages. When I write manually in socket, it works fine.
I have created a python script that prints "Error" messages frequently. I will save the result in a file using:

stdbuf -oL python my_script.py &>> my_file.txt

and read the file from the socket:

nc -lk 9999 | tail -f my_file.txt

Every thing is ok, the socket will read data from file while the file is being updated on the background, But the problem is that my spark program doen't capture the "Error" messages.
As a summary: when I write manually "Error" messages in socket, spark capture them, But it won't capture "Error" message generated by python script from socket.
Actually the program doesn't work if I read file from socket instead of typing in it.
What is the difference?

I have coded a single program in spark to receive data from a local socket and capture lines that has the "Error" in them. I write manually in terminal:

$ nc -lk 9999
Error: some error message
Warning: some warning message

and so on. I need the socket get the system log information automatically. What ever log file in the system that contains Error messages or Warning ones and then my program can extract Error ones from the socket.

I have followed this manual to run spark on my Ubuntu 16.04. I have downloaded Spark 2.3, Java 9 and Scala 2.12.5.

In my /.bashrc:

export JAVA_HOME=/usr/local/java/jdk-9.0.1  
export SPARK_HOME=/usr/bin/spark  
export PATH=$PATH:SPARK_HOME/bin  
export PATH=$PATH:SPARK_HOME/sbin  
export SBT_HOME=/usr/bin/sbt  
export PATH=$PATH:$SBT_HOME/bin 

In my spark-env.sh:

JAVA_HOME=/usr/local/java/jdk-9.0.1  
SPARK_MASTER_IP=10.1.20.241  
SPARK_WORKER_MEMORY=4g  

at the end of file.

but when I run ./spark-shell I get this error:

2018-03-28 17:17:26 WARN  Utils:66 - Your hostname, ali resolves to a loopback address: 127.0.1.1; using 192.168.1.52 instead (on interface enp0s3)
2018-03-28 17:17:26 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address  
WARNING: An illegal reflective access operation has occurred  
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/bin/spark/jars/hadoop-auth-2.6.5.jar) to method sun.security.krb5.Config.getInstance()  
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

2018-03-28 17:17:28 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Failed to initialize compiler: object java.lang.Object in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.  
Failed to initialize compiler: object java.lang.Object in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Exception in thread "main" java.lang.NullPointerException