IDEA运行Spark报错:Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
IDEA运行Spark报错:Could not locate executable nullbinwinutils.exe in the Hadoop binaries.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/12/14 14:45:19 INFO SparkContext: Running Spark version 3.0.0
20/12/14 14:45:19 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:390)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:274)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:807)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:777)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:650)
at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:303)
at com.songzixian.wc.Spark01$.main(Spark01.scala:21)
at com.songzixian.wc.Spark01.main(Spark01.scala)
20/12/14 14:45:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
出现这个问题的原因,并不是程序的错误,而是 windows 系统用到了 hadoop 相关的服
务,解决办法是通过配置关联到 windows 的系统依赖就可以了
在 IDEA 中配置 Run Configuration,添加 HADOOP_HOME 变量
问题得到解决,再次运行程序,就不报错了