IDEA运行Spark报错:Could not locate executable nullbinwinutils.exe in the Hadoop binaries.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/12/14 14:45:19 INFO SparkContext: Running Spark version 3.0.0
20/12/14 14:45:19 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:390)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:274)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:262)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:807)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:777)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:650)
    at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:303)
    at com.songzixian.wc.Spark01$.main(Spark01.scala:21)
    at com.songzixian.wc.Spark01.main(Spark01.scala)
20/12/14 14:45:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

出现这个问题的原因,并不是程序的错误,而是 windows 系统用到了 hadoop 相关的服
务,解决办法是通过配置关联到 windows 的系统依赖就可以了
微信截图_20201214145318.png
在 IDEA 中配置 Run Configuration,添加 HADOOP_HOME 变量
微信截图_20201214145124.png

微信截图_20201214145521.png

微信截图_20201214145623.png
问题得到解决,再次运行程序,就不报错了

Last modification:December 14th, 2020 at 02:59 pm
如果觉得这篇技术文章对你有用,请随意赞赏