使用spark从hive中的数据导入到nebula graph报java.net.SocketTimeoutException: Read timed out

原创

江南独孤客 2024-02-21 16:14:40 博主文章分类：大数据篇 ©著作权

文章标签 scala spark java 文章分类 Hadoop 大数据

©著作权归作者所有：来自51CTO博客作者江南独孤客的原创作品，请联系作者获取转载授权，否则将追究法律责任

以下是报错的内容

24/02/20 17:32:21 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on cdh052.dn..com:33318 (size: 12.4 KB, free: 5.2 GB)
24/02/20 17:32:21 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.39:47580
24/02/20 17:32:21 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.39:47566
24/02/20 17:32:21 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.104:52368
24/02/20 17:32:21 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.28:49376
24/02/20 17:32:21 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.39:47570
24/02/20 17:32:21 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.87:48531
24/02/20 17:32:22 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.39:47552
24/02/20 17:32:22 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.179:38064
24/02/20 17:32:22 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.28.7.38:38647
24/02/20 17:33:17 WARN scheduler.TaskSetManager: Lost task 5.0 in stage 1.0 (TID 178, cdh027.dn.tcjf.com, executor 61): com.vesoft.nebula.client.graph.exception.IOErrorException: java.net.SocketTimeoutException: Read timed out
	at com.vesoft.nebula.client.graph.net.SyncConnection.executeWithParameter(SyncConnection.java:191)
	at com.vesoft.nebula.client.graph.net.Session.executeWithParameter(Session.java:117)
	at com.vesoft.nebula.client.graph.net.Session.execute(Session.java:82)
	at com.vesoft.exchange.common.GraphProvider.submit(GraphProvider.scala:78)
	at com.vesoft.exchange.common.writer.NebulaGraphClientWriter.writeEdges(ServerBaseWriter.scala:163)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$com$vesoft$nebula$exchange$processor$EdgeProcessor$$processEachPartition$1.apply(EdgeProcessor.scala:71)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$com$vesoft$nebula$exchange$processor$EdgeProcessor$$processEachPartition$1.apply(EdgeProcessor.scala:69)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor.com$vesoft$nebula$exchange$processor$EdgeProcessor$$processEachPartition(EdgeProcessor.scala:69)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$process$3.apply(EdgeProcessor.scala:183)
	at com.vesoft.nebula.exchange.processor.EdgeProcessor$$anonfun$process$3.apply(EdgeProcessor.scala:183)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2113)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2113)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

解决方案：

hive和nebula-graph的hosts保持一致即可