找了半天也没有找到一份完整的 Spark 读取 Hive 数据库的代码,好心人能不能提供一份。

找了半天也没有找到一份完整的 Spark 读取 Hive 数据库的代码,好心人能不能提供一份。
1 F281M6Dh8DXpD1g2 Mar 18, 2016 via iPhone 文档看了么? 把 hive 的配置拷到 conf 下面,再 new 一个 hivecontext 就可以了 |
2 anonymoustian OP @liprais 能再详细点吗,看了 spark 的文档,就一两句话。。 |
3 F281M6Dh8DXpD1g2 Mar 18, 2016 @anonymoustian SparkConf cOnf= new SparkConf().setAppName("JavaRFormulaExample"); JavaSparkContext jsc = new JavaSparkContext(conf); HiveContext hiveCOntext= new HiveContext(jsc); |
4 anonymoustian OP |
5 F281M6Dh8DXpD1g2 Mar 18, 2016 @anonymoustian "Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), hdfs-site.xml (for HDFS configuration) file in conf/. Please note when running the query on a YARN cluster (cluster mode), the datanucleus jars under the lib directory and hive-site.xml under conf/ directory need to be available on the driver and all executors launched by the YARN cluster. The convenient way to do this is adding them through the --jars option and --file option of the spark-submit command." 把上述三个文件(hive-site.xml,core-site.xml,hdfs-site.xml)拷到 spark 的 conf 下面就行了 然后读写的时候代码如下 // sc is an existing JavaSparkContext. HiveContext sqlCOntext= new org.apache.spark.sql.hive.HiveContext(sc.sc); // Queries are expressed in HiveQL. sqlContext.sql("select * from YOUR_HIVE_TABLE_NAME").collect(); |
6 anonymoustian OP @liprais spark 的 conf 下面在哪里? |
7 F281M6Dh8DXpD1g2 Mar 18, 2016 |