hive基于hdfs建表 hive配置hdfs

转载

mob64ca140a59b0 2024-02-02 12:36:02

文章标签 hive基于hdfs建表 hadoop hive apache 文章分类 Hive 大数据

hadoop-hive安装配置

下载hive-0.8.1.tar.gz ---------在ahdoop1.0.0这个版本上要使用这个版本以上

把这个文件解压在/hadoop/app目录下

1.添加bin/hive-config.sh，添加jdk支持

exportJAVA_HOME=/usr/java/jdk1.7.0_02

exportHIVE_HOME=/hadoop/app/hive-0.8.1

exportHADOOP_HOME=/hadoop/app/hadoop-1.0.0

Java代码

root@master:/data/soft#hadoop fs –mkdir /tmp

root@master:/data/soft#hadoop fs –mkdir /usr/hive/warehouse

root@master:/data/soft#hadoop fs –chmod g+w /tmp

root@master:/data/soft# hadoop fs –chmod g+w /usr/hive/warehouse ------------最好把这个目录创建

通过我的试验，以上创建目录的步骤是可以省略的，Hive会自动创建需要的目录

3.修改conf/hive-default.xml，这个是hive的关键配置，所以一般不要直接修改，新建hive-site.xml文件，将修改的内容在这个里面配置。

[root@hadoop-m conf]# cat /hadoop/app/hive-0.8.1/conf/hive-site.xml 
 <configuration>
 <property>
   <name>hive.metastore.local</name>
     <value>true</value>
     </property>


 <property>  
   <name>hive.exec.scratchdir</name>  
     <value>/hadoop/app/hive-0.8.1/tmp</value>  
       <description>Scratch space for Hive jobs</description>  
       </property>  
       <property>  
         <name>hive.querylog.location</name>  
           <value>/hadoop/app/hive-0.8.1/querylog</value>  
           </property>  
           <property>  
             <name>hive.hwi.listen.host</name>  
               <value>0.0.0.0</value>  
                 <description>This is the host address the Hive Web Interface will listen on</description>  
                 </property>  
                 <property>  
                   <name>hive.hwi.listen.port</name>  
                     <value>9999</value>  
                       <description>This is the port the Hive Web Interface will listen on</description>  
                       </property>  
 <property>
   <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://192.168.10.33:3306/hive?characterEncoding=UTF-8</value>
       <description>JDBC connect string for a JDBC metastore</description>
       </property>


 <property>
   <name>hive.metastore.warehouse.dir</name>
      <value>hdfs://master:9000/cucrz/hive/warehouse</value>
         <value>/cucrz/hive/warehouse</value>
           <description>location of default database for the warehouse</description>
           </property>


 <property>
 <name>javax.jdo.option.ConnectionDriverName</name>                     
 <value>com.mysql.jdbc.Driver</value>
 </property>


 <property>     
 <name>javax.jdo.option.ConnectionUserName</name>     
 <value>hive</value>           #mysql登陆名
 <description>username to use against metastore database</description>  
 </property>
   
 <property>     
 <name>javax.jdo.option.ConnectionPassword</name>     
 <value>111</value>             #mysql密码
 <description>password to use against metastore database</description>  
 </property>


  </configuration>

进入/home/hadoop/hive-0.8.1目录

用命令：bin/hive开启hive

Hive使用的语法与SQL差不多

检查：

c. 检查$ hivehive> use dw2;OK

Time taken: 3.43 seconds

hive> create table hwz2(id int, name string ) row format delimited FIELDS TERMINATED BY ','; OK Time taken: 2.519 seconds

hive> show tables;

hwz2

Time taken: 0.419 seconds

hive> load data local inpath 'demo.txt' overwrite into table hwz2;

Copying data from file:/home/hadoop/demo.txt

Copying file: file:/home/hadoop/demo.txt

Loading data to table dw2.hwz2

Deleted hdfs://master:9000/user/hive/warehouse/dw2.db/hwz2

Time taken: 0.557 seconds

hive> select * from hwz2;

12 jack

$ hadoop dfs -lsr /user/hive

Warning: $HADOOP_HOME is deprecated.

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2

-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:48 /user/hive/warehouse/dw2.db/hwz2/demo.txt

drwxr-xr-x - hadoop supergroup 0 2012-03-22 12:36 /user/hive/warehouse/hwz

drwxr-xr-x - hadoop supergroup 0 2012-03-22 15:36 /user/hive/warehouse/hwz2

-rw-r--r-- 2 hadoop supergroup 1201 2012-03-22 15:36 /user/hive/warehouse/hwz2/demo.txt

$ hadoop dfs -cat /user/hive/warehouse/dw2.db/hwz2/demo.txt |head

Warning: $HADOOP_HOME is deprecated.

12,jack

d. 在mysql中验证建立的新表

mysql> use hive;

Database changed

mysql> show tables;

+-----------------+

| Tables_in_hive |

+-----------------+

| BUCKETING_COLS |

| CDS |

| COLUMNS_V2 |

| DATABASE_PARAMS |

| DBS |

| PARTITION_KEYS |

| SDS |

| SD_PARAMS |

| SEQUENCE_TABLE |

| SERDES |

| SERDE_PARAMS |

| SORT_COLS |

| TABLE_PARAMS |

| TBLS |

+-----------------+

14 rows in set (0.00 sec)

错误：

[hadoop@master data]$ hive

WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.

Logging initialized using configuration in jar:file:/app/hive/lib/hive-common-0.9.0.jar!/hive-log4j.properties

Hive history file=/tmp/hadoop/hive_job_log_hadoop_201209251016_560764333.txt

当登录hive的时候老是有个警告：

WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.

根据警告的提示只需要将一个参数值从 org.apache.hadoop.metrics.jvm.EventCounter 修改为 org.apache.hadoop.log.metrics.EventCounter 即可。

解决的办法：

-- 将conf下面所有的配置文件都cp 一份

cd $HIVE_HOME/conf

default.xml.template hive-default.xml

default.xml.template hive-site.xml

cp hive-env.sh.template hive-env.sh

cp hive-log4j.properties.template hive-log4j.properties

--修改参数

vi hive-log4j.properties

找到参数项 log4j.appender.EventCounter

将这一项的值修改为： org.apache.hadoop.log.metrics.EventCounter

修改之后就好了再次登录hive 警告就消失了。

[hadoop@master conf]$ hive Logging initialized using configuration in file:/app/hive/conf/hive-log4j.properties Hive history file=/tmp/hadoop/hive_job_log_hadoop_201209251022_944441939.txt hive> exit

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。