1.1 为什么要分层
1.2 数仓分层
1.3 数据集市与数据仓库概念
1.4 数仓命名规范
ODS层命名为ods
DWD层命名为dwd
DWS层命名为dws
ADS层命名为ads
临时表数据库命名为xxx_tmp
备份数据数据库命名为xxx_bak
第2章 数仓搭建环境准备
集群规划
2.1 Hive&MySQL安装
请看hive专栏
2.1.2 修改hive-site.xml
1）关闭元数据检查
vim hive-site.xml
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>2.2 Hive运行引擎Tez
Tez是一个Hive的运行引擎，性能优于MR。为什么优于MR呢？看下图。
用Hive直接编写MR程序，假设有四个有依赖关系的MR作业，上图中，绿色是Reduce Task，云状表示写屏蔽，需要将中间结果持久化写到HDFS。
Tez可以将多个有依赖的作业转换为一个作业，这样只需写一次HDFS，且中间节点较少，从而大大提升作业的计算性能。2.2.1 安装包准备
下载tez的依赖包：http://tez.apache.org 解压缩apache-tez-0.9.1-bin.tar.gz
tar -zxvf apache-tez-0.9.1-bin.tar.gz修改名称
mv apache-tez-0.9.1-bin/ tez2.2.2 在Hive中配置Tez
在hive目录
在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置
vim hive-env.sh添加如下配置
export TEZ_HOME=/export/servers/tez-0.9.1 #是你的tez的解压目录
export TEZ_JARS=""
for jar in `ls $TEZ_HOME |grep jar`; do
export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
done
for jar in `ls $TEZ_HOME/lib`; do
export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
done
export HIVE_AUX_JARS_PATH=/export/servers/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS在hive-site.xml文件中添加如下配置，更改hive计算引擎
<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>2.2.3 配置Tez
在Hive的/export/servers/hive/conf下面创建一个tez-site.xml文件
vim tez-site.xml添加如下内容
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<pperty>
<name>tez.lib.uris</name> <value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFroS}/tez/tez-0.9.1/lib</value>
</property>
<property>
<name>tez.lib.uris.classpath</name> <value>${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib</value>
</property>
<property>
<name>tez.use.cluster.hadoop-libs</name>
<value>true</value>
</property>
<property>
<name>tez.history.logging.service.class</name> <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
</property>
</configuration>2.2.4 上传Tez到集群
将/export/servers/tez-0.9.1上传到HDFS的/tez路径
hadoop fs -mkdir /tez
hadoop fs -put /export/servers/tez-0.9.1/ /tez
hadoop fs -ls /tez记得重启一下群集2.2.5 测试
启动Hive
bin/hive创建LZO表
create table student(
id int,
name string);向表中插入数据
insert into student values(1,"zhangsan");如果没有报错就表示成功了
select * from student;2.2.6 小结
运行Tez时检查到用过多内存而被NodeManager杀死进程问题：
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1546781144082_0005 failed 2 times due to AM Container for appattempt_1546781144082_0005_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://hadoop13:8088/cluster/app/application_1546781144082_0005Then, click on links to logs of each attempt.
Diagnostics: Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.这种问题是从机上运行的Container试图使用过多的内存，而被NodeManager kill掉了。
[摘录] The NodeManager is killing your container. It sounds like you are trying to use hadoop streaming which is running as a child process of the map-reduce task. The NodeManager monitors the entire process tree of the task and if it eats up more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb respectively, we would expect the Nodemanager to kill the task, otherwise your task is stealing memory belonging to other containers, which you don't want.解决方法：
方案一：或者是关掉虚拟内存检查。我们选这个，修改yarn-site.xml
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>方案二：mapred-site.xml中设置Map和Reduce任务的内存配置如下：(value中实际配置的内存需要根据自己机器内存大小及应用情况进行修改)
<property>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024M</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2560M</value>
</property>
