Hive优化之自动合并输出的小文件

1.先在hive-site.xml中设置小文件的标准. 

  hive.merge.smallfiles.avgsize
  536870912
  When the average output file size of a job is less than this number, Hive will start an additional map-reduce job to merge the output files into bigger files.  This is only done for map-only jobs if hive.merge.mapfiles is true, and for map-reduce jobs if hive.merge.mapredfiles is true.

2.为只有map的mapreduce的输出并合并小文件.

  hive.merge.mapfiles
  true
  Merge small files at the end of a map-only job

3.为含有reduce的mapreduce的输出并合并小文件.

  hive.merge.mapredfiles
  true
  Merge small files at the end of a map-reduce job

Hive编程指南 PDF 中文高清版  2015-01/111837.htm

基于Hadoop集群的Hive安装 2013-07/87952.htm

Hive内表和外表的区别 2013-07/87313.htm

Hadoop + Hive + Map +reduce 集群安装部署 2013-07/86959.htm

Hive本地独立模式安装 2013-06/86104.htm

Hive学习之WordCount单词统计 2013-04/82874.htm

Hive运行架构及配置部署 2014-08/105508.htm

Hive 的详细介绍：请点这里
Hive 的下载地址：请点这里

：2015-06/118391.htm