前言
Apache Hadoop 是一款支持数据密集型 分布式应用 并以Apache 2.0许可协议发布的 开源软件框架。它支持在商品硬件构建的大型集群上运行的应用程序。 Hadoop是根据 Google公司发表的 MapReduce和 Google档案系统的论文,由Doug Cutting在2004年完成最初版本。
环境
操作系统:CentOS_6.5_64bit
[root@master conf]# cat /proc/version; lsb_release -a
Linux version 2.6.32-431.5.1.el6.x86_64 (mockbuild@c6b10.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Wed Feb 12 00:41:43 UTC 2014
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final
[root@master conf]# cat /proc/version; lsb_release -a
Linux version 2.6.32-431.5.1.el6.x86_64 (mockbuild@c6b10.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Wed Feb 12 00:41:43 UTC 2014
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final
编译Hadoop版本:hadoop-2.2.0
准备
注:最好可以找到一台可以连接的到互联网的机器进行编译Hadoop,因为准备或者编译的时候都需要从网络安装很多插件。
1. 安装Java JDK(官方网站地址:http://www.oracle.com/technetwork/java/index.html)
由于Hadoop是Doug用java完成的,所以java是必须安装的软件之一。在CentOS环境中可以通过yum进行安装或者自己去oracle官网下载最新版本的JDK。
安装JDK:
[root@master ~]# yum -y install java-1.7.0-openjdk java-1.7.0-openjdk-devel ldapjdk
[root@master ~]# yum -y install java-1.7.0-openjdk java-1.7.0-openjdk-devel ldapjdk
安装完成后目录如下。
[root@master ~]# ll /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/
总用量 216
-r--r--r-- 1 root root 1503 1月 14 20:59 ASSEMBLY_EXCEPTION
drwxr-xr-x 2 root root 4096 3月 11 03:33 bin
drwxr-xr-x 3 root root 4096 3月 11 03:33 include
drwxr-xr-x 4 root root 4096 3月 11 03:33 jre
drwxr-xr-x 3 root root 4096 3月 11 03:33 lib
-r--r--r-- 1 root root 19263 1月 14 20:59 LICENSE
drwxr-xr-x 2 root root 4096 3月 11 03:33 tapset
-r--r--r-- 1 root root 172252 1月 14 20:59 THIRD_PARTY_README
[root@master ~]# ll /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/
总用量 216
-r--r--r-- 1 root root 1503 1月 14 20:59 ASSEMBLY_EXCEPTION
drwxr-xr-x 2 root root 4096 3月 11 03:33 bin
drwxr-xr-x 3 root root 4096 3月 11 03:33 include
drwxr-xr-x 4 root root 4096 3月 11 03:33 jre
drwxr-xr-x 3 root root 4096 3月 11 03:33 lib
-r--r--r-- 1 root root 19263 1月 14 20:59 LICENSE
drwxr-xr-x 2 root root 4096 3月 11 03:33 tapset
-r--r--r-- 1 root root 172252 1月 14 20:59 THIRD_PARTY_README
查看java版本。
[root@master ~]# java -version
java version "1.7.0_51"
OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
[root@master ~]# java -version
java version "1.7.0_51"
OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
2. 安装maven(官方网站地址:http://maven.apache.org/)
Apache Maven ,是一个 软件(特别是 Java软件) 项目管理及 自动构建工具,由 Apache软件基金会所提供。基于项目对象模型(缩写:POM)概念,Maven利用一个中央信息片断能管理一个项目的构建、报告和文档等步骤。曾是 Jakarta项目的子项目,现为独立Apache项目。
yum安装maven
[root@master maven]# yum install apache-maven.noarch
[root@master maven]# yum install apache-maven.noarch
查看maven版本
[root@master maven]# mvn -version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T12:37:52-05:00)
Maven home: /usr/share/apache-maven
Java version: 1.7.0_51, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.5.1.el6.x86_64", arch: "amd64", family: "unix"
[root@master maven]# mvn -version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T12:37:52-05:00)
Maven home: /usr/share/apache-maven
Java version: 1.7.0_51, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.5.1.el6.x86_64", arch: "amd64", family: "unix"
下载安装maven(下载地址:http://maven.apache.org/download.cgi)apache-maven-3.2.1-bin.tar.gz
解压
[root@master ~]# tar -zxvf apache-maven-3.2.1-bin.tar.gz
[root@master ~]# tar -zxvf apache-maven-3.2.1-bin.tar.gz
[root@master ~]# cd apache-maven-3.2.1
[root@master apache-maven-3.2.1]# ll
总用量 40
drwxr-xr-x 2 root root 4096 3月 18 02:23 bin
drwxr-xr-x 2 root root 4096 3月 18 02:23 boot
drwxr-xr-x 3 root root 4096 2月 14 12:34 conf
drwxr-xr-x 3 root root 4096 3月 18 02:23 lib
-rw-r--r-- 1 root root 14865 2月 14 12:40 LICENSE
-rw-r--r-- 1 root root 182 2月 14 12:39 NOTICE
-rw-r--r-- 1 root root 2513 2月 14 12:34 README.txt
[root@master ~]# cd apache-maven-3.2.1
[root@master apache-maven-3.2.1]# ll
总用量 40
drwxr-xr-x 2 root root 4096 3月 18 02:23 bin
drwxr-xr-x 2 root root 4096 3月 18 02:23 boot
drwxr-xr-x 3 root root 4096 2月 14 12:34 conf
drwxr-xr-x 3 root root 4096 3月 18 02:23 lib
-rw-r--r-- 1 root root 14865 2月 14 12:40 LICENSE
-rw-r--r-- 1 root root 182 2月 14 12:39 NOTICE
-rw-r--r-- 1 root root 2513 2月 14 12:34 README.txt
[root@master ~]# cp -r apache-maven-3.2.1 /usr/local/
[root@master ~]# cd /usr/local/apache-maven-3.2.1/
[root@master apache-maven-3.2.1]# ll
总用量 40
drwxr-xr-x 2 root root 4096 3月 18 02:24 bin
drwxr-xr-x 2 root root 4096 3月 18 02:24 boot
drwxr-xr-x 3 root root 4096 3月 18 02:24 conf
drwxr-xr-x 3 root root 4096 3月 18 02:24 lib
-rw-r--r-- 1 root root 14865 3月 18 02:24 LICENSE
-rw-r--r-- 1 root root 182 3月 18 02:24 NOTICE
-rw-r--r-- 1 root root 2513 3月 18 02:24 README.txt
[root@master apache-maven-3.2.1]#
[root@master ~]# cp -r apache-maven-3.2.1 /usr/local/
[root@master ~]# cd /usr/local/apache-maven-3.2.1/
[root@master apache-maven-3.2.1]# ll
总用量 40
drwxr-xr-x 2 root root 4096 3月 18 02:24 bin
drwxr-xr-x 2 root root 4096 3月 18 02:24 boot
drwxr-xr-x 3 root root 4096 3月 18 02:24 conf
drwxr-xr-x 3 root root 4096 3月 18 02:24 lib
-rw-r--r-- 1 root root 14865 3月 18 02:24 LICENSE
-rw-r--r-- 1 root root 182 3月 18 02:24 NOTICE
-rw-r--r-- 1 root root 2513 3月 18 02:24 README.txt
[root@master apache-maven-3.2.1]#
将MAVEN_HOME加入环境变量中。
[root@master apache-maven-3.2.1]# vim /etc/profile
[root@master apache-maven-3.2.1]# vim /etc/profile
在/etc/profile的最后一行加入如下。
export MAVEN_HOME=/usr/local/apache-maven-3.2.1
export PATH=.:$MAVEN_HOME/bin:$PATH
export MAVEN_HOME=/usr/local/apache-maven-3.2.1
export PATH=.:$MAVEN_HOME/bin:$PATH
保存退出。
更新并查看maven版本。
[root@master apache-maven-3.2.1]# source /etc/profile
[root@master apache-maven-3.2.1]# mvn --version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T12:37:52-05:00)
Maven home: /usr/local/apache-maven-3.2.1
Java version: 1.7.0_51, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.5.1.el6.x86_64", arch: "amd64", family: "unix"
[root@master apache-maven-3.2.1]# source /etc/profile
[root@master apache-maven-3.2.1]# mvn --version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T12:37:52-05:00)
Maven home: /usr/local/apache-maven-3.2.1
Java version: 1.7.0_51, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.5.1.el6.x86_64", arch: "amd64", family: "unix"
3. 安装protobuf(官方网站地址:https://code.google.com/p/protobuf/)
protobuf是google提供的一个可以编码格式化结构数据方法,Google大部分的RPC端通信协议都是基于protocol buffers的。同时现Hadoop中master和slave中的RPC通信协议也都是基于它实现的。
安装protobuf之前需要安装几个组件,才可以编译安装protobuf,如已有该些软件,请跳过。
[root@master ~]# yum install gcc gcc-c++ make
[root@master ~]# yum install gcc gcc-c++ make
下载2.5.0版本protobuf(下载地址:https://code.google.com/p/protobuf/downloads/list)protobuf-2.5.0.tar.gz。
[root@master ~]# tar -zxvf protobuf-2.5.0.tar.gz
[root@master ~]# cd protobuf-2.5.0
[root@master protobuf-2.5.0]# ll
总用量 1212
-rw-r--r-- 1 109965 5000 36976 2月 26 2013 aclocal.m4
-rwxr-xr-- 1 109965 5000 1519 2月 26 2013 autogen.sh
-rw-r--r-- 1 109965 5000 25312 2月 26 2013 CHANGES.txt
-rwxr-xr-- 1 109965 5000 44826 2月 26 2013 config.guess
-rw-r--r-- 1 109965 5000 3709 2月 26 2013 config.h.in
-rwxr-xr-- 1 109965 5000 35454 2月 26 2013 config.sub
-rwxr-xr-- 1 109965 5000 599150 2月 26 2013 configure
-rw-r--r-- 1 109965 5000 4829 2月 26 2013 configure.ac
-rw-r--r-- 1 109965 5000 3527 2月 26 2013 CONTRIBUTORS.txt
-rw-r--r-- 1 109965 5000 1732 2月 26 2013 COPYING.txt
-rwxr-xr-- 1 109965 5000 20334 2月 26 2013 depcomp
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 editors
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 examples
-rwxr-xr-- 1 109965 5000 1105 2月 26 2013 generate_descriptor_proto.sh
drwxr-xr-x 15 109965 5000 4096 2月 26 2013 gtest
-rwxr-xr-- 1 109965 5000 13998 2月 26 2013 install-sh
-rw-r--r-- 1 109965 5000 9537 2月 26 2013 INSTALL.txt
drwxr-xr-x 3 109965 5000 4096 2月 26 2013 java
-rw-r--r-- 1 109965 5000 283680 2月 26 2013 ltmain.sh
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 m4
-rw-r--r-- 1 109965 5000 15189 2月 26 2013 Makefile.am
-rw-r--r-- 1 109965 5000 41742 2月 26 2013 Makefile.in
-rwxr-xr-- 1 109965 5000 10346 2月 26 2013 missing
-rw-r--r-- 1 109965 5000 408 2月 26 2013 protobuf-lite.pc.in
-rw-r--r-- 1 109965 5000 429 2月 26 2013 protobuf.pc.in
drwxr-xr-x 3 109965 5000 4096 2月 26 2013 python
-rw-r--r-- 1 109965 5000 5312 2月 26 2013 README.txt
drwxr-xr-x 4 109965 5000 4096 2月 26 2013 src
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 vsprojects
[root@master protobuf-2.5.0]# ./configure --prefix=/usr/local/protoc/
[root@master protobuf-2.5.0]# make && make install
[root@master ~]# tar -zxvf protobuf-2.5.0.tar.gz
[root@master ~]# cd protobuf-2.5.0
[root@master protobuf-2.5.0]# ll
总用量 1212
-rw-r--r-- 1 109965 5000 36976 2月 26 2013 aclocal.m4
-rwxr-xr-- 1 109965 5000 1519 2月 26 2013 autogen.sh
-rw-r--r-- 1 109965 5000 25312 2月 26 2013 CHANGES.txt
-rwxr-xr-- 1 109965 5000 44826 2月 26 2013 config.guess
-rw-r--r-- 1 109965 5000 3709 2月 26 2013 config.h.in
-rwxr-xr-- 1 109965 5000 35454 2月 26 2013 config.sub
-rwxr-xr-- 1 109965 5000 599150 2月 26 2013 configure
-rw-r--r-- 1 109965 5000 4829 2月 26 2013 configure.ac
-rw-r--r-- 1 109965 5000 3527 2月 26 2013 CONTRIBUTORS.txt
-rw-r--r-- 1 109965 5000 1732 2月 26 2013 COPYING.txt
-rwxr-xr-- 1 109965 5000 20334 2月 26 2013 depcomp
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 editors
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 examples
-rwxr-xr-- 1 109965 5000 1105 2月 26 2013 generate_descriptor_proto.sh
drwxr-xr-x 15 109965 5000 4096 2月 26 2013 gtest
-rwxr-xr-- 1 109965 5000 13998 2月 26 2013 install-sh
-rw-r--r-- 1 109965 5000 9537 2月 26 2013 INSTALL.txt
drwxr-xr-x 3 109965 5000 4096 2月 26 2013 java
-rw-r--r-- 1 109965 5000 283680 2月 26 2013 ltmain.sh
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 m4
-rw-r--r-- 1 109965 5000 15189 2月 26 2013 Makefile.am
-rw-r--r-- 1 109965 5000 41742 2月 26 2013 Makefile.in
-rwxr-xr-- 1 109965 5000 10346 2月 26 2013 missing
-rw-r--r-- 1 109965 5000 408 2月 26 2013 protobuf-lite.pc.in
-rw-r--r-- 1 109965 5000 429 2月 26 2013 protobuf.pc.in
drwxr-xr-x 3 109965 5000 4096 2月 26 2013 python
-rw-r--r-- 1 109965 5000 5312 2月 26 2013 README.txt
drwxr-xr-x 4 109965 5000 4096 2月 26 2013 src
drwxr-xr-x 2 109965 5000 4096 2月 26 2013 vsprojects
[root@master protobuf-2.5.0]# ./configure --prefix=/usr/local/protoc/
[root@master protobuf-2.5.0]# make && make install
安装完成后,将protoc写入环境变量。
[root@master protobuf-2.5.0]# vim /etc/profile
[root@master protobuf-2.5.0]# vim /etc/profile
在/etc/profile最后一行中,写入如下两句,并保存退出。
export PROTOC_HOME=/usr/local/protoc
export PATH=.:$PROTOC_HOME/bin:$PATH
export PROTOC_HOME=/usr/local/protoc
export PATH=.:$PROTOC_HOME/bin:$PATH
更新并查看protobuf版本。
[root@master protobuf-2.5.0]# source /etc/profile
[root@master protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
[root@master protobuf-2.5.0]# source /etc/profile
[root@master protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
4. 安装其他依赖
[root@master protobuf-2.5.0]# yum install cmake openssl-devel ncurses-devel
[root@master protobuf-2.5.0]# yum install cmake openssl-devel ncurses-devel
编译安装
从Hadoop官网下载2.2.0稳定版本(下载地址:http://mirrors.cnnic.cn/apache/hadoop/common/stable/)hadoop-2.2.0-src.tar.gz
解压
[root@master hadoop]# tar -zxvf hadoop-2.2.0-src.tar.gz
[root@master hadoop]# tar -zxvf hadoop-2.2.0-src.tar.gz
[root@master hadoop]# cd hadoop-2.2.0-src
root@master hadoop-2.2.0-src]# ll
总用量 108
-rw-r--r-- 1 67974 users 9968 10月 7 01:53 BUILDING.txt
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 dev-support
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-assemblies
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-client
drwxr-xr-x 8 67974 users 4096 10月 7 02:45 hadoop-common-project
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-dist
drwxr-xr-x 6 67974 users 4096 10月 7 02:45 hadoop-hdfs-project
drwxr-xr-x 10 67974 users 4096 10月 7 02:45 hadoop-mapreduce-project
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-maven-plugins
[root@master hadoop]# cd hadoop-2.2.0-src
root@master hadoop-2.2.0-src]# ll
总用量 108
-rw-r--r-- 1 67974 users 9968 10月 7 01:53 BUILDING.txt
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 dev-support
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-assemblies
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-client
drwxr-xr-x 8 67974 users 4096 10月 7 02:45 hadoop-common-project
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-dist
drwxr-xr-x 6 67974 users 4096 10月 7 02:45 hadoop-hdfs-project
drwxr-xr-x 10 67974 users 4096 10月 7 02:45 hadoop-mapreduce-project
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-maven-plugins
注:源码中存在一个BUG详见( https://issues.apache.org/jira/browse/HADOOP-10110 ),解决办法如下。
[root@master hadoop-2.2.0-src]# cd hadoop-common-project/hadoop-auth
[root@master hadoop-auth]# ll
总用量 20
-rw-r--r-- 1 67974 users 629 10月 7 01:52 BUILDING.txt
-rw-r--r-- 1 67974 users 7028 10月 7 01:52 pom.xml
-rw-r--r-- 1 67974 users 545 10月 7 01:52 README.txt
drwxr-xr-x 5 67974 users 4096 10月 7 01:52 src
[root@master hadoop-auth]# vim pom.xml
[root@master hadoop-2.2.0-src]# cd hadoop-common-project/hadoop-auth
[root@master hadoop-auth]# ll
总用量 20
-rw-r--r-- 1 67974 users 629 10月 7 01:52 BUILDING.txt
-rw-r--r-- 1 67974 users 7028 10月 7 01:52 pom.xml
-rw-r--r-- 1 67974 users 545 10月 7 01:52 README.txt
drwxr-xr-x 5 67974 users 4096 10月 7 01:52 src
[root@master hadoop-auth]# vim pom.xml
在pom.xml第55行加入以下内容。
org.mortbay.jetty
jetty-util
test
org.mortbay.jetty
jetty-util
test
保存并退出,BUG解决完成。
退回到起初hadoop-2.2.0-src目录下。
[root@master hadoop-auth]# cd ../../
[root@master hadoop-2.2.0-src]# ll
总用量 108
-rw-r--r-- 1 67974 users 9968 10月 7 01:53 BUILDING.txt
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 dev-support
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-assemblies
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-client
drwxr-xr-x 8 67974 users 4096 10月 7 02:45 hadoop-common-project
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-dist
drwxr-xr-x 6 67974 users 4096 10月 7 02:45 hadoop-hdfs-project
drwxr-xr-x 10 67974 users 4096 10月 7 02:45 hadoop-mapreduce-project
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-maven-plugins
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-minicluster
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-project
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-project-dist
drwxr-xr-x 11 67974 users 4096 10月 7 02:45 hadoop-tools
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-yarn-project
-rw-r--r-- 1 67974 users 15164 10月 7 02:46 LICENSE.txt
-rw-r--r-- 1 67974 users 101 10月 7 02:46 NOTICE.txt
-rw-r--r-- 1 67974 users 16569 10月 7 01:53 pom.xml
-rw-r--r-- 1 67974 users 1366 10月 7 02:46 README.txt
[root@master hadoop-auth]# cd ../../
[root@master hadoop-2.2.0-src]# ll
总用量 108
-rw-r--r-- 1 67974 users 9968 10月 7 01:53 BUILDING.txt
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 dev-support
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-assemblies
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-client
drwxr-xr-x 8 67974 users 4096 10月 7 02:45 hadoop-common-project
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-dist
drwxr-xr-x 6 67974 users 4096 10月 7 02:45 hadoop-hdfs-project
drwxr-xr-x 10 67974 users 4096 10月 7 02:45 hadoop-mapreduce-project
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-maven-plugins
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-minicluster
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-project
drwxr-xr-x 2 67974 users 4096 10月 7 02:45 hadoop-project-dist
drwxr-xr-x 11 67974 users 4096 10月 7 02:45 hadoop-tools
drwxr-xr-x 3 67974 users 4096 10月 7 02:45 hadoop-yarn-project
-rw-r--r-- 1 67974 users 15164 10月 7 02:46 LICENSE.txt
-rw-r--r-- 1 67974 users 101 10月 7 02:46 NOTICE.txt
-rw-r--r-- 1 67974 users 16569 10月 7 01:53 pom.xml
-rw-r--r-- 1 67974 users 1366 10月 7 02:46 README.txt
最后,执行编译命令
[root@master hadoop-2.2.0-src]# mvn package -DskipTests -Pdist,native
[root@master hadoop-2.2.0-src]# mvn package -DskipTests -Pdist,native
编译过程很久,请耐心等待。
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [02:36 min]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [01:17 min]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [ 34.374 s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [ 0.217 s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [ 41.527 s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [ 43.952 s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [ 33.819 s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [ 11.162 s]
[INFO] Apache Hadoop Common .............................. SUCCESS [04:40 min]
[INFO] Apache Hadoop NFS ................................. SUCCESS [ 16.089 s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [ 0.035 s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [02:29 min]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [ 16.069 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [01:15 min]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [ 3.178 s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [ 0.025 s]
[INFO] hadoop-yarn ....................................... SUCCESS [01:17 min]
[INFO] hadoop-yarn-api ................................... SUCCESS [ 41.931 s]
[INFO] hadoop-yarn-common ................................ SUCCESS [ 23.424 s]
[INFO] hadoop-yarn-server ................................ SUCCESS [ 0.077 s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [ 7.492 s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [ 13.799 s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [ 3.021 s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [ 10.077 s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [ 0.305 s]
[INFO] hadoop-yarn-client ................................ SUCCESS [ 4.342 s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [ 0.102 s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [ 3.759 s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [ 0.044 s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [ 19.331 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [ 1.776 s]
[INFO] hadoop-yarn-site .................................. SUCCESS [ 0.088 s]
[INFO] hadoop-yarn-project ............................... SUCCESS [ 10.864 s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [ 16.493 s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [ 2.516 s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [ 8.973 s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [ 4.314 s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [ 5.121 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [ 1.646 s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [ 5.443 s]
[INFO] hadoop-mapreduce .................................. SUCCESS [ 2.021 s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [ 4.114 s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [ 26.639 s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [ 1.992 s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [ 5.204 s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [ 3.827 s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [ 2.590 s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [ 2.658 s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [ 6.587 s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [ 1.103 s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [ 0.023 s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [ 7.099 s]
[INFO] Apache Hadoop Client .............................. SUCCESS [ 2.781 s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [ 0.072 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21:12 min
[INFO] Finished at: 2014-03-18T06:35:50+00:00
[INFO] Final Memory: 141M/461M
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [02:36 min]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [01:17 min]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [ 34.374 s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [ 0.217 s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [ 41.527 s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [ 43.952 s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [ 33.819 s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [ 11.162 s]
[INFO] Apache Hadoop Common .............................. SUCCESS [04:40 min]
[INFO] Apache Hadoop NFS ................................. SUCCESS [ 16.089 s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [ 0.035 s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [02:29 min]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [ 16.069 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [01:15 min]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [ 3.178 s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [ 0.025 s]
[INFO] hadoop-yarn ....................................... SUCCESS [01:17 min]
[INFO] hadoop-yarn-api ................................... SUCCESS [ 41.931 s]
[INFO] hadoop-yarn-common ................................ SUCCESS [ 23.424 s]
[INFO] hadoop-yarn-server ................................ SUCCESS [ 0.077 s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [ 7.492 s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [ 13.799 s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [ 3.021 s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [ 10.077 s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [ 0.305 s]
[INFO] hadoop-yarn-client ................................ SUCCESS [ 4.342 s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [ 0.102 s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [ 3.759 s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [ 0.044 s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [ 19.331 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [ 1.776 s]
[INFO] hadoop-yarn-site .................................. SUCCESS [ 0.088 s]
[INFO] hadoop-yarn-project ............................... SUCCESS [ 10.864 s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [ 16.493 s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [ 2.516 s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [ 8.973 s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [ 4.314 s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [ 5.121 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [ 1.646 s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [ 5.443 s]
[INFO] hadoop-mapreduce .................................. SUCCESS [ 2.021 s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [ 4.114 s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [ 26.639 s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [ 1.992 s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [ 5.204 s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [ 3.827 s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [ 2.590 s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [ 2.658 s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [ 6.587 s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [ 1.103 s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [ 0.023 s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [ 7.099 s]
[INFO] Apache Hadoop Client .............................. SUCCESS [ 2.781 s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [ 0.072 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21:12 min
[INFO] Finished at: 2014-03-18T06:35:50+00:00
[INFO] Final Memory: 141M/461M
[INFO] ------------------------------------------------------------------------
编译安装完成结果信息。
编译完成后根目录。
[root@test hadoop-2.2.0-src]# cd hadoop-dist/target/hadoop-2.2.0/
[root@test hadoop-2.2.0]# ll
总用量
推荐阅读
1 Hadoop中各工程包依赖简述 Google的核心竞争技术是它的计算平台。Google的大牛们用了下面5篇文章,介绍了它们的计算设施。 GoogleCluster:ht ...
[详细]
蜡笔小新 2023-10-17 13:28:20
一、Hadoop来历Hadoop的思想来源于Google在做搜索引擎的时候出现一个很大的问题就是这么多网页我如何才能以最快的速度来搜索到,由于这个问题Google发明 ...
[详细]
蜡笔小新 2023-12-14 18:58:01
本文介绍了在Win10上安装WinPythonHadoop的详细步骤,包括安装Python环境、安装JDK8、安装pyspark、安装Hadoop和Spark、设置环境变量、下载winutils.exe等。同时提醒注意Hadoop版本与pyspark版本的一致性,并建议重启电脑以确保安装成功。 ...
[详细]
蜡笔小新 2023-12-14 11:26:56
本文介绍了解决高并发的核心原则,即将用户访问请求尽量往前推,避免访问CDN、静态服务器、动态服务器、数据库和存储,从而实现高性能、高并发、高可扩展的网站架构。同时提到了Google的成功案例,以及适用于千万级别PV站和亿级PV网站的架构层次。 ...
[详细]
蜡笔小新 2023-12-12 10:56:24
本文介绍了大数据Hadoop生态(20)MapReduce框架原理OutputFormat的开发笔记,包括outputFormat接口实现类、自定义outputFormat步骤和案例。案例中将包含nty的日志输出到nty.log文件,其他日志输出到other.log文件。同时提供了一些相关网址供参考。 ...
[详细]
蜡笔小新 2023-12-10 11:44:06
Maven构建Hadoop工程阅读目录序Maven安装构建示例下载系列索引 序 上一篇,我们编写了第一个MapReduce,并且成功的运行了Job,Hadoop1.x是通过ant ...
[详细]
蜡笔小新 2023-10-17 16:11:18
这篇文章总结的非常到位,故而转之一MapReduce概述MapReduce是一个用于大规模数据处理的分布式计算模型,它最初是由Google工程师设计并实现的ÿ ...
[详细]
蜡笔小新 2023-10-17 12:36:35
本文讨论了Alink回归预测的不完善问题,指出目前主要针对Python做案例,对其他语言支持不足。同时介绍了pom.xml文件的基本结构和使用方法,以及Maven的相关知识。最后,对Alink回归预测的未来发展提出了期待。 ...
[详细]
蜡笔小新 2023-12-14 14:25:33
本文介绍了使用switch语句时的一些用法和注意事项,包括如何实现"fall through"、default语句的作用、在case语句中定义变量时可能出现的问题以及解决方法。同时也提到了C#严格控制switch分支不允许贯穿的规定。通过本文的介绍,读者可以更好地理解和使用switch语句。 ...
[详细]
蜡笔小新 2023-12-13 14:47:39
本文介绍了在mac环境下配置php错误日志的方法,包括修改php.ini文件和httpd.conf文件的操作步骤。同时还介绍了如何修改错误级别,以及相应的错误级别参考链接。 ...
[详细]
蜡笔小新 2023-12-12 11:59:08
本文介绍了在sqoop1.4.*版本中,如何实现自定义分隔符的方法及步骤。通过修改sqoop生成的java文件,并重新编译,可以满足实际开发中对分隔符的需求。具体步骤包括修改java文件中的一行代码,重新编译所需的hadoop包等。详细步骤和编译方法在本文中都有详细说明。 ...
[详细]
蜡笔小新 2023-12-10 11:29:22
nsitionalENhttp:www.w3.orgTRxhtml1DTDxhtml1-transitional.dtd ...
[详细]
蜡笔小新 2023-10-17 10:27:04
nsitionalENhttp:www.w3.orgTRxhtml1DTDxhtml1-transitional.dtd ...
[详细]
蜡笔小新 2023-10-16 18:40:50
本节书摘来自华章社区《Spark核心技术与高级应用》一书中的第1章,第1.2节Spark的重要扩展,作者于俊向海代其锋马海平,更多章节内容可以访问云栖社区“华章社区”公众号查看1. ...
[详细]
蜡笔小新 2023-10-16 18:07:56
我们在之前的文章中已经初步介绍了Cloudera。hadoop基础----hadoop实战(零)-----hadoop的平台版本选择从版本选择这篇文章中我们了解到除了hadoop官方版本外很多 ...
[详细]
蜡笔小新 2023-10-16 14:21:13