环境
三台天翼云主机 (node209, node452, node440)
OS:CentOS 6.5 64位
JDK:Oracle JDK 1.7.0_45
sudo mkdir -p /hadoop/data/yarn/logs
sudo chown -R yarn:yarn /hadoop/data/yarn/logs
NodeManager(node209, node452, node440):
sudo service hadoop-yarn-nodemanager start
MapReduce JobHistory Server(node440):
sudo service hadoop-mapreduce-historyserver start
安装ZooKeeper(集群模式)
Node Type:
node229, node452, node440
1.所有节点安装zookeeper, zookeeper-server
yum install -y zookeeper zookeeper-server
2.所有节点修改zookeeper配置文件
vi /etc/zookeeper/conf/zoo.cfg
增加节点的配置
server.1=node229:2888:3888
server.2=node452:2888:3888
server.3=node440:2888:3888
server.2=node452:2888:3888
server.3=node440:2888:3888
3.所有节点初始化zookeeper-server
每个节点的myid唯一
node229:service zookeeper-server init --myid=1
node452:service zookeeper-server init --myid=2
node440:service zookeeper-server init --myid=3
4.所有节点启动zookeeper
service zookeeper-server start
5.查看zookeeper状态
zookeeper-server status
安装CDH(集群模式,HDFS+YARN)
Node Type:
namenode: node229
datanode: node229, node452, node440
yarn-resourcemanager: node452
yarn-nodemanager: node229, node452, node440
mapreduce-historyserver: node440
yarn-proxyserver: node440
node1:
yum install hadoop-hdfs-namenode
node2:
yum install hadoop-yarn-resourcemanager
node3:
yum install hadoop-mapreduce-historyserver hadoop-yarn-proxyserver
所有节点:
yum install hadoop-client
yum install hadoop-yarn-nodemanager hadoop-hdfs-datanode hadoop-mapreduce
部署CDH
1.部署HDFS
(1) 配置文件
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://node229:8020</value>
</property>
<name>fs.defaultFS</name>
<value>hdfs://node229:8020</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1440</value>
</property>
<name>fs.trash.interval</name>
<value>1440</value>
</property>
hdfs-site.xml
<property>
<name>dfs.permissions.superusergroup</name>
<value>hadoop</value>
</property>
<name>dfs.permissions.superusergroup</name>
<value>hadoop</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/hdfs/namenode</value>
</property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/hdfs/datanode</value>
</property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/hdfs/datanode</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
slaves
node209
node452
node440
(2)创建namenode和datanode文件夹
namenode:
mkdir -p /hadoop/hdfs/namenode
chown -R hdfs:hdfs /hadoop/hdfs/namenode
chmod 700 /hadoop/hdfs/namenode
chown -R hdfs:hdfs /hadoop/hdfs/namenode
chmod 700 /hadoop/hdfs/namenode
datanode:
mkdir -p /hadoop/hdfs/datanode
chown -R hdfs:hdfs /hadoop/hdfs/datanode
chmod 700 /hadoop/hdfs/datanode
mkdir -p /hadoop/hdfs/datanode
chown -R hdfs:hdfs /hadoop/hdfs/datanode
chmod 700 /hadoop/hdfs/datanode
(3)格式化namenode
sudo -u hdfs hadoop namenode -format
(4)启动hdfs
namenode(node209):
service hadoop-hdfs-namenode start
datanode(node209, node452, node440):
service hadoop-hdfs-datanode start
(for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done)
(5)查看hdfs状态
sudo -u hdfs hdfs dfsadmin -report
sudo -u hdfs hadoop fs -ls -R -h /
(6)创建HDFS临时文件夹
sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
2.部署YARN
(1)配置YARN
mapred-site.xml:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>node440:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>node440:19888</value>
</property>
<name>mapreduce.jobhistory.address</name>
<value>node440:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>node440:19888</value>
</property>
yarn-site.xml
<property>
<name>yarn.resourcemanager.address</name>
<value>node452:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>node452:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>node452:8088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>node452:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>node452:8033</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop/data/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/data/yarn/logs</value>
</property>
<property>
<name>yarn.aggregation.enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>node452:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>node452:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>node452:8088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>node452:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>node452:8033</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop/data/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/data/yarn/logs</value>
</property>
<property>
<name>yarn.aggregation.enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
(2)所有nodemanager创建本地目录
sudo mkdir -p /hadoop/data/yarn/local
sudo chown -R yarn:yarn /hadoop/data/yarn/local
sudo chown -R yarn:yarn /hadoop/data/yarn/local
sudo mkdir -p /hadoop/data/yarn/logs
sudo chown -R yarn:yarn /hadoop/data/yarn/logs
(3)创建HDFS目录
sudo -u hdfs hadoop fs -mkdir -p /user/history
sudo -u hdfs hadoop fs -chmod -R 1777 /user/history
sudo -u hdfs hadoop fs -chown yarn /user/history
sudo -u hdfs hadoop fs -chmod -R 1777 /user/history
sudo -u hdfs hadoop fs -chown yarn /user/history
sudo -u hdfs hadoop fs -mkdir -p /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
(4)启动YARN
ResourceManager(node452):
sudo service hadoop-yarn-resourcemanager start
sudo service hadoop-yarn-resourcemanager start
NodeManager(node209, node452, node440):
sudo service hadoop-yarn-nodemanager start
MapReduce JobHistory Server(node440):
sudo service hadoop-mapreduce-historyserver start
(5)创建YARN的HDFS用户目录
sudo -u hdfs hadoop fs -mkdir -p /user/$USER
sudo -u hdfs hadoop fs -chown $USER /user/$USER
sudo -u hdfs hadoop fs -chown $USER /user/$USER
(6)测试
查看节点状态
yarn node -all -list
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar randomwriter input
(7)关闭
sudo service hadoop-yarn-resourcemanager stop
sudo service hadoop-yarn-nodemanager stop
sudo service hadoop-mapreduce-historyserver stop
安装和部署HBase
Node Type:
hbase-master: node229, node440
hbase-regionserver: node229, node452, node440
hbase-thrift: node440
hbase-rest: node229, node452, node440
hbase-regionserver: node229, node452, node440
hbase-thrift: node440
hbase-rest: node229, node452, node440
1.安装HBase
(1)修改配置
/etc/security/limits.conf,增加配置
hdfs - nofile 32768
hbase - nofile 32768
hdfs - nofile 32768
hbase - nofile 32768
hdfs-site.xml,增加配置
<property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
<name>dfs.datanode.max.xcievers</name>
<value>4096</value>
</property>
(2)安装HBase
hbase-master:
sudo yum install hbase hbase-master
hbase-regionserver:
hbase-regionserver:
sudo yum install hbase hbase-regionserver
hbase-thrift:
hbase-thrift:
sudo yum install hbase-thrift
hbase-rest:
hbase-rest:
sudo yum install hbase-rest
(3)配置HBase
hbase-site.xml
<property>
<name>hbase.rest.port</name>
<value>60050</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>node229, node452, node440</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/hadoop/hbase</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://node229:8020/hbase/</value>
<name>hbase.rest.port</name>
<value>60050</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>node229, node452, node440</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.tmp.dir</name>
<value>/hadoop/hbase</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://node229:8020/hbase/</value>
</property>
(4)创建本地目录
mkdir -p /hadoop/hbase
chown -R hbase:hbase /hadoop/hbase
(5)创建hbase的HDFS目录
sudo -u hdfs hadoop fs -mkdir /hbase/
sudo -u hdfs hadoop fs -chown hbase /hbase
(6)启动HBase
hbase-master:
sudo service hbase-master start
hbase-regionserver: sudo service hbase-regionserver start
hbase-thrift: sudo service hbase-thrift start
hbase-rest: sudo service hbase-rest start
http://blog.csdn.net/beckham008/article/details/19028853
相关推荐
1、 请仔细阅读脚本内容是否真实符合本地Linux操作系统和CDH使用版本 2、 参数说明:在Cloudera Manager Server上部署脚本和文档,hostname请参照附件2内的内容填写 3、 执行方法:见附件内详细说明 sh 脚本文件 ...
主要介绍了yum安装CDH5.5 hive、impala的过程详解的相关资料,非常不错 具有一定的参考借鉴价值,需要的朋友可以参考下
3.4 使用本地类库安装CDH5 11 4 HDFS启用HA高可用性(基于Quorum-based Storage) 16 5.CDH安装使用lzo 22 5.1 hadoop_lzo安装 22 5.2 配置MapReduce: 23 5.3相关服务重启 25 6.安装Storm 25 7.附录. 25 7.1 CDH安装...
CDH中yum源制作,制作过程中的每个细节,都在文档中有锁体现,需要的同学们赶快下载,可以节省很多时间哦。
【Centos7.6安装CDH6.1.0】手把手教你搭建CDH6.1.0主节点安装本地yum仓库中的CM组件从节点安装本地yum仓库中的CM组件初始化scm数据库启动CM服务跳转 主节点安装本地yum仓库中的CM组件 root用户下,在mini1上执行: #...
CDH6.2.0、datax-web2.1.2 离线安装文档,从本地yum源配置到数据库安装以及cdh和datax-web的安装,亲测非常详细
CDH 5.14.0 安装笔记,之前安装了5.3.6 之后再安装5.14.0
Centos7.4离线本地yum源安装CDH5.13.生产环境0详细图文教程 1、本文档介绍了如何在Centos上利用本地yum源图文一步步安装CDH5.13.0 2、文档为本人亲自编写,且在真实生产环境部署,请结合本地情况自行参考,本着分享原则...
版本: Impala v0.6 built on Sat Feb 23 18:52:43 PST 2013 使用自定义yum 库Impala的CM详细安装。
Cloudera Hadoop 5&Hadoop高阶管理及调优课程,完整版,提供课件...2、搭建本地Yum部署CDH5的重要组件和优化配置 3、Impala、Oozie和Hue的部署、使用操作及调优 4、Hadoop安全认证及授权管理 5、Hadoop各组件性能调优
本资源包为 cdh6.2.1 redhat7 版本 # 目录结构如下 ├── cdh6 ... └── yum ├── cloudera-manager.repo ├── RPM-GPG-KEY-cloudera └── RPMS └── x86_64 ├── cloudera-manage
cdh6\6.3.0\redhat7\yum\RPMS\noarch cdh6\6.3.0\redhat7\yum\RPMS\x86_64 2个目录 包含hadoop hive hbase bigtop impala spark kudu zookeeper sqoop flume 等等全系列安装rpm包 cdh6\6.3.0\parcels 包含cloudera ...
1 安装方式选择(PATH B方式,配置本地yum源安装): 3 2 软件、安装包下载 4 3 安装Oracle JDK1.8(全部主机) 4 4 安装mysql数据库(server60159) 4 5 配置mysql数据库为InnoDB模式 4 6 创建CDH相关数据库 6 7 ...
CDH是一个整合的大数据平台,文档提供详细的CDH-5.14搭建步骤,包含基本的本地yum源和parcel包和rpm包制作,适合大数据平台初级探索者和学习者
【Centos7.6安装CDH6.1.0】手把手教你搭建CDH6.1.0下载依赖时钟同步安装Python2.7.5跳转 下载依赖 root用户下,在mini1上创建down.sh脚本,脚本内容如下: #!/bin/bash ###########################################...
CDH 5.6 操作系统基本设置 # IP设置 # Hostname 设置 # hosts 设置 # selinux 设置 # 防火墙设置 配置 httpd 服务 # 检查 httpd 是否安装 # 安装 httpd # 启动httpd服务 service httpd restart 设置必要的软件 yum...
在RedHat 7.1 x64环境下通过rpm包及本地yum源离线部署CDH5.12.1
官方下载链接: https://archive.cloudera.com/cdh6/6.2.0/parcels/ 百度网盘下载链接: https://pan.baidu.com/s/1Dm5Elf9uQqn14BUbgU3AFQ 提取码: mws3 主机规划 sparkproject1 manager 8G 50GB sparkproject2 agent...
CDH版本Hadoop 伪分布式环境搭建 服务规划 步骤 第一步:上传压缩包并解压 cd /export/softwares/ tar -zxvf hadoop-2.6.0-cdh5.14.0.tar....yum -y install openssl-devel 第三步:修改配置文件 修改core-site.xml
这包含使用 Hadoop 2.5.0 和 CDH 5.3 在高可用性模式下安装 HDFS 的角色。 这些角色最初基于此处找到的剧本: : 。 这些剧本经过测试可以在 Ubuntu 14.04 Precise 上运行。 但如果需要在 RHEL 上安装,请将apt替换...