一、集群服务器配置说明(整个过程中我会提前把一些小坑填上,有的坑后面没有提到)
ip | 节点名 | os | cores | memory | disk | remark |
172.25.16.1 | cdh1 | centos7.5 | 40 | 128 | 4t | cloudera server、cloudera agent |
172.25.16.2 | cdh2 | centos7.5 | 4 | 64 | 40t | cloudera agent |
172.25.16.3 | cdh3 | centos7.5 | 4 | 64 | 40t | cloudera agent |
172.25.16.4 | cdh4 | centos7.5 | 4 | 64 | 40t | ccloudera agen |
注1:这里主机名尽量小写,后面配置主机是会出警告
二、cdh6.0.1配套软件配置
- jdk1.8.0_131
- mysql5.7
name |
版本 |
来源 |
cm |
6.0.1 |
|
cdh |
6.0.1 |
|
hadoop |
3.0.0-cdh6.0.1 |
cdh自带包 |
hive |
2.1.1-cdh6.0.1 |
cdh自带包 |
hbase |
2.0.0-cdh6.0.1 |
cdh自带包 |
solr |
7.0.0+cdh6.0.1 |
cdh自带包 |
impala |
3.0.0-cdh6.0.1 |
cdh自带包 |
hue |
3.9.0+cdh6.0.1 |
cdh自带包 |
sqoop |
1.4.7-cdh6.0.1 |
cdh自带包 |
kafka |
1.0.0-cdh6.0.1 |
cdh自带包 |
spark |
2.2.0+cdh6.0.1 |
cdh自带包 |
oozie |
5.0.0-beta1-cdh6.0.1 |
cdh自带包 |
kudu |
1.6.0-cdh6.0.1 |
cdh自带包 |
flume-ng |
flume-ng |
cdh自带包 |
注意:其他组件需要单独安装部署,eg:flink,kylin,elk等
三、基础环境配置
1、关闭防火墙[cdh1-4]
1 临时关闭防火墙 2 3 # systemctl stop firewalld 4 5 永久防火墙开机自启动 6 7 # systemctl disable firewalld 8 9 查看防火墙状态 10 11 # systemctl status firewalld
2、selinux关闭[cdh1-4]
1 # vim /etc/sysconfig/selinux selinux=disabled 2 3 重启机器,命令: 4 5 # reboot 6 7 重启后检查: 8 9 # sestatus –v
3、配置各节点之间ssh免密登录[cdh1-4]
1 1. centos默认没有启动ssh无密登录, 2 3 # vim /etc/ssh/sshd_config 4 5 2、去掉其中3行的注释,每台服务器都要配置 6 7 开启 rsaauthentication yes 8 9 开启 pubkeyauthentication yes 10 11 开启 authorizedkeyfile .ssh/authorized_keys 12 13 # ssh-keygen -t rsa 一直按回车 14 15 # ls /root/.ssh/ id_rsa 和 id_rsa.pub # cp id_rsa.pub authorized_keys 两个woker节点执行 16 17 # cat ~/.ssh/authorized_keys | ssh root@172.20.32.1 'cat >> ~/.ssh/authorized_keys' 18 19 3、cdh1上执行 20 21 # scp authorized_keys root@172.20.32.2:/root/.ssh/ 22 23 # scp authorized_keys root@172.20.32.3:/root/.ssh/ 24 25 # scp known_hosts root@172.20.32.2:/root/.ssh/ 26 27 # scp known_hosts root@172.20.32.3:/root/.ssh/ 28 29 4、cdh2-4执行 30 31 # chmod 700 ~/.ssh 32 33 # chmod 600 ~/.ssh/authorized_keys 34 35 5、重启ssh服务 36 37 # systemctl restart sshd.service 38 39 6、验证能否无密码ssh,在master服务器上执行操作 40 41 # ssh cdh1 42 43 7、不同用户之间退出 44 45 # logout
4、hostname config[cdh1-4]
1 master上执行 2 3 # vim /etc/hosts 4 5 172.25.16.1 cdh1 6 7 172.25.16.2 slave1 8 9 172.25.16.3 slave2 10 11 172.25.16.4 slave3 12 13 通过scp 发送给其他机器 14 15 # for i in 2 3 4 ;do scp /etc/hosts $cdh$i:/etc ;done
5、ntp服务器设置[cdh1-4]
自行配置或看我博客ntp配置文章
6、安装jdk[cdh1-4]
1 方法一:自行安装 2 3 1.下载“jdk-8u131-linux-x64.tar.gz”,放到/usr/java/目录下 4 5 2.解压 # tar -xzvf jdk-8u131-linux-x64.tar.gz 6 7 3.在任何目录下均可编辑 8 9 # vim /etc/profile.d/java_home.sh 10 11 export java_home=/usr/java/jdk1.8.0_141 12 13 export jre_home=$java_home/jre 14 15 export path=$path:$java_home/bin 16 17 export classpath=.:$java_home/lib:$jre_home/lib 18 19 4、刷新环境变量 20 21 # source /etc/profile 22 23 方法二:采用cdh自带jdk包 24 25 下载 对应java包 26 27 wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/rpms/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm 28 29 安装rpm包 30 31 rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm 32 33 修改环境变量 34 35 export java_home=/usr/java/jdk1.8.0_141-cloudera export jre_home=$java_home/jre 36 37 export path=$path:$java_home/bin export classpath=.:$java_home/lib:$jre_home/lib
注2!必须在该目录下安装jdk,后续cdh默认加载此路径的jdk**
7.优化虚拟内存需求率[cdh1-4]
1 cdh1 2 3 # cat /proc/sys/vm/swappiness 4 5 显示如下: 30 6 7 临时降低虚拟内存需求率 8 9 # sysctl vm.swappiness=0 10 11 永久降低虚拟内存需求率 12 13 # echo 'vm.swappiness = 0' > /etc/sysctl.d/swappiness.conf 14 15 并运行如下命令使生效 16 17 # sysctl -p 18 19 --------------------------- 20 21 cdh2-4 检查虚拟内存需求率 22 23 # cat /proc/sys/vm/swappiness 24 25 显示如下: 30 临时降低虚拟内存需求率 26 27 # sysctl vm.swappiness=10 28 29 永久降低虚拟内存需求率 # echo 'vm.swappiness = 10' > /etc/sysctl.d/swappiness.conf 30 31 并运行如下命令使生效 32 33 # sysctl -p
8. 解决透明大页面问题[cdh1-4]
1 检查透明大页面问题 2 3 # cat /sys/kernel/mm/transparent_hugepage/defrag 4 5 如果显示为: [always] madvise never 6 7 临时关闭透明大页面问题 8 9 # echo never > /sys/kernel/mm/transparent_hugepage/defrag 10 11 确认配置生效 12 13 # cat /sys/kernel/mm/transparent_hugepage/defrag 14 15 应该显示为: always madvise [never] 16 17 配置开机自动生效 18 19 # echo never > /sys/kernel/mm/transparent_hugepage/enabled
9.mysql安装[任选择一台机器]
1 下载源文件 2 3 # wget https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm 4 5 安装源文件 6 7 # rpm -ivh mysql57-community-release-el7-9.noarch.rpm 8 9 安装数据库 10 11 # yum install mysql-server 12 13 启动数据库 14 15 # systemctl restart mysqld
10 、拷贝mysql-connector-java到各个节点指定目录下[cdh1-4]
1 1、下载文件 2 3 wget https://dev.mysql.com/get/downloads/connector-j/mysql-connector-java-5.1.46.tar.gz 4 5 2、解压文件 6 7 tar zxvf mysql-connector-java-5.1.46.tar.gz 8 3、创建目录 9 10 mkdir -p /usr/share/java/ cd mysql-connector-java-5.1.46 11 12 4、复制jar包 13 14 cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
11、创建数据库[172.25.16.2]
1 mysql -u root -p 2 3 create database db_cdh6_amon default character set utf8 default collate utf8_general_ci; 4 5 create database db_cdh6_hue default character set utf8 default collate utf8_general_ci; 6 7 create database db_cdh6_metastore default character set utf8 default collate utf8_general_ci; 8 9 create database db_cdh6_sentry default character set utf8 default collate utf8_general_ci; 10 11 create database db_cdh6_oozie default character set utf8 default collate utf8_general_ci; 12 13 create database db_cdh6_hive default character set utf8 default collate utf8_general_ci; 14 15 grant all on db_cdh6_scm.* to 'cdh6'@'%' identified by '123123'; 16 17 grant all on db_cdh6_amon.* to 'cdh6'@'%' identified by '123123'; 18 19 grant all on db_cdh6_hue.* to 'cdh6'@'%' identified by '123123'; 20 21 grant all on db_cdh6_metastore.* to 'cdh6'@'%' identified by '123123'; 22 23 grant all on db_cdh6_sentry.* to 'cdh6'@'%' identified by '123123'; 24 25 grant all on db_cdh6_oozie.* to 'cdh6'@'%' identified by '123123'; 26 27 grant all on db_cdh6_hive.* to 'cdh6'@'%' identified by '123123'; 28 29 flush privileges;
四、cloudera manager安装
1、配置 cloudera manager 仓库cdh1-4(可以选择,也可下载rpm自行手动安装)
1 下载 4 wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/rpms/x86_64/cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm 5 6 wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/rpms/x86_64/cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm 7 8 wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/rpms/x86_64/cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
2.安装 cm server 和 agent
1 1、安装依赖 4 yum -y install bind-utils redhat-lsb psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap httpd mod_ssl openssl-devel python-psycopg2 mysql-python 7 2、cdh1安装: 9 10 [root@cdh1~]# yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server 12 13 3、cdh2-4安装: 14 15 [root@cdh2~]# yum install cloudera-manager-daemons cloudera-manager-agent 16 17 [root@cdh3 ~]# yum install cloudera-manager-daemons cloudera-manager-agent 18 19 [root@cdh4 ~]# yum install cloudera-manager-daemons cloudera-manager-agent
3.设置 cloudera manager 数据库
1 mysql在本地时执行: 2 3 [root@master ~]# rm -rf //cloudera-scm-server/db.mgmt.properties 4 5 [root@master ~]# /opt/cloudera/cm/scm_prepare_database.sh mysql db_cdh6_scm -h172.25.16.206 -uroot -p'123123' --scm-host 172.25.16.206 db_cdh6_scm root 123123 6 7 java_home=/usr/java/jdk1.8.0_131-cloudera 8 9 verifying that we can write to /etc/cloudera-scm-server 10 11 creating scm configuration file in /etc/cloudera-scm-server 12 13 executing: /usr/java/jdk1.8.0_131/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.dbcommandexecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. 14 15 [ main] dbcommandexecutor info successfully connected to database. 16 17 all done, your scm database is configured correctly!
注3 hive报错:
1 jdbc driver cannot be found. unable to find the jdbc database... validate hive metastore schema (70)) 已失败 2 3 <property> 4 5 <name>javax.jdo.option.connectionurl</name> 6 7 <value>jdbc:mysql://my_cluster.com:3306/hive1?useunicode=true&characterencoding=utf-8</value> 8 9 </property> 10 11 <property> 12 13 <name>javax.jdo.option.connectiondrivername</name> 14 15 <value>com.mysql.jdbc.driver</value> 16 17 </property>
4.安装 cdh[master]
配置cdh的软件包 parcels(namenode01)
cm安装成功之后,接下来我们就可以通过cm安装cdh的方式构建企业大数据平台。所以首先需要把cdh的parcels包下载到cm主服务器上。
1 [root@master ~]# cd /opt/cloudera/parcel-repo 2 3 [root@master ~]# wget https://archive.cloudera.com/cdh6/6.0.1/parcels/cdh-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel 4 5 [root@master ~]# wget https://archive.cloudera.com/cdh6/6.0.1/parcels/manifest.json 6 7 [root@master ~]# wget https://archive.cloudera.com/cdh6/6.0.1/parcels/cdh-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256 8 9 [root@master ~]# mv cdh-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256 cdh-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
注意!上面最好指定ip,切勿使用hostname很容易出现问题
也可以执行下面的操作
在manifest.json文件中,找到对应版本的秘钥,复制到.sha文件中
1 [root@master ~]# echo "2e650f1f1ea020a3efc98a231b85c2df1a50b030" > cdh-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha 2 3 修改属主属组 4 5 [root@master ~]# chown cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/ * 6 启动 cloudera manager server[master] 7 [root@master ~]# systemctl start cloudera-scm-server 8 如果启动中有什么问题,可以查看日志。[master] 9 [root@master ~]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
五、 cloudera浏览器配置
1、浏览器配置cloudera manager
浏览器打开:ip:7180
1)登录、
账号:admin
密码:admin
2)选择部署版本
选择免费60天测试,正式版等可以实现需求在进行选择。
3)勾选上需要安装的主机
全部勾选
4)部署cdh
可以看到所下载并安装的cdh版本表示前面部署成功,否则需要在线下载。
5)前面配置无误则无须下载,否则表示前面配置有误。分发可能需要一段时间,视集群大小而定(此时集群一定要保证master可以免密码登录到其它机器上)。
6)检查集群状态
检查集群的状况,可根据提示信息修改配置并重新检查。
a、cdh 托管主机之间的 java 版本不一致
.......
有些地方忘记了,自行百度一下,前边步骤对了问题不大
7)选择需要安装的组件,可自定义安装。 这个可以选择少点,安装好cm之后可以慢慢添加服务。
8)给集群各个节点分配角色,目前所知道需要注意的地方是hbase thrift server不要为空,否则hue无法访问hbase。主要看hive的角色
9)设置mysql连接。
注意: 这里很容易报错,都是表权限、名称、密码的错误。
10)下一步
后面的一路“继续”下去即可。安装所需时间较长,请耐心等待。
后续服务自行安装
错误集锦:
检查主机正确性:
从cdh 6开始,postgresql支持的hue要求psycopg2版本至少为2.5.4,有关更多信息,请参见文章。以下主机缺少psycopg2库的兼容版本
安装psycopg2
pip3 install psycopg2==2.5.4
没有pip3 自行安装 python
hive错误,connector连接错误
cp -r mysql-connector-java-5.1.41.jar /usr/share/java/mysql-connector-java.jar
cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 60。
sysctl vm.swappiness=0 echo 0 > /proc/sys/vm/swappiness echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled
exception in thread "main" java.lang.abstractmethoderror: com.mysql.jdbc.jdbc4connection.getschema()ljava/lang/string;
解决:
1 wget https://dev.mysql.com/get/downloads/connector-j/mysql-connector-java-5.1.46.tar.gz 2 3 tar zxvf mysql-connector-java-5.1.46.tar.gz 4 5 mkdir -p /usr/share/java/ cd mysql-connector-java-5.1.46 6 7 cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
有问题留言!!!!