企业官网建设流程全解析

网站欢迎页面flash,wordpress首页定制,分类网站模版,wordpress全站cdn教程节点信息请参考: zookeeper: Apache Hadoop生态组件部署分享-zookeeper hadoop#xff1a;Apache Hadoop生态组件部署分享-Hadoop 说明: 本次选择的tez版本为0.10.2 ,hive版本为3.1.3 .如果你后续不需要ranger其实可以提高版本hive4.0.1tez0.10.4 tez编译 下载地址: https://d…节点信息请参考:zookeeper: Apache Hadoop生态组件部署分享-zookeeperhadoopApache Hadoop生态组件部署分享-Hadoop说明: 本次选择的tez版本为0.10.2 ,hive版本为3.1.3 .如果你后续不需要ranger其实可以提高版本hive4.0.1tez0.10.4tez编译下载地址: https://dlcdn.apache.org/tez/1、解压到指定路径下2、修改pom.xml依赖主要修改内容为: hadoop.version3.3.5/hadoop.version其次保证:protobuf.version为2.5.0(有点忘记有没有修改了)3、开始执行命令编译mvn clean package -DskipTests -Pnoui注: -Pnoui参数是跳过tez ui模块编译(一是编译会遇到非常多问题,二是感觉用不到 目前看cdp产品的tez ui也是打不开的)4、编译成功后你会在apache-tez-0.10.2-src/tez-dist/target路径下找到tar包后续hive会用到Hive集群部署1、 分发解压并修改名称#将安装包分发到其他两个节点ansible hadoopsrcclustero -m copy -a src/opt/softs/apache-hive-3.1.3-bin.tar.gz destsrc/opt/softs/#解压到指定路径ansible hadoopsrccluster -m shell -a tar -xf /opt/softs/apache-hive-3.1.3-bin.tar.gz -C /opt/apache_v00#为了稍显正式修改其名称ansible hadoopsrccluster -m shell -a mv /opt/apache_v00/apache-hive-3.1.3-bin /opt/apache_v00/apache-hive-3.1.32、hive集成tez前的准备工作#在hdfs上创建存放tez安装包的路径hdfs dfs -mkdir -p /user/tez/pkg #将tez安装包上传至指定路径hdfs dfs -put /opt/softs/tez-0.10.2.tar.gz /user/tez/pkg3、tez-site.xml配置/opt/apache_v00/hadoop-3.3.5/etc/hadoop/tez-site.xml?xml version1.0 encodingUTF-8??xml-stylesheet typetext/xsl hrefconfiguration.xsl?configuration!-- 指定依赖位置 --property nametez.lib.uris/name valuehdfs://nameservice1/user/tez/pkg/tez-0.10.2.tar.gz/value/propertyproperty description控制 Tez 运行时是否使用集群环境中已存在的 Hadoop 库如 HDFS、YARN、MapReduce 的 JAR 文件而不是使用 Tez 自带的版本/description nametez.use.cluster.hadoop-libs/name valuefalse/value/propertyproperty nametez.history.logging.service.class/name valueorg.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService/value/property/configuration然后分发至其他节点以及hive-conf路径下#分发至其他节点的hadoop conf下ansible hadoopsrcclustero -m copy -a src/opt/apache_v00/hadoop-3.3.5/etc/hadoop/tez-site.xml dest/opt/apache_v00/hadoop-3.3.5/etc/hadoop/ #分发至其他节点的hive conf下ansible hadoopsrccluster -m copy -a src/opt/apache_v00/hadoop-3.3.5/etc/hadoop/tez-site.xml dest/opt/apache_v00/apache-hive-3.1.3/conf这里需要重启hadoop集群才会生效,这个我们放在后面重启4、hive-site.xml配置/opt/apache_v00/apache-hive-3.1.3/conf/hive-site.xmlconfiguration property description用于metastore的JDBC连接信息/description namejavax.jdo.option.ConnectionURL/name valuejdbc:mysql://192.168.242.230:3306/hive3?allowMultiQueriestrueamp;useSSLfalseamp;verifyServerCertificatefalse/value /property property namejavax.jdo.option.ConnectionDriverName/name valuecom.mysql.jdbc.Driver/value /property property namejavax.jdo.option.ConnectionUserName/name valuehive/value /property property namejavax.jdo.option.ConnectionPassword/name value123456/value /property property description仓库的默认数据库位置/description namehive.metastore.warehouse.dir/name value/user/hive/warehouse/value /property property description绑定要运行HiveServer2 Thrift服务的主机。/description namehive.server2.thrift.bind.host/name valueapache230.hadoop.com/value /property property description是否应对metastore针对数据库通知相关API如get_next_notification执行授权。如果设置为true则只有代理设置中的超级用户才拥有该权限/description namehive.metastore.event.db.notification.api.auth/name valuefalse/value /property property description远程metastore的Thrift URI。metastore客户端用于连接到远程metastore。/description namehive.metastore.uris/name valuethrift://apache230.hadoop.com:9083,thrift://apache231.hadoop.com:9083/value /property property description需要指定 [tez, mr] 中的一种。选择执行引擎。可选项为tezTez引擎默认值mrMapReduce引擎已弃用。 MR是历史遗留引擎在Hive 2系列中已被弃用未来可能在没有进一步警告的情况下被移除。/description namehive.execution.engine/name valuetez/value /property !--strat hiveserver2高可用--property!--HiveServer2 是否支持客户端的动态服务发现。为支持此功能当前每个 HiveServer2 实例在启动时都会使用 ZooKeeper 进行自我注册。JDBC/ODBC 客户端应在连接字符串中使用 ZooKeeper 集群hive.zookeeper.quorum-- namehive.server2.support.dynamic.service.discovery/name valuetrue/value/propertyproperty!--在zk中的znode名称-- namehive.server2.zookeeper.namespace/name valuehiveserver2/value/propertyproperty namehive.zookeeper.quorum/name valueapache230.hadoop.com:2181,apache231.hadoop.com:2181,apache232.hadoop.com:2181/value/propertyproperty namehive.zookeeper.client.port/name value2181/value/property!--end hiveserver2高可用--/configuration分发至其他节点#分发至其他节点的hive conf下ansible hadoopsrcclustero -m copy -a src/opt/apache_v00/apache-hive-3.1.3/conf/hive-site.xml dest/opt/apache_v00/apache-hive-3.1.3/conf/此时修改其他两个节点绑定的hive.server2.thrift.bind.host 属性为各自主机名称/opt/apache_v00/apache-hive-3.1.3/conf/hive-site.xml5、配置hive环境变量cat /etc/profile.d/apache_v00.sh EOFexport ZOOKEEPER_HOME/opt/apache_v00/apache-zookeeper-3.9.2export HADOOP_HOME/opt/apache_v00/hadoop-3.3.5export HIVE_HOME/opt/apache_v00/apache-hive-3.1.3export TEZ_HOME/opt/apache_v00/tez-0.10.3export PATH$PATH:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/binEOFsource /etc/profile.d/apache_v00.sh6、mysql库及权限准备create database hive3;grant all privileges on hive3.* to hive% identified by 123456;flush privileges;7、执行初始化schematool -initSchema -dbType mysql -verbose当看到 schemaTool completed 即完成8、core-site.xml配置vim /opt/apache_v00/hadoop-3.3.5/etc/hadoop/core-site.xml 新增以下内容property!-- 允许从任何主机来的代理请求 -- namehadoop.proxyuser.root.hosts/name value*/value /property!-- 允许root用户代理所有用户 --property namehadoop.proxyuser.root.users/name value*/value /property9、hadoop-env.sh设置A. 分发并解压tez-0.10.2.tar.gzansible hadoopsrcclustero -m copy -a src/opt/softs/tez-0.10.2.tar.gz dest/opt/softs/ansible hadoopsrccluster -m shell -a mkdir /opt/apache_v00/tez-0.10.2ansible hadoopsrccluster -m shell -a tar -xf /opt/softs/tez-0.10.2.tar.gz -C /opt/apache_v00/tez-0.10.2注: hadoopsrcclustero 是ansible定义的主机列表名称,是其他节点,不包含当前机器,hadoopsrccluster 是包含当前机器B.设置hadoop-env.sh/opt/apache_v00/hadoop-3.3.5/etc/hadoop/hadoop-env.sh 新增export TEZ_CONF_DIR$HADOOP_HOME/etc/hadoopexport TEZ_JARS/opt/apache/tez-0.10.2export HADOOP_CLASSPATH$HADOOP_CLASSPATH:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*分发至其他两个节点10、启动服务重启hadoop服务sh /root/apachecluster/start_apachev00.sh 赋权权限hdfs dfs -chmod 777 /tmphdfs dfs -chmod 777 /user #在apache230.hadoop.com节点执行开两个创建,因为是前台执行hive --service metastorehive --service hiveserver2 #在apache231.hadoop.com节点执行(同上)hive --service metastorehive --service hiveserver2 #在apache232.hadoop.com节点执行hive --service hiveserver2这里的/root/apachecluster/start_apachev00.sh 脚本内容其实就是hadoop篇章的启停hadoop的命令然后你可以验证hive功能了连接方式你可以直接连接也可以走zk方式:beeline -u jdbc:hive2://apache230.hadoop.com:2181,apache231.hadoop.com:2181,apache232.hadoop.com:2181/default;serviceDiscoveryModezooKeeper;zooKeeperNamespacehiveserver2beeline -u jdbc:hive2://apache230.hadoop.com:10000/default验证不同用户去执行需要加-n xxx(因集群没有启动只发一下之前记录部署4.0.1的截图)