Hadoop入门·环境搭建
1 步骤
2 分布式集群环境部署
2.1 硬件环境准备
本案例中使用三台服务器(仅作为学习案例),分别为Hadoop102,Hadoop103,Hadoop104,要求如下(资源充足可多分配):
备注:可使用创建模板机进行克隆操作
2.2 资源下载
资源列表:
Apache产品下载网:Apache Distribution Directory
2.3 环境部署
前提:确保服务上述服务器及资源已准备完全。
建议将服务器防火墙关闭(否则需要配置防火墙出站、入站规则),以确保客户端可以访问相应端口。
集群服务安装位置要求:
2.3.1 服务器环境配置
每一台服务器都需要进行操作(Hadoop102,Hadoop103,Hadoop104)。
主机名配置:
确认主机名是否与要求一致: [my@Hadoop102 ~]$ hostname Hadoop102 [my@Hadoop102 ~]$ 若不一致,可进行如下操作,编辑hostname文件,输入需要的主机名称(需重启生效): [my@Hadoop102 ~]$ sudo vi /etc/hostname [sudo] my 的密码: Hadoop102 ~ ~ ~ "/etc/hostname" 1L, 10C |
配置域名解析文件
编辑hosts文件(便于使用域名解析对应主机ip),如下加入Hadoop集群主机列表: [my@Hadoop102 ~]$ sudo vi /etc/hosts [sudo] my 的密码: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #Hadoop集群主机列表 192.168.10.102 Hadoop102 192.168.10.103 Hadoop103 192.168.10.104 Hadoop104 ~ "/etc/hosts" [noeol] 6L, 258C |
关闭防火墙
切换到root用户: [my@Hadoop102 ~]$ su - root 密码: 上一次登录:六 4月 16 23:00:28 CST 2022pts/1 上 关闭防火墙: [root@Hadoop102 ~]# systemctl stop firewalld 查看防火墙状态: [root@Hadoop102 ~]# systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) 关闭防火墙开机自启: [root@Hadoop102 ~]# systemctl disable firewalld [root@Hadoop102 ~]# |
2.3.2 安装jdk
在Hadoop102上安装
上传jdk安装包
连接sftp会话到Hadoop102进行文件上传 sftp> lcd D:\soft\Hadoop sftp> lls hadoop-3.2.3.tar.gz jdk-8u321-linux-x64.tar.gz sftp> cd /opt/Hadoop/ sftp> ls hadoop-3.2.3 sftp> put jdk-8u321-linux-x64.tar.gz Uploading hadoop-3.2.3.tar.gz to /opt/Hadoop/ jdk-8u321-linux-x64.tar.gz 100% 480705KB 48070KB/s 00:00:10 D:/soft/Hadoop/ jdk-8u321-linux-x64.tar.gz: 592241961 bytes transferred in 18 seconds (48070 KB/s) sftp> |
安装jdk
进入安装包目录,将文件解压到对应的路径下: tar zxvf jdk-8u321-linux-x64.tar.gz -C /opt/java 配置JAVA环境变量: [root@Hadoop102 java]# cd /etc/profile.d/ [root@Hadoop102 profile.d]# ls 256term.csh bash_completion.sh colorls.csh lang.csh less.sh vim.csh which2.csh 256term.sh colorgrep.csh colorls.sh lang.sh vim.sh which2.sh abrt-console-notification.sh colorgrep.sh flatpak.sh less.csh PackageKit.sh vte.sh 在/etc/profile.d/目录下新增myenv.sh,并写入指定环境变量: [root@Hadoop102 profile.d]# vi myenv.sh #JAVA JDK 环境变量 export JAVA_HOME=/opt/java/jdk1.8/jdk1.8.0_321 export PATH=$PATH:$JAVA_HOME/bin ~ ~ "myenv.sh" 6L, 220C 使环境变量生效: [root@Hadoop102 profile.d]# source /etc/profile [root@Hadoop102 profile.d]# 查看是否安装成功,出现以下信息代表成功: [root@Hadoop102 profile.d]# java 用法: java [-options] class [args...] (执行类) 或 java [-options] -jar jarfile [args...] (执行 jar 文件) 其中选项包括: -d32 使用 32 位数据模型 (如果可用) -d64 使用 64 位数据模型 (如果可用) -server 选择 "server" VM 默认 VM 是 server. -cp <目录和 zip/jar 文件的类搜索路径> -classpath <目录和 zip/jar 文件的类搜索路径> 用 : 分隔的目录, JAR 档案 和 ZIP 档案列表, 用于搜索类文件。 -D<名称>=<值> 设置系统属性 -verbose:[class|gc|jni] 启用详细输出 -version 输出产品版本并退出 -version:<值> 警告: 此功能已过时, 将在 未来发行版中删除。 需要指定的版本才能运行 -showversion 输出产品版本并继续 -jre-restrict-search | -no-jre-restrict-search 警告: 此功能已过时, 将在 未来发行版中删除。 在版本搜索中包括/排除用户专用 JRE -? -help 输出此帮助消息 -X 输出非标准选项的帮助 -ea[:<packagename>...|:<classname>] -enableassertions[:<packagename>...|:<classname>] 按指定的粒度启用断言 -da[:<packagename>...|:<classname>] -disableassertions[:<packagename>...|:<classname>] 禁用具有指定粒度的断言 -esa | -enablesystemassertions 启用系统断言 -dsa | -disablesystemassertions 禁用系统断言 -agentlib:<libname>[=<选项>] 加载本机代理库 <libname>, 例如 -agentlib:hprof 另请参阅 -agentlib:jdwp=help 和 -agentlib:hprof=help -agentpath:<pathname>[=<选项>] 按完整路径名加载本机代理库 -javaagent:<jarpath>[=<选项>] 加载 Java 编程语言代理, 请参阅 java.lang.instrument -splash:<imagepath> 使用指定的图像显示启动屏幕 有关详细信息, 请参阅 http://www.oracle.com/technetwork/java/javase/documentation/index.html。 [root@Hadoop102 profile.d]# |
2.3.3 安装Hadoop
在Hadoop102上安装
上传Hadoop安装包
连接sftp会话到Hadoop102进行文件上传 sftp> lcd D:\soft\Hadoop sftp> lls hadoop-3.2.3.tar.gz jdk-8u321-linux-x64.tar.gz sftp> cd /opt/Hadoop/ sftp> ls hadoop-3.2.3 sftp> put hadoop-3.2.3.tar.gz Uploading hadoop-3.2.3.tar.gz to /opt/Hadoop/hadoop-3.2.3.tar.gz 100% 480705KB 48070KB/s 00:00:10 D:/soft/Hadoop/hadoop-3.2.3.tar.gz: 492241961 bytes transferred in 10 seconds (48070 KB/s) sftp> |
安装Hadoop
进入安装包目录,将文件解压到对应的路径下: tar zxvf hadoop-3.2.3.tar.gz -C /opt/Hadoop 更改Hadoop文件所有者: chown -R my:my /opt/Hadoop 配置Hadoop环境变量: [root@Hadoop102 Hadoop]# cd /etc/profile.d/ [root@Hadoop102 profile.d]# ls 256term.csh bash_completion.sh colorls.csh lang.csh less.sh vim.csh which2.csh 256term.sh colorgrep.csh colorls.sh lang.sh vim.sh which2.sh abrt-console-notification.sh colorgrep.sh flatpak.sh less.csh PackageKit.sh vte.sh 在/etc/profile.d/目录下新增myenv.sh,并写入指定环境变量: [root@Hadoop102 profile.d]# vi myenv.sh #JAVA JDK 环境变量 export JAVA_HOME=/opt/java/jdk1.8/jdk1.8.0_321 export PATH=$PATH:$JAVA_HOME/bin #Hadoop环境变量 export HADOOP_HOME=/opt/Hadoop/hadoop-3.2.3 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin ~ ~ "myenv.sh" 6L, 220C 使环境变量生效: [root@Hadoop102 profile.d]# source /etc/profile [root@Hadoop102 profile.d]# 查看是否安装成功,出现以下信息代表成功: [root@Hadoop102 profile.d]# hadoop Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS] or hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS] where CLASSNAME is a user-provided Java class OPTIONS is none or any of: buildpaths attempt to add class files from build tree --config dir Hadoop config directory --debug turn on shell script debug mode --help usage information hostnames list[,of,host,names] hosts to use in slave mode hosts filename list of hosts to use in slave mode loglevel level set the log4j level for this command workers turn on worker mode SUBCOMMAND is one of: Admin Commands: daemonlog get/set the log level for each daemon Client Commands: archive create a Hadoop archive checknative check native Hadoop and compression libraries availability classpath prints the class path needed to get the Hadoop jar and the required libraries conftest validate configuration XML files credential interact with credential providers distch distributed metadata changer distcp copy file or directories recursively dtutil operations related to delegation tokens envvars display computed Hadoop environment variables fs run a generic filesystem user client gridmix submit a mix of synthetic job, modeling a profiled from production load jar <jar> run a jar file. NOTE: please use "yarn jar" to launch YARN applications, not this command. jnipath prints the java.library.path kdiag Diagnose Kerberos Problems kerbname show auth_to_local principal conversion key manage keys via the KeyProvider rumenfolder scale a rumen input trace rumentrace convert logs into a rumen trace s3guard manage metadata on S3 trace view and modify Hadoop tracing settings version print the version Daemon Commands: kms run KMS, the Key Management Server SUBCOMMAND may print help when invoked w/o parameters or with -h. [root@Hadoop102 profile.d]# |
2.3.3 配置Hadoop集群
在Hadoop102上配置
配置相关的主要配置文件(位于Hadoop安装目录的etc文件夹下,配置文件在官网Hadoop源码中有默认配置文件, Hadoop默认使用默认配置):
配置core-site.xml
[root@Hadoop102 hadoop]# pwd /opt/Hadoop/hadoop-3.2.3/etc/hadoop [root@Hadoop102 hadoop]# vi core-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!--NameNode位置,及内部通讯端口--> <property> <name>fs.default.name</name> <value>hdfs://Hadoop102:8020</value> </property> <!--hdfs数据文件存储路径--> <property> <name>hadoop.tmp.dir</name> <value>/opt/Hadoop/hadoop-3.2.3/Datas</value> </property> </configuration> ~ "core-site.xml" 30L, 1102C |
配置hdfs-site.xml
[root@Hadoop102 hadoop]# ls capacity-scheduler.xml hadoop-user-functions.sh.example kms-log4j.properties ssl-client.xml.example configuration.xsl hdfs-site.xml kms-site.xml ssl-server.xml.example container-executor.cfg httpfs-env.sh log4j.properties user_ec_policies.xml.template core-site.xml httpfs-log4j.properties mapred-env.cmd workers hadoop-env.cmd httpfs-signature.secret mapred-env.sh yarn-env.cmd hadoop-env.sh httpfs-site.xml mapred-queues.xml.template yarn-env.sh hadoop-metrics2.properties kms-acls.xml mapred-site.xml yarnservice-log4j.properties hadoop-policy.xml kms-env.sh shellprofile.d yarn-site.xml [root@Hadoop102 hadoop]# vi hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!--2nn web前端访问地址--> <property> <name>dfs.namenode.secondary.http-address</name> <value>Hadoop104:9868</value> </property> <!--nn web前端访问地址--> <property> <name>dfs.namenode.http-address</name> <value>Hadoop102:9870</value> </property> </configuration> ~ "hdfs-site.xml" 30L, 1049C |
配置mapred-site.xml
[root@Hadoop102 hadoop]# vi mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <!--mapreduce处理平台配置,此案例使用yarn--> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> ~ ~ ~ ~ ~ ~ "mapred-site.xml" 25L, 900C |
配置yarn-site.xml
[root@Hadoop102 hadoop]# vi yarn-site.xml <?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <!-- Site specific YARN configuration properties --> <!—配置resourcemanager 安装位置--> <property> <name>yarn.resourcemanager.hostname</name> <value>Hadoop103</value> </property> <property> <!—配置nodemanager 使用服务--> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> ~ ~ ~ ~ "yarn-site.xml" 27L, 888C |
配置集群主机列表
works 配置DataNode节点的主机名或IP,用于集群服务启动时启动对应集群中的Hadoop服务 [root@Hadoop102 hadoop]# vi workers Hadoop102 Hadoop103 Hadoop104 ~ ~ ~ "workers" [noeol] 3L, 29C |
配置集群主机间免密登录
生成RSA密钥: ssh-keygen -t rsa 进入用户家目录中的.ssh目录: cd ~/.ssh 将公钥发送到指定主机,实现Hadoop102 ssh连接到Hadoop103、Hadoop104免密登录 ssh-copy-id -i id_rsa.pub my@Hadoop103 ssh-copy-id -i id_rsa.pub my@Hadoop104 |
Hadoop集群配置分发
将Hadoop集群配置分发复制同步到各个集群主机(Hadoop103、Hadoop104,当然也可以每一台主机分别进行与Hadoop102相同的配置),免密登录需单独仿照“配置集群主机间免密登录”进行配置。
全部完全复制可使用scp命令
更新同步配置可使用rsync命令
此处首次全部复制同步,使用scp进行复制分发。 同步安装Hadoop环境及配置: [root@Hadoop102 hadoop]# scp -r /opt/Hadoop my@Hadoop103:/opt [root@Hadoop102 hadoop]# scp -r /opt/Hadoop my@Hadoop104:/opt 同步环境变量: [root@Hadoop102 hadoop]# scp -r /etc/profile.d/ myenv.sh root@Hadoop103: /etc/profile.d/ [root@Hadoop102 hadoop]# scp -r /etc/profile.d/ myenv.sh root@Hadoop104: /etc/profile.d/ 更改文件所有者(各集群主机保持一致) [root@Hadoop103 hadoop]# chown -R my:my /opt/Hadoop/ [root@Hadoop104 hadoop]# chown -R my:my /opt/Hadoop/ 完成同步后,分别检查各主机上Hadoop是否安装成功 |
启动Hadoop 集群服务
初始化HDFS: [my@Hadoop102 hadoop-3.2.3]$ hdfs namenode -format [my@Hadoop102 hadoop-3.2.3]$ ll 总用量 180 drwxr-xr-x. 2 my my 203 3月 20 09:58 bin drwxrwxr-x. 3 my my 17 4月 17 18:20 Datas drwxr-xr-x. 3 my my 20 3月 20 09:20 etc drwxr-xr-x. 2 my my 106 3月 20 09:58 include drwxr-xr-x. 3 my my 20 3月 20 09:58 lib drwxr-xr-x. 4 my my 288 3月 20 09:58 libexec -rw-rw-r--. 1 my my 150571 3月 10 13:39 LICENSE.txt drwxrwxr-x. 2 my my 35 4月 17 18:20 logs -rw-rw-r--. 1 my my 21943 3月 10 13:39 NOTICE.txt -rw-rw-r--. 1 my my 1361 3月 10 13:39 README.txt drwxr-xr-x. 3 my my 4096 3月 20 09:20 sbin drwxr-xr-x. 4 my my 31 3月 20 10:17 share 启动HDFS集群服务: [my@Hadoop102 ~]$ $HADOOP_HOME/sbin/start-dfs.sh Starting namenodes on [Hadoop102] Starting datanodes Starting secondary namenodes [Hadoop104] 启动yarn服务: [my@Hadoop103 ~]$ /$HADOOP_HOME/sbin/start-yarn.sh Starting resourcemanager Starting nodemanagers 查看运行服务(确认对应服务位置是否与规划一致): [my@Hadoop102 ~]$ jps 9328 Jps 8774 DataNode 9225 NodeManager 8653 NameNode [my@Hadoop102 ~]$ [my@Hadoop103 ~]$ jps 7989 ResourceManager 7670 DataNode 8104 NodeManager 8473 Jps [my@Hadoop103 ~]$ [my@Hadoop104 ~]$ jps 7968 NodeManager 7590 DataNode 7704 SecondaryNameNode 8104 Jps [my@Hadoop104 ~]$ |
启动成功以后在网页上输入对应HDFS网页浏览地址,验证是否成功:
3 备注
3.1 常见错误
3.1.1 集群配置文件错误
启动时若对应xml配置有误,则在集群启动是边会抛出指定配置文件错误,如:
ration>; expected . at [row,col,system-id]: [24,15,"file:/opt/Hadoop/hadoop-3.2.3/etc/hadoop/mapred-site.xml"] 2022-04-17 18:17:48,721 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at Hadoop102/192.168.10.102 ************************************************************/ 2022-04-17 18:17:48,767 ERROR conf.Configuration: error parsing conf mapred-site.xml com.ctc.wstx.exc.WstxParsingException: Unexpected close tag ; expected . at [row,col,system-id]: [24,15,"file:/opt/Hadoop/hadoop-3.2.3/etc/hadoop/mapred-site.xml"] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:634) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:504) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:488) at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3352) at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3279) at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2900) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1121) at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3336) at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3130) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3023) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2984) at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2862) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2844) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1812) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1789) at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183) at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145) at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102) Exception in thread "Thread-1" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Unexpected cl ose tag ; expected . at [row,col,system-id]: [24,15,"file:/opt/Hadoop/hadoop-3.2.3/etc/hadoop/mapred-site.xml"] at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3040) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2984) at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2862) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2844) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1812) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1789) at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183) at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145) at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102) Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag ; expected . at [row,col,system-id]: [24,15,"file:/opt/Hadoop/hadoop-3.2.3/etc/hadoop/mapred-site.xml"] at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:634) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:504) at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:488) at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3352) at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3279) at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2900) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1121) at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3336) at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3130) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3023) ... 10 more |
这时需更改修正指定配置文件错误,并将其分发更新到各个集群服务器上:
[my@Hadoop102 hadoop-3.2.3]$ rsync -av /opt/Hadoop/hadoop-3.2.3/etc/hadoop/mapred-site.xml Hadoop103:/opt/Hadoo p/hadoop-3.2.3/etc/hadoop/ my@hadoop103's password: sending incremental file list mapred-site.xml sent 296 bytes received 43 bytes 96.86 bytes/sec total size is 900 speedup is 2.65 [my@Hadoop102 hadoop-3.2.3]$ rsync -av /opt/Hadoop/hadoop-3.2.3/etc/hadoop/mapred-site.xml Hadoop104:/opt/Hadoo p/hadoop-3.2.3/etc/hadoop/ The authenticity of host 'hadoop104 (192.168.10.104)' can't be established. ECDSA key fingerprint is SHA256:fE+HBwM03RQA+TNPrpQvWYHV46mYltvqrh9psMUXwos. ECDSA key fingerprint is MD5:38:8a:6a:6c:a2:f9:43:2e:e4:99:58:53:aa:84:cc:13. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'hadoop104,192.168.10.104' (ECDSA) to the list of known hosts. my@hadoop104's password: sending incremental file list mapred-site.xml sent 296 bytes received 43 bytes 61.64 bytes/sec total size is 900 speedup is 2.65 初始化HDFS: [my@Hadoop102 hadoop-3.2.3]$ hdfs namenode -format 2022-04-17 18:20:30,242 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = Hadoop102/192.168.10.102 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 3.2.3 STARTUP_MSG: classpath = /opt/Hadoop/hadoop-3.2.3/etc/hadoop:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib /kerb-client-1 .0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/opt/Hadoop/hadoop-3 .2.3/share/had oop/common/lib/kerb-common-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/op t/Hadoop/hadoo p-3.2.3/share/hadoop/common/lib/jetty-http-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/li b/jersey-core1.19.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/hadoop-annotations-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3 /share/hadoop/ common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop /common/lib/jc ip-annotations-1.0-1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/woodstox-core-5.3.0.jar:/opt/Hadoop/h adoop-3.2.3/sh are/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/common s-collections3.2.2.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/netty-3.10.6.Final.jar:/opt/Hadoop/hadoop-3.2.3/shar e/hadoop/commo n/lib/zookeeper-3.4.14.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jersey-server-1.19.jar:/opt/Hadoop/ hadoop-3.2.3/s hare/hadoop/common/lib/paranamer-2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/javax.servlet-api-3.1 .0.jar:/opt/Ha doop/hadoop-3.2.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/com mon/lib/javax. activation-api-1.2.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/guava-27.0-jre.jar:/opt/Hadoop/hadoop -3.2.3/share/h adoop/common/lib/asm-5.0.4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/opt /Hadoop/hadoop -3.2.3/share/hadoop/common/lib/curator-framework-2.13.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/ac cessors-smart2.4.7.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/re2j-1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/c ommon/lib/kerb y-config-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jackson-databind-2.10.5.1.jar:/opt/Hadoop/h adoop-3.2.3/sh are/hadoop/common/lib/commons-beanutils-1.9.4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/failureacces s-1.0.jar:/opt /Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/comm on/lib/commons -cli-1.2.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jersey-json-1.19.jar:/opt/Hadoop/hadoop-3.2.3/sha re/hadoop/comm on/lib/hadoop-auth-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/op t/Hadoop/hadoo p-3.2.3/share/hadoop/common/lib/dnsjava-2.1.7.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jackson-xc-1 .9.13.jar:/opt /Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/j axb-api-2.2.11 .jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/kerby-util-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoo p/common/lib/k erb-core-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/httpclient-4.5.13.jar:/opt/Hadoop/hadoop-3. 2.3/share/hado op/common/lib/nimbus-jose-jwt-9.8.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/curator-client-2.13.0. jar:/opt/Hadoo p/hadoop-3.2.3/share/hadoop/common/lib/commons-text-1.4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/sl f4j-api-1.7.25 .jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jackson-annotations-2.10.5.jar:/opt/Hadoop/hadoop-3.2.3/s hare/hadoop/co mmon/lib/jaxb-impl-2.2.3-1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jersey-servlet-1.19.jar:/opt/Ha doop/hadoop-3. 2.3/share/hadoop/common/lib/jetty-xml-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/slf 4j-log4j12-1.7 .25.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jetty-server-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3 .2.3/share/had oop/common/lib/gson-2.2.4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/opt/Had oop/hadoop-3.2 .3/share/hadoop/common/lib/spotbugs-annotations-3.1.9.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/chec ker-qual-2.5.2 .jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jetty-webapp-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2. 3/share/hadoop /common/lib/metrics-core-3.2.4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/token-provider-1.0.1.jar:/o pt/Hadoop/hado op-3.2.3/share/hadoop/common/lib/jetty-security-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/comm on/lib/snappyjava-1.0.5.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/commons-net-3.6.jar:/opt/Hadoop/hadoop-3.2.3/sh are/hadoop/com mon/lib/kerb-server-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/opt/Hadoop /hadoop-3.2.3/ share/hadoop/common/lib/commons-compress-1.21.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/httpcore-4.4 .13.jar:/opt/H adoop/hadoop-3.2.3/share/hadoop/common/lib/jsr305-3.0.2.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/lo g4j-1.2.17.jar :/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop /common/lib/je tty-util-ajax-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/opt/H adoop/hadoop-3 .2.3/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/common s-codec-1.11.j ar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/commons-io-2.8.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/ common/lib/jet ty-servlet-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/avro-1.7.7.jar:/opt/Hadoop/had oop-3.2.3/shar e/hadoop/common/lib/kerby-asn1-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/commons-lang3-3.7.jar :/opt/Hadoop/h adoop-3.2.3/share/hadoop/common/lib/json-smart-2.4.7.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/kerbutil-1.0.1.jar :/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/co mmon/lib/kerbsimplekdc-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/Hadoop/hadoo p-3.2.3/share/ hadoop/common/lib/jackson-core-2.10.5.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/kerby-xdr-1.0.1.jar: /opt/Hadoop/ha doop-3.2.3/share/hadoop/common/lib/jsch-0.1.55.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jettison-1. 1.jar:/opt/Had oop/hadoop-3.2.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/l ib/commons-log ging-1.1.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/opt/Hadoop /hadoop-3.2.3/ share/hadoop/common/lib/error_prone_annotations-2.2.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/jett y-io-9.4.40.v2 0210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/Hadoop/hadoop3.2.3/share/ha doop/common/lib/jetty-util-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/lib/stax2-api-4.2. 1.jar:/opt/Had oop/hadoop-3.2.3/share/hadoop/common/hadoop-kms-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/common/hadoop-n fs-3.2.3.jar:/ opt/Hadoop/hadoop-3.2.3/share/hadoop/common/hadoop-common-3.2.3-tests.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop /common/hadoop -common-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/ker b-client-1.0.1 .jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/opt/Hadoop/hadoop-3.2.3/s hare/hadoop/hd fs/lib/kerb-common-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/opt/Hadoop/h adoop-3.2.3/sh are/hadoop/hdfs/lib/jetty-http-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jersey-core1.19.jar:/opt/ Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/hadoop-annotations-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hd fs/lib/listena blefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jcip-ann otations-1.0-1 .jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/woodstox-core-5.3.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hado op/hdfs/lib/ht race-core4-4.1.0-incubating.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/o pt/Hadoop/hado op-3.2.3/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/zookeeper3.4.14.jar:/op t/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/l ib/paranamer-2 .3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/Hadoop/hadoop-3.2.3/share/had oop/hdfs/lib/j avax.servlet-api-3.1.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/Hado op/hadoop-3.2. 3/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/javax.activation-a pi-1.2.0.jar:/ opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib /asm-5.0.4.jar :/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoo p/hdfs/lib/cur ator-framework-2.13.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/accessors-smart-2.4.7.jar:/opt/Hadoop/ hadoop-3.2.3/s hare/hadoop/hdfs/lib/re2j-1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/opt/Ha doop/hadoop-3. 2.3/share/hadoop/hdfs/lib/jackson-databind-2.10.5.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/commonsbeanutils-1.9. 4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoo p/hdfs/lib/com mons-cli-1.2.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/opt/Hadoop/hadoop-3.2.3/s hare/hadoop/hd fs/lib/hadoop-auth-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/ Hadoop/hadoop3.2.3/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-xc-1.9.13. jar:/opt/Hadoo p/hadoop-3.2.3/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-u til-1.0.1.jar: /opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/l ib/httpclient4.5.13.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar:/opt/Hadoop/hadoop-3.2.3/sh are/hadoop/hdf s/lib/curator-client-2.13.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-text-1.4.jar:/opt/Hadoop /hadoop-3.2.3/ share/hadoop/hdfs/lib/netty-all-4.1.68.Final.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-annotat ions-2.10.5.ja r:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hd fs/lib/jerseyservlet-1.19.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-xml-9.4.40.v20210413.jar:/opt/Hadoop/hado op-3.2.3/share /hadoop/hdfs/lib/jetty-server-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/gson-2.2.4.ja r:/opt/Hadoop/ hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/spotb ugs-annotation s-3.1.9.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/opt/Hadoop/hadoop-3.2.3/shar e/hadoop/hdfs/ lib/jetty-webapp-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/ opt/Hadoop/had oop-3.2.3/share/hadoop/hdfs/lib/jetty-security-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/ lib/snappy-jav a-1.0.5.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-net-3.6.jar:/opt/Hadoop/hadoop-3.2.3/share/h adoop/hdfs/lib /kerb-server-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/opt/Hadoop/hadoop-3 .2.3/share/had oop/hdfs/lib/commons-compress-1.21.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/opt/ Hadoop/hadoop3.2.3/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/o pt/Hadoop/hado op-3.2.3/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-uti l-ajax-9.4.40. v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/shar e/hadoop/hdfs/ lib/commons-daemon-1.0.13.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/opt/Ha doop/hadoop-3. 2.3/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-io-2.8. 0.jar:/opt/Had oop/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-servlet-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop /hdfs/lib/avro -1.7.7.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/h adoop/hdfs/lib /commons-lang3-3.7.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/json-smart-2.4.7.jar:/opt/Hadoop/hadoop-3 .2.3/share/had oop/hdfs/lib/kerb-util-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/opt/Hadoo p/hadoop-3.2.3 /share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-jaxrs-1. 9.13.jar:/opt/ Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/okio-1.6.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson -core-2.10.5.j ar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdf s/lib/jsch-0.1 .55.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jettison-1.1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/h dfs/lib/protob uf-java-2.5.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/Hadoop/hadoop-3 .2.3/share/had oop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/error_prone_ann otations-2.2.0 .jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-io-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/shar e/hadoop/hdfs/ lib/audience-annotations-0.5.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/opt/Hadoop/ hadoop-3.2.3/s hare/hadoop/hdfs/lib/jetty-util-9.4.40.v20210413.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/stax2-api-4 .2.1.jar:/opt/ Hadoop/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/hadoop-h dfs-3.2.3-test s.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-client-3.2.3-tests.jar:/opt/Hadoop/hadoop-3.2.3/sh are/hadoop/hdf s/hadoop-hdfs-nfs-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.3.jar:/opt/Hadoop/ha doop-3.2.3/sha re/hadoop/hdfs/hadoop-hdfs-native-client-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-httpf s-3.2.3.jar:/o pt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.3-tests.jar:/opt/Hadoop/hadoop-3.2.3/sha re/hadoop/hdfs /hadoop-hdfs-client-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.3-tests.jar:/opt/H adoop/hadoop-3 .2.3/share/hadoop/mapreduce/lib/junit-4.13.2.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/lib/hamcrest-c ore-1.3.jar:/o pt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.3.jar:/opt/Hadoop/hadoop-3. 2.3/share/hado op/mapreduce/hadoop-mapreduce-examples-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapredu ce-client-app3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.2.3.jar:/opt/Had oop/hadoop-3.2 .3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/ma preduce/hadoop -mapreduce-client-jobclient-3.2.3-tests.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-cl ient-uploader3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.2.3.jar:/opt/Hadoop/ha doop-3.2.3/sha re/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/hadoop -mapreduce-cli ent-common-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.2.3.jar: /opt/Hadoop/ha doop-3.2.3/share/hadoop/yarn:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/opt/Hadoop/h adoop-3.2.3/sh are/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/json-io-2.5.1. jar:/opt/Hadoo p/hadoop-3.2.3/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/e hcache-3.3.1.j ar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop /yarn/lib/jack son-jaxrs-json-provider-2.10.5.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/Hadoo p/hadoop-3.2.3 /share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/jakarta.xml.bind-api2.3.2.jar:/opt /Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/li b/objenesis-1. 0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/snakeyaml-1.26.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/y arn/lib/bcprov -jdk15on-1.60.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.10.5.jar:/op t/Hadoop/hadoo p-3.2.3/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/ya rn/lib/jackson -jaxrs-base-2.10.5.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/fst-2.50.jar:/opt/Hadoop/hadoop-3.2.3/sha re/hadoop/yarn /lib/mssql-jdbc-6.2.1.jre7.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/guice-4.0.jar:/opt/Hadoop/hadoop3.2.3/share/ha doop/yarn/lib/java-util-1.9.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/jersey-client-1.19.jar:/opt/Ha doop/hadoop-3. 2.3/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/jakarta.activati on-api-1.2.1.j ar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-services-api-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share /hadoop/yarn/h adoop-yarn-server-common-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-client-3.2.3.jar:/opt /Hadoop/hadoop -3.2.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/shar e/hadoop/yarn/ hadoop-yarn-submarine-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3 .2.3.jar:/opt/ Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-common-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/h adoop-yarn-reg istry-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.2.3.jar:/opt /Hadoop/hadoop -3.2.3/share/hadoop/yarn/hadoop-yarn-server-router-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoopyarn-server-ap plicationhistoryservice-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-api-3.2.3.jar:/opt/Had oop/hadoop-3.2 .3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-y arn-server-tes ts-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.2.3.jar:/opt/Hadoop/ha doop-3.2.3/sha re/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/h adoop-yarn-ser vices-core-3.2.3.jar:/opt/Hadoop/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.2.3 .jar STARTUP_MSG: build = https://github.com/apache/hadoop -r abe5358143720085498613d399be3bbf01e0f131; compiled b y 'ubuntu' on 2022-03-20T01:18Z STARTUP_MSG: java = 1.8.0_321 ************************************************************/ 2022-04-17 18:20:30,255 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2022-04-17 18:20:30,438 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-98655ee1-8960-41ad-bc6d-2bb95651b12a 2022-04-17 18:20:31,781 INFO namenode.FSEditLog: Edit logging is async:true 2022-04-17 18:20:31,851 INFO namenode.FSNamesystem: KeyProvider: null 2022-04-17 18:20:31,853 INFO namenode.FSNamesystem: fsLock is fair: true 2022-04-17 18:20:31,853 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false 2022-04-17 18:20:31,872 INFO namenode.FSNamesystem: fsOwner = my (auth:SIMPLE) 2022-04-17 18:20:31,872 INFO namenode.FSNamesystem: supergroup = supergroup 2022-04-17 18:20:31,872 INFO namenode.FSNamesystem: isPermissionEnabled = true 2022-04-17 18:20:31,872 INFO namenode.FSNamesystem: HA Enabled: false 2022-04-17 18:20:31,952 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profi ling 2022-04-17 18:20:31,968 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, coun ted=60, effect ed=1000 2022-04-17 18:20:31,968 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-ch eck=true 2022-04-17 18:20:31,974 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00: 00.000 2022-04-17 18:20:31,981 INFO blockmanagement.BlockManager: The block deletion will start around 2022 四月 17 18:2 0:31 2022-04-17 18:20:31,985 INFO util.GSet: Computing capacity for map BlocksMap 2022-04-17 18:20:31,985 INFO util.GSet: VM type = 64-bit 2022-04-17 18:20:31,986 INFO util.GSet: 2.0% max memory 481.4 MB = 9.6 MB 2022-04-17 18:20:31,987 INFO util.GSet: capacity = 2^20 = 1048576 entries 2022-04-17 18:20:32,001 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled 2022-04-17 18:20:32,001 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false 2022-04-17 18:20:32,011 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990 000128746033 2022-04-17 18:20:32,011 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0 2022-04-17 18:20:32,011 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000 2022-04-17 18:20:32,012 INFO blockmanagement.BlockManager: defaultReplication = 3 2022-04-17 18:20:32,013 INFO blockmanagement.BlockManager: maxReplication = 512 2022-04-17 18:20:32,013 INFO blockmanagement.BlockManager: minReplication = 1 2022-04-17 18:20:32,013 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 2022-04-17 18:20:32,013 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms 2022-04-17 18:20:32,013 INFO blockmanagement.BlockManager: encryptDataTransfer = false 2022-04-17 18:20:32,013 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 2022-04-17 18:20:32,050 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911 2022-04-17 18:20:32,050 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215 2022-04-17 18:20:32,050 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215 2022-04-17 18:20:32,050 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215 2022-04-17 18:20:32,067 INFO util.GSet: Computing capacity for map INodeMap 2022-04-17 18:20:32,068 INFO util.GSet: VM type = 64-bit 2022-04-17 18:20:32,068 INFO util.GSet: 1.0% max memory 481.4 MB = 4.8 MB 2022-04-17 18:20:32,068 INFO util.GSet: capacity = 2^19 = 524288 entries 2022-04-17 18:20:32,069 INFO namenode.FSDirectory: ACLs enabled? false 2022-04-17 18:20:32,069 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true 2022-04-17 18:20:32,069 INFO namenode.FSDirectory: XAttrs enabled? true 2022-04-17 18:20:32,069 INFO namenode.NameNode: Caching file names occurring more than 10 times 2022-04-17 18:20:32,074 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccess TimeOnlyChange : false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536 2022-04-17 18:20:32,079 INFO snapshot.SnapshotManager: SkipList is disabled 2022-04-17 18:20:32,097 INFO util.GSet: Computing capacity for map cachedBlocks 2022-04-17 18:20:32,098 INFO util.GSet: VM type = 64-bit 2022-04-17 18:20:32,098 INFO util.GSet: 0.25% max memory 481.4 MB = 1.2 MB 2022-04-17 18:20:32,098 INFO util.GSet: capacity = 2^17 = 131072 entries 2022-04-17 18:20:32,108 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 2022-04-17 18:20:32,108 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 2022-04-17 18:20:32,108 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 2022-04-17 18:20:32,112 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 2022-04-17 18:20:32,112 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache ent ry expiry time is 600000 millis 2022-04-17 18:20:32,120 INFO util.GSet: Computing capacity for map NameNodeRetryCache 2022-04-17 18:20:32,120 INFO util.GSet: VM type = 64-bit 2022-04-17 18:20:32,120 INFO util.GSet: 0.029999999329447746% max memory 481.4 MB = 147.9 KB 2022-04-17 18:20:32,120 INFO util.GSet: capacity = 2^14 = 16384 entries 2022-04-17 18:20:32,173 INFO namenode.FSImage: Allocated new BlockPoolId: BP-56363176-192.168.10.102-1650190832 146 2022-04-17 18:20:32,204 INFO common.Storage: Storage directory /opt/Hadoop/hadoop-3.2.3/Datas/dfs/name has been successfully formatted. 2022-04-17 18:20:32,254 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/Hadoop/hadoop-3.2.3/Datas/d fs/name/curren t/fsimage.ckpt_0000000000000000000 using no compression 2022-04-17 18:20:32,406 INFO namenode.FSImageFormatProtobuf: Image file /opt/Hadoop/hadoop-3.2.3/Datas/dfs/name /current/fsima ge.ckpt_0000000000000000000 of size 397 bytes saved in 0 seconds . 2022-04-17 18:20:32,420 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 2022-04-17 18:20:32,438 INFO namenode.FSNamesystem: Stopping services started for active state 2022-04-17 18:20:32,438 INFO namenode.FSNamesystem: Stopping services started for standby state 2022-04-17 18:20:32,442 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown. 2022-04-17 18:20:32,443 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at Hadoop102/192.168.10.102 ************************************************************/ [my@Hadoop102 hadoop-3.2.3]$ ll 总用量 180 drwxr-xr-x. 2 my my 203 3月 20 09:58 bin drwxrwxr-x. 3 my my 17 4月 17 18:20 Datas drwxr-xr-x. 3 my my 20 3月 20 09:20 etc drwxr-xr-x. 2 my my 106 3月 20 09:58 include drwxr-xr-x. 3 my my 20 3月 20 09:58 lib drwxr-xr-x. 4 my my 288 3月 20 09:58 libexec -rw-rw-r--. 1 my my 150571 3月 10 13:39 LICENSE.txt drwxrwxr-x. 2 my my 35 4月 17 18:20 logs -rw-rw-r--. 1 my my 21943 3月 10 13:39 NOTICE.txt -rw-rw-r--. 1 my my 1361 3月 10 13:39 README.txt drwxr-xr-x. 3 my my 4096 3月 20 09:20 sbin drwxr-xr-x. 4 my my 31 3月 20 10:17 share |
3.1.2 集群未配置SSH免密登录
未配置免密登录,启动HDFS集群服务时会抛出以下错误: [my@Hadoop102 current]$ $HADOOP_HOME/sbin/start-dfs.sh Starting namenodes on [Hadoop102] Hadoop102: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Starting datanodes Hadoop104: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Hadoop102: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Hadoop103: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Starting secondary namenodes [Hadoop104] Hadoop104: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [my@Hadoop102 current]$ $HADOOP_HOME/sbin/start-dfs.sh Starting namenodes on [Hadoop102] Hadoop102: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Starting datanodes Hadoop103: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Hadoop104: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Hadoop102: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). 1Starting secondary namenodes [Hadoop104] Hadoop104: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). |
此时需要配置RSA密钥,进行免密认证:
[my@Hadoop102 current]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/my/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/my/.ssh/id_rsa. Your public key has been saved in /home/my/.ssh/id_rsa.pub. The key fingerprint is: SHA256:hrIi7CAZx5vqX0VertHrvbDi7lO1eKMPqECOaaJ6auw my@Hadoop102 The key's randomart image is: +---[RSA 2048]----+ | | | | | . . | | . o.+ . | |. o.. .+Soo . | |.+=o o..+o.+ | |B=+o.. o.+o . | |B=..o .o..= | |XE.. .++oo.+. | +----[SHA256]-----+ [my@Hadoop102 current]$ cd ~ [my@Hadoop102 ~]$ ll 总用量 0 drwxr-xr-x. 2 my my 6 4月 16 13:59 公共 drwxr-xr-x. 2 my my 6 4月 16 13:59 模板 drwxr-xr-x. 2 my my 6 4月 16 13:59 视频 drwxr-xr-x. 2 my my 6 4月 16 13:59 图片 drwxr-xr-x. 2 my my 6 4月 16 13:59 文档 drwxr-xr-x. 2 my my 6 4月 16 13:59 下载 drwxr-xr-x. 2 my my 6 4月 16 13:59 音乐 drwxr-xr-x. 2 my my 6 4月 16 13:59 桌面 [my@Hadoop102 ~]$ ls -a .. .bash_history .bash_profile ..ccaacchhee .esd_auth ..llooccaall ..sssshh 公共 视频 文档 音乐 .... .bash_logout .bashrc ..ccoonnffiigg .ICEauthority ..mmoozziillllaa .viminfo 模板 图片 下载 桌面 [my@Hadoop102 ~]$ cd .ssh [my@Hadoop102 .ssh]$ ls id_rsa id_rsa.pub known_hosts [my@Hadoop102 .ssh]$ ssh-copy-id -i id_rsa.pub Hadoop103 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already instal led /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys my@hadoop103's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'Hadoop103'" and check to make sure that only the key(s) you wanted were added. [my@Hadoop102 .ssh]$ ssh-copy-id -i id_rsa.pub Hadoop104 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already instal led /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys my@hadoop104's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'Hadoop104'" and check to make sure that only the key(s) you wanted were added. [my@Hadoop102 .ssh]$ $HADOOP_HOME/sbin/start-dfs.sh Starting namenodes on [Hadoop102] Hadoop102: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Starting datanodes Hadoop102: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Hadoop103: WARNING: /opt/Hadoop/hadoop-3.2.3/logs does not exist. Creating. Hadoop104: WARNING: /opt/Hadoop/hadoop-3.2.3/logs does not exist. Creating. Starting secondary namenodes [Hadoop104] [my@Hadoop102 .ssh]$ ssh-copy-id -i id_rsa.pub Hadoop102 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already instal led /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys my@hadoop102's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'Hadoop102'" and check to make sure that only the key(s) you wanted were added. |
3.1.3 使用其它非集群用户启动集群服务
[root@Hadoop102 hadoop-3.2.3]# $HADOOP_HOME/sbin/start-dfs.sh Starting namenodes on [Hadoop102] ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation. Starting datanodes ERROR: Attempting to operate on hdfs datanode as root ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation. Starting secondary namenodes [Hadoop104] ERROR: Attempting to operate on hdfs secondarynamenode as root ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation. 此时使用原本配置的集成用户启动集群服务便可。 |
文章浏览阅读5.4k次,点赞2次,收藏20次。ONVIF Discovery 协议介绍
文章浏览阅读54次。题目地址:http://acm.hdu.edu.cn/showproblem.php?pid=1172题目概述: 对于每组数据,先给出一个n,然后n行每行一个四位数,然后两个数b,c表示与答案有b个数相同,c个数在同一位置上,对于所有的n个给出的四位数,如果推出的答案唯一则输出,否则输出“Not sure”(不含引号)大致思路: 首先我想的是用搜索。穷极一生之力写了100+行的...
文章浏览阅读2w次,点赞6次,收藏37次。刚工作时候第一次看到接触到视图,感觉很不能理解,我当时就在想为什么要用视图呢?明明可以关联表查出来数据的,为什么又单独创建一个视图查询数据呢?工作久了我有以下几点心得分享给大家.1.安全性 因为视图是虚拟的,物理上是不存在的,只是存储了数据的集合,视图中数据是根据基表的更新而更新,用户不可以更改/删除视图。2.方便,简化数据操作当我们业务需求要查出多张表的数据,这时我们可能会关联多张表查询处理.如..._哪种情况更适合建立视图
文章浏览阅读643次,点赞4次,收藏6次。Anaconda的安装及基本概念Anaconda中通过conda进行python包的管理Anaconda中通过conda进行虚拟环境的管理为形成良好的编程习惯,为每一个项目单独创建一个虚拟环境对开发来说至关重要。这一习惯即使对非数据data项目(如Web App)也能十分有效的帮助管理开发环境。_anaconda csdn
文章浏览阅读6.4k次。 软件缺陷的描述是是软件缺陷报告的基础部分,也是测试人员就一个软件问题与开发小组交流的最初且最好的机会。一个好的描述,需要使用简单的、准确的、专业的语言来抓住缺陷的本质。否则,它就会使信息含糊不清,可能会误导开发人员。准确报告软件缺陷是非常重要的,因为: 清晰准确的软件缺陷描述可以减少软件缺陷从开发人员返回的数量 提高软件缺陷修复的速度,使每一个小组能够有效的_描述缺陷发生的可能性的是
文章浏览阅读6.6k次。# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.# yarn lockfile v1accepts@~1.3.5, accepts@~1.3.7:version "1.3.7"resolved "https://registry.yarnpkg.com/accepts/-/accepts-1.3.7.tgz#531bc..._hsck735.cc
文章浏览阅读1.3k次。idea卸载Material Theme UI后,git配色无法恢复。解决办法:一个一个地点回来(restore default)。_idea material theme ui 卸载
文章浏览阅读1k次。最近在做appStore时,涉及到一个第三方app下载视频后将车机存储撑爆的问题,从而影响到我们车机自带的app使用,问题很是严重,于是就需要提供缓存清理以及垃圾清理的功能,当用户使用第三方app时,存储空间达到一定量的时候会提示用户去清理,否则不允许使用第三方app。缓存获取获取各个应用程序的缓存大小,可以通过使用PackageManager.getPackageSizeInfo方法来获取,但是..._android packageinstaller缓存数据很大
文章浏览阅读1.8w次。vue给对象动态添加属性和值一、背景介绍:在vue中请求接口中,一个请求方法可能对应后台两个请求接口,所以请求参数就会有所不同。需要我们先设置共同的参数,然后根据条件动态添加参数属性。二、案例let that = this; let params = { "type":that.addQueTab..._vue3 键值对动态添加与获取
文章浏览阅读3.1w次,点赞55次,收藏16次。http://www.51hei.com/bbs/dpj-25003-1.html这里参考了这个帖子,在我身上发生的问题就是我在声明一个函数的时候忘记加分号了,会报错特别多奇奇怪怪的东西,吓死我了。解决办法就是加上分号就好了..._main.c(34): error c132: 'lcd1602_busycheck': not in formal parameter list
文章浏览阅读1.2k次。今天帮助哥们解决了一个比较蛋疼的问题,就是在有的情况下会出现设置activity的windowSoftInputMode="adjustResize"时,会失效的情况。历尽千辛万苦,终于在stackflow上找到解决方法。在activity的根布局上添加fitsSystemWindows="true".然后adjustResize就可以成功的起作用了。但是在这种情况下,你的titlebar会下移s..._adjustresize
文章浏览阅读139次。// 消息自定义// 1、#defien WM_MSG WM_USER// 2、.h文件中添加事件: afx_msg LERSULT OnFun(WPARAM wparam, LPARAM lparam);// 3、宏定义消息路由: ON_MESSAGE(WM_MSG, OnFun)_c++ lersult