大数据之Hadoop集群中Yarn常用命令

2022/6/17 23:28:18

编程Tag： 常用命令 yarn application 06 Hadoop hadoop103 1654496324557

本文主要是介绍大数据之Hadoop集群中Yarn常用命令，对大家解决编程问题具有一定的参考价值，需要的程序猿们随着小编来一起学习吧！

Yarn状态的查询，除了可以在hadoop103:8088页面查看以外，还可以通过命令操作。常见的命令操作如下所示。

需求：执行WordCount案例，并通过Yarn命令查看任务运行情况。

# 启动集群
[hadoop@hadoop102 bin]$ myhadoop.sh start
# 运行wordcount案例
[hadoop@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

yar application查看任务

1、列出所有Application：

[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -list
2022-06-06 16:51:19,896 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
                Application-Id      Application-Name        Application-Type          User           Queue                   State              Final-State             Progress                        Tracking-URL

2、根据Application状态过滤：yarn application -list -appStates [ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED]

[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
2022-06-06 16:53:49,886 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):2
                Application-Id      Application-Name        Application-Type          User           Queue                   State              Final-State             Progress                        Tracking-URL
application_1654496324557_0001            word count               MAPREDUCE        hadoop         default                FINISHED                   FAILED                 100% http://hadoop102:19888/jobhistory/job/job_1654496324557_0001
application_1654496324557_0002            word count               MAPREDUCE        hadoop         default                FINISHED                   FAILED                 100% http://hadoop102:19888/jobhistory/job/job_1654496324557_0002

2、Kill掉Application

[hadoop@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1654496324557_0001
2022-06-06 16:57:34,082 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Application application_1654496324557_0001 has already finished

yarn logs查看日志

1、查询Application日志：yarn logs -applicationId <ApplicationId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1654496324557_0001

2、查询Container日志：yarn logs -applicationId <ApplicationId> -containerId <ContainerId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1654496324557_0001 -containerId container_1654496324557_0001_01_000001

yarn applicationattempt查看尝试的任务

1、列出所有Application尝试的列表：yarn applicationattempt -list <ApplicationId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1654496324557_0001
2022-06-06 17:05:10,914 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of application attempts :1
         ApplicationAttempt-Id                 State                        AM-Container-Id                            Tracking-URL
appattempt_1654496324557_0001_000001                FINISHED    container_1654496324557_0001_01_000001  http://hadoop103:8088/proxy/application_1654496324557_0001/

2、打印ApplicationAttempt状态：yarn applicationattempt -status <ApplicationAttemptId>

[hadoop@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1654496324557_0001_000001
2022-06-06 17:08:56,570 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Application Attempt Report : 
        ApplicationAttempt-Id : appattempt_1654496324557_0001_000001
        State : FINISHED
        AMContainer : container_1654496324557_0001_01_000001
        Tracking-URL : http://hadoop103:8088/proxy/application_1654496324557_0001/
        RPC Port : 32771
        AM Host : hadoop103
        Diagnostics : Task failed task_1654496324557_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

yarn container查看容器

1、列出所有Container：yarn container -list <ApplicationAttemptId>

[hadoop@hadoop103 ~]$ yarn container -list appattempt_1654496324557_0004_000001
2022-06-06 17:14:40,313 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of containers :2
                  Container-Id            Start Time             Finish Time                   State                    Host       Node Http Address                                 LOG-URL
container_1654496324557_0004_01_000001  星期一 六月 06 17:13:55 +0800 2022                       N/A                 RUNNING         hadoop102:46450     http://hadoop102:8042   http://hadoop102:8042/node/containerlogs/container_1654496324557_0004_01_000001/hadoop
container_1654496324557_0004_01_000004  星期一 六月 06 17:14:38 +0800 2022                       N/A                 RUNNING         hadoop103:44058     http://hadoop103:8042   http://hadoop103:8042/node/containerlogs/container_1654496324557_0004_01_000004/hadoop

2、打印Container状态：yarn container -status <ContainerId>

[hadoop@hadoop103 ~]$ yarn container -status container_1654496324557_0006_01_000001
2022-06-06 17:20:51,357 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Container Report : 
        Container-Id : container_1654496324557_0006_01_000001
        Start-Time : 1654507199188
        Finish-Time : 0
        State : RUNNING
        Execution-Type : GUARANTEED
        LOG-URL : http://hadoop103:8042/node/containerlogs/container_1654496324557_0006_01_000001/hadoop
        Host : hadoop103:44058
        NodeHttpAddress : http://hadoop103:8042
        Diagnostics : null

注：只有任务跑的途中才能看到container的状态

yarn node查看节点状态

[hadoop@hadoop103 ~]$ yarn node -list -all
2022-06-06 17:25:20,336 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total Nodes:3
         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
 hadoop104:38945                RUNNING    hadoop104:8042                                  0
 hadoop103:44058                RUNNING    hadoop103:8042                                  0
 hadoop102:46450                RUNNING    hadoop102:8042                                  0

yarn rmadmin更新配置

加载队列配置：yarn rmadmin -refreshQueues

[hadoop@hadoop103 ~]$ yarn rmadmin -refreshQueues
2022-06-06 17:27:10,830 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8033

yarn queue查看队列

打印队列信息：yarn queue -status <QueueName>

[hadoop@hadoop103 ~]$ yarn queue -status default
2022-06-06 17:28:54,282 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Queue Information : 
Queue Name : default
        State : RUNNING
        Capacity : 100.0%
        Current Capacity : .0%
        Maximum Capacity : 100.0%
        Default Node Label expression : <DEFAULT_PARTITION>
        Accessible Node Labels : *
        Preemption : disabled
        Intra-queue Preemption : disabled

下篇文章：[]() 相关文章： [大数据之Hadoop集群中MapReduce的Join操作](https://www.cnblogs.com/sw-code/p/16387043.html) [大数据之Hadoop集群的HDFS压力测试](https://www.cnblogs.com/sw-code/p/16387034.html)

这篇关于大数据之Hadoop集群中Yarn常用命令的文章就介绍到这儿，希望我们推荐的文章对大家有所帮助，也希望大家多多支持为之网！