flink启动等待10分钟问题
Locked
 
|
各位Flink社区大佬，您好！
我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>16384</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm bigdata6 ./flink_bigdata6.jar 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
|
Hi
Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 祝好 唐云 ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:04 To: user-zh <[hidden email]> Subject: flink启动等待10分钟问题 各位Flink社区大佬，您好！ 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>16384</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm bigdata6 ./flink_bigdata6.jar 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
|
|
您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！
2019-08-21 20:00:25,428 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2019-08-21 20:00:25,430 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2019-08-21 20:00:25,433 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: root Yarn client user obtainer: root 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1566383236573_0003 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink-yarn 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs://host51:9000/flink/ha-yarn 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, host51:2181 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, DETACHED 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-attempts, 10 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 8192m 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 2048m 2019-08-21 20:00:25,465 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:25,469 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint. 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2019-08-21 20:00:25,561 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE) 2019-08-21 20:00:25,580 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2019-08-21 20:00:25,757 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at host53:0 2019-08-21 20:00:26,187 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:26,251 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:26,439 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@host53:59609] 2019-08-21 20:00:26,447 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@host53:59609 2019-08-21 20:00:27,011 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. 2019-08-21 20:00:27,161 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=host53 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_77 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.16.0-0.bpo.4-amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 2019-08-21 20:00:27,171 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=host51:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 2019-08-21 20:00:27,184 WARN org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2019-08-21 20:00:27,186 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server host51/10.10.10.51:2181 2019-08-21 20:00:27,187 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed 2019-08-21 20:00:27,187 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to host51/10.10.10.51:2181, initiating session 2019-08-21 20:00:27,193 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server host51/10.10.10.51:2181, sessionid = 0x1001a86de810008, negotiated timeout = 40000 2019-08-21 20:00:27,193 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a 2019-08-21 20:00:27,195 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2019-08-21 20:00:27,198 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max backlog: 1000 2019-08-21 20:00:27,213 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2019-08-21 20:00:27,215 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at host53:0 2019-08-21 20:00:27,239 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:27,247 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:27,255 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] 2019-08-21 20:00:27,257 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metrics@host53:59941 2019-08-21 20:00:27,263 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 2019-08-21 20:00:27,296 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:27,298 WARN org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2019-08-21 20:00:27,298 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for file uploads. 2019-08-21 20:00:27,320 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting rest endpoint. 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out 2019-08-21 20:00:27,771 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at host53:46401 2019-08-21 20:00:27,772 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. 2019-08-21 20:00:27,795 WARN org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. 2019-08-21 20:00:27,807 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web frontend listening at http://host53:46401. 2019-08-21 20:00:27,836 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://host53:46401 was granted leadership with leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 2019-08-21 20:00:27,932 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager . 2019-08-21 20:00:28,021 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at akka://flink/user/dispatcher . 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. 2019-08-21 20:00:28,045 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be 2019-08-21 20:00:28,048 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering all persisted jobs. 2019-08-21 20:00:28,054 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at host51/10.10.10.51:8030 2019-08-21 20:00:28,078 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2019-08-21 20:00:28,086 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,099 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=0) for StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2019-08-21 20:00:28,153 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,154 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms. 2019-08-21 20:00:28,187 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Initialized ZooKeeperCompletedCheckpointStore in '/checkpoints/e5ea83238d721e932b4645589660406c'. 2019-08-21 20:00:28,196 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]). 2019-08-21 20:00:28,199 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2019-08-21 20:00:28,200 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. 2019-08-21 20:00:28,203 INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 2019-08-21 20:00:28,211 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 2019-08-21 20:00:28,214 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2019-08-21 20:00:28,225 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to fetch 0 checkpoints from storage. 2019-08-21 20:00:28,233 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. 2019-08-21 20:00:28,248 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at akka.tcp://flink@host53:59609/user/jobmanager_0. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under job master id a4576f0320d71bc112c96c94ebbd4991. 2019-08-21 20:00:28,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state CREATED to RUNNING. 2019-08-21 20:00:28,266 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,280 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] 2019-08-21 20:00:28,287 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,292 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,295 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Registration at ResourceManager attempt 1 (timeout=100ms) 2019-08-21 20:00:28,304 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. 2019-08-21 20:00:28,304 INFO org.apache.flink.yarn.YarnResourceManager - Registering job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,310 INFO org.apache.flink.yarn.YarnResourceManager - Registered job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,313 INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: 97537c0d61bbf091e37cb6c45b784568. 2019-08-21 20:00:28,314 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,315 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id faa6a521758e5be4b3dc1955ca0f51b4. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,321 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 1. 2019-08-21 20:00:28,322 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id a2b419aea4a373686eeed6c46163a528. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id cbc11737ce6729cfe45f24c43180b5c8. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 2. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id f6566567c173a55b2bc52b72bcb24ff8. 2019-08-21 20:00:31,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host53:59938 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host52:33246 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000002 - Remaining pending container requests: 2 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 1. 2019-08-21 20:00:34,009 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,010 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,021 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host53:59938 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000003 - Remaining pending container requests: 1 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 0. 2019-08-21 20:00:34,088 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,089 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,090 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host52:33246 2019-08-21 20:00:36,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:37,856 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000002 (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,573 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000003 (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,782 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,783 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,784 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,787 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,806 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,992 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,996 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,997 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,005 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,006 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:41,605 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:42,466 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 ms). ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:08 收件人: "user-zh"<[hidden email]>; 主题: Re: flink启动等待10分钟问题 Hi Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 祝好 唐云 ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:04 To: user-zh <[hidden email]> Subject: flink启动等待10分钟问题 各位Flink社区大佬，您好！ 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>16384</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm bigdata6 ./flink_bigdata6.jar 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
|
|
启动时间是 20:00:25，task都处于running甚至第一次checkpoint completed的时间是20:00:42，一共才17秒，何来10分钟的问题？
________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:18 To: user-zh <[hidden email]> Subject: 回复： flink启动等待10分钟问题 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ 2019-08-21 20:00:25,428 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2019-08-21 20:00:25,430 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2019-08-21 20:00:25,433 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: root Yarn client user obtainer: root 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1566383236573_0003 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink-yarn 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs://host51:9000/flink/ha-yarn 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, host51:2181 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, DETACHED 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-attempts, 10 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 8192m 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 2048m 2019-08-21 20:00:25,465 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:25,469 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint. 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2019-08-21 20:00:25,561 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE) 2019-08-21 20:00:25,580 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2019-08-21 20:00:25,757 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at host53:0 2019-08-21 20:00:26,187 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:26,251 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:26,439 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@host53:59609] 2019-08-21 20:00:26,447 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@host53:59609 2019-08-21 20:00:27,011 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. 2019-08-21 20:00:27,161 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=host53 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_77 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.16.0-0.bpo.4-amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 2019-08-21 20:00:27,171 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=host51:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 2019-08-21 20:00:27,184 WARN org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2019-08-21 20:00:27,186 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server host51/10.10.10.51:2181 2019-08-21 20:00:27,187 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed 2019-08-21 20:00:27,187 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to host51/10.10.10.51:2181, initiating session 2019-08-21 20:00:27,193 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server host51/10.10.10.51:2181, sessionid = 0x1001a86de810008, negotiated timeout = 40000 2019-08-21 20:00:27,193 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a 2019-08-21 20:00:27,195 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2019-08-21 20:00:27,198 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max backlog: 1000 2019-08-21 20:00:27,213 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2019-08-21 20:00:27,215 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at host53:0 2019-08-21 20:00:27,239 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:27,247 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:27,255 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] 2019-08-21 20:00:27,257 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metrics@host53:59941 2019-08-21 20:00:27,263 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 2019-08-21 20:00:27,296 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:27,298 WARN org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2019-08-21 20:00:27,298 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for file uploads. 2019-08-21 20:00:27,320 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting rest endpoint. 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out 2019-08-21 20:00:27,771 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at host53:46401 2019-08-21 20:00:27,772 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. 2019-08-21 20:00:27,795 WARN org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. 2019-08-21 20:00:27,807 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web frontend listening at http://host53:46401. 2019-08-21 20:00:27,836 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://host53:46401 was granted leadership with leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 2019-08-21 20:00:27,932 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager . 2019-08-21 20:00:28,021 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at akka://flink/user/dispatcher . 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. 2019-08-21 20:00:28,045 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be 2019-08-21 20:00:28,048 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering all persisted jobs. 2019-08-21 20:00:28,054 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at host51/10.10.10.51:8030 2019-08-21 20:00:28,078 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2019-08-21 20:00:28,086 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,099 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=0) for StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2019-08-21 20:00:28,153 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,154 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms. 2019-08-21 20:00:28,187 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Initialized ZooKeeperCompletedCheckpointStore in '/checkpoints/e5ea83238d721e932b4645589660406c'. 2019-08-21 20:00:28,196 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]). 2019-08-21 20:00:28,199 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2019-08-21 20:00:28,200 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. 2019-08-21 20:00:28,203 INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 2019-08-21 20:00:28,211 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 2019-08-21 20:00:28,214 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2019-08-21 20:00:28,225 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to fetch 0 checkpoints from storage. 2019-08-21 20:00:28,233 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. 2019-08-21 20:00:28,248 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at akka.tcp://flink@host53:59609/user/jobmanager_0. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under job master id a4576f0320d71bc112c96c94ebbd4991. 2019-08-21 20:00:28,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state CREATED to RUNNING. 2019-08-21 20:00:28,266 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,280 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] 2019-08-21 20:00:28,287 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,292 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,295 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Registration at ResourceManager attempt 1 (timeout=100ms) 2019-08-21 20:00:28,304 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. 2019-08-21 20:00:28,304 INFO org.apache.flink.yarn.YarnResourceManager - Registering job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,310 INFO org.apache.flink.yarn.YarnResourceManager - Registered job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,313 INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: 97537c0d61bbf091e37cb6c45b784568. 2019-08-21 20:00:28,314 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,315 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id faa6a521758e5be4b3dc1955ca0f51b4. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,321 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 1. 2019-08-21 20:00:28,322 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id a2b419aea4a373686eeed6c46163a528. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id cbc11737ce6729cfe45f24c43180b5c8. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 2. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id f6566567c173a55b2bc52b72bcb24ff8. 2019-08-21 20:00:31,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host53:59938 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host52:33246 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000002 - Remaining pending container requests: 2 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 1. 2019-08-21 20:00:34,009 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,010 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,021 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host53:59938 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000003 - Remaining pending container requests: 1 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 0. 2019-08-21 20:00:34,088 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,089 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,090 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host52:33246 2019-08-21 20:00:36,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:37,856 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000002 (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,573 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000003 (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,782 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,783 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,784 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,787 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,806 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,992 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,996 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,997 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,005 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,006 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:41,605 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:42,466 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 ms). ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:08 收件人: "user-zh"<[hidden email]>; 主题: Re: flink启动等待10分钟问题 Hi Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 祝好 唐云 ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:04 To: user-zh <[hidden email]> Subject: flink启动等待10分钟问题 各位Flink社区大佬，您好！ 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>16384</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm bigdata6 ./flink_bigdata6.jar 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
|
|
In reply to this post by 守护
Hi,
从你的 LOG 里看从 Cluster 起来 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint. 到全部 RUNNING 2019-08-21 20:00:40,006 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:41,605 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. 一共经过了 16 秒，你观察到的 10 分钟是从执行 flink run 之后的时间吗？可能是提交到 YARN 上的过程网络延迟高？ Best, tison. 々守护々 <[hidden email]> 于2019年8月22日周四 上午11:18写道： > 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ > > > 2019-08-21 20:00:25,428 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > -------------------------------------------------------------------------------- > 2019-08-21 20:00:25,430 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered > UNIX signal handlers for [TERM, HUP, INT] > 2019-08-21 20:00:25,433 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon > is running as: root Yarn client user obtainer: root > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.cluster-id, > application_1566383236573_0003 > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.rpc.address, localhost > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.rpc.port, 6123 > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.zookeeper.path.root, /flink-yarn > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.storageDir, > hdfs://host51:9000/flink/ha-yarn > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.zookeeper.quorum, host51:2181 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: internal.cluster.execution-mode, DETACHED > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability, zookeeper > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: parallelism.default, 1 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: yarn.application-attempts, 10 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.numberOfTaskSlots, 2 > 2019-08-21 20:00:25,439 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.heap.size, 8192m > 2019-08-21 20:00:25,439 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.heap.size, 2048m > 2019-08-21 20:00:25,465 WARN > org.apache.flink.configuration.Configuration - Config uses > deprecated configuration key 'web.port' instead of proper key > 'rest.bind-port' > 2019-08-21 20:00:25,469 INFO > org.apache.flink.runtime.clusterframework.BootstrapTools - Setting > directories for temporary files to: > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 > 2019-08-21 20:00:25,485 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting > YarnJobClusterEntrypoint. > 2019-08-21 20:00:25,485 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install > default filesystem. > 2019-08-21 20:00:25,561 INFO > org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user > set to root (auth:SIMPLE) > 2019-08-21 20:00:25,580 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > Initializing cluster services. > 2019-08-21 20:00:25,757 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to > start actor system at host53:0 > 2019-08-21 20:00:26,187 INFO akka.event.slf4j.Slf4jLogger > - Slf4jLogger started > 2019-08-21 20:00:26,251 INFO akka.remote.Remoting > - Starting remoting > 2019-08-21 20:00:26,439 INFO akka.remote.Remoting > - Remoting started; listening on addresses > :[akka.tcp://flink@host53:59609] > 2019-08-21 20:00:26,447 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor > system started at akka.tcp://flink@host53:59609 > 2019-08-21 20:00:27,011 INFO > org.apache.flink.runtime.blob.FileSystemBlobStore - Creating > highly available BLOB storage directory at > hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob > 2019-08-21 20:00:27,097 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing > default ACL for ZK connections > 2019-08-21 20:00:27,097 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Using > '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. > 2019-08-21 20:00:27,161 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > - Starting > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, > built on 03/23/2017 10:13 GMT > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:host.name=host53 > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.version=1.8.0_77 > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.vendor=Oracle Corporation > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.io.tmpdir=/tmp > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.compiler=<NA> > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.name=Linux > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.arch=amd64 > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.version=3.16.0-0.bpo.4-amd64 > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.name=root > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.home=/root > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 > 2019-08-21 20:00:27,171 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Initiating client connection, connectString=host51:2181 > sessionTimeout=60000 > watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 > 2019-08-21 20:00:27,184 WARN > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL > configuration failed: javax.security.auth.login.LoginException: No JAAS > configuration section named 'Client' was found in specified JAAS > configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue > connection to Zookeeper server without SASL authentication, if Zookeeper > server allows it. > 2019-08-21 20:00:27,186 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > Opening socket connection to server host51/10.10.10.51:2181 > 2019-08-21 20:00:27,187 ERROR > org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - > Authentication failed > 2019-08-21 20:00:27,187 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket > connection established to host51/10.10.10.51:2181, initiating session > 2019-08-21 20:00:27,193 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > Session establishment complete on server host51/10.10.10.51:2181, > sessionid = 0x1001a86de810008, negotiated timeout = 40000 > 2019-08-21 20:00:27,193 INFO org.apache.flink.runtime.blob.BlobServer > - Created BLOB server storage directory > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a > 2019-08-21 20:00:27,195 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager > - State change: CONNECTED > 2019-08-21 20:00:27,198 INFO org.apache.flink.runtime.blob.BlobServer > - Started BLOB server at 0.0.0.0:58591 - max concurrent > requests: 50 - max backlog: 1000 > 2019-08-21 20:00:27,213 INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics > reporter configured, no metrics will be exposed/reported. > 2019-08-21 20:00:27,215 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to > start actor system at host53:0 > 2019-08-21 20:00:27,239 INFO akka.event.slf4j.Slf4jLogger > - Slf4jLogger started > 2019-08-21 20:00:27,247 INFO akka.remote.Remoting > - Starting remoting > 2019-08-21 20:00:27,255 INFO akka.remote.Remoting > - Remoting started; listening on addresses > :[akka.tcp://flink-metrics@host53:59941] > 2019-08-21 20:00:27,257 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor > system started at akka.tcp://flink-metrics@host53:59941 > 2019-08-21 20:00:27,263 INFO > org.apache.flink.runtime.blob.TransientBlobCache - Created > BLOB cache storage directory > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 > 2019-08-21 20:00:27,296 WARN > org.apache.flink.configuration.Configuration - Config uses > deprecated configuration key 'web.port' instead of proper key > 'rest.bind-port' > 2019-08-21 20:00:27,298 WARN > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload > directory > /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does > not exist, or has been deleted externally. Previously uploaded files are no > longer available. > 2019-08-21 20:00:27,298 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created > directory > /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for > file uploads. > 2019-08-21 20:00:27,320 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting > rest endpoint. > 2019-08-21 20:00:27,616 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of main cluster component log file: > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log > 2019-08-21 20:00:27,616 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of main cluster component stdout file: > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out > 2019-08-21 20:00:27,771 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest > endpoint listening at host53:46401 > 2019-08-21 20:00:27,772 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. > 2019-08-21 20:00:27,795 WARN > org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The > version of ZooKeeper being used doesn't support Container nodes. > CreateMode.PERSISTENT will be used instead. > 2019-08-21 20:00:27,807 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web > frontend listening at http://host53:46401. > 2019-08-21 20:00:27,836 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - > http://host53:46401 was granted leadership with > leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 > 2019-08-21 20:00:27,932 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.yarn.YarnResourceManager at > akka://flink/user/resourcemanager . > 2019-08-21 20:00:28,021 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at > akka://flink/user/dispatcher . > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. > 2019-08-21 20:00:28,045 INFO > org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher > akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with > fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be > 2019-08-21 20:00:28,048 INFO > org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering > all persisted jobs. > 2019-08-21 20:00:28,054 INFO org.apache.hadoop.yarn.client.RMProxy > - Connecting to ResourceManager at host51/ > 10.10.10.51:8030 > 2019-08-21 20:00:28,078 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at > akka://flink/user/jobmanager_0 . > 2019-08-21 20:00:28,086 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,099 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Using > restart strategy > FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, > delayBetweenRestartAttempts=0) for StreamKafkaMain > (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,122 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > recovers via failover strategy: full graph restart > 2019-08-21 20:00:28,153 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Running > initialization on master for job StreamKafkaMain > (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,154 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Successfully ran initialization on master in 0 ms. > 2019-08-21 20:00:28,187 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Initialized > ZooKeeperCompletedCheckpointStore in > '/checkpoints/e5ea83238d721e932b4645589660406c'. > 2019-08-21 20:00:28,196 INFO org.apache.flink.yarn.YarnResourceManager > - Recovered 0 containers from previous attempts ([]). > 2019-08-21 20:00:28,199 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > yarn.client.max-cached-nodemanagers-proxies : 0 > 2019-08-21 20:00:28,200 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. > 2019-08-21 20:00:28,203 INFO > org.apache.flink.runtime.jobmaster.JobMaster - No state > backend has been configured, using default (Memory / JobManager) > MemoryStateBackend (data in heap memory / checkpoints to JobManager) > (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: > 5242880) > 2019-08-21 20:00:28,211 INFO org.apache.flink.yarn.YarnResourceManager > - ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager > was granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 > 2019-08-21 20:00:28,214 INFO > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - > Starting the SlotManager. > 2019-08-21 20:00:28,225 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Recovering checkpoints from ZooKeeper. > 2019-08-21 20:00:28,229 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Found 0 checkpoints in ZooKeeper. > 2019-08-21 20:00:28,229 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Trying to fetch 0 checkpoints from storage. > 2019-08-21 20:00:28,233 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. > 2019-08-21 20:00:28,248 INFO > org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager > runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was > granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at > akka.tcp://flink@host53:59609/user/jobmanager_0. > 2019-08-21 20:00:28,260 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > 2019-08-21 20:00:28,260 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Starting > execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under > job master id a4576f0320d71bc112c96c94ebbd4991. > 2019-08-21 20:00:28,261 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state > CREATED to RUNNING. > 2019-08-21 20:00:28,266 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,280 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] > 2019-08-21 20:00:28,287 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,288 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] > 2019-08-21 20:00:28,288 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,292 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,295 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Connecting > to ResourceManager akka.tcp://flink@host53 > :59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) > 2019-08-21 20:00:28,300 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Resolved > ResourceManager address, beginning registration > 2019-08-21 20:00:28,300 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Registration at ResourceManager attempt 1 (timeout=100ms) > 2019-08-21 20:00:28,304 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService > /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. > 2019-08-21 20:00:28,304 INFO org.apache.flink.yarn.YarnResourceManager > - Registering job manager > [hidden email]://flink@host53:59609/user/jobmanager_0 > for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:28,310 INFO org.apache.flink.yarn.YarnResourceManager > - Registered job manager a4576f0320d71bc112c96c94ebbd4991 > @akka.tcp://flink@host53:59609/user/jobmanager_0 for job > e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:28,313 INFO > org.apache.flink.runtime.jobmaster.JobMaster - JobManager > successfully registered at ResourceManager, leader id: > 97537c0d61bbf091e37cb6c45b784568. > 2019-08-21 20:00:28,314 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,315 INFO org.apache.flink.yarn.YarnResourceManager > - Request slot with profile ResourceProfile{cpuCores=-1.0, > heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, > networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with > allocation id faa6a521758e5be4b3dc1955ca0f51b4. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,321 INFO org.apache.flink.yarn.YarnResourceManager > - Requesting new TaskExecutor container with resources > <memory:8192, vCores:2>. Number pending requests 1. > 2019-08-21 20:00:28,322 INFO org.apache.flink.yarn.YarnResourceManager > - Request slot with profile ResourceProfile{cpuCores=-1.0, > heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, > networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with > allocation id a2b419aea4a373686eeed6c46163a528. > 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager > - Request slot with profile ResourceProfile{cpuCores=-1.0, > heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, > networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with > allocation id cbc11737ce6729cfe45f24c43180b5c8. > 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager > - Requesting new TaskExecutor container with resources > <memory:8192, vCores:2>. Number pending requests 2. > 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager > - Request slot with profile ResourceProfile{cpuCores=-1.0, > heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, > networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with > allocation id f6566567c173a55b2bc52b72bcb24ff8. > 2019-08-21 20:00:31,597 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint > triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > RUNNING but SCHEDULED instead. Aborting checkpoint. > 2019-08-21 20:00:33,749 INFO > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > new token for : host53:59938 > 2019-08-21 20:00:33,749 INFO > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > new token for : host52:33246 > 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager > - Received new container: > container_1566383236573_0003_01_000002 - Remaining pending container > requests: 2 > 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager > - Removing container request Capability[<memory:8192, > vCores:2>]Priority[1]. Pending container requests 1. > 2019-08-21 20:00:34,009 INFO org.apache.flink.yarn.YarnResourceManager > - Creating container launch context for TaskManagers > 2019-08-21 20:00:34,010 INFO org.apache.flink.yarn.YarnResourceManager > - Starting TaskManagers > 2019-08-21 20:00:34,021 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : host53:59938 > 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager > - Received new container: > container_1566383236573_0003_01_000003 - Remaining pending container > requests: 1 > 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager > - Removing container request Capability[<memory:8192, > vCores:2>]Priority[1]. Pending container requests 0. > 2019-08-21 20:00:34,088 INFO org.apache.flink.yarn.YarnResourceManager > - Creating container launch context for TaskManagers > 2019-08-21 20:00:34,089 INFO org.apache.flink.yarn.YarnResourceManager > - Starting TaskManagers > 2019-08-21 20:00:34,090 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : host52:33246 > 2019-08-21 20:00:36,597 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint > triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > RUNNING but SCHEDULED instead. Aborting checkpoint. > 2019-08-21 20:00:37,856 INFO org.apache.flink.yarn.YarnResourceManager > - Registering TaskManager with ResourceID > container_1566383236573_0003_01_000002 (akka.tcp://flink@host53:34608/user/taskmanager_0) > at ResourceManager > 2019-08-21 20:00:39,573 INFO org.apache.flink.yarn.YarnResourceManager > - Registering TaskManager with ResourceID > container_1566383236573_0003_01_000003 (akka.tcp://flink@host52:32931/user/taskmanager_0) > at ResourceManager > 2019-08-21 20:00:39,659 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,659 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > (dataPort=52456) > 2019-08-21 20:00:39,664 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,664 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > (dataPort=52456) > 2019-08-21 20:00:39,665 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > (dataPort=39650) > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > (dataPort=39650) > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: > (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, > end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) > -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt > #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,670 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,670 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (1/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,671 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (2/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (3/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (4/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (1/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (2/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (3/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,676 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,676 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (4/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,782 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,783 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,784 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,787 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,806 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,809 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,809 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,991 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,992 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,996 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,997 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:40,005 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:40,006 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:41,605 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering > checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:42,466 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed > checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 > ms). > > > > ------------------ 原始邮件 ------------------ > 发件人: "Yun Tang"<[hidden email]>; > 发送时间: 2019年8月22日(星期四) 中午11:08 > 收件人: "user-zh"<[hidden email]>; > > 主题: Re: flink启动等待10分钟问题 > > > > Hi > > Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot > request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request > timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 > > 祝好 > 唐云 > ________________________________ > From: 々守护々 <[hidden email]> > Sent: Thursday, August 22, 2019 11:04 > To: user-zh <[hidden email]> > Subject: flink启动等待10分钟问题 > > 各位Flink社区大佬，您好！ > 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： > <property> > <name>yarn.nodemanager.resource.memory-mb</name> > <value>16384</value> > </property> > <property> > <name>yarn.scheduler.minimum-allocation-mb</name> > <value>1024</value> > </property> > <property> > <name>yarn.scheduler.maximum-allocation-mb</name> > <value>8192</value> > </property> > > > > 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p > 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm > bigdata6 ./flink_bigdata6.jar > 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
|
|
In reply to this post by Yun Tang
是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:23 收件人: "user-zh"<[hidden email]>; 主题: Re: 回复： flink启动等待10分钟问题 ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:18 To: user-zh <[hidden email]> Subject: 回复： flink启动等待10分钟问题 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ 2019-08-21 20:00:25,428 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2019-08-21 20:00:25,430 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2019-08-21 20:00:25,433 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: root Yarn client user obtainer: root 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1566383236573_0003 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink-yarn 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs://host51:9000/flink/ha-yarn 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, host51:2181 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, DETACHED 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-attempts, 10 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 8192m 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 2048m 2019-08-21 20:00:25,465 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:25,469 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint. 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2019-08-21 20:00:25,561 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE) 2019-08-21 20:00:25,580 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2019-08-21 20:00:25,757 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at host53:0 2019-08-21 20:00:26,187 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:26,251 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:26,439 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@host53:59609] 2019-08-21 20:00:26,447 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@host53:59609 2019-08-21 20:00:27,011 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. 2019-08-21 20:00:27,161 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=host53 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_77 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.16.0-0.bpo.4-amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 2019-08-21 20:00:27,171 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=host51:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 2019-08-21 20:00:27,184 WARN org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2019-08-21 20:00:27,186 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server host51/10.10.10.51:2181 2019-08-21 20:00:27,187 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed 2019-08-21 20:00:27,187 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to host51/10.10.10.51:2181, initiating session 2019-08-21 20:00:27,193 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server host51/10.10.10.51:2181, sessionid = 0x1001a86de810008, negotiated timeout = 40000 2019-08-21 20:00:27,193 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a 2019-08-21 20:00:27,195 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2019-08-21 20:00:27,198 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max backlog: 1000 2019-08-21 20:00:27,213 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2019-08-21 20:00:27,215 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at host53:0 2019-08-21 20:00:27,239 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:27,247 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:27,255 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] 2019-08-21 20:00:27,257 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metrics@host53:59941 2019-08-21 20:00:27,263 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 2019-08-21 20:00:27,296 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:27,298 WARN org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2019-08-21 20:00:27,298 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for file uploads. 2019-08-21 20:00:27,320 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting rest endpoint. 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out 2019-08-21 20:00:27,771 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at host53:46401 2019-08-21 20:00:27,772 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. 2019-08-21 20:00:27,795 WARN org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. 2019-08-21 20:00:27,807 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web frontend listening at http://host53:46401. 2019-08-21 20:00:27,836 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://host53:46401 was granted leadership with leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 2019-08-21 20:00:27,932 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager . 2019-08-21 20:00:28,021 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at akka://flink/user/dispatcher . 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. 2019-08-21 20:00:28,045 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be 2019-08-21 20:00:28,048 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering all persisted jobs. 2019-08-21 20:00:28,054 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at host51/10.10.10.51:8030 2019-08-21 20:00:28,078 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2019-08-21 20:00:28,086 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,099 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=0) for StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2019-08-21 20:00:28,153 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,154 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms. 2019-08-21 20:00:28,187 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Initialized ZooKeeperCompletedCheckpointStore in '/checkpoints/e5ea83238d721e932b4645589660406c'. 2019-08-21 20:00:28,196 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]). 2019-08-21 20:00:28,199 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2019-08-21 20:00:28,200 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. 2019-08-21 20:00:28,203 INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 2019-08-21 20:00:28,211 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 2019-08-21 20:00:28,214 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2019-08-21 20:00:28,225 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to fetch 0 checkpoints from storage. 2019-08-21 20:00:28,233 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. 2019-08-21 20:00:28,248 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at akka.tcp://flink@host53:59609/user/jobmanager_0. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under job master id a4576f0320d71bc112c96c94ebbd4991. 2019-08-21 20:00:28,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state CREATED to RUNNING. 2019-08-21 20:00:28,266 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,280 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] 2019-08-21 20:00:28,287 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,292 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,295 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Registration at ResourceManager attempt 1 (timeout=100ms) 2019-08-21 20:00:28,304 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. 2019-08-21 20:00:28,304 INFO org.apache.flink.yarn.YarnResourceManager - Registering job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,310 INFO org.apache.flink.yarn.YarnResourceManager - Registered job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,313 INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: 97537c0d61bbf091e37cb6c45b784568. 2019-08-21 20:00:28,314 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,315 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id faa6a521758e5be4b3dc1955ca0f51b4. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,321 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 1. 2019-08-21 20:00:28,322 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id a2b419aea4a373686eeed6c46163a528. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id cbc11737ce6729cfe45f24c43180b5c8. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 2. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id f6566567c173a55b2bc52b72bcb24ff8. 2019-08-21 20:00:31,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host53:59938 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host52:33246 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000002 - Remaining pending container requests: 2 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 1. 2019-08-21 20:00:34,009 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,010 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,021 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host53:59938 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000003 - Remaining pending container requests: 1 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 0. 2019-08-21 20:00:34,088 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,089 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,090 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host52:33246 2019-08-21 20:00:36,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:37,856 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000002 (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,573 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000003 (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,782 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,783 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,784 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,787 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,806 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,992 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,996 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,997 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,005 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,006 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:41,605 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:42,466 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 ms). ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:08 收件人: "user-zh"<[hidden email]>; 主题: Re: flink启动等待10分钟问题 Hi Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 祝好 唐云 ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:04 To: user-zh <[hidden email]> Subject: flink启动等待10分钟问题 各位Flink社区大佬，您好！ 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>16384</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm bigdata6 ./flink_bigdata6.jar 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
回复: 回复： flink启动等待10分钟问题
|
|
当时你的集群资源是否充足?
[hidden email] 发件人： 々守护々 发送时间： 2019-08-22 11:32 收件人： user-zh 主题： 回复： 回复： flink启动等待10分钟问题 是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:23 收件人: "user-zh"<[hidden email]>; 主题: Re: 回复： flink启动等待10分钟问题 启动时间是 20:00:25，task都处于running甚至第一次checkpoint completed的时间是20:00:42，一共才17秒，何来10分钟的问题？ ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:18 To: user-zh <[hidden email]> Subject: 回复： flink启动等待10分钟问题 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ 2019-08-21 20:00:25,428 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2019-08-21 20:00:25,430 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2019-08-21 20:00:25,433 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: root Yarn client user obtainer: root 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1566383236573_0003 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink-yarn 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs://host51:9000/flink/ha-yarn 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, host51:2181 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, DETACHED 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-attempts, 10 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 8192m 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 2048m 2019-08-21 20:00:25,465 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:25,469 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint. 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2019-08-21 20:00:25,561 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE) 2019-08-21 20:00:25,580 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2019-08-21 20:00:25,757 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at host53:0 2019-08-21 20:00:26,187 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:26,251 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:26,439 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@host53:59609] 2019-08-21 20:00:26,447 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@host53:59609 2019-08-21 20:00:27,011 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. 2019-08-21 20:00:27,161 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=host53 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_77 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.16.0-0.bpo.4-amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 2019-08-21 20:00:27,171 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=host51:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 2019-08-21 20:00:27,184 WARN org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2019-08-21 20:00:27,186 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server host51/10.10.10.51:2181 2019-08-21 20:00:27,187 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed 2019-08-21 20:00:27,187 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to host51/10.10.10.51:2181, initiating session 2019-08-21 20:00:27,193 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server host51/10.10.10.51:2181, sessionid = 0x1001a86de810008, negotiated timeout = 40000 2019-08-21 20:00:27,193 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a 2019-08-21 20:00:27,195 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2019-08-21 20:00:27,198 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max backlog: 1000 2019-08-21 20:00:27,213 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2019-08-21 20:00:27,215 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at host53:0 2019-08-21 20:00:27,239 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:27,247 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:27,255 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] 2019-08-21 20:00:27,257 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metrics@host53:59941 2019-08-21 20:00:27,263 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 2019-08-21 20:00:27,296 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:27,298 WARN org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2019-08-21 20:00:27,298 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for file uploads. 2019-08-21 20:00:27,320 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting rest endpoint. 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out 2019-08-21 20:00:27,771 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at host53:46401 2019-08-21 20:00:27,772 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. 2019-08-21 20:00:27,795 WARN org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. 2019-08-21 20:00:27,807 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web frontend listening at http://host53:46401. 2019-08-21 20:00:27,836 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://host53:46401 was granted leadership with leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 2019-08-21 20:00:27,932 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager . 2019-08-21 20:00:28,021 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at akka://flink/user/dispatcher . 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. 2019-08-21 20:00:28,045 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be 2019-08-21 20:00:28,048 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering all persisted jobs. 2019-08-21 20:00:28,054 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at host51/10.10.10.51:8030 2019-08-21 20:00:28,078 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2019-08-21 20:00:28,086 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,099 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=0) for StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2019-08-21 20:00:28,153 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,154 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms. 2019-08-21 20:00:28,187 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Initialized ZooKeeperCompletedCheckpointStore in '/checkpoints/e5ea83238d721e932b4645589660406c'. 2019-08-21 20:00:28,196 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]). 2019-08-21 20:00:28,199 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2019-08-21 20:00:28,200 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. 2019-08-21 20:00:28,203 INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 2019-08-21 20:00:28,211 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 2019-08-21 20:00:28,214 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2019-08-21 20:00:28,225 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to fetch 0 checkpoints from storage. 2019-08-21 20:00:28,233 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. 2019-08-21 20:00:28,248 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at akka.tcp://flink@host53:59609/user/jobmanager_0. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under job master id a4576f0320d71bc112c96c94ebbd4991. 2019-08-21 20:00:28,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state CREATED to RUNNING. 2019-08-21 20:00:28,266 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,280 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] 2019-08-21 20:00:28,287 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,292 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,295 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Registration at ResourceManager attempt 1 (timeout=100ms) 2019-08-21 20:00:28,304 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. 2019-08-21 20:00:28,304 INFO org.apache.flink.yarn.YarnResourceManager - Registering job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,310 INFO org.apache.flink.yarn.YarnResourceManager - Registered job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,313 INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: 97537c0d61bbf091e37cb6c45b784568. 2019-08-21 20:00:28,314 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,315 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id faa6a521758e5be4b3dc1955ca0f51b4. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,321 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 1. 2019-08-21 20:00:28,322 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id a2b419aea4a373686eeed6c46163a528. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id cbc11737ce6729cfe45f24c43180b5c8. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 2. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id f6566567c173a55b2bc52b72bcb24ff8. 2019-08-21 20:00:31,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host53:59938 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host52:33246 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000002 - Remaining pending container requests: 2 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 1. 2019-08-21 20:00:34,009 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,010 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,021 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host53:59938 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000003 - Remaining pending container requests: 1 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 0. 2019-08-21 20:00:34,088 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,089 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,090 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host52:33246 2019-08-21 20:00:36,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:37,856 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000002 (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,573 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000003 (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,782 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,783 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,784 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,787 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,806 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,992 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,996 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,997 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,005 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,006 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:41,605 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:42,466 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 ms). ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:08 收件人: "user-zh"<[hidden email]>; 主题: Re: flink启动等待10分钟问题 Hi Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 祝好 唐云 ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:04 To: user-zh <[hidden email]> Subject: flink启动等待10分钟问题 各位Flink社区大佬，您好！ 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>16384</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm bigdata6 ./flink_bigdata6.jar 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
|
|
In reply to this post by 守护
user-zh 不支持贴图，你用下第三方存储然后贴个链接吧，或者我记得可以传邮件附件
Best, tison. 々守护々 <[hidden email]> 于2019年8月22日周四 上午11:33写道： > 是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application > <http://hadoop.changba.com/cluster/app/application_1566383236573_0003> > > > ------------------ 原始邮件 ------------------ > *发件人:* "Yun Tang"<[hidden email]>; > *发送时间:* 2019年8月22日(星期四) 中午11:23 > *收件人:* "user-zh"<[hidden email]>; > *主题:* Re: 回复： flink启动等待10分钟问题 > > 启动时间是 20:00:25，task都处于running甚至第一次checkpoint > completed的时间是20:00:42，一共才17秒，何来10分钟的问题？ > ________________________________ > From: 々守护々 <[hidden email]> > Sent: Thursday, August 22, 2019 11:18 > To: user-zh <[hidden email]> > Subject: 回复： flink启动等待10分钟问题 > > 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ > > > 2019-08-21 20:00:25,428 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > -------------------------------------------------------------------------------- > 2019-08-21 20:00:25,430 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered > UNIX signal handlers for [TERM, HUP, INT] > 2019-08-21 20:00:25,433 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon > is running as: root Yarn client user obtainer: root > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.cluster-id, > application_1566383236573_0003 > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.rpc.address, localhost > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.rpc.port, 6123 > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.zookeeper.path.root, /flink-yarn > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.storageDir, > hdfs://host51:9000/flink/ha-yarn > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.zookeeper.quorum, host51:2181 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: internal.cluster.execution-mode, DETACHED > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability, zookeeper > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: parallelism.default, 1 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: yarn.application-attempts, 10 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.numberOfTaskSlots, 2 > 2019-08-21 20:00:25,439 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.heap.size, 8192m > 2019-08-21 20:00:25,439 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.heap.size, 2048m > 2019-08-21 20:00:25,465 WARN > org.apache.flink.configuration.Configuration - Config uses > deprecated configuration key 'web.port' instead of proper key > 'rest.bind-port' > 2019-08-21 20:00:25,469 INFO > org.apache.flink.runtime.clusterframework.BootstrapTools - Setting > directories for temporary files to: > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 > 2019-08-21 20:00:25,485 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting > YarnJobClusterEntrypoint. > 2019-08-21 20:00:25,485 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install > default filesystem. > 2019-08-21 20:00:25,561 INFO > org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user > set to root (auth:SIMPLE) > 2019-08-21 20:00:25,580 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > Initializing cluster services. > 2019-08-21 20:00:25,757 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to > start actor system at host53:0 > 2019-08-21 20:00:26,187 INFO > akka.event.slf4j.Slf4jLogger - Slf4jLogger > started > 2019-08-21 20:00:26,251 INFO > akka.remote.Remoting - Starting > remoting > 2019-08-21 20:00:26,439 INFO > akka.remote.Remoting - Remoting > started; listening on addresses :[akka.tcp://flink@host53:59609] > 2019-08-21 20:00:26,447 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor > system started at akka.tcp://flink@host53:59609 > 2019-08-21 20:00:27,011 INFO > org.apache.flink.runtime.blob.FileSystemBlobStore - Creating > highly available BLOB storage directory at > hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob > 2019-08-21 20:00:27,097 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing > default ACL for ZK connections > 2019-08-21 20:00:27,097 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Using > '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. > 2019-08-21 20:00:27,161 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > - Starting > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, > built on 03/23/2017 10:13 GMT > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:host.name=host53 > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.version=1.8.0_77 > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.vendor=Oracle Corporation > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.io.tmpdir=/tmp > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.compiler=<NA> > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.name=Linux > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.arch=amd64 > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.version=3.16.0-0.bpo.4-amd64 > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.name=root > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.home=/root > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 > 2019-08-21 20:00:27,171 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Initiating client connection, connectString=host51:2181 > sessionTimeout=60000 > watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 > 2019-08-21 20:00:27,184 WARN > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL > configuration failed: javax.security.auth.login.LoginException: No JAAS > configuration section named 'Client' was found in specified JAAS > configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue > connection to Zookeeper server without SASL authentication, if Zookeeper > server allows it. > 2019-08-21 20:00:27,186 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > Opening socket connection to server host51/10.10.10.51:2181 > 2019-08-21 20:00:27,187 ERROR > org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - > Authentication failed > 2019-08-21 20:00:27,187 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket > connection established to host51/10.10.10.51:2181, initiating session > 2019-08-21 20:00:27,193 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > Session establishment complete on server host51/10.10.10.51:2181, > sessionid = 0x1001a86de810008, negotiated timeout = 40000 > 2019-08-21 20:00:27,193 INFO > org.apache.flink.runtime.blob.BlobServer - Created > BLOB server storage directory > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a > 2019-08-21 20:00:27,195 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager > - State change: CONNECTED > 2019-08-21 20:00:27,198 INFO > org.apache.flink.runtime.blob.BlobServer - Started > BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max backlog: > 1000 > 2019-08-21 20:00:27,213 INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics > reporter configured, no metrics will be exposed/reported. > 2019-08-21 20:00:27,215 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to > start actor system at host53:0 > 2019-08-21 20:00:27,239 INFO > akka.event.slf4j.Slf4jLogger - Slf4jLogger > started > 2019-08-21 20:00:27,247 INFO > akka.remote.Remoting - Starting > remoting > 2019-08-21 20:00:27,255 INFO > akka.remote.Remoting - Remoting > started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] > 2019-08-21 20:00:27,257 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor > system started at akka.tcp://flink-metrics@host53:59941 > 2019-08-21 20:00:27,263 INFO > org.apache.flink.runtime.blob.TransientBlobCache - Created > BLOB cache storage directory > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 > 2019-08-21 20:00:27,296 WARN > org.apache.flink.configuration.Configuration - Config uses > deprecated configuration key 'web.port' instead of proper key > 'rest.bind-port' > 2019-08-21 20:00:27,298 WARN > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload > directory > /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does > not exist, or has been deleted externally. Previously uploaded files are no > longer available. > 2019-08-21 20:00:27,298 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created > directory > /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for > file uploads. > 2019-08-21 20:00:27,320 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting > rest endpoint. > 2019-08-21 20:00:27,616 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of main cluster component log file: > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log > 2019-08-21 20:00:27,616 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of main cluster component stdout file: > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out > 2019-08-21 20:00:27,771 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest > endpoint listening at host53:46401 > 2019-08-21 20:00:27,772 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. > 2019-08-21 20:00:27,795 WARN > org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The > version of ZooKeeper being used doesn't support Container nodes. > CreateMode.PERSISTENT will be used instead. > 2019-08-21 20:00:27,807 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web > frontend listening at http://host53:46401. > 2019-08-21 20:00:27,836 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - > http://host53:46401 was granted leadership with > leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 > 2019-08-21 20:00:27,932 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.yarn.YarnResourceManager at > akka://flink/user/resourcemanager . > 2019-08-21 20:00:28,021 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at > akka://flink/user/dispatcher . > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. > 2019-08-21 20:00:28,045 INFO > org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher > akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with > fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be > 2019-08-21 20:00:28,048 INFO > org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering > all persisted jobs. > 2019-08-21 20:00:28,054 INFO > org.apache.hadoop.yarn.client.RMProxy - Connecting > to ResourceManager at host51/10.10.10.51:8030 > 2019-08-21 20:00:28,078 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at > akka://flink/user/jobmanager_0 . > 2019-08-21 20:00:28,086 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,099 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Using > restart strategy > FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, > delayBetweenRestartAttempts=0) for StreamKafkaMain > (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,122 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > recovers via failover strategy: full graph restart > 2019-08-21 20:00:28,153 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Running > initialization on master for job StreamKafkaMain > (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,154 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Successfully ran initialization on master in 0 ms. > 2019-08-21 20:00:28,187 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Initialized > ZooKeeperCompletedCheckpointStore in > '/checkpoints/e5ea83238d721e932b4645589660406c'. > 2019-08-21 20:00:28,196 INFO > org.apache.flink.yarn.YarnResourceManager - Recovered 0 > containers from previous attempts ([]). > 2019-08-21 20:00:28,199 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > yarn.client.max-cached-nodemanagers-proxies : 0 > 2019-08-21 20:00:28,200 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. > 2019-08-21 20:00:28,203 INFO > org.apache.flink.runtime.jobmaster.JobMaster - No state > backend has been configured, using default (Memory / JobManager) > MemoryStateBackend (data in heap memory / checkpoints to JobManager) > (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: > 5242880) > 2019-08-21 20:00:28,211 INFO > org.apache.flink.yarn.YarnResourceManager - > ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was > granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 > 2019-08-21 20:00:28,214 INFO > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - > Starting the SlotManager. > 2019-08-21 20:00:28,225 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Recovering checkpoints from ZooKeeper. > 2019-08-21 20:00:28,229 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Found 0 checkpoints in ZooKeeper. > 2019-08-21 20:00:28,229 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Trying to fetch 0 checkpoints from storage. > 2019-08-21 20:00:28,233 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. > 2019-08-21 20:00:28,248 INFO > org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager > runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was > granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at > akka.tcp://flink@host53:59609/user/jobmanager_0. > 2019-08-21 20:00:28,260 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > 2019-08-21 20:00:28,260 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Starting > execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under > job master id a4576f0320d71bc112c96c94ebbd4991. > 2019-08-21 20:00:28,261 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state > CREATED to RUNNING. > 2019-08-21 20:00:28,266 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,280 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] > 2019-08-21 20:00:28,287 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,288 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] > 2019-08-21 20:00:28,288 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,292 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,295 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Connecting > to ResourceManager akka.tcp://flink@host53 > :59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) > 2019-08-21 20:00:28,300 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Resolved > ResourceManager address, beginning registration > 2019-08-21 20:00:28,300 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Registration at ResourceManager attempt 1 (timeout=100ms) > 2019-08-21 20:00:28,304 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService > /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. > 2019-08-21 20:00:28,304 INFO > org.apache.flink.yarn.YarnResourceManager - Registering > job manager [hidden email]://flink@host53:59609/user/jobmanager_0 > for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:28,310 INFO > org.apache.flink.yarn.YarnResourceManager - Registered > job manager [hidden email]://flink@host53:59609/user/jobmanager_0 > for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:28,313 INFO > org.apache.flink.runtime.jobmaster.JobMaster - JobManager > successfully registered at ResourceManager, leader id: > 97537c0d61bbf091e37cb6c45b784568. > 2019-08-21 20:00:28,314 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,315 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > faa6a521758e5be4b3dc1955ca0f51b4. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,321 INFO > org.apache.flink.yarn.YarnResourceManager - Requesting > new TaskExecutor container with resources <memory:8192, vCores:2>. Number > pending requests 1. > 2019-08-21 20:00:28,322 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > a2b419aea4a373686eeed6c46163a528. > 2019-08-21 20:00:28,323 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > cbc11737ce6729cfe45f24c43180b5c8. > 2019-08-21 20:00:28,323 INFO > org.apache.flink.yarn.YarnResourceManager - Requesting > new TaskExecutor container with resources <memory:8192, vCores:2>. Number > pending requests 2. > 2019-08-21 20:00:28,323 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > f6566567c173a55b2bc52b72bcb24ff8. > 2019-08-21 20:00:31,597 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint > triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > RUNNING but SCHEDULED instead. Aborting checkpoint. > 2019-08-21 20:00:33,749 INFO > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > new token for : host53:59938 > 2019-08-21 20:00:33,749 INFO > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > new token for : host52:33246 > 2019-08-21 20:00:33,765 INFO > org.apache.flink.yarn.YarnResourceManager - Received > new container: container_1566383236573_0003_01_000002 - Remaining pending > container requests: 2 > 2019-08-21 20:00:33,765 INFO > org.apache.flink.yarn.YarnResourceManager - Removing > container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending > container requests 1. > 2019-08-21 20:00:34,009 INFO > org.apache.flink.yarn.YarnResourceManager - Creating > container launch context for TaskManagers > 2019-08-21 20:00:34,010 INFO > org.apache.flink.yarn.YarnResourceManager - Starting > TaskManagers > 2019-08-21 20:00:34,021 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : host53:59938 > 2019-08-21 20:00:34,066 INFO > org.apache.flink.yarn.YarnResourceManager - Received > new container: container_1566383236573_0003_01_000003 - Remaining pending > container requests: 1 > 2019-08-21 20:00:34,066 INFO > org.apache.flink.yarn.YarnResourceManager - Removing > container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending > container requests 0. > 2019-08-21 20:00:34,088 INFO > org.apache.flink.yarn.YarnResourceManager - Creating > container launch context for TaskManagers > 2019-08-21 20:00:34,089 INFO > org.apache.flink.yarn.YarnResourceManager - Starting > TaskManagers > 2019-08-21 20:00:34,090 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : host52:33246 > 2019-08-21 20:00:36,597 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint > triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > RUNNING but SCHEDULED instead. Aborting checkpoint. > 2019-08-21 20:00:37,856 INFO > org.apache.flink.yarn.YarnResourceManager - Registering > TaskManager with ResourceID container_1566383236573_0003_01_000002 > (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager > 2019-08-21 20:00:39,573 INFO > org.apache.flink.yarn.YarnResourceManager - Registering > TaskManager with ResourceID container_1566383236573_0003_01_000003 > (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager > 2019-08-21 20:00:39,659 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,659 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > (dataPort=52456) > 2019-08-21 20:00:39,664 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,664 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > (dataPort=52456) > 2019-08-21 20:00:39,665 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > (dataPort=39650) > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > (dataPort=39650) > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: > (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, > end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) > -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt > #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,670 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,670 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (1/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,671 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (2/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (3/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (4/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (1/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (2/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (3/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,676 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,676 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (4/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,782 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,783 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,784 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,787 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,806 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,809 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,809 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,991 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,992 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,996 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,997 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:40,005 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:40,006 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:41,605 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering > checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:42,466 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed > checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 > ms). > > > > ------------------ 原始邮件 ------------------ > 发件人: "Yun Tang"<[hidden email]>; > 发送时间: 2019年8月22日(星期四) 中午11:08 > 收件人: "user-zh"<[hidden email]>; > > 主题: Re: flink启动等待10分钟问题 > > > > Hi > > Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot > request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request > timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 > > 祝好 > 唐云 > ________________________________ > From: 々守护々 <[hidden email]> > Sent: Thursday, August 22, 2019 11:04 > To: user-zh <[hidden email]> > Subject: flink启动等待10分钟问题 > > 各位Flink社区大佬，您好！ > 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： > <property> > <name>yarn.nodemanager.resource.memory-mb</name> > <value>16384</value> > </property> > <property> > <name>yarn.scheduler.minimum-allocation-mb</name> > <value>1024</value> > </property> > <property> > <name>yarn.scheduler.maximum-allocation-mb</name> > <value>8192</value> > </property> > > > > 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p > 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm > bigdata6 ./flink_bigdata6.jar > 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ > |
|
|
你说的【停在那儿了】是说 flink run 的终端输出不动了吗？你看一下这个终端输出里 YARN 是什么时候 accept 你的应用的，我怀疑是
YARN 集群忙导致 10 分钟没响应。 Best, tison. Zili Chen <[hidden email]> 于2019年8月22日周四 上午11:35写道： > user-zh 不支持贴图，你用下第三方存储然后贴个链接吧，或者我记得可以传邮件附件 > > Best, > tison. > > > 々守护々 <[hidden email]> 于2019年8月22日周四 上午11:33写道： > >> 是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application >> <http://hadoop.changba.com/cluster/app/application_1566383236573_0003> >> >> >> ------------------ 原始邮件 ------------------ >> *发件人:* "Yun Tang"<[hidden email]>; >> *发送时间:* 2019年8月22日(星期四) 中午11:23 >> *收件人:* "user-zh"<[hidden email]>; >> *主题:* Re: 回复： flink启动等待10分钟问题 >> >> 启动时间是 20:00:25，task都处于running甚至第一次checkpoint >> completed的时间是20:00:42，一共才17秒，何来10分钟的问题？ >> ________________________________ >> From: 々守护々 <[hidden email]> >> Sent: Thursday, August 22, 2019 11:18 >> To: user-zh <[hidden email]> >> Subject: 回复： flink启动等待10分钟问题 >> >> 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ >> >> >> 2019-08-21 20:00:25,428 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - >> -------------------------------------------------------------------------------- >> 2019-08-21 20:00:25,430 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered >> UNIX signal handlers for [TERM, HUP, INT] >> 2019-08-21 20:00:25,433 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon >> is running as: root Yarn client user obtainer: root >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.cluster-id, >> application_1566383236573_0003 >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: jobmanager.rpc.address, localhost >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: jobmanager.rpc.port, 6123 >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.zookeeper.path.root, /flink-yarn >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.storageDir, >> hdfs://host51:9000/flink/ha-yarn >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.zookeeper.quorum, host51:2181 >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: internal.cluster.execution-mode, DETACHED >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability, zookeeper >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: parallelism.default, 1 >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: yarn.application-attempts, 10 >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: taskmanager.numberOfTaskSlots, 2 >> 2019-08-21 20:00:25,439 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: taskmanager.heap.size, 8192m >> 2019-08-21 20:00:25,439 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: jobmanager.heap.size, 2048m >> 2019-08-21 20:00:25,465 WARN >> org.apache.flink.configuration.Configuration - Config uses >> deprecated configuration key 'web.port' instead of proper key >> 'rest.bind-port' >> 2019-08-21 20:00:25,469 INFO >> org.apache.flink.runtime.clusterframework.BootstrapTools - Setting >> directories for temporary files to: >> /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 >> 2019-08-21 20:00:25,485 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting >> YarnJobClusterEntrypoint. >> 2019-08-21 20:00:25,485 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install >> default filesystem. >> 2019-08-21 20:00:25,561 INFO >> org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user >> set to root (auth:SIMPLE) >> 2019-08-21 20:00:25,580 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - >> Initializing cluster services. >> 2019-08-21 20:00:25,757 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to >> start actor system at host53:0 >> 2019-08-21 20:00:26,187 INFO >> akka.event.slf4j.Slf4jLogger - Slf4jLogger >> started >> 2019-08-21 20:00:26,251 INFO >> akka.remote.Remoting - Starting >> remoting >> 2019-08-21 20:00:26,439 INFO >> akka.remote.Remoting - Remoting >> started; listening on addresses :[akka.tcp://flink@host53:59609] >> 2019-08-21 20:00:26,447 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor >> system started at akka.tcp://flink@host53:59609 >> 2019-08-21 20:00:27,011 INFO >> org.apache.flink.runtime.blob.FileSystemBlobStore - Creating >> highly available BLOB storage directory at >> hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob >> 2019-08-21 20:00:27,097 INFO >> org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing >> default ACL for ZK connections >> 2019-08-21 20:00:27,097 INFO >> org.apache.flink.runtime.util.ZooKeeperUtils - Using >> '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. >> 2019-08-21 20:00:27,161 INFO >> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl >> - Starting >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, >> built on 03/23/2017 10:13 GMT >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:host.name=host53 >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.version=1.8.0_77 >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.vendor=Oracle Corporation >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.io.tmpdir=/tmp >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.compiler=<NA> >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:os.name=Linux >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:os.arch=amd64 >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:os.version=3.16.0-0.bpo.4-amd64 >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:user.name=root >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:user.home=/root >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 >> 2019-08-21 20:00:27,171 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - >> Initiating client connection, connectString=host51:2181 >> sessionTimeout=60000 >> watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 >> 2019-08-21 20:00:27,184 WARN >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL >> configuration failed: javax.security.auth.login.LoginException: No JAAS >> configuration section named 'Client' was found in specified JAAS >> configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue >> connection to Zookeeper server without SASL authentication, if Zookeeper >> server allows it. >> 2019-08-21 20:00:27,186 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - >> Opening socket connection to server host51/10.10.10.51:2181 >> 2019-08-21 20:00:27,187 ERROR >> org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - >> Authentication failed >> 2019-08-21 20:00:27,187 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket >> connection established to host51/10.10.10.51:2181, initiating session >> 2019-08-21 20:00:27,193 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - >> Session establishment complete on server host51/10.10.10.51:2181, >> sessionid = 0x1001a86de810008, negotiated timeout = 40000 >> 2019-08-21 20:00:27,193 INFO >> org.apache.flink.runtime.blob.BlobServer - Created >> BLOB server storage directory >> /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a >> 2019-08-21 20:00:27,195 INFO >> org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager >> - State change: CONNECTED >> 2019-08-21 20:00:27,198 INFO >> org.apache.flink.runtime.blob.BlobServer - Started >> BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max >> backlog: 1000 >> 2019-08-21 20:00:27,213 INFO >> org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics >> reporter configured, no metrics will be exposed/reported. >> 2019-08-21 20:00:27,215 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to >> start actor system at host53:0 >> 2019-08-21 20:00:27,239 INFO >> akka.event.slf4j.Slf4jLogger - Slf4jLogger >> started >> 2019-08-21 20:00:27,247 INFO >> akka.remote.Remoting - Starting >> remoting >> 2019-08-21 20:00:27,255 INFO >> akka.remote.Remoting - Remoting >> started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] >> 2019-08-21 20:00:27,257 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor >> system started at akka.tcp://flink-metrics@host53:59941 >> 2019-08-21 20:00:27,263 INFO >> org.apache.flink.runtime.blob.TransientBlobCache - Created >> BLOB cache storage directory >> /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 >> 2019-08-21 20:00:27,296 WARN >> org.apache.flink.configuration.Configuration - Config uses >> deprecated configuration key 'web.port' instead of proper key >> 'rest.bind-port' >> 2019-08-21 20:00:27,298 WARN >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload >> directory >> /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does >> not exist, or has been deleted externally. Previously uploaded files are no >> longer available. >> 2019-08-21 20:00:27,298 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created >> directory >> /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for >> file uploads. >> 2019-08-21 20:00:27,320 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting >> rest endpoint. >> 2019-08-21 20:00:27,616 INFO >> org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined >> location of main cluster component log file: >> /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log >> 2019-08-21 20:00:27,616 INFO >> org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined >> location of main cluster component stdout file: >> /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out >> 2019-08-21 20:00:27,771 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest >> endpoint listening at host53:46401 >> 2019-08-21 20:00:27,772 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. >> 2019-08-21 20:00:27,795 WARN >> org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The >> version of ZooKeeper being used doesn't support Container nodes. >> CreateMode.PERSISTENT will be used instead. >> 2019-08-21 20:00:27,807 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web >> frontend listening at http://host53:46401. >> 2019-08-21 20:00:27,836 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - >> http://host53:46401 was granted leadership with >> leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 >> 2019-08-21 20:00:27,932 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting >> RPC endpoint for org.apache.flink.yarn.YarnResourceManager at >> akka://flink/user/resourcemanager . >> 2019-08-21 20:00:28,021 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting >> RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at >> akka://flink/user/dispatcher . >> 2019-08-21 20:00:28,033 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. >> 2019-08-21 20:00:28,033 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. >> 2019-08-21 20:00:28,033 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. >> 2019-08-21 20:00:28,045 INFO >> org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher >> akka.tcp://flink@host53:59609/user/dispatcher was granted leadership >> with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be >> 2019-08-21 20:00:28,048 INFO >> org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering >> all persisted jobs. >> 2019-08-21 20:00:28,054 INFO >> org.apache.hadoop.yarn.client.RMProxy - Connecting >> to ResourceManager at host51/10.10.10.51:8030 >> 2019-08-21 20:00:28,078 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting >> RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at >> akka://flink/user/jobmanager_0 . >> 2019-08-21 20:00:28,086 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - >> Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). >> 2019-08-21 20:00:28,099 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Using >> restart strategy >> FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, >> delayBetweenRestartAttempts=0) for StreamKafkaMain >> (e5ea83238d721e932b4645589660406c). >> 2019-08-21 20:00:28,122 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >> recovers via failover strategy: full graph restart >> 2019-08-21 20:00:28,153 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Running >> initialization on master for job StreamKafkaMain >> (e5ea83238d721e932b4645589660406c). >> 2019-08-21 20:00:28,154 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - >> Successfully ran initialization on master in 0 ms. >> 2019-08-21 20:00:28,187 INFO >> org.apache.flink.runtime.util.ZooKeeperUtils - Initialized >> ZooKeeperCompletedCheckpointStore in >> '/checkpoints/e5ea83238d721e932b4645589660406c'. >> 2019-08-21 20:00:28,196 INFO >> org.apache.flink.yarn.YarnResourceManager - Recovered 0 >> containers from previous attempts ([]). >> 2019-08-21 20:00:28,199 INFO >> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - >> yarn.client.max-cached-nodemanagers-proxies : 0 >> 2019-08-21 20:00:28,200 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. >> 2019-08-21 20:00:28,203 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - No state >> backend has been configured, using default (Memory / JobManager) >> MemoryStateBackend (data in heap memory / checkpoints to JobManager) >> (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: >> 5242880) >> 2019-08-21 20:00:28,211 INFO >> org.apache.flink.yarn.YarnResourceManager - >> ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was >> granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 >> 2019-08-21 20:00:28,214 INFO >> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - >> Starting the SlotManager. >> 2019-08-21 20:00:28,225 INFO >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - >> Recovering checkpoints from ZooKeeper. >> 2019-08-21 20:00:28,229 INFO >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - >> Found 0 checkpoints in ZooKeeper. >> 2019-08-21 20:00:28,229 INFO >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - >> Trying to fetch 0 checkpoints from storage. >> 2019-08-21 20:00:28,233 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. >> 2019-08-21 20:00:28,248 INFO >> org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager >> runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was >> granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at >> akka.tcp://flink@host53:59609/user/jobmanager_0. >> 2019-08-21 20:00:28,260 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. >> 2019-08-21 20:00:28,260 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Starting >> execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under >> job master id a4576f0320d71bc112c96c94ebbd4991. >> 2019-08-21 20:00:28,261 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >> StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state >> CREATED to RUNNING. >> 2019-08-21 20:00:28,266 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,280 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] >> 2019-08-21 20:00:28,287 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,288 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] >> 2019-08-21 20:00:28,288 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) >> (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,292 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,295 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Connecting >> to ResourceManager akka.tcp://flink@host53 >> :59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) >> 2019-08-21 20:00:28,300 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Resolved >> ResourceManager address, beginning registration >> 2019-08-21 20:00:28,300 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - >> Registration at ResourceManager attempt 1 (timeout=100ms) >> 2019-08-21 20:00:28,304 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService >> /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. >> 2019-08-21 20:00:28,304 INFO >> org.apache.flink.yarn.YarnResourceManager - Registering >> job manager [hidden email]://flink@host53:59609/user/jobmanager_0 >> for job e5ea83238d721e932b4645589660406c. >> 2019-08-21 20:00:28,310 INFO >> org.apache.flink.yarn.YarnResourceManager - Registered >> job manager [hidden email]://flink@host53:59609/user/jobmanager_0 >> for job e5ea83238d721e932b4645589660406c. >> 2019-08-21 20:00:28,313 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - JobManager >> successfully registered at ResourceManager, leader id: >> 97537c0d61bbf091e37cb6c45b784568. >> 2019-08-21 20:00:28,314 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,315 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> faa6a521758e5be4b3dc1955ca0f51b4. >> 2019-08-21 20:00:28,316 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,316 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,316 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,321 INFO >> org.apache.flink.yarn.YarnResourceManager - Requesting >> new TaskExecutor container with resources <memory:8192, vCores:2>. Number >> pending requests 1. >> 2019-08-21 20:00:28,322 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> a2b419aea4a373686eeed6c46163a528. >> 2019-08-21 20:00:28,323 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> cbc11737ce6729cfe45f24c43180b5c8. >> 2019-08-21 20:00:28,323 INFO >> org.apache.flink.yarn.YarnResourceManager - Requesting >> new TaskExecutor container with resources <memory:8192, vCores:2>. Number >> pending requests 2. >> 2019-08-21 20:00:28,323 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> f6566567c173a55b2bc52b72bcb24ff8. >> 2019-08-21 20:00:31,597 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint >> triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> >> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, >> rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: >> (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state >> RUNNING but SCHEDULED instead. Aborting checkpoint. >> 2019-08-21 20:00:33,749 INFO >> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received >> new token for : host53:59938 >> 2019-08-21 20:00:33,749 INFO >> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received >> new token for : host52:33246 >> 2019-08-21 20:00:33,765 INFO >> org.apache.flink.yarn.YarnResourceManager - Received >> new container: container_1566383236573_0003_01_000002 - Remaining pending >> container requests: 2 >> 2019-08-21 20:00:33,765 INFO >> org.apache.flink.yarn.YarnResourceManager - Removing >> container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending >> container requests 1. >> 2019-08-21 20:00:34,009 INFO >> org.apache.flink.yarn.YarnResourceManager - Creating >> container launch context for TaskManagers >> 2019-08-21 20:00:34,010 INFO >> org.apache.flink.yarn.YarnResourceManager - Starting >> TaskManagers >> 2019-08-21 20:00:34,021 INFO >> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - >> Opening proxy : host53:59938 >> 2019-08-21 20:00:34,066 INFO >> org.apache.flink.yarn.YarnResourceManager - Received >> new container: container_1566383236573_0003_01_000003 - Remaining pending >> container requests: 1 >> 2019-08-21 20:00:34,066 INFO >> org.apache.flink.yarn.YarnResourceManager - Removing >> container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending >> container requests 0. >> 2019-08-21 20:00:34,088 INFO >> org.apache.flink.yarn.YarnResourceManager - Creating >> container launch context for TaskManagers >> 2019-08-21 20:00:34,089 INFO >> org.apache.flink.yarn.YarnResourceManager - Starting >> TaskManagers >> 2019-08-21 20:00:34,090 INFO >> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - >> Opening proxy : host52:33246 >> 2019-08-21 20:00:36,597 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint >> triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> >> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, >> rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: >> (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state >> RUNNING but SCHEDULED instead. Aborting checkpoint. >> 2019-08-21 20:00:37,856 INFO >> org.apache.flink.yarn.YarnResourceManager - Registering >> TaskManager with ResourceID container_1566383236573_0003_01_000002 >> (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager >> 2019-08-21 20:00:39,573 INFO >> org.apache.flink.yarn.YarnResourceManager - Registering >> TaskManager with ResourceID container_1566383236573_0003_01_000003 >> (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager >> 2019-08-21 20:00:39,659 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,659 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (attempt #0) to container_1566383236573_0003_01_000002 @ host53 >> (dataPort=52456) >> 2019-08-21 20:00:39,664 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,664 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (attempt #0) to container_1566383236573_0003_01_000002 @ host53 >> (dataPort=52456) >> 2019-08-21 20:00:39,665 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (attempt #0) to container_1566383236573_0003_01_000003 @ host52 >> (dataPort=39650) >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (attempt #0) to container_1566383236573_0003_01_000003 @ host52 >> (dataPort=39650) >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) >> (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: >> (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, >> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) >> -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt >> #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,670 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,670 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (1/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,671 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (2/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (3/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,673 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (4/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,673 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,673 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (1/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (2/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (3/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,676 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,676 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (4/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,782 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,783 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,784 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,787 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,806 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,809 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,809 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) >> (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,991 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,992 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,996 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,997 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:40,005 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:40,006 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:41,605 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering >> checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. >> 2019-08-21 20:00:42,466 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed >> checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 >> ms). >> >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: "Yun Tang"<[hidden email]>; >> 发送时间: 2019年8月22日(星期四) 中午11:08 >> 收件人: "user-zh"<[hidden email]>; >> >> 主题: Re: flink启动等待10分钟问题 >> >> >> >> Hi >> >> Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot >> request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request >> timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 >> >> 祝好 >> 唐云 >> ________________________________ >> From: 々守护々 <[hidden email]> >> Sent: Thursday, August 22, 2019 11:04 >> To: user-zh <[hidden email]> >> Subject: flink启动等待10分钟问题 >> >> 各位Flink社区大佬，您好！ >> 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： >> <property> >> <name>yarn.nodemanager.resource.memory-mb</name> >> <value>16384</value> >> </property> >> <property> >> <name>yarn.scheduler.minimum-allocation-mb</name> >> <value>1024</value> >> </property> >> <property> >> <name>yarn.scheduler.maximum-allocation-mb</name> >> <value>8192</value> >> </property> >> >> >> >> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p >> 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm >> bigdata6 ./flink_bigdata6.jar >> 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ >> > |
|
|
In reply to this post by tangjunliang@huitongjy.com
是的，集群的资源充足，附件有截图，就停在那不往下走了。 ------------------ 原始邮件 ------------------ 发件人: "[hidden email]"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:34 收件人: "user-zh"<[hidden email]>; 主题: 回复: 回复： flink启动等待10分钟问题 [hidden email] 发件人： 々守护々 发送时间： 2019-08-22 11:32 收件人： user-zh 主题： 回复： 回复： flink启动等待10分钟问题 是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:23 收件人: "user-zh"<[hidden email]>; 主题: Re: 回复： flink启动等待10分钟问题 启动时间是 20:00:25，task都处于running甚至第一次checkpoint completed的时间是20:00:42，一共才17秒，何来10分钟的问题？ ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:18 To: user-zh <[hidden email]> Subject: 回复： flink启动等待10分钟问题 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ 2019-08-21 20:00:25,428 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2019-08-21 20:00:25,430 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2019-08-21 20:00:25,433 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: root Yarn client user obtainer: root 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1566383236573_0003 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink-yarn 2019-08-21 20:00:25,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs://host51:9000/flink/ha-yarn 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, host51:2181 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, DETACHED 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-attempts, 10 2019-08-21 20:00:25,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 8192m 2019-08-21 20:00:25,439 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 2048m 2019-08-21 20:00:25,465 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:25,469 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint. 2019-08-21 20:00:25,485 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2019-08-21 20:00:25,561 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE) 2019-08-21 20:00:25,580 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2019-08-21 20:00:25,757 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at host53:0 2019-08-21 20:00:26,187 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:26,251 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:26,439 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@host53:59609] 2019-08-21 20:00:26,447 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@host53:59609 2019-08-21 20:00:27,011 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections 2019-08-21 20:00:27,097 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. 2019-08-21 20:00:27,161 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=host53 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_77 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2019-08-21 20:00:27,169 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.16.0-0.bpo.4-amd64 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 2019-08-21 20:00:27,170 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 2019-08-21 20:00:27,171 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=host51:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 2019-08-21 20:00:27,184 WARN org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2019-08-21 20:00:27,186 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server host51/10.10.10.51:2181 2019-08-21 20:00:27,187 ERROR org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed 2019-08-21 20:00:27,187 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to host51/10.10.10.51:2181, initiating session 2019-08-21 20:00:27,193 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server host51/10.10.10.51:2181, sessionid = 0x1001a86de810008, negotiated timeout = 40000 2019-08-21 20:00:27,193 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a 2019-08-21 20:00:27,195 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2019-08-21 20:00:27,198 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max backlog: 1000 2019-08-21 20:00:27,213 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2019-08-21 20:00:27,215 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at host53:0 2019-08-21 20:00:27,239 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-08-21 20:00:27,247 INFO akka.remote.Remoting - Starting remoting 2019-08-21 20:00:27,255 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] 2019-08-21 20:00:27,257 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metrics@host53:59941 2019-08-21 20:00:27,263 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 2019-08-21 20:00:27,296 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port' 2019-08-21 20:00:27,298 WARN org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2019-08-21 20:00:27,298 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created directory /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for file uploads. 2019-08-21 20:00:27,320 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting rest endpoint. 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log 2019-08-21 20:00:27,616 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out 2019-08-21 20:00:27,771 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at host53:46401 2019-08-21 20:00:27,772 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. 2019-08-21 20:00:27,795 WARN org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. 2019-08-21 20:00:27,807 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web frontend listening at http://host53:46401. 2019-08-21 20:00:27,836 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://host53:46401 was granted leadership with leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 2019-08-21 20:00:27,932 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager . 2019-08-21 20:00:28,021 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at akka://flink/user/dispatcher . 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. 2019-08-21 20:00:28,033 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. 2019-08-21 20:00:28,045 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be 2019-08-21 20:00:28,048 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering all persisted jobs. 2019-08-21 20:00:28,054 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at host51/10.10.10.51:8030 2019-08-21 20:00:28,078 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2019-08-21 20:00:28,086 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,099 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=0) for StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,122 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2019-08-21 20:00:28,153 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job StreamKafkaMain (e5ea83238d721e932b4645589660406c). 2019-08-21 20:00:28,154 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms. 2019-08-21 20:00:28,187 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Initialized ZooKeeperCompletedCheckpointStore in '/checkpoints/e5ea83238d721e932b4645589660406c'. 2019-08-21 20:00:28,196 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]). 2019-08-21 20:00:28,199 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2019-08-21 20:00:28,200 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. 2019-08-21 20:00:28,203 INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 2019-08-21 20:00:28,211 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 2019-08-21 20:00:28,214 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2019-08-21 20:00:28,225 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 0 checkpoints in ZooKeeper. 2019-08-21 20:00:28,229 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to fetch 0 checkpoints from storage. 2019-08-21 20:00:28,233 INFO org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. 2019-08-21 20:00:28,248 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at akka.tcp://flink@host53:59609/user/jobmanager_0. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. 2019-08-21 20:00:28,260 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under job master id a4576f0320d71bc112c96c94ebbd4991. 2019-08-21 20:00:28,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state CREATED to RUNNING. 2019-08-21 20:00:28,266 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,280 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] 2019-08-21 20:00:28,287 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] 2019-08-21 20:00:28,288 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] 2019-08-21 20:00:28,289 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,291 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,292 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from CREATED to SCHEDULED. 2019-08-21 20:00:28,295 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration 2019-08-21 20:00:28,300 INFO org.apache.flink.runtime.jobmaster.JobMaster - Registration at ResourceManager attempt 1 (timeout=100ms) 2019-08-21 20:00:28,304 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. 2019-08-21 20:00:28,304 INFO org.apache.flink.yarn.YarnResourceManager - Registering job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,310 INFO org.apache.flink.yarn.YarnResourceManager - Registered job manager [hidden email]://flink@host53:59609/user/jobmanager_0 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:28,313 INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: 97537c0d61bbf091e37cb6c45b784568. 2019-08-21 20:00:28,314 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,315 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id faa6a521758e5be4b3dc1955ca0f51b4. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,316 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. 2019-08-21 20:00:28,321 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 1. 2019-08-21 20:00:28,322 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id a2b419aea4a373686eeed6c46163a528. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id cbc11737ce6729cfe45f24c43180b5c8. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 2. 2019-08-21 20:00:28,323 INFO org.apache.flink.yarn.YarnResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job e5ea83238d721e932b4645589660406c with allocation id f6566567c173a55b2bc52b72bcb24ff8. 2019-08-21 20:00:31,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host53:59938 2019-08-21 20:00:33,749 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : host52:33246 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000002 - Remaining pending container requests: 2 2019-08-21 20:00:33,765 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 1. 2019-08-21 20:00:34,009 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,010 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,021 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host53:59938 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_1566383236573_0003_01_000003 - Remaining pending container requests: 1 2019-08-21 20:00:34,066 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 0. 2019-08-21 20:00:34,088 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers 2019-08-21 20:00:34,089 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers 2019-08-21 20:00:34,090 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : host52:33246 2019-08-21 20:00:36,597 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state RUNNING but SCHEDULED instead. Aborting checkpoint. 2019-08-21 20:00:37,856 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000002 (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,573 INFO org.apache.flink.yarn.YarnResourceManager - Registering TaskManager with ResourceID container_1566383236573_0003_01_000003 (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,659 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,664 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,665 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,666 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,670 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,671 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,672 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,673 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (1/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (2/4) (attempt #0) to container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,675 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (3/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from SCHEDULED to DEPLOYING. 2019-08-21 20:00:39,676 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Print to Std. Out (4/4) (attempt #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) 2019-08-21 20:00:39,782 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,783 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,784 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,787 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,806 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,809 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,991 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,992 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,996 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:39,997 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,005 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:40,006 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. 2019-08-21 20:00:41,605 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. 2019-08-21 20:00:42,466 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 ms). ------------------ 原始邮件 ------------------ 发件人: "Yun Tang"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:08 收件人: "user-zh"<[hidden email]>; 主题: Re: flink启动等待10分钟问题 Hi Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 祝好 唐云 ________________________________ From: 々守护々 <[hidden email]> Sent: Thursday, August 22, 2019 11:04 To: user-zh <[hidden email]> Subject: flink启动等待10分钟问题 各位Flink社区大佬，您好！ 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>16384</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>1024</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value> </property> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm bigdata6 ./flink_bigdata6.jar 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ |
|
|
图看不到，可以用邮件附件或者找个第三方图库...
Best, tison. 々守护々 <[hidden email]> 于2019年8月22日周四 上午11:39写道： > 是的，集群的资源充足，附件有截图，就停在那不往下走了。 > > > ------------------ 原始邮件 ------------------ > *发件人:* "[hidden email]"<[hidden email]>; > *发送时间:* 2019年8月22日(星期四) 中午11:34 > *收件人:* "user-zh"<[hidden email]>; > *主题:* 回复: 回复： flink启动等待10分钟问题 > > 当时你的集群资源是否充足? > > > > [hidden email] > > 发件人： 々守护々 > 发送时间： 2019-08-22 11:32 > 收件人： user-zh > 主题： 回复： 回复： flink启动等待10分钟问题 > 是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application > > > ------------------ 原始邮件 ------------------ > 发件人: "Yun Tang"<[hidden email]>; > 发送时间: 2019年8月22日(星期四) 中午11:23 > 收件人: "user-zh"<[hidden email]>; > 主题: Re: 回复： flink启动等待10分钟问题 > > 启动时间是 20:00:25，task都处于running甚至第一次checkpoint > completed的时间是20:00:42，一共才17秒，何来10分钟的问题？ > ________________________________ > From: 々守护々 <[hidden email]> > Sent: Thursday, August 22, 2019 11:18 > To: user-zh <[hidden email]> > Subject: 回复： flink启动等待10分钟问题 > > 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ > > > 2019-08-21 20:00:25,428 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > -------------------------------------------------------------------------------- > 2019-08-21 20:00:25,430 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered > UNIX signal handlers for [TERM, HUP, INT] > 2019-08-21 20:00:25,433 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon > is running as: root Yarn client user obtainer: root > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.cluster-id, > application_1566383236573_0003 > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.rpc.address, localhost > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.rpc.port, 6123 > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.zookeeper.path.root, /flink-yarn > 2019-08-21 20:00:25,437 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.storageDir, > hdfs://host51:9000/flink/ha-yarn > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability.zookeeper.quorum, host51:2181 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: internal.cluster.execution-mode, DETACHED > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: high-availability, zookeeper > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: parallelism.default, 1 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: yarn.application-attempts, 10 > 2019-08-21 20:00:25,438 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.numberOfTaskSlots, 2 > 2019-08-21 20:00:25,439 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: taskmanager.heap.size, 8192m > 2019-08-21 20:00:25,439 INFO > org.apache.flink.configuration.GlobalConfiguration - Loading > configuration property: jobmanager.heap.size, 2048m > 2019-08-21 20:00:25,465 WARN > org.apache.flink.configuration.Configuration - Config uses > deprecated configuration key 'web.port' instead of proper key > 'rest.bind-port' > 2019-08-21 20:00:25,469 INFO > org.apache.flink.runtime.clusterframework.BootstrapTools - Setting > directories for temporary files to: > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 > 2019-08-21 20:00:25,485 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting > YarnJobClusterEntrypoint. > 2019-08-21 20:00:25,485 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install > default filesystem. > 2019-08-21 20:00:25,561 INFO > org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user > set to root (auth:SIMPLE) > 2019-08-21 20:00:25,580 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > Initializing cluster services. > 2019-08-21 20:00:25,757 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to > start actor system at host53:0 > 2019-08-21 20:00:26,187 INFO > akka.event.slf4j.Slf4jLogger - Slf4jLogger > started > 2019-08-21 20:00:26,251 INFO > akka.remote.Remoting - Starting > remoting > 2019-08-21 20:00:26,439 INFO > akka.remote.Remoting - Remoting > started; listening on addresses :[akka.tcp://flink@host53:59609] > 2019-08-21 20:00:26,447 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor > system started at akka.tcp://flink@host53:59609 > 2019-08-21 20:00:27,011 INFO > org.apache.flink.runtime.blob.FileSystemBlobStore - Creating > highly available BLOB storage directory at > hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob > 2019-08-21 20:00:27,097 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing > default ACL for ZK connections > 2019-08-21 20:00:27,097 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Using > '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. > 2019-08-21 20:00:27,161 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > - Starting > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, > built on 03/23/2017 10:13 GMT > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:host.name=host53 > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.version=1.8.0_77 > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.vendor=Oracle Corporation > 2019-08-21 20:00:27,169 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.io.tmpdir=/tmp > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:java.compiler=<NA> > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.name=Linux > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.arch=amd64 > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:os.version=3.16.0-0.bpo.4-amd64 > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.name=root > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.home=/root > 2019-08-21 20:00:27,170 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client > environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 > 2019-08-21 20:00:27,171 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Initiating client connection, connectString=host51:2181 > sessionTimeout=60000 > watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 > 2019-08-21 20:00:27,184 WARN > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL > configuration failed: javax.security.auth.login.LoginException: No JAAS > configuration section named 'Client' was found in specified JAAS > configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue > connection to Zookeeper server without SASL authentication, if Zookeeper > server allows it. > 2019-08-21 20:00:27,186 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > Opening socket connection to server host51/10.10.10.51:2181 > 2019-08-21 20:00:27,187 ERROR > org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - > Authentication failed > 2019-08-21 20:00:27,187 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket > connection established to host51/10.10.10.51:2181, initiating session > 2019-08-21 20:00:27,193 INFO > org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > Session establishment complete on server host51/10.10.10.51:2181, > sessionid = 0x1001a86de810008, negotiated timeout = 40000 > 2019-08-21 20:00:27,193 INFO > org.apache.flink.runtime.blob.BlobServer - Created > BLOB server storage directory > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a > 2019-08-21 20:00:27,195 INFO > org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager > - State change: CONNECTED > 2019-08-21 20:00:27,198 INFO > org.apache.flink.runtime.blob.BlobServer - Started > BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max backlog: > 1000 > 2019-08-21 20:00:27,213 INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics > reporter configured, no metrics will be exposed/reported. > 2019-08-21 20:00:27,215 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to > start actor system at host53:0 > 2019-08-21 20:00:27,239 INFO > akka.event.slf4j.Slf4jLogger - Slf4jLogger > started > 2019-08-21 20:00:27,247 INFO > akka.remote.Remoting - Starting > remoting > 2019-08-21 20:00:27,255 INFO > akka.remote.Remoting - Remoting > started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] > 2019-08-21 20:00:27,257 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor > system started at akka.tcp://flink-metrics@host53:59941 > 2019-08-21 20:00:27,263 INFO > org.apache.flink.runtime.blob.TransientBlobCache - Created > BLOB cache storage directory > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 > 2019-08-21 20:00:27,296 WARN > org.apache.flink.configuration.Configuration - Config uses > deprecated configuration key 'web.port' instead of proper key > 'rest.bind-port' > 2019-08-21 20:00:27,298 WARN > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload > directory > /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does > not exist, or has been deleted externally. Previously uploaded files are no > longer available. > 2019-08-21 20:00:27,298 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created > directory > /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for > file uploads. > 2019-08-21 20:00:27,320 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting > rest endpoint. > 2019-08-21 20:00:27,616 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of main cluster component log file: > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log > 2019-08-21 20:00:27,616 INFO > org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined > location of main cluster component stdout file: > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out > 2019-08-21 20:00:27,771 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest > endpoint listening at host53:46401 > 2019-08-21 20:00:27,772 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. > 2019-08-21 20:00:27,795 WARN > org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The > version of ZooKeeper being used doesn't support Container nodes. > CreateMode.PERSISTENT will be used instead. > 2019-08-21 20:00:27,807 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web > frontend listening at http://host53:46401. > 2019-08-21 20:00:27,836 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - > http://host53:46401 was granted leadership with > leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 > 2019-08-21 20:00:27,932 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.yarn.YarnResourceManager at > akka://flink/user/resourcemanager . > 2019-08-21 20:00:28,021 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at > akka://flink/user/dispatcher . > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. > 2019-08-21 20:00:28,033 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. > 2019-08-21 20:00:28,045 INFO > org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher > akka.tcp://flink@host53:59609/user/dispatcher was granted leadership with > fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be > 2019-08-21 20:00:28,048 INFO > org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering > all persisted jobs. > 2019-08-21 20:00:28,054 INFO > org.apache.hadoop.yarn.client.RMProxy - Connecting > to ResourceManager at host51/10.10.10.51:8030 > 2019-08-21 20:00:28,078 INFO > org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at > akka://flink/user/jobmanager_0 . > 2019-08-21 20:00:28,086 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,099 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Using > restart strategy > FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, > delayBetweenRestartAttempts=0) for StreamKafkaMain > (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,122 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > recovers via failover strategy: full graph restart > 2019-08-21 20:00:28,153 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Running > initialization on master for job StreamKafkaMain > (e5ea83238d721e932b4645589660406c). > 2019-08-21 20:00:28,154 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Successfully ran initialization on master in 0 ms. > 2019-08-21 20:00:28,187 INFO > org.apache.flink.runtime.util.ZooKeeperUtils - Initialized > ZooKeeperCompletedCheckpointStore in > '/checkpoints/e5ea83238d721e932b4645589660406c'. > 2019-08-21 20:00:28,196 INFO > org.apache.flink.yarn.YarnResourceManager - Recovered 0 > containers from previous attempts ([]). > 2019-08-21 20:00:28,199 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > yarn.client.max-cached-nodemanagers-proxies : 0 > 2019-08-21 20:00:28,200 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. > 2019-08-21 20:00:28,203 INFO > org.apache.flink.runtime.jobmaster.JobMaster - No state > backend has been configured, using default (Memory / JobManager) > MemoryStateBackend (data in heap memory / checkpoints to JobManager) > (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: > 5242880) > 2019-08-21 20:00:28,211 INFO > org.apache.flink.yarn.YarnResourceManager - > ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was > granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 > 2019-08-21 20:00:28,214 INFO > org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - > Starting the SlotManager. > 2019-08-21 20:00:28,225 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Recovering checkpoints from ZooKeeper. > 2019-08-21 20:00:28,229 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Found 0 checkpoints in ZooKeeper. > 2019-08-21 20:00:28,229 INFO > org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > Trying to fetch 0 checkpoints from storage. > 2019-08-21 20:00:28,233 INFO > org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - > Starting ZooKeeperLeaderElectionService > ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. > 2019-08-21 20:00:28,248 INFO > org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager > runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was > granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at > akka.tcp://flink@host53:59609/user/jobmanager_0. > 2019-08-21 20:00:28,260 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > 2019-08-21 20:00:28,260 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Starting > execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under > job master id a4576f0320d71bc112c96c94ebbd4991. > 2019-08-21 20:00:28,261 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state > CREATED to RUNNING. > 2019-08-21 20:00:28,266 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,280 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] > 2019-08-21 20:00:28,287 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,288 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] > 2019-08-21 20:00:28,288 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > serve slot request, no ResourceManager connected. Adding as pending request > [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] > 2019-08-21 20:00:28,289 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED > to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,291 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,292 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from CREATED to SCHEDULED. > 2019-08-21 20:00:28,295 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Connecting > to ResourceManager akka.tcp://flink@host53 > :59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) > 2019-08-21 20:00:28,300 INFO > org.apache.flink.runtime.jobmaster.JobMaster - Resolved > ResourceManager address, beginning registration > 2019-08-21 20:00:28,300 INFO > org.apache.flink.runtime.jobmaster.JobMaster - > Registration at ResourceManager attempt 1 (timeout=100ms) > 2019-08-21 20:00:28,304 INFO > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > Starting ZooKeeperLeaderRetrievalService > /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. > 2019-08-21 20:00:28,304 INFO > org.apache.flink.yarn.YarnResourceManager - Registering > job manager [hidden email]://flink@host53:59609/user/jobmanager_0 > for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:28,310 INFO > org.apache.flink.yarn.YarnResourceManager - Registered > job manager [hidden email]://flink@host53:59609/user/jobmanager_0 > for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:28,313 INFO > org.apache.flink.runtime.jobmaster.JobMaster - JobManager > successfully registered at ResourceManager, leader id: > 97537c0d61bbf091e37cb6c45b784568. > 2019-08-21 20:00:28,314 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,315 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > faa6a521758e5be4b3dc1955ca0f51b4. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,316 INFO > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting > new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile > ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > 2019-08-21 20:00:28,321 INFO > org.apache.flink.yarn.YarnResourceManager - Requesting > new TaskExecutor container with resources <memory:8192, vCores:2>. Number > pending requests 1. > 2019-08-21 20:00:28,322 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > a2b419aea4a373686eeed6c46163a528. > 2019-08-21 20:00:28,323 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > cbc11737ce6729cfe45f24c43180b5c8. > 2019-08-21 20:00:28,323 INFO > org.apache.flink.yarn.YarnResourceManager - Requesting > new TaskExecutor container with resources <memory:8192, vCores:2>. Number > pending requests 2. > 2019-08-21 20:00:28,323 INFO > org.apache.flink.yarn.YarnResourceManager - Request > slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > e5ea83238d721e932b4645589660406c with allocation id > f6566567c173a55b2bc52b72bcb24ff8. > 2019-08-21 20:00:31,597 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint > triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > RUNNING but SCHEDULED instead. Aborting checkpoint. > 2019-08-21 20:00:33,749 INFO > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > new token for : host53:59938 > 2019-08-21 20:00:33,749 INFO > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > new token for : host52:33246 > 2019-08-21 20:00:33,765 INFO > org.apache.flink.yarn.YarnResourceManager - Received > new container: container_1566383236573_0003_01_000002 - Remaining pending > container requests: 2 > 2019-08-21 20:00:33,765 INFO > org.apache.flink.yarn.YarnResourceManager - Removing > container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending > container requests 1. > 2019-08-21 20:00:34,009 INFO > org.apache.flink.yarn.YarnResourceManager - Creating > container launch context for TaskManagers > 2019-08-21 20:00:34,010 INFO > org.apache.flink.yarn.YarnResourceManager - Starting > TaskManagers > 2019-08-21 20:00:34,021 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : host53:59938 > 2019-08-21 20:00:34,066 INFO > org.apache.flink.yarn.YarnResourceManager - Received > new container: container_1566383236573_0003_01_000003 - Remaining pending > container requests: 1 > 2019-08-21 20:00:34,066 INFO > org.apache.flink.yarn.YarnResourceManager - Removing > container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending > container requests 0. > 2019-08-21 20:00:34,088 INFO > org.apache.flink.yarn.YarnResourceManager - Creating > container launch context for TaskManagers > 2019-08-21 20:00:34,089 INFO > org.apache.flink.yarn.YarnResourceManager - Starting > TaskManagers > 2019-08-21 20:00:34,090 INFO > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > Opening proxy : host52:33246 > 2019-08-21 20:00:36,597 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint > triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > RUNNING but SCHEDULED instead. Aborting checkpoint. > 2019-08-21 20:00:37,856 INFO > org.apache.flink.yarn.YarnResourceManager - Registering > TaskManager with ResourceID container_1566383236573_0003_01_000002 > (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager > 2019-08-21 20:00:39,573 INFO > org.apache.flink.yarn.YarnResourceManager - Registering > TaskManager with ResourceID container_1566383236573_0003_01_000003 > (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager > 2019-08-21 20:00:39,659 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,659 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > (dataPort=52456) > 2019-08-21 20:00:39,664 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,664 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > (dataPort=52456) > 2019-08-21 20:00:39,665 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > (dataPort=39650) > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > (dataPort=39650) > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,666 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: > (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, > end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) > -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt > #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,670 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,670 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (1/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,671 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (2/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (3/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,672 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Sink: Print to Std. Out (4/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,673 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (1/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (2/4) (attempt #0) to > container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,675 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (3/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,676 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from SCHEDULED to DEPLOYING. > 2019-08-21 20:00:39,676 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying > Map -> Sink: Print to Std. Out (4/4) (attempt #0) to > container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > 2019-08-21 20:00:39,782 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,783 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,784 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,787 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,806 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) > (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,809 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) > (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,809 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: > (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,991 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,992 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print > to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > DEPLOYING to RUNNING. > 2019-08-21 20:00:39,996 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:39,997 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched > from DEPLOYING to RUNNING. > 2019-08-21 20:00:40,005 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) > (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:40,006 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > Custom Source -> Flat Map -> Filter -> Filter -> Map -> > Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> > select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) > (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. > 2019-08-21 20:00:41,605 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering > checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. > 2019-08-21 20:00:42,466 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed > checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 > ms). > > > > ------------------ 原始邮件 ------------------ > 发件人: "Yun Tang"<[hidden email]>; > 发送时间: 2019年8月22日(星期四) 中午11:08 > 收件人: "user-zh"<[hidden email]>; > > 主题: Re: flink启动等待10分钟问题 > > > > Hi > > Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot > request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request > timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 > > 祝好 > 唐云 > ________________________________ > From: 々守护々 <[hidden email]> > Sent: Thursday, August 22, 2019 11:04 > To: user-zh <[hidden email]> > Subject: flink启动等待10分钟问题 > > 各位Flink社区大佬，您好！ > 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： > <property> > <name>yarn.nodemanager.resource.memory-mb</name> > <value>16384</value> > </property> > <property> > <name>yarn.scheduler.minimum-allocation-mb</name> > <value>1024</value> > </property> > <property> > <name>yarn.scheduler.maximum-allocation-mb</name> > <value>8192</value> > </property> > > > > 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p > 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm > bigdata6 ./flink_bigdata6.jar > 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ > |
|
|
In reply to this post by tison
这个是终端打印的日志,十分钟后才启动- Submitting application master application_1566383236573_0004 2019-08-22 11:28:21,766 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - This YARN session requires 34816MB of memory in the cluster. There are currently only 26624MB available. The Flink YARN client will try to allocate the YARN session, but maybe not all TaskManagers are connecting from the beginning because the resources are currently not available in the cluster. The allocation might take more time than usual because the Flink YARN client needs to wait until the resources become available. 2019-08-22 11:28:21,766 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - There is not enough memory available in the YARN cluster. The TaskManager(s) require 8192MB each. NodeManagers available: [14336, 12288] After allocating the JobManager (2048MB) and (2/4) TaskManagers, the following NodeManagers are available: [4096, 4096] The Flink YARN client will try to allocate the YARN session, but maybe not all TaskManagers are connecting from the beginning because the resources are currently not available in the cluster. The allocation might take more time than usual because the Flink YARN client needs to wait until the resources become available. 2019-08-22 11:28:21,766 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - There is not enough memory available in the YARN cluster. The TaskManager(s) require 8192MB each. NodeManagers available: [14336, 12288] After allocating the JobManager (2048MB) and (3/4) TaskManagers, the following NodeManagers are available: [4096, 4096] The Flink YARN client will try to allocate the YARN session, but maybe not all TaskManagers are connecting from the beginning because the resources are currently not available in the cluster. The allocation might take more time than usual because the Flink YARN client needs to wait until the resources become available. 2019-08-22 11:28:21,767 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster specification: ClusterSpecification{masterMemoryMB=2048, taskManagerMemoryMB=8192, numberTaskManagers=4, slotsPerTaskManager=2} 2019-08-22 11:28:22,317 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - The configuration directory ('/usr/flink/flink-1.8.1/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them. 2019-08-22 11:38:02,185 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting application master application_1566383236573_0004 2019-08-22 11:38:02,226 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1566383236573_0004 2019-08-22 11:38:02,226 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for the cluster to be allocated 2019-08-22 11:38:02,228 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying cluster, current state ACCEPTED 2019-08-22 11:38:07,244 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN application has been deployed successfully. 2019-08-22 11:38:07,244 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink YARN client has been started in detached mode. In order to stop Flink on YARN, use the following command or a YARN web interface to stop it: yarn application -kill application_1566383236573_0004 Please also note that the temporary files of the YARN session in the home directory will not be removed. Job has been submitted with JobID c16c4fc1556ccb2ceaaa2f0e8f32ec88 ------------------ 原始邮件 ------------------ 发件人: "Zili Chen"<[hidden email]>; 发送时间: 2019年8月22日(星期四) 中午11:37 收件人: "user-zh"<[hidden email]>; 主题: Re: 回复： flink启动等待10分钟问题 YARN 集群忙导致 10 分钟没响应。 Best, tison. Zili Chen <[hidden email]> 于2019年8月22日周四 上午11:35写道： > user-zh 不支持贴图，你用下第三方存储然后贴个链接吧，或者我记得可以传邮件附件 > > Best, > tison. > > > 々守护々 <[hidden email]> 于2019年8月22日周四 上午11:33写道： > >> 是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application >> <http://hadoop.changba.com/cluster/app/application_1566383236573_0003> >> >> >> ------------------ 原始邮件 ------------------ >> *发件人:* "Yun Tang"<[hidden email]>; >> *发送时间:* 2019年8月22日(星期四) 中午11:23 >> *收件人:* "user-zh"<[hidden email]>; >> *主题:* Re: 回复： flink启动等待10分钟问题 >> >> 启动时间是 20:00:25，task都处于running甚至第一次checkpoint >> completed的时间是20:00:42，一共才17秒，何来10分钟的问题？ >> ________________________________ >> From: 々守护々 <[hidden email]> >> Sent: Thursday, August 22, 2019 11:18 >> To: user-zh <[hidden email]> >> Subject: 回复： flink启动等待10分钟问题 >> >> 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ >> >> >> 2019-08-21 20:00:25,428 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - >> -------------------------------------------------------------------------------- >> 2019-08-21 20:00:25,430 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered >> UNIX signal handlers for [TERM, HUP, INT] >> 2019-08-21 20:00:25,433 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon >> is running as: root Yarn client user obtainer: root >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.cluster-id, >> application_1566383236573_0003 >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: jobmanager.rpc.address, localhost >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: jobmanager.rpc.port, 6123 >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.zookeeper.path.root, /flink-yarn >> 2019-08-21 20:00:25,437 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.storageDir, >> hdfs://host51:9000/flink/ha-yarn >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability.zookeeper.quorum, host51:2181 >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: internal.cluster.execution-mode, DETACHED >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: high-availability, zookeeper >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: parallelism.default, 1 >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: yarn.application-attempts, 10 >> 2019-08-21 20:00:25,438 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: taskmanager.numberOfTaskSlots, 2 >> 2019-08-21 20:00:25,439 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: taskmanager.heap.size, 8192m >> 2019-08-21 20:00:25,439 INFO >> org.apache.flink.configuration.GlobalConfiguration - Loading >> configuration property: jobmanager.heap.size, 2048m >> 2019-08-21 20:00:25,465 WARN >> org.apache.flink.configuration.Configuration - Config uses >> deprecated configuration key 'web.port' instead of proper key >> 'rest.bind-port' >> 2019-08-21 20:00:25,469 INFO >> org.apache.flink.runtime.clusterframework.BootstrapTools - Setting >> directories for temporary files to: >> /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 >> 2019-08-21 20:00:25,485 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting >> YarnJobClusterEntrypoint. >> 2019-08-21 20:00:25,485 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install >> default filesystem. >> 2019-08-21 20:00:25,561 INFO >> org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user >> set to root (auth:SIMPLE) >> 2019-08-21 20:00:25,580 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - >> Initializing cluster services. >> 2019-08-21 20:00:25,757 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to >> start actor system at host53:0 >> 2019-08-21 20:00:26,187 INFO >> akka.event.slf4j.Slf4jLogger - Slf4jLogger >> started >> 2019-08-21 20:00:26,251 INFO >> akka.remote.Remoting - Starting >> remoting >> 2019-08-21 20:00:26,439 INFO >> akka.remote.Remoting - Remoting >> started; listening on addresses :[akka.tcp://flink@host53:59609] >> 2019-08-21 20:00:26,447 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor >> system started at akka.tcp://flink@host53:59609 >> 2019-08-21 20:00:27,011 INFO >> org.apache.flink.runtime.blob.FileSystemBlobStore - Creating >> highly available BLOB storage directory at >> hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob >> 2019-08-21 20:00:27,097 INFO >> org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing >> default ACL for ZK connections >> 2019-08-21 20:00:27,097 INFO >> org.apache.flink.runtime.util.ZooKeeperUtils - Using >> '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. >> 2019-08-21 20:00:27,161 INFO >> org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl >> - Starting >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, >> built on 03/23/2017 10:13 GMT >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:host.name=host53 >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.version=1.8.0_77 >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.vendor=Oracle Corporation >> 2019-08-21 20:00:27,169 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.io.tmpdir=/tmp >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:java.compiler=<NA> >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:os.name=Linux >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:os.arch=amd64 >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:os.version=3.16.0-0.bpo.4-amd64 >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:user.name=root >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:user.home=/root >> 2019-08-21 20:00:27,170 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client >> environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 >> 2019-08-21 20:00:27,171 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - >> Initiating client connection, connectString=host51:2181 >> sessionTimeout=60000 >> watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 >> 2019-08-21 20:00:27,184 WARN >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - SASL >> configuration failed: javax.security.auth.login.LoginException: No JAAS >> configuration section named 'Client' was found in specified JAAS >> configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue >> connection to Zookeeper server without SASL authentication, if Zookeeper >> server allows it. >> 2019-08-21 20:00:27,186 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - >> Opening socket connection to server host51/10.10.10.51:2181 >> 2019-08-21 20:00:27,187 ERROR >> org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - >> Authentication failed >> 2019-08-21 20:00:27,187 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket >> connection established to host51/10.10.10.51:2181, initiating session >> 2019-08-21 20:00:27,193 INFO >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - >> Session establishment complete on server host51/10.10.10.51:2181, >> sessionid = 0x1001a86de810008, negotiated timeout = 40000 >> 2019-08-21 20:00:27,193 INFO >> org.apache.flink.runtime.blob.BlobServer - Created >> BLOB server storage directory >> /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a >> 2019-08-21 20:00:27,195 INFO >> org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager >> - State change: CONNECTED >> 2019-08-21 20:00:27,198 INFO >> org.apache.flink.runtime.blob.BlobServer - Started >> BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max >> backlog: 1000 >> 2019-08-21 20:00:27,213 INFO >> org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics >> reporter configured, no metrics will be exposed/reported. >> 2019-08-21 20:00:27,215 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to >> start actor system at host53:0 >> 2019-08-21 20:00:27,239 INFO >> akka.event.slf4j.Slf4jLogger - Slf4jLogger >> started >> 2019-08-21 20:00:27,247 INFO >> akka.remote.Remoting - Starting >> remoting >> 2019-08-21 20:00:27,255 INFO >> akka.remote.Remoting - Remoting >> started; listening on addresses :[akka.tcp://flink-metrics@host53:59941] >> 2019-08-21 20:00:27,257 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor >> system started at akka.tcp://flink-metrics@host53:59941 >> 2019-08-21 20:00:27,263 INFO >> org.apache.flink.runtime.blob.TransientBlobCache - Created >> BLOB cache storage directory >> /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 >> 2019-08-21 20:00:27,296 WARN >> org.apache.flink.configuration.Configuration - Config uses >> deprecated configuration key 'web.port' instead of proper key >> 'rest.bind-port' >> 2019-08-21 20:00:27,298 WARN >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload >> directory >> /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload does >> not exist, or has been deleted externally. Previously uploaded files are no >> longer available. >> 2019-08-21 20:00:27,298 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created >> directory >> /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for >> file uploads. >> 2019-08-21 20:00:27,320 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting >> rest endpoint. >> 2019-08-21 20:00:27,616 INFO >> org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined >> location of main cluster component log file: >> /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log >> 2019-08-21 20:00:27,616 INFO >> org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined >> location of main cluster component stdout file: >> /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out >> 2019-08-21 20:00:27,771 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest >> endpoint listening at host53:46401 >> 2019-08-21 20:00:27,772 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. >> 2019-08-21 20:00:27,795 WARN >> org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The >> version of ZooKeeper being used doesn't support Container nodes. >> CreateMode.PERSISTENT will be used instead. >> 2019-08-21 20:00:27,807 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web >> frontend listening at http://host53:46401. >> 2019-08-21 20:00:27,836 INFO >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - >> http://host53:46401 was granted leadership with >> leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 >> 2019-08-21 20:00:27,932 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting >> RPC endpoint for org.apache.flink.yarn.YarnResourceManager at >> akka://flink/user/resourcemanager . >> 2019-08-21 20:00:28,021 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting >> RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at >> akka://flink/user/dispatcher . >> 2019-08-21 20:00:28,033 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. >> 2019-08-21 20:00:28,033 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. >> 2019-08-21 20:00:28,033 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. >> 2019-08-21 20:00:28,045 INFO >> org.apache.flink.runtime.dispatcher.MiniDispatcher - Dispatcher >> akka.tcp://flink@host53:59609/user/dispatcher was granted leadership >> with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be >> 2019-08-21 20:00:28,048 INFO >> org.apache.flink.runtime.dispatcher.MiniDispatcher - Recovering >> all persisted jobs. >> 2019-08-21 20:00:28,054 INFO >> org.apache.hadoop.yarn.client.RMProxy - Connecting >> to ResourceManager at host51/10.10.10.51:8030 >> 2019-08-21 20:00:28,078 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting >> RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at >> akka://flink/user/jobmanager_0 . >> 2019-08-21 20:00:28,086 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - >> Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). >> 2019-08-21 20:00:28,099 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Using >> restart strategy >> FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, >> delayBetweenRestartAttempts=0) for StreamKafkaMain >> (e5ea83238d721e932b4645589660406c). >> 2019-08-21 20:00:28,122 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >> recovers via failover strategy: full graph restart >> 2019-08-21 20:00:28,153 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Running >> initialization on master for job StreamKafkaMain >> (e5ea83238d721e932b4645589660406c). >> 2019-08-21 20:00:28,154 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - >> Successfully ran initialization on master in 0 ms. >> 2019-08-21 20:00:28,187 INFO >> org.apache.flink.runtime.util.ZooKeeperUtils - Initialized >> ZooKeeperCompletedCheckpointStore in >> '/checkpoints/e5ea83238d721e932b4645589660406c'. >> 2019-08-21 20:00:28,196 INFO >> org.apache.flink.yarn.YarnResourceManager - Recovered 0 >> containers from previous attempts ([]). >> 2019-08-21 20:00:28,199 INFO >> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - >> yarn.client.max-cached-nodemanagers-proxies : 0 >> 2019-08-21 20:00:28,200 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. >> 2019-08-21 20:00:28,203 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - No state >> backend has been configured, using default (Memory / JobManager) >> MemoryStateBackend (data in heap memory / checkpoints to JobManager) >> (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: >> 5242880) >> 2019-08-21 20:00:28,211 INFO >> org.apache.flink.yarn.YarnResourceManager - >> ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was >> granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 >> 2019-08-21 20:00:28,214 INFO >> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - >> Starting the SlotManager. >> 2019-08-21 20:00:28,225 INFO >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - >> Recovering checkpoints from ZooKeeper. >> 2019-08-21 20:00:28,229 INFO >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - >> Found 0 checkpoints in ZooKeeper. >> 2019-08-21 20:00:28,229 INFO >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - >> Trying to fetch 0 checkpoints from storage. >> 2019-08-21 20:00:28,233 INFO >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService - >> Starting ZooKeeperLeaderElectionService >> ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. >> 2019-08-21 20:00:28,248 INFO >> org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager >> runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was >> granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 at >> akka.tcp://flink@host53:59609/user/jobmanager_0. >> 2019-08-21 20:00:28,260 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. >> 2019-08-21 20:00:28,260 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Starting >> execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) under >> job master id a4576f0320d71bc112c96c94ebbd4991. >> 2019-08-21 20:00:28,261 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job >> StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state >> CREATED to RUNNING. >> 2019-08-21 20:00:28,266 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,280 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] >> 2019-08-21 20:00:28,287 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,288 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] >> 2019-08-21 20:00:28,288 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot >> serve slot request, no ResourceManager connected. Adding as pending request >> [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] >> 2019-08-21 20:00:28,289 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) >> (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from CREATED >> to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,291 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,292 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched >> from CREATED to SCHEDULED. >> 2019-08-21 20:00:28,295 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Connecting >> to ResourceManager akka.tcp://flink@host53 >> :59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) >> 2019-08-21 20:00:28,300 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - Resolved >> ResourceManager address, beginning registration >> 2019-08-21 20:00:28,300 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - >> Registration at ResourceManager attempt 1 (timeout=100ms) >> 2019-08-21 20:00:28,304 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - >> Starting ZooKeeperLeaderRetrievalService >> /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. >> 2019-08-21 20:00:28,304 INFO >> org.apache.flink.yarn.YarnResourceManager - Registering >> job manager [hidden email]://flink@host53:59609/user/jobmanager_0 >> for job e5ea83238d721e932b4645589660406c. >> 2019-08-21 20:00:28,310 INFO >> org.apache.flink.yarn.YarnResourceManager - Registered >> job manager [hidden email]://flink@host53:59609/user/jobmanager_0 >> for job e5ea83238d721e932b4645589660406c. >> 2019-08-21 20:00:28,313 INFO >> org.apache.flink.runtime.jobmaster.JobMaster - JobManager >> successfully registered at ResourceManager, leader id: >> 97537c0d61bbf091e37cb6c45b784568. >> 2019-08-21 20:00:28,314 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,315 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> faa6a521758e5be4b3dc1955ca0f51b4. >> 2019-08-21 20:00:28,316 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,316 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,316 INFO >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting >> new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. >> 2019-08-21 20:00:28,321 INFO >> org.apache.flink.yarn.YarnResourceManager - Requesting >> new TaskExecutor container with resources <memory:8192, vCores:2>. Number >> pending requests 1. >> 2019-08-21 20:00:28,322 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> a2b419aea4a373686eeed6c46163a528. >> 2019-08-21 20:00:28,323 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> cbc11737ce6729cfe45f24c43180b5c8. >> 2019-08-21 20:00:28,323 INFO >> org.apache.flink.yarn.YarnResourceManager - Requesting >> new TaskExecutor container with resources <memory:8192, vCores:2>. Number >> pending requests 2. >> 2019-08-21 20:00:28,323 INFO >> org.apache.flink.yarn.YarnResourceManager - Request >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job >> e5ea83238d721e932b4645589660406c with allocation id >> f6566567c173a55b2bc52b72bcb24ff8. >> 2019-08-21 20:00:31,597 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint >> triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> >> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, >> rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: >> (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state >> RUNNING but SCHEDULED instead. Aborting checkpoint. >> 2019-08-21 20:00:33,749 INFO >> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received >> new token for : host53:59938 >> 2019-08-21 20:00:33,749 INFO >> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received >> new token for : host52:33246 >> 2019-08-21 20:00:33,765 INFO >> org.apache.flink.yarn.YarnResourceManager - Received >> new container: container_1566383236573_0003_01_000002 - Remaining pending >> container requests: 2 >> 2019-08-21 20:00:33,765 INFO >> org.apache.flink.yarn.YarnResourceManager - Removing >> container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending >> container requests 1. >> 2019-08-21 20:00:34,009 INFO >> org.apache.flink.yarn.YarnResourceManager - Creating >> container launch context for TaskManagers >> 2019-08-21 20:00:34,010 INFO >> org.apache.flink.yarn.YarnResourceManager - Starting >> TaskManagers >> 2019-08-21 20:00:34,021 INFO >> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - >> Opening proxy : host53:59938 >> 2019-08-21 20:00:34,066 INFO >> org.apache.flink.yarn.YarnResourceManager - Received >> new container: container_1566383236573_0003_01_000003 - Remaining pending >> container requests: 1 >> 2019-08-21 20:00:34,066 INFO >> org.apache.flink.yarn.YarnResourceManager - Removing >> container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending >> container requests 0. >> 2019-08-21 20:00:34,088 INFO >> org.apache.flink.yarn.YarnResourceManager - Creating >> container launch context for TaskManagers >> 2019-08-21 20:00:34,089 INFO >> org.apache.flink.yarn.YarnResourceManager - Starting >> TaskManagers >> 2019-08-21 20:00:34,090 INFO >> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - >> Opening proxy : host52:33246 >> 2019-08-21 20:00:36,597 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint >> triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> >> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, >> rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: >> (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state >> RUNNING but SCHEDULED instead. Aborting checkpoint. >> 2019-08-21 20:00:37,856 INFO >> org.apache.flink.yarn.YarnResourceManager - Registering >> TaskManager with ResourceID container_1566383236573_0003_01_000002 >> (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager >> 2019-08-21 20:00:39,573 INFO >> org.apache.flink.yarn.YarnResourceManager - Registering >> TaskManager with ResourceID container_1566383236573_0003_01_000003 >> (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager >> 2019-08-21 20:00:39,659 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,659 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (attempt #0) to container_1566383236573_0003_01_000002 @ host53 >> (dataPort=52456) >> 2019-08-21 20:00:39,664 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,664 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (attempt #0) to container_1566383236573_0003_01_000002 @ host53 >> (dataPort=52456) >> 2019-08-21 20:00:39,665 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (attempt #0) to container_1566383236573_0003_01_000003 @ host52 >> (dataPort=39650) >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (attempt #0) to container_1566383236573_0003_01_000003 @ host52 >> (dataPort=39650) >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) >> (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,666 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: >> (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, >> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) >> -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) (attempt >> #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,670 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,670 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (1/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,671 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (2/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (3/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,672 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from >> SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,673 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Sink: Print to Std. Out (4/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,673 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,673 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (1/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (2/4) (attempt #0) to >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,675 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (3/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,676 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched >> from SCHEDULED to DEPLOYING. >> 2019-08-21 20:00:39,676 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying >> Map -> Sink: Print to Std. Out (4/4) (attempt #0) to >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) >> 2019-08-21 20:00:39,782 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,783 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,784 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,787 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,806 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (2/4) >> (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,809 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (1/4) >> (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,809 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> select: >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) >> (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,991 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,992 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: Print >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from >> DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,996 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:39,997 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) switched >> from DEPLOYING to RUNNING. >> 2019-08-21 20:00:40,005 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (4/4) >> (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:40,006 INFO >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) -> >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) (3/4) >> (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. >> 2019-08-21 20:00:41,605 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering >> checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. >> 2019-08-21 20:00:42,466 INFO >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed >> checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 >> ms). >> >> >> >> ------------------ 原始邮件 ------------------ >> 发件人: "Yun Tang"<[hidden email]>; >> 发送时间: 2019年8月22日(星期四) 中午11:08 >> 收件人: "user-zh"<[hidden email]>; >> >> 主题: Re: flink启动等待10分钟问题 >> >> >> >> Hi >> >> Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot >> request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request >> timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 >> >> 祝好 >> 唐云 >> ________________________________ >> From: 々守护々 <[hidden email]> >> Sent: Thursday, August 22, 2019 11:04 >> To: user-zh <[hidden email]> >> Subject: flink启动等待10分钟问题 >> >> 各位Flink社区大佬，您好！ >> 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： >> <property> >> <name>yarn.nodemanager.resource.memory-mb</name> >> <value>16384</value> >> </property> >> <property> >> <name>yarn.scheduler.minimum-allocation-mb</name> >> <value>1024</value> >> </property> >> <property> >> <name>yarn.scheduler.maximum-allocation-mb</name> >> <value>8192</value> >> </property> >> >> >> >> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster -p >> 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm >> bigdata6 ./flink_bigdata6.jar >> 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ >> > |
|
|
基本上你是卡在上传用户 jar 这一步了，提交任务到部署成功是一瞬间的
2019-08-22 11:38:02,185 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting application master application_1566383236573_0004 2019-08-22 11:38:02,226 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1566383236573_0004 中间隔开的时间 FLINK 干的事情就是上传用户依赖的 jar 包 Best, tison. 々守护々 <[hidden email]> 于2019年8月22日周四 上午11:49写道： > 这个是终端打印的日志,十分钟后才启动- Submitting application master > application_1566383236573_0004 > > > > 2019-08-22 11:28:21,766 WARN > org.apache.flink.yarn.AbstractYarnClusterDescriptor - This YARN > session requires 34816MB of memory in the cluster. There are currently only > 26624MB available. > The Flink YARN client will try to allocate the YARN session, but maybe not > all TaskManagers are connecting from the beginning because the resources > are currently not available in the cluster. The allocation might take more > time than usual because the Flink YARN client needs to wait until the > resources become available. > 2019-08-22 11:28:21,766 WARN > org.apache.flink.yarn.AbstractYarnClusterDescriptor - There is > not enough memory available in the YARN cluster. The TaskManager(s) require > 8192MB each. NodeManagers available: [14336, 12288] > After allocating the JobManager (2048MB) and (2/4) TaskManagers, the > following NodeManagers are available: [4096, 4096] > The Flink YARN client will try to allocate the YARN session, but maybe not > all TaskManagers are connecting from the beginning because the resources > are currently not available in the cluster. The allocation might take more > time than usual because the Flink YARN client needs to wait until the > resources become available. > 2019-08-22 11:28:21,766 WARN > org.apache.flink.yarn.AbstractYarnClusterDescriptor - There is > not enough memory available in the YARN cluster. The TaskManager(s) require > 8192MB each. NodeManagers available: [14336, 12288] > After allocating the JobManager (2048MB) and (3/4) TaskManagers, the > following NodeManagers are available: [4096, 4096] > The Flink YARN client will try to allocate the YARN session, but maybe not > all TaskManagers are connecting from the beginning because the resources > are currently not available in the cluster. The allocation might take more > time than usual because the Flink YARN client needs to wait until the > resources become available. > 2019-08-22 11:28:21,767 INFO > org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster > specification: ClusterSpecification{masterMemoryMB=2048, > taskManagerMemoryMB=8192, numberTaskManagers=4, slotsPerTaskManager=2} > 2019-08-22 11:28:22,317 WARN > org.apache.flink.yarn.AbstractYarnClusterDescriptor - The > configuration directory ('/usr/flink/flink-1.8.1/conf') contains both LOG4J > and Logback configuration files. Please delete or rename one of them. > 2019-08-22 11:38:02,185 INFO > org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting > application master application_1566383236573_0004 > 2019-08-22 11:38:02,226 INFO > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted > application application_1566383236573_0004 > 2019-08-22 11:38:02,226 INFO > org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for > the cluster to be allocated > 2019-08-22 11:38:02,228 INFO > org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying > cluster, current state ACCEPTED > 2019-08-22 11:38:07,244 INFO > org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN > application has been deployed successfully. > 2019-08-22 11:38:07,244 INFO > org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink > YARN client has been started in detached mode. In order to stop Flink on > YARN, use the following command or a YARN web interface to stop it: > yarn application -kill application_1566383236573_0004 > Please also note that the temporary files of the YARN session in the home > directory will not be removed. > Job has been submitted with JobID c16c4fc1556ccb2ceaaa2f0e8f32ec88 > > ------------------ 原始邮件 ------------------ > *发件人:* "Zili Chen"<[hidden email]>; > *发送时间:* 2019年8月22日(星期四) 中午11:37 > *收件人:* "user-zh"<[hidden email]>; > *主题:* Re: 回复： flink启动等待10分钟问题 > > 你说的【停在那儿了】是说 flink run 的终端输出不动了吗？你看一下这个终端输出里 YARN 是什么时候 accept 你的应用的，我怀疑是 > YARN 集群忙导致 10 分钟没响应。 > > Best, > tison. > > > Zili Chen <[hidden email]> 于2019年8月22日周四 上午11:35写道： > > > user-zh 不支持贴图，你用下第三方存储然后贴个链接吧，或者我记得可以传邮件附件 > > > > Best, > > tison. > > > > > > 々守护々 <[hidden email]> 于2019年8月22日周四 上午11:33写道： > > > >> 是这样的，在启动的是后客户端日志就停在那了，等待10分钟后才能生成yarn的application > >> <http://hadoop.changba.com/cluster/app/application_1566383236573_0003> > >> > >> > >> ------------------ 原始邮件 ------------------ > >> *发件人:* "Yun Tang"<[hidden email]>; > >> *发送时间:* 2019年8月22日(星期四) 中午11:23 > >> *收件人:* "user-zh"<[hidden email]>; > >> *主题:* Re: 回复： flink启动等待10分钟问题 > >> > >> 启动时间是 20:00:25，task都处于running甚至第一次checkpoint > >> completed的时间是20:00:42，一共才17秒，何来10分钟的问题？ > >> ________________________________ > >> From: 々守护々 <[hidden email]> > >> Sent: Thursday, August 22, 2019 11:18 > >> To: user-zh <[hidden email]> > >> Subject: 回复： flink启动等待10分钟问题 > >> > >> 您好，这个是我jobmanager启动日志，请帮忙看看，谢谢！ > >> > >> > >> 2019-08-21 20:00:25,428 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > >> > -------------------------------------------------------------------------------- > >> 2019-08-21 20:00:25,430 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > Registered > >> UNIX signal handlers for [TERM, HUP, INT] > >> 2019-08-21 20:00:25,433 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN > daemon > >> is running as: root Yarn client user obtainer: root > >> 2019-08-21 20:00:25,437 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: high-availability.cluster-id, > >> application_1566383236573_0003 > >> 2019-08-21 20:00:25,437 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: jobmanager.rpc.address, localhost > >> 2019-08-21 20:00:25,437 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: jobmanager.rpc.port, 6123 > >> 2019-08-21 20:00:25,437 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: high-availability.zookeeper.path.root, > /flink-yarn > >> 2019-08-21 20:00:25,437 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: high-availability.storageDir, > >> hdfs://host51:9000/flink/ha-yarn > >> 2019-08-21 20:00:25,438 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: high-availability.zookeeper.quorum, host51:2181 > >> 2019-08-21 20:00:25,438 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: internal.cluster.execution-mode, DETACHED > >> 2019-08-21 20:00:25,438 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: high-availability, zookeeper > >> 2019-08-21 20:00:25,438 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: parallelism.default, 1 > >> 2019-08-21 20:00:25,438 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: yarn.application-attempts, 10 > >> 2019-08-21 20:00:25,438 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: taskmanager.numberOfTaskSlots, 2 > >> 2019-08-21 20:00:25,439 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: taskmanager.heap.size, 8192m > >> 2019-08-21 20:00:25,439 INFO > >> org.apache.flink.configuration.GlobalConfiguration - Loading > >> configuration property: jobmanager.heap.size, 2048m > >> 2019-08-21 20:00:25,465 WARN > >> org.apache.flink.configuration.Configuration - Config > uses > >> deprecated configuration key 'web.port' instead of proper key > >> 'rest.bind-port' > >> 2019-08-21 20:00:25,469 INFO > >> org.apache.flink.runtime.clusterframework.BootstrapTools - Setting > >> directories for temporary files to: > >> > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003 > >> 2019-08-21 20:00:25,485 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting > >> YarnJobClusterEntrypoint. > >> 2019-08-21 20:00:25,485 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install > >> default filesystem. > >> 2019-08-21 20:00:25,561 INFO > >> org.apache.flink.runtime.security.modules.HadoopModule - Hadoop > user > >> set to root (auth:SIMPLE) > >> 2019-08-21 20:00:25,580 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - > >> Initializing cluster services. > >> 2019-08-21 20:00:25,757 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying > to > >> start actor system at host53:0 > >> 2019-08-21 20:00:26,187 INFO > >> akka.event.slf4j.Slf4jLogger - > Slf4jLogger > >> started > >> 2019-08-21 20:00:26,251 INFO > >> akka.remote.Remoting - Starting > >> remoting > >> 2019-08-21 20:00:26,439 INFO > >> akka.remote.Remoting - Remoting > >> started; listening on addresses :[akka.tcp://flink@host53:59609] > >> 2019-08-21 20:00:26,447 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor > >> system started at akka.tcp://flink@host53:59609 > >> 2019-08-21 20:00:27,011 INFO > >> org.apache.flink.runtime.blob.FileSystemBlobStore - Creating > >> highly available BLOB storage directory at > >> hdfs://host51:9000/flink/ha-yarn/application_1566383236573_0003/blob > >> 2019-08-21 20:00:27,097 INFO > >> org.apache.flink.runtime.util.ZooKeeperUtils - > Enforcing > >> default ACL for ZK connections > >> 2019-08-21 20:00:27,097 INFO > >> org.apache.flink.runtime.util.ZooKeeperUtils - Using > >> '/flink-yarn/application_1566383236573_0003' as Zookeeper namespace. > >> 2019-08-21 20:00:27,161 INFO > >> > org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl > >> - Starting > >> 2019-08-21 20:00:27,169 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> > environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, > >> built on 03/23/2017 10:13 GMT > >> 2019-08-21 20:00:27,169 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:host.name=host53 > >> 2019-08-21 20:00:27,169 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:java.version=1.8.0_77 > >> 2019-08-21 20:00:27,169 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:java.vendor=Oracle Corporation > >> 2019-08-21 20:00:27,169 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> > environment:java.library.path=:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/hadoop/hadoop-2.7.7/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:java.io.tmpdir=/tmp > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:java.compiler=<NA> > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:os.name=Linux > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:os.arch=amd64 > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:os.version=3.16.0-0.bpo.4-amd64 > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:user.name=root > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> environment:user.home=/root > >> 2019-08-21 20:00:27,170 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > Client > >> > environment:user.dir=/usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/container_1566383236573_0003_01_000001 > >> 2019-08-21 20:00:27,171 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - > >> Initiating client connection, connectString=host51:2181 > >> sessionTimeout=60000 > >> > watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@319854f0 > >> 2019-08-21 20:00:27,184 WARN > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > SASL > >> configuration failed: javax.security.auth.login.LoginException: No JAAS > >> configuration section named 'Client' was found in specified JAAS > >> configuration file: '/tmp/jaas-4274640356638794035.conf'. Will continue > >> connection to Zookeeper server without SASL authentication, if Zookeeper > >> server allows it. > >> 2019-08-21 20:00:27,186 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > >> Opening socket connection to server host51/10.10.10.51:2181 > >> 2019-08-21 20:00:27,187 ERROR > >> org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - > >> Authentication failed > >> 2019-08-21 20:00:27,187 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > Socket > >> connection established to host51/10.10.10.51:2181, initiating session > >> 2019-08-21 20:00:27,193 INFO > >> org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - > >> Session establishment complete on server host51/10.10.10.51:2181, > >> sessionid = 0x1001a86de810008, negotiated timeout = 40000 > >> 2019-08-21 20:00:27,193 INFO > >> org.apache.flink.runtime.blob.BlobServer - Created > >> BLOB server storage directory > >> > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-eaa2bb2b-cddb-42e3-8ce5-bf2f83e8567a > >> 2019-08-21 20:00:27,195 INFO > >> > org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager > >> - State change: CONNECTED > >> 2019-08-21 20:00:27,198 INFO > >> org.apache.flink.runtime.blob.BlobServer - Started > >> BLOB server at 0.0.0.0:58591 - max concurrent requests: 50 - max > >> backlog: 1000 > >> 2019-08-21 20:00:27,213 INFO > >> org.apache.flink.runtime.metrics.MetricRegistryImpl - No > metrics > >> reporter configured, no metrics will be exposed/reported. > >> 2019-08-21 20:00:27,215 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying > to > >> start actor system at host53:0 > >> 2019-08-21 20:00:27,239 INFO > >> akka.event.slf4j.Slf4jLogger - > Slf4jLogger > >> started > >> 2019-08-21 20:00:27,247 INFO > >> akka.remote.Remoting - Starting > >> remoting > >> 2019-08-21 20:00:27,255 INFO > >> akka.remote.Remoting - Remoting > >> started; listening on addresses :[akka.tcp://flink-metrics@host53 > :59941] > >> 2019-08-21 20:00:27,257 INFO > >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor > >> system started at akka.tcp://flink-metrics@host53:59941 > >> 2019-08-21 20:00:27,263 INFO > >> org.apache.flink.runtime.blob.TransientBlobCache - Created > >> BLOB cache storage directory > >> > /usr/hadoop/hadoop-2.7.7/tmp/nm-local-dir/usercache/root/appcache/application_1566383236573_0003/blobStore-405a4797-824f-48da-bb50-2ee2e230d249 > >> 2019-08-21 20:00:27,296 WARN > >> org.apache.flink.configuration.Configuration - Config > uses > >> deprecated configuration key 'web.port' instead of proper key > >> 'rest.bind-port' > >> 2019-08-21 20:00:27,298 WARN > >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload > >> directory > >> /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload > does > >> not exist, or has been deleted externally. Previously uploaded files > are no > >> longer available. > >> 2019-08-21 20:00:27,298 INFO > >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created > >> directory > >> /tmp/flink-web-57c9cddd-64af-4e3d-96a9-4f9e78d20b86/flink-web-upload for > >> file uploads. > >> 2019-08-21 20:00:27,320 INFO > >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - > Starting > >> rest endpoint. > >> 2019-08-21 20:00:27,616 INFO > >> org.apache.flink.runtime.webmonitor.WebMonitorUtils - > Determined > >> location of main cluster component log file: > >> > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.log > >> 2019-08-21 20:00:27,616 INFO > >> org.apache.flink.runtime.webmonitor.WebMonitorUtils - > Determined > >> location of main cluster component stdout file: > >> > /usr/hadoop/hadoop-2.7.7/logs/userlogs/application_1566383236573_0003/container_1566383236573_0003_01_000001/jobmanager.out > >> 2019-08-21 20:00:27,771 INFO > >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest > >> endpoint listening at host53:46401 > >> 2019-08-21 20:00:27,772 INFO > >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService > - > >> Starting ZooKeeperLeaderElectionService > >> ZooKeeperLeaderElectionService{leaderPath='/leader/rest_server_lock'}. > >> 2019-08-21 20:00:27,795 WARN > >> org.apache.flink.shaded.curator.org.apache.curator.utils.ZKPaths - The > >> version of ZooKeeper being used doesn't support Container nodes. > >> CreateMode.PERSISTENT will be used instead. > >> 2019-08-21 20:00:27,807 INFO > >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web > >> frontend listening at http://host53:46401. > >> 2019-08-21 20:00:27,836 INFO > >> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - > >> http://host53:46401 was granted leadership with > >> leaderSessionID=8487757f-fd82-42b2-a1a6-adcc69e1eba1 > >> 2019-08-21 20:00:27,932 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > >> RPC endpoint for org.apache.flink.yarn.YarnResourceManager at > >> akka://flink/user/resourcemanager . > >> 2019-08-21 20:00:28,021 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > >> RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at > >> akka://flink/user/dispatcher . > >> 2019-08-21 20:00:28,033 INFO > >> > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > >> Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > >> 2019-08-21 20:00:28,033 INFO > >> > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > >> Starting ZooKeeperLeaderRetrievalService /leader/dispatcher_lock. > >> 2019-08-21 20:00:28,033 INFO > >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService > - > >> Starting ZooKeeperLeaderElectionService > >> ZooKeeperLeaderElectionService{leaderPath='/leader/dispatcher_lock'}. > >> 2019-08-21 20:00:28,045 INFO > >> org.apache.flink.runtime.dispatcher.MiniDispatcher - > Dispatcher > >> akka.tcp://flink@host53:59609/user/dispatcher was granted leadership > >> with fencing token 6216d17e-56b1-43bb-9bb3-f9fa5b8921be > >> 2019-08-21 20:00:28,048 INFO > >> org.apache.flink.runtime.dispatcher.MiniDispatcher - > Recovering > >> all persisted jobs. > >> 2019-08-21 20:00:28,054 INFO > >> org.apache.hadoop.yarn.client.RMProxy - > Connecting > >> to ResourceManager at host51/10.10.10.51:8030 > >> 2019-08-21 20:00:28,078 INFO > >> org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting > >> RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at > >> akka://flink/user/jobmanager_0 . > >> 2019-08-21 20:00:28,086 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - > >> Initializing job StreamKafkaMain (e5ea83238d721e932b4645589660406c). > >> 2019-08-21 20:00:28,099 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - Using > >> restart strategy > >> FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, > >> delayBetweenRestartAttempts=0) for StreamKafkaMain > >> (e5ea83238d721e932b4645589660406c). > >> 2019-08-21 20:00:28,122 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > >> recovers via failover strategy: full graph restart > >> 2019-08-21 20:00:28,153 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - Running > >> initialization on master for job StreamKafkaMain > >> (e5ea83238d721e932b4645589660406c). > >> 2019-08-21 20:00:28,154 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - > >> Successfully ran initialization on master in 0 ms. > >> 2019-08-21 20:00:28,187 INFO > >> org.apache.flink.runtime.util.ZooKeeperUtils - > Initialized > >> ZooKeeperCompletedCheckpointStore in > >> '/checkpoints/e5ea83238d721e932b4645589660406c'. > >> 2019-08-21 20:00:28,196 INFO > >> org.apache.flink.yarn.YarnResourceManager - > Recovered 0 > >> containers from previous attempts ([]). > >> 2019-08-21 20:00:28,199 INFO > >> > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > >> yarn.client.max-cached-nodemanagers-proxies : 0 > >> 2019-08-21 20:00:28,200 INFO > >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService > - > >> Starting ZooKeeperLeaderElectionService > >> > ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}. > >> 2019-08-21 20:00:28,203 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - No state > >> backend has been configured, using default (Memory / JobManager) > >> MemoryStateBackend (data in heap memory / checkpoints to JobManager) > >> (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, > maxStateSize: > >> 5242880) > >> 2019-08-21 20:00:28,211 INFO > >> org.apache.flink.yarn.YarnResourceManager - > >> ResourceManager akka.tcp://flink@host53:59609/user/resourcemanager was > >> granted leadership with fencing token 97537c0d61bbf091e37cb6c45b784568 > >> 2019-08-21 20:00:28,214 INFO > >> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - > >> Starting the SlotManager. > >> 2019-08-21 20:00:28,225 INFO > >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > >> Recovering checkpoints from ZooKeeper. > >> 2019-08-21 20:00:28,229 INFO > >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > >> Found 0 checkpoints in ZooKeeper. > >> 2019-08-21 20:00:28,229 INFO > >> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - > >> Trying to fetch 0 checkpoints from storage. > >> 2019-08-21 20:00:28,233 INFO > >> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService > - > >> Starting ZooKeeperLeaderElectionService > >> > ZooKeeperLeaderElectionService{leaderPath='/leader/e5ea83238d721e932b4645589660406c/job_manager_lock'}. > >> 2019-08-21 20:00:28,248 INFO > >> org.apache.flink.runtime.jobmaster.JobManagerRunner - > JobManager > >> runner for job StreamKafkaMain (e5ea83238d721e932b4645589660406c) was > >> granted leadership with session id 12c96c94-ebbd-4991-a457-6f0320d71bc1 > at > >> akka.tcp://flink@host53:59609/user/jobmanager_0. > >> 2019-08-21 20:00:28,260 INFO > >> > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > >> Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock. > >> 2019-08-21 20:00:28,260 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - Starting > >> execution of job StreamKafkaMain (e5ea83238d721e932b4645589660406c) > under > >> job master id a4576f0320d71bc112c96c94ebbd4991. > >> 2019-08-21 20:00:28,261 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > >> StreamKafkaMain (e5ea83238d721e932b4645589660406c) switched from state > >> CREATED to RUNNING. > >> 2019-08-21 20:00:28,266 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (1/4) > >> (ee39866835f7b7387f518e352787a532) switched from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,280 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > >> serve slot request, no ResourceManager connected. Adding as pending > request > >> [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] > >> 2019-08-21 20:00:28,287 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (2/4) > >> (f1a893fc5e2440331c8e6db1874ad220) switched from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,288 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > >> serve slot request, no ResourceManager connected. Adding as pending > request > >> [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] > >> 2019-08-21 20:00:28,288 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (3/4) > >> (4756a0450881325af5fb396a18dbb8ae) switched from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,289 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > >> serve slot request, no ResourceManager connected. Adding as pending > request > >> [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] > >> 2019-08-21 20:00:28,289 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (4/4) > >> (78f520582607e26b365fca483fc98d4c) switched from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,289 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot > >> serve slot request, no ResourceManager connected. Adding as pending > request > >> [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] > >> 2019-08-21 20:00:28,289 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> > select: > >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > >> (0071e1c7c4f70646914329f81dcbd349) switched from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,291 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > CREATED > >> to SCHEDULED. > >> 2019-08-21 20:00:28,291 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > CREATED > >> to SCHEDULED. > >> 2019-08-21 20:00:28,291 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > CREATED > >> to SCHEDULED. > >> 2019-08-21 20:00:28,291 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > CREATED > >> to SCHEDULED. > >> 2019-08-21 20:00:28,291 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) > switched > >> from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,291 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) > switched > >> from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,291 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) > switched > >> from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,292 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) > switched > >> from CREATED to SCHEDULED. > >> 2019-08-21 20:00:28,295 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - > Connecting > >> to ResourceManager akka.tcp://flink@host53 > >> :59609/user/resourcemanager(97537c0d61bbf091e37cb6c45b784568) > >> 2019-08-21 20:00:28,300 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - Resolved > >> ResourceManager address, beginning registration > >> 2019-08-21 20:00:28,300 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - > >> Registration at ResourceManager attempt 1 (timeout=100ms) > >> 2019-08-21 20:00:28,304 INFO > >> > org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - > >> Starting ZooKeeperLeaderRetrievalService > >> /leader/e5ea83238d721e932b4645589660406c/job_manager_lock. > >> 2019-08-21 20:00:28,304 INFO > >> org.apache.flink.yarn.YarnResourceManager - > Registering > >> job manager [hidden email]://flink@host53 > :59609/user/jobmanager_0 > >> for job e5ea83238d721e932b4645589660406c. > >> 2019-08-21 20:00:28,310 INFO > >> org.apache.flink.yarn.YarnResourceManager - > Registered > >> job manager [hidden email]://flink@host53 > :59609/user/jobmanager_0 > >> for job e5ea83238d721e932b4645589660406c. > >> 2019-08-21 20:00:28,313 INFO > >> org.apache.flink.runtime.jobmaster.JobMaster - > JobManager > >> successfully registered at ResourceManager, leader id: > >> 97537c0d61bbf091e37cb6c45b784568. > >> 2019-08-21 20:00:28,314 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - > Requesting > >> new slot [SlotRequestId{13970fab9312acf9d5063b1d1a33aa0e}] and profile > >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > >> 2019-08-21 20:00:28,315 INFO > >> org.apache.flink.yarn.YarnResourceManager - Request > >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > >> e5ea83238d721e932b4645589660406c with allocation id > >> faa6a521758e5be4b3dc1955ca0f51b4. > >> 2019-08-21 20:00:28,316 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - > Requesting > >> new slot [SlotRequestId{fdbb2e71c9bc795452687396abff7bf8}] and profile > >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > >> 2019-08-21 20:00:28,316 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - > Requesting > >> new slot [SlotRequestId{0621470f796aa3ea4291d1b8b423e428}] and profile > >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > >> 2019-08-21 20:00:28,316 INFO > >> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - > Requesting > >> new slot [SlotRequestId{f6123820ffb8b3e906b876e9dea4497b}] and profile > >> ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, > >> nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. > >> 2019-08-21 20:00:28,321 INFO > >> org.apache.flink.yarn.YarnResourceManager - > Requesting > >> new TaskExecutor container with resources <memory:8192, vCores:2>. > Number > >> pending requests 1. > >> 2019-08-21 20:00:28,322 INFO > >> org.apache.flink.yarn.YarnResourceManager - Request > >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > >> e5ea83238d721e932b4645589660406c with allocation id > >> a2b419aea4a373686eeed6c46163a528. > >> 2019-08-21 20:00:28,323 INFO > >> org.apache.flink.yarn.YarnResourceManager - Request > >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > >> e5ea83238d721e932b4645589660406c with allocation id > >> cbc11737ce6729cfe45f24c43180b5c8. > >> 2019-08-21 20:00:28,323 INFO > >> org.apache.flink.yarn.YarnResourceManager - > Requesting > >> new TaskExecutor container with resources <memory:8192, vCores:2>. > Number > >> pending requests 2. > >> 2019-08-21 20:00:28,323 INFO > >> org.apache.flink.yarn.YarnResourceManager - Request > >> slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, > >> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job > >> e5ea83238d721e932b4645589660406c with allocation id > >> f6566567c173a55b2bc52b72bcb24ff8. > >> 2019-08-21 20:00:31,597 INFO > >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > Checkpoint > >> triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > >> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > >> rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > >> (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > >> RUNNING but SCHEDULED instead. Aborting checkpoint. > >> 2019-08-21 20:00:33,749 INFO > >> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > >> new token for : host53:59938 > >> 2019-08-21 20:00:33,749 INFO > >> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received > >> new token for : host52:33246 > >> 2019-08-21 20:00:33,765 INFO > >> org.apache.flink.yarn.YarnResourceManager - Received > >> new container: container_1566383236573_0003_01_000002 - Remaining > pending > >> container requests: 2 > >> 2019-08-21 20:00:33,765 INFO > >> org.apache.flink.yarn.YarnResourceManager - Removing > >> container request Capability[<memory:8192, vCores:2>]Priority[1]. > Pending > >> container requests 1. > >> 2019-08-21 20:00:34,009 INFO > >> org.apache.flink.yarn.YarnResourceManager - Creating > >> container launch context for TaskManagers > >> 2019-08-21 20:00:34,010 INFO > >> org.apache.flink.yarn.YarnResourceManager - Starting > >> TaskManagers > >> 2019-08-21 20:00:34,021 INFO > >> > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > >> Opening proxy : host53:59938 > >> 2019-08-21 20:00:34,066 INFO > >> org.apache.flink.yarn.YarnResourceManager - Received > >> new container: container_1566383236573_0003_01_000003 - Remaining > pending > >> container requests: 1 > >> 2019-08-21 20:00:34,066 INFO > >> org.apache.flink.yarn.YarnResourceManager - Removing > >> container request Capability[<memory:8192, vCores:2>]Priority[1]. > Pending > >> container requests 0. > >> 2019-08-21 20:00:34,088 INFO > >> org.apache.flink.yarn.YarnResourceManager - Creating > >> container launch context for TaskManagers > >> 2019-08-21 20:00:34,089 INFO > >> org.apache.flink.yarn.YarnResourceManager - Starting > >> TaskManagers > >> 2019-08-21 20:00:34,090 INFO > >> > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - > >> Opening proxy : host52:33246 > >> 2019-08-21 20:00:36,597 INFO > >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > Checkpoint > >> triggering task Source: Custom Source -> Flat Map -> Filter -> Filter -> > >> Map -> Timestamps/Watermarks -> from: (request, curuserid, timelong, > >> rowtime) -> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: > >> (rowtime) (1/4) of job e5ea83238d721e932b4645589660406c is not in state > >> RUNNING but SCHEDULED instead. Aborting checkpoint. > >> 2019-08-21 20:00:37,856 INFO > >> org.apache.flink.yarn.YarnResourceManager - > Registering > >> TaskManager with ResourceID container_1566383236573_0003_01_000002 > >> (akka.tcp://flink@host53:34608/user/taskmanager_0) at ResourceManager > >> 2019-08-21 20:00:39,573 INFO > >> org.apache.flink.yarn.YarnResourceManager - > Registering > >> TaskManager with ResourceID container_1566383236573_0003_01_000003 > >> (akka.tcp://flink@host52:32931/user/taskmanager_0) at ResourceManager > >> 2019-08-21 20:00:39,659 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (1/4) > >> (ee39866835f7b7387f518e352787a532) switched from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,659 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (1/4) > >> (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > >> (dataPort=52456) > >> 2019-08-21 20:00:39,664 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (2/4) > >> (f1a893fc5e2440331c8e6db1874ad220) switched from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,664 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (2/4) > >> (attempt #0) to container_1566383236573_0003_01_000002 @ host53 > >> (dataPort=52456) > >> 2019-08-21 20:00:39,665 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (3/4) > >> (4756a0450881325af5fb396a18dbb8ae) switched from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,666 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (3/4) > >> (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > >> (dataPort=39650) > >> 2019-08-21 20:00:39,666 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (4/4) > >> (78f520582607e26b365fca483fc98d4c) switched from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,666 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Source: Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (4/4) > >> (attempt #0) to container_1566383236573_0003_01_000003 @ host52 > >> (dataPort=39650) > >> 2019-08-21 20:00:39,666 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> > select: > >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > >> (0071e1c7c4f70646914329f81dcbd349) switched from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,666 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> window: (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: > >> (COUNT(*) AS pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, > >> end('w$) AS w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS > w$proctime) > >> -> select: (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > (attempt > >> #0) to container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > >> 2019-08-21 20:00:39,670 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > >> SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,670 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Sink: Print to Std. Out (1/4) (attempt #0) to > >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > >> 2019-08-21 20:00:39,671 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > >> SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,672 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Sink: Print to Std. Out (2/4) (attempt #0) to > >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > >> 2019-08-21 20:00:39,672 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > >> SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,672 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Sink: Print to Std. Out (3/4) (attempt #0) to > >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > >> 2019-08-21 20:00:39,672 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > >> SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,673 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Sink: Print to Std. Out (4/4) (attempt #0) to > >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > >> 2019-08-21 20:00:39,673 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) > switched > >> from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,673 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Map -> Sink: Print to Std. Out (1/4) (attempt #0) to > >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > >> 2019-08-21 20:00:39,675 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) > switched > >> from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,675 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Map -> Sink: Print to Std. Out (2/4) (attempt #0) to > >> container_1566383236573_0003_01_000003 @ host52 (dataPort=39650) > >> 2019-08-21 20:00:39,675 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) > switched > >> from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,675 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Map -> Sink: Print to Std. Out (3/4) (attempt #0) to > >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > >> 2019-08-21 20:00:39,676 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) > switched > >> from SCHEDULED to DEPLOYING. > >> 2019-08-21 20:00:39,676 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - > Deploying > >> Map -> Sink: Print to Std. Out (4/4) (attempt #0) to > >> container_1566383236573_0003_01_000002 @ host53 (dataPort=52456) > >> 2019-08-21 20:00:39,782 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (4/4) (0e8c4210bd81e581fbf354b86397ce41) switched from > >> DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,783 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (3/4) (9543bbde6538ea9d407af3012fede5c8) switched from > >> DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,784 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (4/4) (9aa5bb28f3ba75afc1de47bd9a4e5d01) > switched > >> from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,787 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (3/4) (f109918ce195d1c88c2c63a44934217f) > switched > >> from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,806 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (2/4) > >> (f1a893fc5e2440331c8e6db1874ad220) switched from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,809 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (1/4) > >> (ee39866835f7b7387f518e352787a532) switched from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,809 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - window: > >> (TumblingGroupWindow('w$, 'rowtime, 60000.millis)), select: (COUNT(*) AS > >> pv, COUNT(DISTINCT curuserid) AS uv, start('w$) AS w$start, end('w$) AS > >> w$end, rowtime('w$) AS w$rowtime, proctime('w$) AS w$proctime) -> > select: > >> (pv, uv, utc2local(w$end) AS EXPR$2) -> to: Row (1/1) > >> (0071e1c7c4f70646914329f81dcbd349) switched from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,991 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (1/4) (0a206444b10553cce42fa34ea0b15497) switched from > >> DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,992 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: > >> to Std. Out (2/4) (a5d1bb759ee6c8001313fe39c7982a8b) switched from > >> DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,996 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (1/4) (39d437bceb033f5bd37747e9142c7f0e) > switched > >> from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:39,997 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> > >> Sink: Print to Std. Out (2/4) (aa4f64edc4f10da2e8bdbf6ebf0aeb83) > switched > >> from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:40,005 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (4/4) > >> (78f520582607e26b365fca483fc98d4c) switched from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:40,006 INFO > >> org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: > >> Custom Source -> Flat Map -> Filter -> Filter -> Map -> > >> Timestamps/Watermarks -> from: (request, curuserid, timelong, rowtime) > -> > >> select: (rowtime, 0 AS $f1, curuserid) -> time attribute: (rowtime) > (3/4) > >> (4756a0450881325af5fb396a18dbb8ae) switched from DEPLOYING to RUNNING. > >> 2019-08-21 20:00:41,605 INFO > >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > Triggering > >> checkpoint 1 @ 1566388841597 for job e5ea83238d721e932b4645589660406c. > >> 2019-08-21 20:00:42,466 INFO > >> org.apache.flink.runtime.checkpoint.CheckpointCoordinator - > Completed > >> checkpoint 1 for job e5ea83238d721e932b4645589660406c (7994 bytes in 821 > >> ms). > >> > >> > >> > >> ------------------ 原始邮件 ------------------ > >> 发件人: "Yun Tang"<[hidden email]>; > >> 发送时间: 2019年8月22日(星期四) 中午11:08 > >> 收件人: "user-zh"<[hidden email]>; > >> > >> 主题: Re: flink启动等待10分钟问题 > >> > >> > >> > >> Hi > >> > >> Flink on YARN作业启动时间长，有很多原因，例如资源不够在等待，container申请的时候又退出了。默认的slot > >> request的timeout时间是5min，感觉你的作业应该是可能遇到了一个slot request > >> timeout，然后又重新申请。最好能提供一下jobmanager的日志才好进一步分析。 > >> > >> 祝好 > >> 唐云 > >> ________________________________ > >> From: 々守护々 <[hidden email]> > >> Sent: Thursday, August 22, 2019 11:04 > >> To: user-zh <[hidden email]> > >> Subject: flink启动等待10分钟问题 > >> > >> 各位Flink社区大佬，您好！ > >> 我使用Flink SQL (Flink 1.8.1),在hadoop 集群中调整了一下yarn-site.xml文件的内存参数： > >> <property> > >> <name>yarn.nodemanager.resource.memory-mb</name> > >> <value>16384</value> > >> </property> > >> <property> > >> <name>yarn.scheduler.minimum-allocation-mb</name> > >> <value>1024</value> > >> </property> > >> <property> > >> <name>yarn.scheduler.maximum-allocation-mb</name> > >> <value>8192</value> > >> </property> > >> > >> > >> > >> 然后我去启动flink on yarn任务，启动参数：./flink-1.8.1/bin/flink run -m yarn-cluster > -p > >> 4 -d -c streaming.StreamKafkaMain -yn 4 -ys 2 -yjm 2048 -ytm 8192 -ynm > >> bigdata6 ./flink_bigdata6.jar > >> 最终能能启动成功，就是启动要等待10分钟，这个等待时间太长了，请问各位大佬有什么解决的方法吗？ > >> > > > > |
«
Return to Apache Flink 中文用户邮件列表
|
1 view|%1 views
| Free forum by Nabble | Edit this page |