In this article, we will explain how to resolve the the CREDENTIAL SERVICE unstable in the Azure Hadoop clusters.
ERROR:

stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/CREDENTIAL_SERVICE/0.1/package/scripts/credential_server.py", line 125, in <module>
CredentialServerInstall().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/CREDENTIAL_SERVICE/0.1/package/scripts/credential_server.py", line 63, in start
Execute(kinit_if_needed, user = params.cs_user)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -kt /usr/lib/ -credentialservice/conf/hcs.service.keytab hcs/ <hostname>. <domain name>.com@ .GO. <domain name>.COM;' returned 1. kinit: Client 'hcs/ <hostname>. <domain name>.com@ .GO. <domain name>.COM' not found in Kerberos database while getting initial credentials
stdout:
2022-11-15 06:36:57,518 - Stack Feature Version Info: Cluster Stack=4.1, Command Stack=None, Command Version=4.1.7.7 -> 4.1.7.7
2022-11-15 06:36:57,531 - Using hadoop conf dir: /usr/hdp/4.1.7.7/hadoop/conf
2022-11-15 06:36:57,714 - Stack Feature Version Info: Cluster Stack=4.1, Command Stack=None, Command Version=4.1.7.7 -> 4.1.7.7
2022-11-15 06:36:57,718 - Using hadoop conf dir: /usr/hdp/4.1.7.7/hadoop/conf
2022-11-15 06:36:57,719 - Skipping creation of User and Group as host is sys prepped or ignore_groupsusers_create flag is on
2022-11-15 06:36:57,719 - Skipping setting dfs cluster admin and tez view acls as host is sys prepped
2022-11-15 06:36:57,719 - FS Type: HDFS
2022-11-15 06:36:57,719 - Directory['/etc/hadoop'] {'mode': 0755}
2022-11-15 06:36:57,726 - File['/usr/hdp/4.1.7.7/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2022-11-15 06:36:57,727 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2022-11-15 06:36:57,739 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2022-11-15 06:36:57,746 - Skipping Execute[('setenforce', '0')] due to not_if
2022-11-15 06:36:57,746 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2022-11-15 06:36:57,748 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2022-11-15 06:36:57,748 - Changing owner for /var/run/hadoop from 202 to root
2022-11-15 06:36:57,748 - Changing group for /var/run/hadoop from 1003 to root
2022-11-15 06:36:57,748 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2022-11-15 06:36:57,748 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2022-11-15 06:36:57,751 - File['/usr/hdp/4.1.7.7/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2022-11-15 06:36:57,752 - File['/usr/hdp/4.1.7.7/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2022-11-15 06:36:57,755 - File['/usr/hdp/4.1.7.7/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
Skipping copying of fast-hdfs-resource.jar as host is sys prepped
2022-11-15 06:36:57,763 - File['/usr/hdp/4.1.7.7/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2022-11-15 06:36:57,763 - File['/usr/hdp/4.1.7.7/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2022-11-15 06:36:57,764 - File['/usr/hdp/4.1.7.7/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2022-11-15 06:36:57,766 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2022-11-15 06:36:57,770 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2022-11-15 06:36:57,775 - Skipping unlimited key JCE policy check and setup since the host is sys prepped
2022-11-15 06:36:57,781 - Skipping stack-select on CREDENTIAL_SERVICE because it does not exist in the stack-select package structure.
2022-11-15 06:36:58,051 - Log folder for current script is not defined
2022-11-15 06:36:58,052 - Directory['/etc/ -credentialservice/conf'] {'owner': 'hcs', 'group': 'hadoop', 'create_parents': True}
2022-11-15 06:36:58,054 - XmlConfig['cs-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/ -credentialservice/conf', 'mode': 0664, 'configuration_attributes': [EMPTY], 'owner': 'hcs', 'configurations': ...}
2022-11-15 06:36:58,062 - Generating config: /etc/ -credentialservice/conf/cs-site.xml
2022-11-15 06:36:58,062 - File['/etc/ -credentialservice/conf/cs-site.xml'] {'owner': 'hcs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'}
2022-11-15 06:36:58,072 - Execute['/usr/bin/kinit -kt /usr/lib/ -credentialservice/conf/hcs.service.keytab hcs/ <hostname>. <domain name>.com@ .GO. <domain name>.COM;'] {'user': 'hcs'}
2022-11-15 06:36:58,117 - Skipping stack-select on CREDENTIAL_SERVICE because it does not exist in the stack-select package structure.

 Command failed after 1 tries
Solution

Here we provided simple resolution for CREDENTIAL_SERVICE unstable in the Azure clusters.
Step1 : Login the Ambari with Admin priviliges
Step 2: Click on CREDENTIAL_SERVICE, stop the service first after that start the service.
Step 3: Restart the Ambari agents as well.
Tag: CREDENTIAL SERVICE

[Resolved] How to resolve the CREDENTIAL SERVICE unstable in the Azure Hadoop clusters | Big Data | Hadoop | Azure

ERROR:

Solution
