Hadoop Admin Roles and Responsibilities:
Hadoop Administrator career is an excellent career and lot of growth opportunities because less amount of people and Hadoop is huge demand technology.
Hadoop Administrator is responsible for Hadoop Install and monitoring Cluster Management.
Roles and Responsibilities:
- Capacity Planning and Hardware requirement of the nodes, Network architecture and Planning.
- Hadoop Software Installation and configuration whether Cloudera Distribution or Horton Works distribution etc.
- Configuring Name Node, Data Nodes to ensure its high availability.
- Tuning of Hadoop Cluster and creating new users in Hadoop, handling permissions, performance upgrades.
- Hadoop Backup and Recovery tasks
- Every day finding out which jobs are taking more time, if users say that jobs are stuck to find out the reason.
- Health check of Hadoop cluster Monitoring
- Deployment in Hadoop Cluster and maintaining it.
- Support and maintenace of Hadoop Storage (HDFS)
- Security administration during installation and basic knowledge on Kerberos, Apache Knoz and Apache Ranger etc.
- Data migration between clusters if needed ex: using Falcon tool.
- Manage Hadoop Log files and analyzing failed jobs
- Troubleshoot Network and applications
- Knowledge on Scripting Skills on Linux environment
- Knowledge on Oozie, Hive , HCatalog and Hadoop Eco – System
Day to Day Activities of Hadoop Admin:
- Monitoring Console whether Cloudera Manager or Horton works and job tracker UI.
- HDFS Maintenance and Support
- Health check of Hadoop cluster monitoring
- Managing Hadoop log files and find out errors
- Managing users, permissions etc.
- Troubleshoot Network errors and application errors.
Skill sets required to become a Hadoop Administrator :
- Strong Knowledge on Linux/Unix
- Knowledge on Shell Scripting/Python Scripting
- Hands on Experience of Cluster Monitoring tools like Ambari, Gangila etc.
- Networking and Memory management
Summary: Hadoop Administration is one of the best careers in terms of growth and opportunities. Nowadays the Hadoop market is on rising. If you have knowledge on Linux and Database then admin it can be an advantage.