How to Install Kakfa in Linux/Ubuntu (Single node cluster) with Pictures

Apache Kafka is one of the distributed messaging systems. Here is step by step processing to install of Apache Kafka in Linux/Ubuntu operating system.



Prerequisites:

To install Kafka required Zookeeper and java to run. Mandatory for JDK 1.7 or above version for Kafka installation using below commands:

$ sudo apt update
$ sudo apt install default - jdk

Step 1: Download the Kafka binary files from official website like Download from apache website.

https://archive.apache.org/dist/kafka/

Step 2:  Extract the tarball using the below command:

$ tar -xzvf kafka_2.12.-0.10.1.1tgz


Step 3: After extraction, we see Kafka directory

Step 4: Update the KAFKA_HOME & PATH variable in bashrc file

 export KAFKA_HOME = /home/your_path/INSTALL/kafka_2.12-0.10.1.1
 export PATH=$PATH:$KAFKA_HOME/bin


Step 5: Ater bashrc changes, open a new terminal and check the bashrc changes using below command:

$ echo $KAFKA_HOME






After installing Apache Kafka on Linux/Ubuntu Start Kafka Server. Before start, the Kafka server, start Zookeeper server on your single node cluster using below commands:

$ cd/usr/local/kafka
$ bin/zookeeper-server-start.sh config/zookeeper.properties

After the start, the Zookeeper server then start the Kafka server

$ bin/kafka-server-start.sh config/server.properties

After starting Kafka server then create topics after that will go with message passing from producer to consumer. Above steps Kafka installation for single/pseudo cluster setup on top of the Hadoop ecosystem.

Summary: Apache Kafka installation in Linux/Ubuntu operating system it is very simple and uses it. If you need installation in Clouder to need to download separately bundle in Cloudera manager to set up in multi-node cluster setup. In Hortonworks need to the installation of Kafka in Ambari.

 

Java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver in Spark Scala

While writing Apache Spark in Scala / Python (PySpark) programming language to read data from Oracle Data Base using Scala / Python in Linux operating system/ Amazon Web Services, sometimes will get below error in




spark.driver.extraClassPath in either executor class or driver class.

Caused by: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver

at scala.tools.nsc.interpreter.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:83)

at Java .lang.ClassLoader.loadClass(ClassLoader.java.424)

at Java.lang.ClassLoader.loadClass(ClassLoader.java.357)

at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:35)

at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$anofun$createConnectionFactory$1.api

at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$anofun$createConnectionFactory$1.api

at scala.Option.foreach ( Option .scla:236)

at org . apache . spark . sql . execution . datasources . jdbc.JdbcUtils $ anofun $ createConnection Factory $ (JdbcUtils.scala)

at <init> ( < console >:46)

at . <init> (<console>:52)

at. <clinit> (<console>)

at. <init> (<console>:7)

at. <clinit> (<console>)

at $print (<console>)

Solution:

After getting this error will provide a simple solution in Spark Scala. Sometimes these are coming to Python (PySpark).

import related jars to both executor class and driver class. First, we need to edit the configuration file as spark defaults in spark-default.conf file.

Adding below two jar files path in spark-default.conf file.

spark.driver.extraClassPath /home/hadoop/ojdbc7.jar
spark.executor.extraClassPath /home/hadoop/ojdbc7.jar

Above two jar files path in configurations with exact version is matched to your Spark version otherwise will get the compatible issue.

Sometimes these two classpaths get an error then will add in your code either Scala or Pyspark programming –conf before Spark driver, executor jar files in bottom of the page example.

If will get the same issue again then will follow the below solution:
Step 1: Download Spark ODBC jar files from the official Maven website.

Step 2: Copy the download jar files into the below path in the share location in Spark.

/usr/lib/spark/jars

For Example –  PySpark programming code snippet for more information.

 

pyspark --driver-classpath /home/hadoop/odbc7.jar --jars #  jar file path
from pyspark import SparkContext, Spark conf # import Spark Context and configuration

from pyspark.sql import SparkContext #Sql context

sqlContext = sqlContext (sc)

dbconnection = sqlContext . read . format ("jdbc") . options (url: "Give your jdbc connection url path").load()
#Data Base connection to read data from Spark with the help of jdbc

 

Sample Talend Questions – MCQs





1. Which of the following in the design workspace indicates an error with a component in Talend

A) A red ‘X’                                                                      B) A red exclamation point

C)A green ‘ I’                                                                    D)A yellow  exclamation point

2. Which of the following components can be used to implement lookup in Talend

A)tJoin                                                                         B) tLookUp

C)tMap                                                                        D)tUnite

3.tMap offers following match modes for a lookup in Talend studio

A)Unique match                                             B)Only unique match

C)First match                                                   D)All matches

4.tMap offers following join model in Talend tool

A)Left Outer Join                                     B)Right Outer Join

C)Inner Join                                                D)Full Outer Join

5. Which of the following the components is used to execute a job infinite times in Talend

A)tInfiniteLoop                                          B) tFileWatcher

C) tForEach                                                  D)tRunJon

6.How to access parameters in the Global Map in Talend ETL tool

A)globalMap.put(“Key”, Object)

B)globalMap.get(“Key”, Object)

C)globalMap.put(“key”)

D)globalMap.get(“key”)

7. How do you reference the value of Context Variable FileName in configuration while Talend programming

A)Context.FileName                                B)context.FileName

C)FileName.value                                     D)$context.FileName

8. While Installing your Talend solutions, you have to set the following variable is mandatory?

A)JAVA_HOME                           B)TALEND_HOME

C)TIS_HOME                                 D)JRE_HOME

9. What is the use of a tReplicate component? Choose one best answer?

A)To duplicate the configuration of an existing component

B)To copy the input row to an output row without processing it

C)To duplicate a sub job

D)To send duplicates of an output row to multiple target components

10. How do you see the configuration of an error message for a component in Talend studio?

A)Right-click the component and then click show problem

B)From the errors view

C)Place the mouse pointer over error symbol in the design workspace

D)From the problems view

11. How do you create a row between two components in Talend

A) Drag the target component to source component

B)Right-click the source component click Row followed by the row type and then click the target component

C)Drag the source component onto target components

D)Right-click the source component and then click the target component

12. How do you ensure that a subjob completes before a second subjob runs in Talend?

A)Using RunIf trigger

B)Using the main connection

C)Using onComponentOk or OnComponentError trigger

D)Using onSubJobOk or onSubJobError trigger

13. Which of the following the components will be used to load JSON file to MySQL database in Talend?

A)tMySQLInput

B)tFileInputJSON

C)tMySQLOutput

D)tMap

14. How do you run a job in Talend Studio?

A)Click the Run button in the Run view

B)Click the Run button in the Job view

C)Click the Run button in the File Menu

D)Click the Start button in the Run view
15. What is the best practice for arranging components on the design workspace in Talend studio?

A)Bottom to Top

B)Right to Left

C)Top to Bottom

D)Matching the flow of data

16. From which tab in component view would you specify the component label in Talend

A)View

B)Advanced settings

C)Basic settings

D)Documentation

17. How to place your component in a job in Talend Studio?

A) Click it on Edit Menu

B) Click it in the Repository and then click in the design workspace

C) Click it from Repository to the design workspace

D)Click it in the Palette and then click in the design workspace

7 Best Programming Languages for Mobile Apps




The Best Programming languages for Developing Mobile Applications:

Nowadays, mobile applications are the most necessary part of our lifestyle. Smartphones with applications are essential and mandatory for information and needy in real life with a huge demand for mobile applications to provide work opportunity for any programmer to divide into mobile application developing. We will be mentioning below some of the programming languages are mandatory to learn and earn money from mobile apps.

Majorly two types of an operating system in a smartphone.

One is Android and another one is iOS.

Below programming languages are the best languages for developing mobile applications both Android and iOS.

1. Java

Java is one of the best programming languages for developing mobile applications. Simple to learn and develop mobile apps bit of easy. It is an official language for developing Andriod apps but still huge demand in the present market. Android and Eclipse is major role while developing mobile apps in Java with OOPS concept. Java has a different type of frameworks to developing and it is emerging technology in the market.

2.Python:

Python is a most trending programming language and fastest growing in the present market for all applications not only mobile application. Python is very simple to learn and develop the application. It has an interactive language for supporting the OOPS concept.

3.JavaScript:

Java and JavaScript are also different programming languages while developing mobile applications. It is high demand programming language nowadays. Purely scripting language for developing mobile applications using NodeJS especially for apps.

4. Swift:

Swift is a programming language for developing a mobile application developed by Apple in 2014. It is used for only iOS and MacOS based on Objective C for developing.

5.C++:


C++ is a programming language that is object-oriented based on C programming language. It is used for mobile application in a simple manner. Nowadays Amazon, Google, and Microsoft used for mobile application tools.

6.KOTLIN






Kotlin is one of the most trending programming languages for developing a mobile application. Founded by JetBrains for android developing. It is also from the Android studio. Kotlin is simple to learn and develop mobile apps.

7.Objective – C:

Objective – C was a very popular programming language among Apple developers before swift came into the market. Many developers are still using Objective C for iOS development.

Top 7 Easiest Programming Languages to Learn in 2019





In present, the IT market needs the most talented programming language, skilled employees. It tells that which programming language is best for the future.

Now, here are the Top 7 best and easiest programming languages you should try to learn.

1.Python:

At present situation the top easiest programming language is Python. You will think about why? the latest reports that Python showed 456 – percent growth in last year. Coming to business analysis IBM, Netflix uses Python. It is considered for deployment automation and web development mostly. Python is among the easiest programming language to learn and deploy. Majorly two reasons it is the easiest language.

The first reason is the uses very few lines to code for complex code. Secondly, it is a scripting language. So these two reasons are the Python is the easiest language to learn in 2019.

2.JavaScript:

JavaScript is one of the languages are on highly demanding language. JavaScript is a pure scripting language to build web applications anyway of its complexity. Nowadays it is used as Front-end and Back-end of websites. It is also learning simply in 2019 for future scope.

Note: JavaScript is not related to Java. Java is a purely programming language but JavaScript is a Scripting language.

3.Java

Java is among the simplest programming language. You must and should learn because it is highly portable and run anywhere(Platform independent). Java is existence from many years for web applications and Android applications. There is a high demand for Java developers in the present market.

4.Ruby:

Ruby is a friendly scripting language compare with all scripting language. It is also used for web applications and android application development. Easily readable language and similar to Python so it is simple to learn in 2019.

5. Go




Go is a Google open source programming language and limited structured typing. Go is mostly used for network applications and web servers.

6. C and C++:

C and C++ languages are great to learn because it is impacted by many other languages. Easily learn Java also because it logically jumps simply. It is used for software development, web applications. It is also learning most preferable for 2019.

7.C#

C# is a good choice for whoever coming to IT sector like freshers to be simple startup time. It is used for web development to console applications. This syntax is also the same as C and C++. Little bit complex compiler.

How to Install Hadoop Single Node Cluster

How to Install Hadoop Single Node Cluster on Ubuntu.

Step 1: Update the “System Software Repositories” using sudo apt-get update

First step update the packages of Ubuntu

Step 2: JAVA 1.8 JRE INSTALLATION using below command.

JAVA is prerequisite for Installation so first install JRE then will go with JDK

 

Step 3: JAVA 1.8 JDK INSTALL using below command

 

Step 4: How to check JAVA version on Linux using below command

Step 5: After that We must and should Install SSH(Secure Shell) using below command:

SSH for secure less communication in name node and secondary name node for frequently communication

Step 6: Check  SSH Installation using below command

After installation of SSH will check using ssh localhost command whether the communication is working or not.

Step 7: Download Hadoop-2.6.0 tarball from Apache Mirrors.

After completion of Hadoop prerequisites then download the hadoop tarball

Step 8: Extract the tar ball using below command

 

Step 9: Update Environment variables and Path for HADOOP_HOME and JAVA_HOME:

 

 

Step 10: To check Path variable is there or not after that edit the Configuration files as part of Hadoop Installation.

 

 

Step 11: First open “Core-site.xml” file, add the properties

Core-site file for Name node information

Step 12: Open “hdfs-site.xml” file and add the properties

Hdfs site xml file related to replication factor and data node information.

Step 13: Open “yarn-site.xml” file and add the properties to configure ‘Resource Manager’ & ‘Node Manage’ details:

Step 14: Update JAVA_HOME path in ‘ hadoop-env.sh’ file

Step 15:Update JAVA_HOME path in ‘ mapred-env.sh‘ file

Step 16: Open ‘mapred-site.xml‘ and update the yarn into that file

Step 17: Open slaves file and check whether the hostname is localhost or not

Step 18: Before starting Name Node, we must and should format the name node using below command: hadoop namenode -format

Step 19: To start all the daemons of hadoop using below command:

start-all.sh

Step 20: How to check daemons whether work or not using jps command


Step 21: After that all to access the Name Node information in GUI:

http://localhost:50070