Spark MLlib Interview Questions

StringIndexer is used when a label does not have an integer value. Models generally prefer prediction columns to be of integer type rather than string type. It is similar to the LabelEncoder of sklearn.

Regression is performed when we are predicting a value. For example, consider a scenario wherein you want to predict the number of jumps given and the number of steps a person has to follow. In this case, we will use regression. We will use particular steps as a feature and the number of jumps will be used as output.

A local vector contains both integer-type and 0-based indices. It also contains double-typed values, which are stored on a single machine. In MLlib, two types of local vectors are supported, namely, Dense and Sparse vectors. A sparse vector is one in which most of the entries are zero.

It is used to set configuration and the parameters while submitting a Spark job. These parameters include variables such as the Spark cluster’s IP address, the Spark executor’s memory and the number of cores to be used.

• Easy-to use library
• Can be used with huge amounts of data points
• Suitable for working with SQL queries

K-means clustering clusters around the centroid, that is, it splits the data points from the start into k clusters. On the contrary, the bisecting-k means algorithm splits the data points into sub-clusters.

Normal distribution, uniform distribution and gamma distribution are some of the distribution generating functions that are supported by PySpark mllib.

• Logistic Regression with SGD
• SVM with SGD

Yes, Spark supports SVM with SGD. It is a stochastic gradient descent optimiser that is used to optimise a model for a given data set. It is an iterative method.

• Algorithm
• Transformation
• Pipeline
• Persistence
• Utilities.