Hadoop is Java Based Open Source framework used for Storing and Processing large data sets in a distributed computing environment (Eg. AWS Cloud). Core of Hadoop is HDFS and Map Reduce. HDFS is Hadoop Distributed File System used for Storing large data sets in cloud. Map Reduce is for processing large data sets in cloud.
Hadoop is the solution which was used to overcome the challenges faced by big data. As we know, Big data is nothing but massive amount of data which is being generated every second. This data is huge in volume and thereby we cannot store this huge data in one single storage area. We cannot process big data using traditional ways as there are different types of data and the processing time is more.
Using Hadoop was the solution. Hadoop is a framework that manages big data storage in a distributed way and processes it parallelly.
Hadoop is the framework that has the storage and the processing unit. The storage unit of Hadoop is called HDFS - Hadoop Distributed File System. The processing unit is called MapReduce. Hadoop Distributed File System (HDFS) is specially designed for storing huge datasets in commodity hardware. HDFS is the storage unit of Hadoop and it helps Hadoop store Big data in an efficient way by distributing the data amongst many individual databases.
Hadoop is a framework, while HDFS is a file system. Hadoop framework stores files in the Hadoop Distributes File System. Hadoop is the framework written in java whereas HDFS is one of the most important unit of it. Hadoop has two important units: storage unit and processing unit. Storage unit is the HDFS one which stores the data in distributed manner over different nodes. YARN is the processing unit of Hadoop which process the data available in the HDFS. As HDFS is the storage unit of the Hadoop framework, it is hard to compare this two.
HDFS + MAP Reduce= HADOOP
HDFS- Storage component.
Map Reduce- Processing component.
Hadoop is an Apache open source framework written in java that allows distributed processing of large datasets across clusters of computers using simple programming models. The Hadoop framework application works in an environment that provides distributed storage and computation across clusters of computers. Hadoop is designed to scale up from single server to thousands of machines, each offering local computation and storage.
The Hadoop Distributed File System (HDFS) is based on the Google File System (GFS) and provides a distributed file system that is designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. It is highly fault-tolerant and is designed to be deployed on low-cost hardware. It provides high throughput access to application data and is suitable for applications having large datasets.
- Log in to post comments