Improving performance of apriori algorithm using hadoop
1 like194 views
This paper discusses improving the performance of the apriori algorithm for spatial data mining using the Hadoop framework, highlighting its advantages over the FP-Growth algorithm, particularly for dynamic and large datasets. It proposes a system that uses parallel processing to enhance mining speed and perform essential operations like clustering and classification based on mineral resources data. The results show improved execution times and support for various association rules in mineral resource mining.
1 of 3
Downloaded 12 times
![IJRET: International Journal of Research in Engineering and Technology eISSN: 2319-1163 | pISSN: 2321-7308
_______________________________________________________________________________________
Volume: 03 Issue: 12 | Dec-2014, Available @ http://www.ijret.org 277
IMPROVING PERFORMANCE OF APRIORI ALGORITHM USING
HADOOP
Ravindra Bachate1
, Hyder Ali Hingoliwala2
1
Department of Computer Engineering, JSPM’s JSCOE, Pune, Maharashtra, India
2
Department of Computer Engineering, JSPM’s JSCOE, Pune, Maharashtra, India
Abstract
Spatial data is a data having a geological information. This paper explores the use of Hadoop framework to improve the
performance of Apriori algorithm for spatial data mining. FP growth algorithm is better than Apriori but it fails in certain
situations. By applying the Apriori algorithm parallely using Hadoop framework to spatial data, we can perform well as compare
to FP growth. This paper includes clustering based on geological location, classification based on mineral resource type and
spatial coherence between mineral resources. Spatial data mining find out the different association rules by observing the spatial
data by using Apriori algorithm. The result of the paper will indicate the accurate prediction of occurrence of commodity with
respect to other commodity of mineral resources.
Keywords: Hadoop, data mining, association rules, clustering, spatial coherence
--------------------------------------------------------------------***----------------------------------------------------------------------
1. INTRODUCTION
FP growth algorithm is a very popular algorithm used for
the association mining due its performance and less storage
requirement. With these pros, it has some limitations also
which are considered rarely. This paper focuses on the
limitations of FP growth association mining algorithm and
trying to suggest improvements in Apriori algorithm which
can overcome the limitations of FP Growth association
mining algorithm. As we aware about the hardware cost, it’s
decreasing day by day, so there is no need to concentrate
more on storage requirement. FP growth has two limitations
– 1. It is difficult to use for interactive mining where the
user may change support value as per requirement. 2. It is
not suitable where the data has been increasing with time.
So to deal with limitations of FP growth, this paper suggest
to implement Apriori algorithm using Hadoop for
association rule mining.
1.1 Mineral Resources Data System
Mineral Resources Data System is a collection of data
describing metallic and nonmetallic mineral resources in the
world [7]. It includes resource name, location, commodity,
geologic characteristics, resource description, production,
reserves, and references. As MRDS contains mineral
resources data around the world, it is large and complex. If
data size goes beyond the Tera Byte, it is difficult to process
and mine using FP growth algorithm. The performance of
FP growth algorithm hampers when adding the new records
and also by changing the support value. As the mineral
resources data set is collected from various regions and
people, we need to perform ETL (extract, transform and
load) operations on the data for processing and mining.
1.2 Hadoop Map Reduce
To deal with unstructured and big data like mineral
resources data system (MRDS), we need a best technology
which can cope with it. There are two options available,
parallel DBMS and Hadoop Map Reduce technology.
Hadoop has another projects also which can be used for the
mining purpose like Hive. But internally again all the
projects using Hadoop Map Reduce technique [5]. It gives a
better data processing performance with minimum cost and
time as compare to parallel DBMS because it works with
commodity hardware. Hadoop stores data in the form of
blocks on Hadoop Distributed File System i.e. HDFS. The
Hadoop framework provides a solution for problems of
massive data processing; because it runs applications on
large cluster built of commodity hardware with failure
tolerance [4].Unstructured data can be processed with
Hadoop Map Reduce technique which is not possible with
RDBMS. Map Reduce provides flexibility and fault
tolerance which is not with parallel DBMS. Map Reduce
provides automatic parallelization, data partitioning, task
scheduling, handling machine failures and manages inter-
machine communication. Hadoop is totally transparent from
the end user. The rate of growing an unstructured data is
much more as compare to the structured data. The
unstructured data includes media files, heavy text files, csv
files, log files etc.
2. RELATED WORK
Association rule mining algorithm includes to find coverage,
support, confidence, lift and interesting [4]. Coverage
defines the proportion of case data specified on the Left Haft
Hand Side of the rule. Support of an association rule means
percentage of task relevant data for which the pattern is true.
Confidence gives the trustworthiness associated with the
patterns discovered. Lift is a measure of the importance of
the association. The term interesting gives the strength of
associations between sets of items in the association rule.](https://image.slidesharecdn.com/improvingperformanceofapriorialgorithmusinghadoop-160827113756/85/Improving-performance-of-apriori-algorithm-using-hadoop-1-320.jpg)
![IJRET: International Journal of Research in Engineering and Technology eISSN: 2319-1163 | pISSN: 2321-7308
_______________________________________________________________________________________
Volume: 03 Issue: 12 | Dec-2014, Available @ http://www.ijret.org 278
Hongyong Yu, Deshuai Wang [1] proposed a system for
data processing and mining log data of SaaS cloud using
Hadoop. This paper focuses on Hadoop’s Map Reduce
technique and the algorithm used for data mining by
Hongyong Yu, Deshuai Wang. They suggest by applying
Apriori algorithm concurrently in the distributed system,
performance of Apriori algorithm can be increased in
proportional to the number of nodes in the distributed
system [1].
Every day, 2.5 quintillion bytes of data are created and 90
percent of the data in the world today were produced within
the past two years [3].The commonly used software
technology cannot cope with massive data and big challenge
is to extract an important information from it. Big data has
large volume, heterogeneous format and decentralized data
control. The example of big data applications are Facebook,
Twitter and Google. It is a big challenge to manage and
mining a massive data because of its volume, different file
formats and growing rate of the data in the world. There are
many challenges with big data such as storage, processing,
variety and cost.
3. PROPOSED SYSTEM
This paper proposes a system which overcomes the problem
faced in FP growth association mining algorithm. To
improve the performance of Apriori algorithm, it is
implemented parallely using Hadoop map reduce technique.
The proposed system has three modules
1. Spatial Clustering
2. Spatial Classification
3. Spatial Coherence
Before implementing these modules, we need to perform
Extract, Transfer and Load operations on the raw data.
Because the data available may be in the various form and
having an unnecessary information into to it. So first we
need to extract the required data from raw data and then
transfer it in to the csv format. To process this data, it should
be loaded on to the HDFS.
Fig-1 Proposed System
3.1 Spatial Clustering
There are various algorithms available for clustering like k
means but here in this paper, Hadoop partition technique is
used to perform spatial clustering based on the geological
location like Alaska, California etc[2]. It runs parallel on
different nodes concurrently. Because of this, time required
for spatial clustering is less as compared to k-means
clustering algorithm. In this paper, U.S. mineral resources
dataset is taken. In effect, there are 50 clusters formed with
respect to the states in United State.
These clusters have all the records which belongs to
respective state. Intention behind to form the clusters is to
make it specialize and reduce the processing time and cost.
3.2 Spatial Classification
Here, the same technique is used for implementing a spatial
classification which is used for spatial clustering. In this
module, again the data set is clustered according to the type
of commodity which is required for the next module. In
spatial classification, the records having same commodity
put in to the separate cluster to make it more specialize. It
means, e.g. Alaska state may have another sub clusters with
commodity type like gold, silver, copper etc.
3.3 Spatial Coherence
Objective of this paper is to find out the spatial coherence.
Spatial coherence means if we are mining for gold then what
are the possibilities of getting other mineral resources at the
same location. For this, association mining is required. As
we have discussed above, FP Growth algorithms fails in two
situations. So to overcome this, this paper introduces
parallel Apriori algorithm for spatial association mining. In
Apriori algorithm, first of all, we have to find out the
candidate keys. After this, we have to perform two task, one
is to find out the support values and second is to find out the
confidence. Formula’s to calculate a support and confidence
for gold with copper.](https://image.slidesharecdn.com/improvingperformanceofapriorialgorithmusinghadoop-160827113756/85/Improving-performance-of-apriori-algorithm-using-hadoop-2-320.jpg)
![IJRET: International Journal of Research in Engineering and Technology eISSN: 2319-1163 | pISSN: 2321-7308
_______________________________________________________________________________________
Volume: 03 Issue: 12 | Dec-2014, Available @ http://www.ijret.org 279
1. Support Formula
Where N is all the records in a cluster
2. Confidence Formula
Here, a formulas for calculating support and confidence of
gold is given. Support of (gold => copper) gives the
probability of having gold with copper in all the records
whereas confidence gives the probability of having of gold
with copper with respect to all the gold records in a cluster.
The value of support is always less than the confidence.
4. RESULTS
To implement this idea, we have taken a dataset of MRDS
which is a spatial data set of mineral resources in U.S.[7].
This sample data has around 3.5 lack records.
Table -1: Support and Confidence
Commodity Support (%) Confidence (%)
(Gold -> Copper) 2.89 9.30
(Gold -> Copper, Lead,
Silver)
0.72 2.3
(Gold -> Silver) 2.17 6.97
(Gold -> null) 24.63 79.06
Here, the Apriori algorithm is performed on single node and
multi node Hadoop to compare the performance. Also we
find the association rules for this spatial data for a single
cluster.
Table -2: Execution on Single and Multinode System
No. of Node Execution Time in Sec
1 5.2
2 3.1
3 2.3
The result in Table-1 shows the different association rules
with respect to gold.Table-2 shows the performance of
Apriori algorithm using single node and multi node system.
It is observed that if we implement the Apriori Algorithm
parallel on Hadoop, performance is improved.
5. CONCLUSION
FP growth algorithm has two limitations, it cannot be used
for dynamic data size and where the support needs to change
according to situation. By applying the Apriori Algorithm
concurrently using Hadoop, we can overcome the problems
faced by FP growth association mining algorithm. Also we
can improve the speed of association rule mining for spatial
data as compare to the FP growth algorithm as Hadoop is a
distributed system.
ACKNOWLEDGEMENTS
I express true sense of gratitude towards my project guide
Prof. H.A. Hingoliwala, Associate Professor Computer
Department for his invaluable co-operation and guidance
that he gave me throughout my project. I specially thank our
P.G coordinator Prof. M. D. Ingle for inspiring me and
providing me all the lab facilities. I would also like to
express my appreciation and thanks to HOD Prof. S.M.
Shinde & JSCOE Principal Dr. M.G. Jadhav and all my
friends who knowingly or unknowingly have assisted me
throughout my hard work
REFERENCES
[1]. Hongyong Yu, Deshuai Wang, “Mass Log Data
Processing and Mining Based on Hadoop and Cloud
Computing” .The 7th International Conference on Computer
Science & Education (ICCSE 2012)July 14-17, 2012.
Melbourne, Australia.
[2]. Duck-Ho Bae Coll. of Inf. & Commun., Hanyang Univ.,
Seoul, South Korea Ji-Haeng Baek ; Hyun-Kyo Oh ; Ju-
Won Song ; Sang-Wook Kim, “SD-Miner: A SPATIAL
DATA MINING SYSTEM” Network Infrastructure and
Digital Content, 2009.
[3]. Xindong Wu,Fellow, IEEE,Xingquan Zhu, Senior
Member, IEEE, Gong-Qing Wu, and Wei Ding,Senior
Member, IEEE, “Data mining with big data,” IEEE
TRANSACTIONS ON KNOWLEDGE AND DATA
ENGINEERING, VOL. 26, NO. 1, JANUARY 2014.
[4]. Binbin He, Ying Cui, Jianhua Chen, Pingjing Xie, “A
Spatial Data Mining Method for Mineral Resources
Potential Assessment,” IEEE 978-1-4244-8351-
8/11,2011IEEE.
[5]. Hadoop: The definitive Guide, 3rd ed., O’Reilly, Tom
White, 2012
[6]. Hadoop, http://hadoop.apache.org/
[7]. MRDS, http://tin.er.usgs.gov/mrds/](https://image.slidesharecdn.com/improvingperformanceofapriorialgorithmusinghadoop-160827113756/85/Improving-performance-of-apriori-algorithm-using-hadoop-3-320.jpg)
Ad
Recommended
AN ENHANCED FREQUENT PATTERN GROWTH BASED ON MAPREDUCE FOR MINING ASSOCIATION...
AN ENHANCED FREQUENT PATTERN GROWTH BASED ON MAPREDUCE FOR MINING ASSOCIATION...IJDKP This paper presents the enhanced frequent pattern growth algorithm (efp-growth) which utilizes the MapReduce framework to improve performance in mining frequent itemsets from large transaction databases. The proposed efp-growth algorithm is shown to outperform the original fp-growth algorithm in terms of execution time, particularly with large datasets. Experimental results confirm the method's efficiency in discovering frequent patterns while addressing challenges associated with processing massive small files.
Ijetcas14 316
Ijetcas14 316Iasir Journals This document proposes a model to parallelize the frequent itemset mining process using GPUs instead of multi-core processors. It aims to speed up the mining process and allow it to handle large datasets more efficiently. The model parallelizes the FP-growth algorithm at different levels without generating the FP-tree. It first sorts the transaction database in parallel using GPUs for preprocessing. It then groups the transactions based on the first item and mines for frequent itemsets within each group in parallel on the GPU. Preliminary results show the sorting step is significantly faster when parallelized on the GPU compared to serial processing. The overall goal is to efficiently mine large datasets using the low-cost and high-performance capabilities of GPUs.
43_Sameer_Kumar_Das2
43_Sameer_Kumar_Das2Mr.Sameer Kumar Das This document provides an overview of the Pig tool, which is a scripting language for exploring large datasets within the Apache Hadoop ecosystem. It discusses how Pig allows processing of terabytes of data through just a few lines of code by customizing all parts of the processing path, including storing, filtering, grouping, and joining data. The document then presents a sample problem using a publicly available million song dataset to demonstrate loading and storing the data with Pig, finding the song density, and filtering the results. It analyzes the input and output, showing that the input dataset contained 1 million records which were filtered down to the top 50 songs with the highest sound densities.
IRJET - A Prognosis Approach for Stock Market Prediction based on Term Streak...
IRJET - A Prognosis Approach for Stock Market Prediction based on Term Streak...IRJET Journal This document discusses using machine learning algorithms and Hadoop to predict stock market performance based on historical stock data. It proposes a model that would collect stock data from various sources, preprocess the data to clean it, cluster the data using MapReduce, and then use a support vector machine algorithm to analyze the clustered data and generate stock predictions. The model is designed to take advantage of Hadoop's ability to process large datasets in parallel across multiple servers or clusters. The goal is to more accurately predict stock prices and identify market trends based on analyzing huge amounts of historical stock market data.
Paper id 25201498
Paper id 25201498IJRAT This document summarizes a research paper that analyzes and evaluates the performance of processing large data sets using Hadoop. It discusses how Hadoop Distributed File System (HDFS) and MapReduce provide parallel and distributed processing of large structured and unstructured data at scale. The paper also presents the results of experiments conducted on Hadoop to classify and cluster large data sets using machine learning algorithms. The experiments showed that Hadoop can process large data sets more efficiently and reliably compared to processing on a single computer.
A Survey of Machine Learning Techniques for Self-tuning Hadoop Performance
A Survey of Machine Learning Techniques for Self-tuning Hadoop Performance IJECEIAES This paper surveys machine learning techniques aimed at optimizing the performance of the Apache Hadoop framework, which is crucial for processing big data due to its complex configuration parameters. It discusses existing machine learning applications, identifies critical issues in the Hadoop system, and proposes a deep learning approach to enhance performance. Various algorithms, including random forests and support vector regression, are analyzed for their effectiveness in self-tuning Hadoop configurations.
A sql implementation on the map reduce framework
A sql implementation on the map reduce frameworkeldariof Tenzing is a SQL query engine built on top of MapReduce for analyzing large datasets at Google. It provides low latency queries over heterogeneous data sources at petabyte scale. Tenzing supports a full SQL implementation with extensions and optimizations to achieve high performance comparable to commercial parallel databases by leveraging MapReduce and traditional database techniques. It is used by over 1,000 Google employees to query over 1.5 petabytes of data and serve 10,000 queries daily.
IRJET- Review of Existing Methods in K-Means Clustering Algorithm
IRJET- Review of Existing Methods in K-Means Clustering AlgorithmIRJET Journal The document reviews existing methods for the k-means clustering algorithm. It discusses how k-means clustering works and some of its limitations when dealing with large datasets, such as being dependent on the initial choice of centroids. It then proposes using Hadoop to overcome big data challenges and calculate preliminary centroids for k-means clustering in a distributed manner. Finally, it reviews different techniques that have been proposed in other research to improve k-means clustering, such as methods for selecting better initial centroids or determining the optimal number of clusters.
Implementation of p pic algorithm in map reduce to handle big data
Implementation of p pic algorithm in map reduce to handle big dataeSAT Publishing House This document presents an implementation of the p-PIC clustering algorithm using the MapReduce framework to handle big data. P-PIC is a parallel version of the Power Iteration Clustering (PIC) algorithm that is able to cluster large datasets in a distributed environment. The document first provides background on PIC and challenges with scaling to big data. It then describes how p-PIC addresses these challenges using MPI for parallelization. The design of implementing p-PIC within MapReduce is presented, including the map and reduce functions. Experimental results on synthetic datasets up to 100,000 records show that p-PIC using MapReduce has increased performance and scalability compared to the original p-PIC implementation using MPI.
Berlin Hadoop Get Together Apache Drill
Berlin Hadoop Get Together Apache Drill MapR Technologies Apache Drill is an open source interactive query engine for large scale datasets. It allows for interactive, ad-hoc queries over data stored in both traditional and NoSQL data stores. Drill uses a pluggable architecture that allows it to work with different data sources and formats. It aims to provide low-latency queries over large datasets through its distributed execution engine and columnar data layout.
Web Oriented FIM for large scale dataset using Hadoop
Web Oriented FIM for large scale dataset using Hadoopdbpublications This document discusses a new parallel mining algorithm utilizing Hadoop's map-reduce framework to enhance frequent itemset mining on large datasets. It proposes a data partitioning technique to improve performance by reducing redundant transactions and balancing computing loads across nodes. The study concludes that this approach can significantly outperform traditional mining algorithms and suggests future exploration with Apache Spark for further performance improvements.
IRJET- Performing Load Balancing between Namenodes in HDFS
IRJET- Performing Load Balancing between Namenodes in HDFSIRJET Journal This document proposes a new architecture for HDFS to address the single point of failure issue of the NameNode. The current HDFS architecture uses a single NameNode that manages file system metadata. If it fails, the entire system fails. The proposed architecture uses multiple interconnected NameNodes that maintain mirrors of each other's metadata using the Chord system. This allows load balancing between NameNodes and prevents failure if one NameNode goes down, as other NameNodes can handle the load and client requests/responses. The goal is to improve scalability, availability and reduce downtime of the NameNode in HDFS through this new distributed architecture.
Parallel Key Value Pattern Matching Model
Parallel Key Value Pattern Matching Modelijsrd.com The document presents a parallel key value pattern matching model for frequent itemset mining, detailing how it improves efficiency and reduces computational costs in data mining tasks by utilizing a MapReduce approach. The paper discusses the limitations of existing models and introduces a new approach that divides mining tasks into independent parallel tasks, enhancing performance compared to previous methods like the xmodel and pmodel. Experimental results show that the proposed model significantly decreases processing time, making it suitable for large-scale data mining applications.
Distributed Algorithm for Frequent Pattern Mining using HadoopMap Reduce Fram...
Distributed Algorithm for Frequent Pattern Mining using HadoopMap Reduce Fram...idescitation The document presents a parallel and distributed frequent pattern mining algorithm (DPFPM) using the Hadoop MapReduce framework to efficiently mine large databases while minimizing communication overhead. The proposed algorithm partitions data across multiple nodes, locally generates frequent patterns, and merges results at the final stage, significantly improving execution time and scalability compared to existing approaches. Experimental results demonstrate the effectiveness of DPFPM in handling very large datasets with enhanced performance.
Effect of countries in performance of hadoop.
Effect of countries in performance of hadoop.Computer Science Journals This document discusses analyzing the performance of Hadoop with different sized datasets and various counters. Experiments were conducted on a Hadoop cluster to analyze how CPU time, number of blocks, heap size, HDFS bytes read, and spilled records are affected by the size of data from 49.7 MB to 71.5 MB. The results showed that CPU time increases slightly as data and number of blocks increase. CPU time also increases as HDFS bytes read and spilled records increase. Tuning the heap size and number of blocks can help optimize performance. Hadoop is concluded to efficiently handle and analyze big data.
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...IRJET Journal This document presents two variations of a job-driven scheduling scheme called JOSS for efficiently executing MapReduce jobs on remote outsourced data across multiple data centers. The goal of JOSS is to improve data locality for map and reduce tasks, avoid job starvation, and improve job performance. Extensive experiments show that the two JOSS variations, called JOSS-T and JOSS-J, outperform other scheduling algorithms in terms of data locality and network overhead without significant overhead. JOSS-T performs best for workloads of small jobs, while JOSS-J provides the shortest workload time for jobs of varying sizes distributed across data centers.
Survey Paper on Big Data and Hadoop
Survey Paper on Big Data and HadoopIRJET Journal This document discusses big data and Hadoop. It defines big data as large datasets that are difficult to process using traditional methods due to their volume, variety, and velocity. Hadoop is presented as an open-source software framework for distributed storage and processing of large datasets across clusters of commodity servers. The key components of Hadoop are the Hadoop Distributed File System (HDFS) for storage and MapReduce as a programming model for distributed processing. A number of other technologies in Hadoop's ecosystem are also described such as HBase, Avro, Pig, Hive, Sqoop, Zookeeper and Mahout. The document concludes that Hadoop provides solutions for efficiently processing and analyzing big data.
An effective classification approach for big data with parallel generalized H...
An effective classification approach for big data with parallel generalized H...riyaniaes The paper presents an efficient classification and dimensionality reduction approach for big data using the parallel generalized Hebbian algorithm (GHA), implemented on the Spark Radoop platform. This method is shown to outperform existing techniques in handling high-dimensional datasets, addressing the challenges posed by the massive volume and complexity of big data. The research includes extensive comparisons with other machine learning algorithms and evaluates performance using multiple large datasets.
An efficient data mining framework on hadoop using java persistence api
An efficient data mining framework on hadoop using java persistence apiJoão Gabriel Lima This document describes a proposed framework for efficient data mining on Hadoop using Java Persistence API (JPA) and MySQL Cluster. The framework indexes data using ZSCORE binning and inverted indexing to enable efficient querying. It then stores mining models and results persistently in MySQL Cluster using JPA for object-relational mapping. The document evaluates the performance of the framework by implementing a decision tree algorithm on Hadoop and comparing the JPA approach to a traditional JDBC implementation.
A Big-Data Process Consigned Geographically by Employing Mapreduce Frame Work
A Big-Data Process Consigned Geographically by Employing Mapreduce Frame WorkIRJET Journal This document discusses frameworks for processing big data that is distributed across geographic locations. It begins by introducing the challenges of geo-distributed big data processing and then describes several MapReduce-based frameworks like G-Hadoop and G-MR that can process pre-located geo-distributed data. It also covers Spark-based systems like Iridium and frameworks that partition data across geographic locations, such as KOALA grid-based systems. The document analyzes key aspects of geo-distributed big data processing systems like data distribution, task scheduling, and fault tolerance.
Improved Map reduce Framework using High Utility Transactional Databases
Improved Map reduce Framework using High Utility Transactional DatabasesInternational Journal of Engineering Inventions www.ijeijournal.com The document proposes improved MapReduce algorithms (UP-Growth and UP-Growth+) for mining high utility itemsets from transactional databases. Existing algorithms often generate a huge number of candidate itemsets, degrading performance. The proposed algorithms use a UP-Tree structure and MapReduce framework on Hadoop to more efficiently identify high utility itemsets from large datasets in distributed storage. Experimental results show the improved algorithms outperform other methods, especially for databases with long transactions or low minimum utility thresholds. The goal is to address limitations of existing approaches for low-memory systems and databases with null transactions.
A data aware caching 2415
A data aware caching 2415SANTOSH WAYAL This document discusses a proposed data-aware caching framework called Dache that could be used with big data applications built on MapReduce. Dache aims to cache intermediate data generated during MapReduce jobs to avoid duplicate computations. When tasks run, they would first check the cache for existing results before running the actual computations. The goal is to improve efficiency by reducing redundant work. The document outlines the objectives and scope of extending MapReduce with Dache, provides background on MapReduce and Hadoop, and concludes that initial experiments show Dache can eliminate duplicate tasks in incremental jobs.
Hadoop interview questions
Hadoop interview questionsbarbie0909 This document outlines key concepts and interview questions related to Hadoop and its MapReduce framework. It explains processes such as how MapReduce works, shuffling, distributed cache, and the roles of components like namenode and heartbeat in HDFS. Additionally, it discusses the use of combiners for efficiency, data node failures, and the function of the partitioner in distributing map output.
Leveraging Map Reduce With Hadoop for Weather Data Analytics
Leveraging Map Reduce With Hadoop for Weather Data Analytics iosrjce The document discusses a project that leverages the MapReduce programming model with Hadoop to process large volumes of weather data collected from multiple sensors. It addresses the complexity of processing high-velocity sensor data for accurate weather prediction and presents a proposed analytical engine that enhances data processing speed without scalability issues. The implementation details include the use of mappers and reducers for temperature data analysis, focusing on efficiency and resource management within the Hadoop framework.
Graphlab Ted Dunning Clustering
Graphlab Ted Dunning ClusteringMapR Technologies The document discusses a large-scale, single-pass k-means clustering method designed to efficiently cluster massive datasets while achieving high-quality results. It details the development process undertaken by a team from a customer bank during a two-week hackathon, resulting in significant speed improvements for clustering algorithms. Key features include the use of synthetic data, open-source licensing, and the implementation of shared memory matrices and streamlined search interfaces to enhance performance.
Introduction to HADOOP
Introduction to HADOOPShital Kat The document discusses big data and Hadoop as a framework for processing large datasets. It describes how Hadoop uses HDFS for storage and MapReduce for parallel processing. HDFS uses a master/slave architecture with a NameNode and DataNodes. MapReduce jobs are managed by a JobTracker and executed on TaskTrackers. The document provides an example of using MapReduce to find common friends between users. It concludes that Hadoop is capable of solving big data challenges through scalable and fault-tolerant distributed processing.
IRJET - Weather Log Analysis based on Hadoop Technology
IRJET - Weather Log Analysis based on Hadoop TechnologyIRJET Journal This document discusses using Hadoop technology to analyze large amounts of weather data. It aims to analyze and predict temperature, which can help agriculture and governments respond to disasters. The system collects weather data from various sources, stores it in Hadoop's distributed file system, filters out irrelevant data, uses MapReduce algorithms to extract patterns, and displays the results in graphs. Analyzing huge volumes of weather data on a single system is inefficient, so Hadoop provides a better solution by allowing distributed processing across clustered systems.
InternReport
InternReportSwetha Tanamala This document summarizes an intern's report on their internship at Altisource working with big data on the Hadoop platform. The intern researched how data is stored in RDBMS versus Hadoop, learned query languages like HiveQL and MySQL, and gained knowledge of MapReduce, Spark, and Sqoop. As part of a project, the intern analyzed Altisource data using Apache Hadoop and Spark. The intern also set up a 3-node cluster on Google Cloud and used it to analyze NASA clickstream and other data, proving concepts like linear increase in processing time with data size. The intern concluded they gained exposure to querying languages and distributed computing concepts during the internship.
Paratransit Services Research
Paratransit Services ResearchP.C.D.I Healthcare and Consultants of Texas L.L.C. This literature review examines the barriers to medical transportation faced by African Americans with end-stage renal disease (ESRD) and cognitive disorders, emphasizing the need for improved paratransit services. It highlights disparities in healthcare access, stressing the importance of public transportation in addressing health challenges within this population. The study calls for advocacy and policy changes to enhance transportation options and healthcare access for minority groups at risk for health complications.
More Related Content
What's hot (20)
Implementation of p pic algorithm in map reduce to handle big data
Implementation of p pic algorithm in map reduce to handle big dataeSAT Publishing House This document presents an implementation of the p-PIC clustering algorithm using the MapReduce framework to handle big data. P-PIC is a parallel version of the Power Iteration Clustering (PIC) algorithm that is able to cluster large datasets in a distributed environment. The document first provides background on PIC and challenges with scaling to big data. It then describes how p-PIC addresses these challenges using MPI for parallelization. The design of implementing p-PIC within MapReduce is presented, including the map and reduce functions. Experimental results on synthetic datasets up to 100,000 records show that p-PIC using MapReduce has increased performance and scalability compared to the original p-PIC implementation using MPI.
Berlin Hadoop Get Together Apache Drill
Berlin Hadoop Get Together Apache Drill MapR Technologies Apache Drill is an open source interactive query engine for large scale datasets. It allows for interactive, ad-hoc queries over data stored in both traditional and NoSQL data stores. Drill uses a pluggable architecture that allows it to work with different data sources and formats. It aims to provide low-latency queries over large datasets through its distributed execution engine and columnar data layout.
Web Oriented FIM for large scale dataset using Hadoop
Web Oriented FIM for large scale dataset using Hadoopdbpublications This document discusses a new parallel mining algorithm utilizing Hadoop's map-reduce framework to enhance frequent itemset mining on large datasets. It proposes a data partitioning technique to improve performance by reducing redundant transactions and balancing computing loads across nodes. The study concludes that this approach can significantly outperform traditional mining algorithms and suggests future exploration with Apache Spark for further performance improvements.
IRJET- Performing Load Balancing between Namenodes in HDFS
IRJET- Performing Load Balancing between Namenodes in HDFSIRJET Journal This document proposes a new architecture for HDFS to address the single point of failure issue of the NameNode. The current HDFS architecture uses a single NameNode that manages file system metadata. If it fails, the entire system fails. The proposed architecture uses multiple interconnected NameNodes that maintain mirrors of each other's metadata using the Chord system. This allows load balancing between NameNodes and prevents failure if one NameNode goes down, as other NameNodes can handle the load and client requests/responses. The goal is to improve scalability, availability and reduce downtime of the NameNode in HDFS through this new distributed architecture.
Parallel Key Value Pattern Matching Model
Parallel Key Value Pattern Matching Modelijsrd.com The document presents a parallel key value pattern matching model for frequent itemset mining, detailing how it improves efficiency and reduces computational costs in data mining tasks by utilizing a MapReduce approach. The paper discusses the limitations of existing models and introduces a new approach that divides mining tasks into independent parallel tasks, enhancing performance compared to previous methods like the xmodel and pmodel. Experimental results show that the proposed model significantly decreases processing time, making it suitable for large-scale data mining applications.
Distributed Algorithm for Frequent Pattern Mining using HadoopMap Reduce Fram...
Distributed Algorithm for Frequent Pattern Mining using HadoopMap Reduce Fram...idescitation The document presents a parallel and distributed frequent pattern mining algorithm (DPFPM) using the Hadoop MapReduce framework to efficiently mine large databases while minimizing communication overhead. The proposed algorithm partitions data across multiple nodes, locally generates frequent patterns, and merges results at the final stage, significantly improving execution time and scalability compared to existing approaches. Experimental results demonstrate the effectiveness of DPFPM in handling very large datasets with enhanced performance.
Effect of countries in performance of hadoop.
Effect of countries in performance of hadoop.Computer Science Journals This document discusses analyzing the performance of Hadoop with different sized datasets and various counters. Experiments were conducted on a Hadoop cluster to analyze how CPU time, number of blocks, heap size, HDFS bytes read, and spilled records are affected by the size of data from 49.7 MB to 71.5 MB. The results showed that CPU time increases slightly as data and number of blocks increase. CPU time also increases as HDFS bytes read and spilled records increase. Tuning the heap size and number of blocks can help optimize performance. Hadoop is concluded to efficiently handle and analyze big data.
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...IRJET Journal This document presents two variations of a job-driven scheduling scheme called JOSS for efficiently executing MapReduce jobs on remote outsourced data across multiple data centers. The goal of JOSS is to improve data locality for map and reduce tasks, avoid job starvation, and improve job performance. Extensive experiments show that the two JOSS variations, called JOSS-T and JOSS-J, outperform other scheduling algorithms in terms of data locality and network overhead without significant overhead. JOSS-T performs best for workloads of small jobs, while JOSS-J provides the shortest workload time for jobs of varying sizes distributed across data centers.
Survey Paper on Big Data and Hadoop
Survey Paper on Big Data and HadoopIRJET Journal This document discusses big data and Hadoop. It defines big data as large datasets that are difficult to process using traditional methods due to their volume, variety, and velocity. Hadoop is presented as an open-source software framework for distributed storage and processing of large datasets across clusters of commodity servers. The key components of Hadoop are the Hadoop Distributed File System (HDFS) for storage and MapReduce as a programming model for distributed processing. A number of other technologies in Hadoop's ecosystem are also described such as HBase, Avro, Pig, Hive, Sqoop, Zookeeper and Mahout. The document concludes that Hadoop provides solutions for efficiently processing and analyzing big data.
An effective classification approach for big data with parallel generalized H...
An effective classification approach for big data with parallel generalized H...riyaniaes The paper presents an efficient classification and dimensionality reduction approach for big data using the parallel generalized Hebbian algorithm (GHA), implemented on the Spark Radoop platform. This method is shown to outperform existing techniques in handling high-dimensional datasets, addressing the challenges posed by the massive volume and complexity of big data. The research includes extensive comparisons with other machine learning algorithms and evaluates performance using multiple large datasets.
An efficient data mining framework on hadoop using java persistence api
An efficient data mining framework on hadoop using java persistence apiJoão Gabriel Lima This document describes a proposed framework for efficient data mining on Hadoop using Java Persistence API (JPA) and MySQL Cluster. The framework indexes data using ZSCORE binning and inverted indexing to enable efficient querying. It then stores mining models and results persistently in MySQL Cluster using JPA for object-relational mapping. The document evaluates the performance of the framework by implementing a decision tree algorithm on Hadoop and comparing the JPA approach to a traditional JDBC implementation.
A Big-Data Process Consigned Geographically by Employing Mapreduce Frame Work
A Big-Data Process Consigned Geographically by Employing Mapreduce Frame WorkIRJET Journal This document discusses frameworks for processing big data that is distributed across geographic locations. It begins by introducing the challenges of geo-distributed big data processing and then describes several MapReduce-based frameworks like G-Hadoop and G-MR that can process pre-located geo-distributed data. It also covers Spark-based systems like Iridium and frameworks that partition data across geographic locations, such as KOALA grid-based systems. The document analyzes key aspects of geo-distributed big data processing systems like data distribution, task scheduling, and fault tolerance.
Improved Map reduce Framework using High Utility Transactional Databases
Improved Map reduce Framework using High Utility Transactional DatabasesInternational Journal of Engineering Inventions www.ijeijournal.com The document proposes improved MapReduce algorithms (UP-Growth and UP-Growth+) for mining high utility itemsets from transactional databases. Existing algorithms often generate a huge number of candidate itemsets, degrading performance. The proposed algorithms use a UP-Tree structure and MapReduce framework on Hadoop to more efficiently identify high utility itemsets from large datasets in distributed storage. Experimental results show the improved algorithms outperform other methods, especially for databases with long transactions or low minimum utility thresholds. The goal is to address limitations of existing approaches for low-memory systems and databases with null transactions.
A data aware caching 2415
A data aware caching 2415SANTOSH WAYAL This document discusses a proposed data-aware caching framework called Dache that could be used with big data applications built on MapReduce. Dache aims to cache intermediate data generated during MapReduce jobs to avoid duplicate computations. When tasks run, they would first check the cache for existing results before running the actual computations. The goal is to improve efficiency by reducing redundant work. The document outlines the objectives and scope of extending MapReduce with Dache, provides background on MapReduce and Hadoop, and concludes that initial experiments show Dache can eliminate duplicate tasks in incremental jobs.
Hadoop interview questions
Hadoop interview questionsbarbie0909 This document outlines key concepts and interview questions related to Hadoop and its MapReduce framework. It explains processes such as how MapReduce works, shuffling, distributed cache, and the roles of components like namenode and heartbeat in HDFS. Additionally, it discusses the use of combiners for efficiency, data node failures, and the function of the partitioner in distributing map output.
Leveraging Map Reduce With Hadoop for Weather Data Analytics
Leveraging Map Reduce With Hadoop for Weather Data Analytics iosrjce The document discusses a project that leverages the MapReduce programming model with Hadoop to process large volumes of weather data collected from multiple sensors. It addresses the complexity of processing high-velocity sensor data for accurate weather prediction and presents a proposed analytical engine that enhances data processing speed without scalability issues. The implementation details include the use of mappers and reducers for temperature data analysis, focusing on efficiency and resource management within the Hadoop framework.
Graphlab Ted Dunning Clustering
Graphlab Ted Dunning ClusteringMapR Technologies The document discusses a large-scale, single-pass k-means clustering method designed to efficiently cluster massive datasets while achieving high-quality results. It details the development process undertaken by a team from a customer bank during a two-week hackathon, resulting in significant speed improvements for clustering algorithms. Key features include the use of synthetic data, open-source licensing, and the implementation of shared memory matrices and streamlined search interfaces to enhance performance.
Introduction to HADOOP
Introduction to HADOOPShital Kat The document discusses big data and Hadoop as a framework for processing large datasets. It describes how Hadoop uses HDFS for storage and MapReduce for parallel processing. HDFS uses a master/slave architecture with a NameNode and DataNodes. MapReduce jobs are managed by a JobTracker and executed on TaskTrackers. The document provides an example of using MapReduce to find common friends between users. It concludes that Hadoop is capable of solving big data challenges through scalable and fault-tolerant distributed processing.
IRJET - Weather Log Analysis based on Hadoop Technology
IRJET - Weather Log Analysis based on Hadoop TechnologyIRJET Journal This document discusses using Hadoop technology to analyze large amounts of weather data. It aims to analyze and predict temperature, which can help agriculture and governments respond to disasters. The system collects weather data from various sources, stores it in Hadoop's distributed file system, filters out irrelevant data, uses MapReduce algorithms to extract patterns, and displays the results in graphs. Analyzing huge volumes of weather data on a single system is inefficient, so Hadoop provides a better solution by allowing distributed processing across clustered systems.
InternReport
InternReportSwetha Tanamala This document summarizes an intern's report on their internship at Altisource working with big data on the Hadoop platform. The intern researched how data is stored in RDBMS versus Hadoop, learned query languages like HiveQL and MySQL, and gained knowledge of MapReduce, Spark, and Sqoop. As part of a project, the intern analyzed Altisource data using Apache Hadoop and Spark. The intern also set up a 3-node cluster on Google Cloud and used it to analyze NASA clickstream and other data, proving concepts like linear increase in processing time with data size. The intern concluded they gained exposure to querying languages and distributed computing concepts during the internship.
Improved Map reduce Framework using High Utility Transactional Databases
Improved Map reduce Framework using High Utility Transactional DatabasesInternational Journal of Engineering Inventions www.ijeijournal.com
Viewers also liked (16)
Paratransit Services Research
Paratransit Services ResearchP.C.D.I Healthcare and Consultants of Texas L.L.C. This literature review examines the barriers to medical transportation faced by African Americans with end-stage renal disease (ESRD) and cognitive disorders, emphasizing the need for improved paratransit services. It highlights disparities in healthcare access, stressing the importance of public transportation in addressing health challenges within this population. The study calls for advocacy and policy changes to enhance transportation options and healthcare access for minority groups at risk for health complications.
Chapters 1 and 2 Summary
Chapters 1 and 2 Summarylivvy milner This document discusses the rise of early civilizations from prehistoric times to around 200 BC. It begins by explaining what archaeology and anthropology teach us about prehistoric humans and the characteristics of early civilizations. Key developments include the emergence of Homo sapiens, the Paleolithic and Neolithic ages, and the first river valley civilizations in Mesopotamia, Egypt, the Indus Valley, and others. The document then examines the spread of civilizations in the eastern Mediterranean, Mexico, Peru, China, and elsewhere up to around 200 BC.
Chapter 7 Summary
Chapter 7 Summarylivvy milner The Roman Republic was established in 509 BC and eventually declined as wealthy landowners took over plebeian land. This led to civil war and the rise of triumvirates including Julius Caesar, Pompey, and Crassus. After Caesar became dictator, he was assassinated, leading to further civil war. Ultimately, Octavian defeated Antony to become the first emperor, ending the republic and establishing the Roman Empire. Rome's success was due to factors like its geography along the Mediterranean, adoption of ideas and technologies from Greeks and Etruscans, strong military, and practical political system.
CHEM2500OutlineSample
CHEM2500OutlineSampleCole French, CET, B.Sc Chemistry Student This document provides information about the CHEM 2500H Elements of Physical Chemistry I course offered in the Fall at Trent University. The course covers quantum mechanics and kinetics, including energy levels of atoms and molecules, molecular orbitals, and reaction rates and mechanisms. It meets on Wednesdays and Fridays, with additional laboratory sessions. Students will complete assignments, a lab, a midterm, and a final exam. The required textbook and schedule of topics are also outlined.
Major 2 p pt
Major 2 p ptRahul Agarwal This document discusses the implementation of data usage mining using the Apriori and FP-Growth algorithms. It describes web usage mining as the automatic discovery of usage patterns from web server logs. The goal is to analyze behavioral patterns and profiles of website users. It then discusses the Apriori and FP-Growth algorithms used to implement the mining and find frequent patterns in the data. Finally, it outlines the deliverables and possibilities for extending the work.
Webinar: Inside Social (April 2016)
Webinar: Inside Social (April 2016)Universum Webinars The document discusses the role of social media in employer branding and recruitment, emphasizing the need for organizations to adapt their strategies to engage with talent through various online channels. It highlights the increasing importance of authentic storytelling and the optimization of social media strategies to differentiate employer brands and enhance engagement with potential candidates. Furthermore, it provides case studies illustrating successful social media campaigns by companies like ExxonMobil and Unilever, demonstrating the effectiveness of targeted campaigns and data-led strategies.
RF 55f - Fondation Raoul Follereau - Rapport Annuel 2008
RF 55f - Fondation Raoul Follereau - Rapport Annuel 2008Bernard hardy Le rapport annuel 2008 de la fondation Raoul Follereau décrit ses missions contre la lèpre et l'exclusion, avec des actions menées dans 38 pays à travers 379 projets. La fondation se concentre sur la guérison, l'éducation et la réinsertion des populations marginalisées, en s'appuyant sur la générosité de ses donateurs. Elle met également l'accent sur l'importance de la transparence et de la bonne gestion des fonds pour accomplir ses objectifs sociaux.
Frontend performance metrics
Frontend performance metricsАртем Захарченко This document discusses Javascript performance metrics and optimization. It covers:
1. Measuring performance is important and should be done often.
2. There are many frontend frameworks and Javascript is the most popular language, leading to performance being critical.
3. Several studies show that even small improvements to page speed can significantly increase user engagement and conversions.
4. The document provides various tools and techniques for measuring and improving performance, including RAIL principles, Lighthouse, WebPageTest, and analyzing network requests, rendering, and computation.
Sunk Costs
Sunk Coststutor2u Sunk costs refer to business costs that cannot be recovered if a firm exits an industry. High sunk costs act as a barrier to entry for new firms because they risk significant losses if they later exit. Examples of sunk costs include asset write-offs, redundancy costs from closing facilities, and loss of goodwill among customers. Industries with low sunk costs like fast food restaurants have low barriers to entry and exit and high competition.
Trabajo ctm
Trabajo ctmMatias ascanio El documento describe las principales tecnologías utilizadas en los estudios ambientales, incluyendo la teledetección, programas de simulación, satélites meteorológicos, SIG, GPS y programas telemáticos de cooperación internacional. Estas tecnologías permiten obtener grandes cantidades de datos ambientales, desarrollar modelos predictivos y tomar mejores decisiones sobre el medio ambiente y los recursos del planeta.
Fixed, variable and Incremental Cost
Fixed, variable and Incremental CostRahat Inayat Ali Fixed costs remain constant regardless of production levels and include expenses like rent, insurance, and salaries. Variable costs change based on production volume and include materials and labor. The total cost is the sum of fixed and variable costs. Incremental cost refers to the additional total cost of increasing production by one unit, and is related to marginal cost, which is the change in total cost from an additional unit of output.
Utterback_Douglas - Resume
Utterback_Douglas - ResumeDouglas Utterback Douglas Utterback has over 10 years of experience managing warehouses and retail spaces totaling over 50,000 square feet. He has led teams of up to 20 employees and volunteers, directing functions like receiving, processing, pricing, and sales. Utterback excels at developing systems and standard operating procedures to increase efficiency and maximize inventory flow and profits. He has strong leadership, customer service, and multi-department coordination skills.
Building a Modern Windows App
Building a Modern Windows AppBrent Edwards The document discusses building modern Windows apps, focusing on the MVVM pattern, universal applications, and dealing with code differences across platforms. It covers essential topics such as navigation, application settings, contracts, and unit testing, highlighting approaches to handle platform variations. The document includes links for further information and resources, along with a call for feedback on the session.
Apriori algorithm
Apriori algorithmAshis Kumar Chanda The document discusses the Apriori algorithm, a fundamental data mining technique used to discover frequent itemsets within large datasets. It explains the algorithm's two main steps: the join step and the prune step, while also addressing its drawbacks such as the generation of huge candidate sets and multiple database scans. Additionally, it outlines several improvement mechanisms and references key literature in the field.
Александр Кашеверов — Коротко про WEB: HTML, CSS, JS.
Александр Кашеверов — Коротко про WEB: HTML, CSS, JS.DataArt Доклад Александра Кашеверова предоставляет обзор основ веб-технологий, включая HTML, CSS и JavaScript, а также их историческое развитие и значимость. Обсуждаются различные инструменты и фреймворки, такие как jQuery, Angular и ReactJS, и их роль в формировании современного веба. В документе также представлены примеры кода и ссылки на дополнительные ресурсы для обучения.
Ad
Similar to Improving performance of apriori algorithm using hadoop (20)
LOAD BALANCING LARGE DATA SETS IN A HADOOP CLUSTER
LOAD BALANCING LARGE DATA SETS IN A HADOOP CLUSTERijdpsjournal This document presents a study on load balancing for large data sets within a Hadoop cluster, focusing on the use of MapReduce to enhance parallel processing and Yarn for task monitoring. It highlights the challenges of processing increasing data volumes and the need for real-time monitoring and efficient task distribution among nodes to optimize server performance. The research aims to propose methods for better load management, ensuring that data processing is handled effectively across a network of connected computers.
MAP/REDUCE DESIGN AND IMPLEMENTATION OF APRIORIALGORITHM FOR HANDLING VOLUMIN...
MAP/REDUCE DESIGN AND IMPLEMENTATION OF APRIORIALGORITHM FOR HANDLING VOLUMIN...acijjournal The paper discusses the design and implementation of the Apriori algorithm using Map/Reduce in a Hadoop distributed computing environment for analyzing voluminous structured data. It highlights the efficiency of processing large data sets using clustered nodes and provides an overview of related work in distributed data mining. The experimental setup, configuration, and results are detailed, showcasing the algorithm's effectiveness in handling complex data analytics tasks.
Cache mechanism to avoid dulpication of same thing in hadoop system to speed ...
Cache mechanism to avoid dulpication of same thing in hadoop system to speed ...eSAT Journals This document proposes mechanisms to improve the efficiency of the Hadoop distributed file system and MapReduce framework. It suggests using locality-sensitive hashing to colocate related files on the same data nodes, which would improve data locality. It also proposes implementing a cache to store the results of MapReduce tasks, so that duplicate computations can be avoided when the same task is run again on the same data. Implementing these mechanisms could help speed up execution times in Hadoop by reducing unnecessary data transmission and repetitive task executions.
BIGDATA- Survey on Scheduling Methods in Hadoop MapReduce Framework
BIGDATA- Survey on Scheduling Methods in Hadoop MapReduce FrameworkMahantesh Angadi This seminar presentation provides an overview of scheduling methods in the Hadoop MapReduce framework. It begins with motivations for using distributed computing for big data and introduces Hadoop and MapReduce. The presentation then surveys several proposed scheduling methods, including static methods like FIFO and adaptive methods like the Fair Scheduler. It summarizes five research papers on scheduling, discussing proposed approaches like optimizing job completion time and learning node capabilities for heterogeneous clusters. The presentation concludes that scheduling algorithms should improve data locality and use prediction to efficiently schedule jobs on heterogeneous Hadoop clusters.
Mining High Utility Patterns in Large Databases using Mapreduce Framework
Mining High Utility Patterns in Large Databases using Mapreduce FrameworkIRJET Journal This document discusses mining high utility patterns from large databases using the MapReduce framework. It proposes using the d2HUP algorithm to efficiently mine high utility patterns from partitioned big data in parallel. The algorithm traverses a reverse set enumeration tree using depth-first search to identify high utility patterns based on a minimum utility threshold, without generating candidates. It partitions the data using MapReduce and mines patterns from each partition individually. The results are then combined to obtain the final high utility patterns. The proposed approach aims to improve efficiency over existing methods that are not scalable to large datasets.
Performance Improvement of Heterogeneous Hadoop Cluster using Ranking Algorithm
Performance Improvement of Heterogeneous Hadoop Cluster using Ranking AlgorithmIRJET Journal This document proposes using a ranking algorithm and sampling algorithm to improve the performance of a heterogeneous Hadoop cluster. The ranking algorithm prioritizes data distribution based on node frequency, so that higher frequency nodes are processed first. The sampling algorithm randomly selects nodes for data distribution instead of evenly distributing across all nodes. The proposed approach reduces computation time and improves overall cluster performance compared to the existing approach of evenly distributing data across nodes of varying sizes. Results show the proposed approach reduces execution time for various file sizes compared to the existing approach.
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCEAM Publications,India The document surveys big data processing techniques using Hadoop and MapReduce, highlighting their significance in managing large datasets that exceed traditional database capabilities. It discusses the architecture of Hadoop, its components, and how data processing is handled through the MapReduce model, emphasizing efficiency and fault tolerance. The paper concludes by addressing challenges in integrating cloud computing with big data processing and outlines the tools developed for effective data management.
IRJET- Analysis for EnhancedForecastof Expense Movement in Stock Exchange
IRJET- Analysis for EnhancedForecastof Expense Movement in Stock ExchangeIRJET Journal This document discusses using machine learning techniques and Hadoop to analyze stock market data and enhance forecasts of expense movements. It proposes training a model using deep learning on data gathered from online sources via APIs. The unstructured data would be cleaned and partitioned using preprocessing before being processed in parallel using Hadoop. The trained model would then be used to predict future stock performance. Related work discussed using ordered weighted averaging to rank baseball players and hedge fund managers based on cross-evaluation scores.
B017320612
B017320612IOSR Journals This document discusses leveraging MapReduce with Hadoop to analyze weather data. It proposes building a data analytical engine using MapReduce on Hadoop to process massive amounts of temperature data from sensors. The document describes implementing MapReduce jobs to analyze National Climatic Data Center temperature data, with mappers filtering and assigning data to key-value pairs and reducers calculating averages, maximums, and minimums on the data. Overall, the document examines using Hadoop and MapReduce to scalably process large volumes of sensor weather data.
IJSRED-V2I3P84
IJSRED-V2I3P84IJSRED This document analyzes ways to improve the efficiency of Hadoop clusters for big data analysis. It first discusses how Hadoop uses the MapReduce framework to parallelize processing of large datasets across clusters of commodity hardware. It then describes three common techniques for providing fault tolerance in Hadoop - data replication, heartbeat messages, and checkpointing and recovery. Data replication stores duplicate copies of data on multiple nodes to allow recovery from failures. Heartbeat messages check that processing nodes remain active. Checkpointing saves processing state periodically to allow restarting from the last checkpoint if a failure occurs. The document aims to suggest methods for improving overall performance and fault tolerance of Hadoop clusters for big data analysis.
IRJET- Deduplication Detection for Similarity in Document Analysis Via Vector...
IRJET- Deduplication Detection for Similarity in Document Analysis Via Vector...IRJET Journal The document discusses techniques for detecting similarity and deduplication in document analysis using vector analysis. It proposes analyzing documents by extracting abstract content, separating words and combining them in a word cloud to determine frequency. This approach aims to identify whether documents are duplicates by analyzing word vectors at the word, sentence and paragraph level while also applying techniques like stemming, stopping words and semantic similarity.
IRJET- Deduplication Detection for Similarity in Document Analysis Via Vector...
IRJET- Deduplication Detection for Similarity in Document Analysis Via Vector...IRJET Journal The document discusses techniques for detecting similarity and deduplication in document analysis using vector analysis. It proposes analyzing documents by extracting abstract content, separating words and combining them in a word cloud to determine frequency. This approach aims to identify whether documents are duplicates by analyzing word vectors at the word, sentence and paragraph level while also applying techniques like stemming, stopping words and semantic similarity.
BIGDATA- Survey on Scheduling Methods in Hadoop MapReduce
BIGDATA- Survey on Scheduling Methods in Hadoop MapReduceMahantesh Angadi The document summarizes a technical seminar presentation on scheduling methods in the Hadoop MapReduce framework. The presentation covers the motivation for Hadoop and MapReduce, provides an introduction to big data and Hadoop, and describes HDFS and the MapReduce programming model. It then discusses challenges in MapReduce scheduling and surveys the literature on existing scheduling methods. The presentation surveys five papers on proposed MapReduce scheduling methods, summarizing the key points of each. It concludes that improving data locality can enhance performance and that future work could consider scheduling algorithms for heterogeneous clusters.
Review: Data Driven Traffic Flow Forecasting using MapReduce in Distributed M...
Review: Data Driven Traffic Flow Forecasting using MapReduce in Distributed M...AM Publications This paper discusses a data-driven traffic flow forecasting system utilizing a MapReduce framework and Bayesian network approach for distributed modeling. It highlights the need for efficient data processing and storage in urban traffic management and proposes solutions via Gaussian Mixture Models and Expectation Maximization algorithms. The use of Hadoop for handling large traffic datasets is emphasized, illustrating its benefits in computational efficiency and scalability.
REVIEW: Frequent Pattern Mining Techniques
REVIEW: Frequent Pattern Mining TechniquesEditor IJMTER This document reviews various frequent pattern mining techniques vital for extracting interesting trends from large datasets, emphasizing methods such as Apriori and FP-tree algorithms. It discusses the challenges of setting minimum support thresholds and explores advancements in parallel and distributed mining across different computing environments. The aim is to develop efficient, scalable mining techniques suitable for high-volume data analysis.
Big Data Processing with Hadoop : A Review
Big Data Processing with Hadoop : A ReviewIRJET Journal 1. This document provides an overview of big data processing with Hadoop. It defines big data and describes the challenges of volume, velocity, variety and variability.
2. Traditional data processing approaches are inadequate for big data due to its scale. Hadoop provides a distributed file system called HDFS and a MapReduce framework to address this.
3. HDFS uses a master-slave architecture with a NameNode and DataNodes to store and retrieve file blocks. MapReduce allows distributed processing of large datasets across clusters through mapping and reducing functions.
IJET-V2I6P25
IJET-V2I6P25IJET - International Journal of Engineering and Techniques This document provides a comprehensive review of big data and its analysis using the Hadoop framework, emphasizing the challenges associated with data processing, particularly focusing on the joining strategies in MapReduce programming. It discusses the concepts of reduce-side and map-side joining approaches, their implementation, and issues like data skewness that can impact performance. The paper outlines various optimization techniques and highlights the necessity for further advancements in big data analytics and frameworks.
B1803031217
B1803031217IOSR Journals This document provides a survey of distributed heterogeneous big data mining adaptation in the cloud. It discusses how big data is large, heterogeneous, and distributed, making it difficult to analyze with traditional tools. The cloud helps overcome these issues by providing scalable infrastructure on demand. However, directly applying Hadoop MapReduce in the cloud is inefficient due to its assumption of homogeneous nodes. The document surveys different approaches for improving MapReduce performance in heterogeneous cloud environments through techniques like optimized task scheduling and resource allocation.
Big Data with Hadoop – For Data Management, Processing and Storing
Big Data with Hadoop – For Data Management, Processing and StoringIRJET Journal This document discusses big data and Hadoop. It begins with defining big data and explaining its characteristics of volume, variety, velocity, and veracity. It then provides an overview of Hadoop, describing its core components of HDFS for storage and MapReduce for processing. Key technologies in Hadoop's ecosystem are also summarized like Hive, Pig, and HBase. The document concludes by outlining some challenges of big data like issues of heterogeneity and incompleteness of data.
Multi-Cloud Services
Multi-Cloud ServicesIRJET Journal This document discusses the design and implementation of an automation framework for data management and storage on cloud platforms using automation tools like Ansible. It proposes using Hadoop for large-scale data distribution and storage, AWS cloud for infrastructure provisioning, and Ansible for automating the deployment and management of the Hadoop cluster. The methodology involves requirements analysis, system design, implementation, and testing. The results indicate the system meets specifications by automatically creating cloud storage spaces as needed by users. The framework has potential for future extension by integrating a front-end interface and additional features.
Ad
More from eSAT Journals (20)
Mechanical properties of hybrid fiber reinforced concrete for pavements
Mechanical properties of hybrid fiber reinforced concrete for pavementseSAT Journals The study investigates the mechanical properties of hybrid fiber reinforced concrete (HFRC) by adding steel and polypropylene fibers to improve its performance. Results indicate that HFRC enhances compressive, split tensile, and flexural strengths significantly compared to mono fiber solutions, with notable improvements of 17%, 52.87%, and 34.25%, respectively. This research highlights the advantages of fiber hybridization in addressing concrete's weaknesses in tensile strength and ductility, making it more suitable for pavement applications.
Material management in construction – a case study
Material management in construction – a case studyeSAT Journals This case study examines the challenges and improvements in material management within construction projects, highlighting issues such as procurement delays and quality control. It investigates methodologies for inventory control, purchasing procedures, and cost analysis, suggesting the use of technologies like RFID for better tracking and management. The findings advocate for a restructured management approach to enhance efficiency and reduce costs through proper resource allocation and the integration of advanced data management tools.
Managing drought short term strategies in semi arid regions a case study
Managing drought short term strategies in semi arid regions a case studyeSAT Journals The document discusses drought management strategies in Gulbarga, Karnataka, highlighting the need for multidisciplinary approaches to address the recurring issue of drought in the semi-arid region. It examines the geographical, demographic, and agricultural characteristics of Gulbarga, emphasizing the impact of droughts on livelihoods and the importance of integrated water management and relief measures. The paper advocates for short-term strategies, including labor-oriented schemes and effective planning to mitigate the effects of drought on vulnerable populations.
Life cycle cost analysis of overlay for an urban road in bangalore
Life cycle cost analysis of overlay for an urban road in bangaloreeSAT Journals The document presents a life cycle cost analysis of overlay methods for urban roads in Bangalore, focusing on concrete and bituminous pavements. It evaluates the structural adequacy of existing pavements, discusses various fiber-reinforced concrete designs, and compares their costs using net present value methods. Findings indicate that while concrete overlays have higher initial costs, they can be more economical over time compared to bituminous overlays.
Laboratory studies of dense bituminous mixes ii with reclaimed asphalt materials
Laboratory studies of dense bituminous mixes ii with reclaimed asphalt materialseSAT Journals The study investigates the use of reclaimed asphalt pavement (RAP) in dense bituminous mixes to address India's growing roadway demands and budget constraints, aiming to rehabilitate existing pavements sustainably. Experimental results show that mixing 20% and 30% RAP with virgin aggregates yields higher stability and reduces costs by 19% and 30%, respectively, compared to conventional mixes. The findings support recycling as a practical solution to improve pavement performance while conserving resources and lowering construction expenses.
Laboratory investigation of expansive soil stabilized with natural inorganic ...
Laboratory investigation of expansive soil stabilized with natural inorganic ...eSAT Journals This document summarizes a study on stabilizing expansive black cotton soil with the natural inorganic stabilizer RBI-81. Laboratory tests were conducted to evaluate the effect of RBI-81 on the soil's engineering properties. The tests showed that with 2% RBI-81 and 28 days of curing, the unconfined compressive strength increased by around 250% and the CBR value improved by approximately 400% compared to the untreated soil. Overall, the study found that RBI-81 effectively improved the strength properties of the black cotton soil and its suitability as a soil stabilizer was supported.
Influence of reinforcement on the behavior of hollow concrete block masonry p...
Influence of reinforcement on the behavior of hollow concrete block masonry p...eSAT Journals The document presents a study on the influence of reinforcement on the behavior of hollow concrete block masonry prisms under compression, employing both experimental and analytical approaches. It details the development of a three-dimensional finite element model using ANSYS to predict ultimate compressive strength and validate findings through comparison with experimental results. The research concludes that the introduction of reinforcement significantly enhances the axial load capacity and ductility of masonry prisms.
Influence of compaction energy on soil stabilized with chemical stabilizer
Influence of compaction energy on soil stabilized with chemical stabilizereSAT Journals This document summarizes a study on the influence of compaction energy on soil stabilized with a chemical stabilizer. Laboratory tests were conducted on locally available loamy soil treated with a patented polymer liquid stabilizer and compacted at four different energy levels. The study found that increasing the compaction effort increased the density of both untreated and treated soil, but the rate of increase was lower for stabilized soil. Treating the soil with the stabilizer improved its unconfined compressive strength and resilient modulus, and reduced accumulated plastic strain, with these properties further improved by higher compaction efforts. The stabilized soil exhibited strength and performance benefits compared to the untreated soil.
Geographical information system (gis) for water resources management
Geographical information system (gis) for water resources managementeSAT Journals This document describes a hydrological framework developed in the form of a Hydrologic Information System (HIS) to meet the information needs of various government departments related to water management in a state. The HIS consists of a hydrological database coupled with tools for collecting and analyzing spatial and non-spatial water resources data. It also incorporates a hydrological model to indirectly assess water balance components over space and time. A web-based GIS portal was created to allow users to access and visualize the hydrological data, as well as outputs from the SWAT hydrological model. The framework is intended to facilitate integrated water resources planning and management across different administrative levels.
Forest type mapping of bidar forest division, karnataka using geoinformatics ...
Forest type mapping of bidar forest division, karnataka using geoinformatics ...eSAT Journals The study explores the use of satellite remote sensing techniques to generate forest type maps and assess tree plantation in the Bidar forest division, Karnataka, covering an area of 5814.60 sq.kms. It reveals that approximately 84% of the area is agricultural land, with only 1.778% comprised of dry deciduous forest and 1.38% mixed plantation, raising concerns about environmental stability. The research highlights the significance of GIS technologies for future plantation efforts, emphasizing the importance of maintaining forest coverage for ecological balance.
Factors influencing compressive strength of geopolymer concrete
Factors influencing compressive strength of geopolymer concreteeSAT Journals This document investigates the factors affecting the compressive strength of fly ash-based geopolymer concrete, focusing on sodium hydroxide (NaOH) concentration, the ratio of NaOH to sodium silicate, and the alkaline liquid to fly ash ratio. The research indicates that higher NaOH concentrations lead to improved compressive strength, while increasing the alkaline liquid to fly ash ratio results in diminished strength. The use of superplasticizers also enhances workability, with water replacement impacting compressive strength negatively.
Experimental investigation on circular hollow steel columns in filled with li...
Experimental investigation on circular hollow steel columns in filled with li...eSAT Journals This document presents an experimental investigation of circular hollow steel columns infilled with lightweight concrete, with and without GFRP, under cyclic loading. The study utilizes artificial neural networks (ANN) to predict the ultimate load-carrying capacity and axial shortening of the specimens, highlighting the impact of different variables such as cross-sectional area and length of the steel tube. The findings indicate that ANN modeling can effectively predict load capacities for composite columns, facilitating the design of seismic-resistant structures.
Experimental behavior of circular hsscfrc filled steel tubular columns under ...
Experimental behavior of circular hsscfrc filled steel tubular columns under ...eSAT Journals This document summarizes an experimental study that tested circular concrete-filled steel tube columns with varying parameters. 45 specimens were tested with different fiber percentages (0-2%), tube diameter-to-wall-thickness ratios (D/t from 15-25), and length-to-diameter (L/d) ratios (from 2.97-7.04). The results found that columns filled with fiber-reinforced concrete exhibited higher stiffness, equal ductility, and enhanced energy absorption compared to those filled with plain concrete. The load carrying capacity increased with fiber content up to 1.5% but not at 2.0%. The analytical predictions of failure load closely matched the experimental values.
Evaluation of punching shear in flat slabs
Evaluation of punching shear in flat slabseSAT Journals The document discusses the evaluation of punching shear in flat slabs, which are commonly used in construction due to their advantages. It emphasizes the need to consider unbalanced moments at slab-column junctions, which can lead to critical failures, and presents generalized design sheets in accordance with IS 456-2000 to assist engineers in calculating necessary shear reinforcement. The study concludes that the developed design sheets help in assessing punching shear and can guide the initial design phase for flat slabs.
Evaluation of performance of intake tower dam for recent earthquake in india
Evaluation of performance of intake tower dam for recent earthquake in indiaeSAT Journals The document evaluates the dynamic behavior of intake towers during earthquakes, highlighting the impact of parameters such as depth of submergence and wall thickness. It involves a comprehensive analysis using a 3-dimensional model that incorporates fluid-structure interaction and conducts modal and time history analyses based on seismic data from India. The findings aim to enhance the understanding of the structural response of intake towers to seismic activities.
Evaluation of operational efficiency of urban road network using travel time ...
Evaluation of operational efficiency of urban road network using travel time ...eSAT Journals This document evaluates the operational efficiency of an urban road network in Tiruchirappalli, India using travel time reliability measures. Traffic volume and travel times were collected using video data from 8-10 AM on various roads. Average travel times, 95th percentile travel times, and buffer time indexes were calculated to assess reliability. Non-motorized vehicles were found to most impact reliability on one road. A relationship between buffer time index and traffic volume was developed. Finally, a travel time model was created and validated based on length, speed, and volume.
Estimation of surface runoff in nallur amanikere watershed using scs cn method
Estimation of surface runoff in nallur amanikere watershed using scs cn methodeSAT Journals The document discusses the estimation of surface runoff in the Nallur Amanikere watershed using the Soil Conservation Service Curve Number (SCS-CN) method, integrating remote sensing and GIS techniques for data analysis. Key findings indicate that runoff potential varies with different land use/land cover and soil conditions, with maximum runoff estimated at 1869.57 mm in 2005. The study highlights the significance of GIS in watershed management and proposes conservation techniques for sustainable resource development.
Estimation of morphometric parameters and runoff using rs & gis techniques
Estimation of morphometric parameters and runoff using rs & gis techniqueseSAT Journals This document summarizes a study that used remote sensing and GIS techniques to estimate morphometric parameters and runoff for the Yagachi catchment area in India over a 10-year period. Morphometric analysis was conducted to understand the hydrological response at the micro-watershed level. Daily runoff was estimated using the SCS curve number model. The results showed a positive correlation between rainfall and runoff. Land use/land cover changes between 2001-2010 were found to impact estimated runoff amounts. Remote sensing approaches provided an effective means to model runoff for this large, ungauged area.
Effect of variation of plastic hinge length on the results of non linear anal...
Effect of variation of plastic hinge length on the results of non linear anal...eSAT Journals This paper examines the sensitivity of pushover analysis results to variations in plastic hinge lengths using different formulations and geometric models of a G+2 storied RCC framed structure. It highlights significant discrepancies between analytical predictions and experimental results, showing that the choice of hinge properties and geometric models substantially affects base shear and displacement outcomes. The study concludes that a more effective analytical methodology is needed for better alignment with experimental pushover curves.
Effect of use of recycled materials on indirect tensile strength of asphalt c...
Effect of use of recycled materials on indirect tensile strength of asphalt c...eSAT Journals The study investigates the impact of using recycled materials, specifically Reclaimed Asphalt Pavement (RAP) and foundry waste, on the indirect tensile strength of asphalt concrete mixes. Laboratory tests showed that the use of 60% RAP resulted in higher stability values compared to conventional mixes, although the optimum binder content decreased with increased RAP. This research emphasizes the benefits of recycling asphalt pavements, which contributes to environmental conservation and resource efficiency in road construction.
Recently uploaded (20)
Generative AI & Scientific Research : Catalyst for Innovation, Ethics & Impact
Generative AI & Scientific Research : Catalyst for Innovation, Ethics & ImpactAlqualsaDIResearchGr Invited keynote at the Artificial Intelligence Symposium on AI-powered Research Innovation, taking place at ENSEM (L'École Nationale Supérieure d'Électricité et de Mécanique), Casablanca on June 21, 2025. I’ll be giving a keynote titled: "Generative AI & Scientific Research: Catalyst for Innovation, Ethics & Impact". Looking forward to engaging with researchers and doctoral students on how Generative AI is reshaping the future of science, from discovery to governance — with both opportunities and responsibilities in focus.
#AI hashtag#GenerativeAI #ScientificResearch #Innovation #Ethics #Keynote #AIinScience #GAI #ResearchInnovation #Casablanca
1. Thinking, Creative Thinking, Innovation
2. Societies Evolution from 1.0 to 5.0
3. AI - 3P Approach, Use Cases & Innovation
4. GAI & Creativity
5. TrustWorthy AI
6. Guidelines on The Responsible use of GAI In Research
Introduction to Natural Language Processing - Stages in NLP Pipeline, Challen...
Introduction to Natural Language Processing - Stages in NLP Pipeline, Challen...resming1 Lecture delivered in 2021. This gives an introduction to Natural Language Processing. It describes the use cases of NLP in daily life. It discusses the stages in NLP Pipeline. It highlights the challenges involved covering the different levels of ambiguity that could arise. It also gives a brief note on the present scenario with the latest language models, tools and frameworks/libraries for NLP.
Tesla-Stock-Analysis-and-Forecast.pptx (1).pptx
Tesla-Stock-Analysis-and-Forecast.pptx (1).pptxmoonsony54 this is data science ppt for tesla stock (linear regression)
دراسة حاله لقرية تقع في جنوب غرب السودان
دراسة حاله لقرية تقع في جنوب غرب السودانمحمد قصص فتوتة دراسة حاله لقرية تقع في جنوب غرب السودان لتوفير إحتياجاتها من الطاقه من مصادر الطاقات المتجدده المتوفره بها
Microwatt: Open Tiny Core, Big Possibilities
Microwatt: Open Tiny Core, Big PossibilitiesIBM Microwatt is a lightweight, open-source core based on the OpenPOWER ISA.
It’s designed for FPGAs and easy experimentation in chip design.
Ideal for education, prototyping, and custom silicon development.
Fully open, it empowers developers to learn, modify, and innovate.
Rapid Prototyping for XR: Lecture 5 - Cross Platform Development
Rapid Prototyping for XR: Lecture 5 - Cross Platform DevelopmentMark Billinghurst This is lecture 5 in the course about Rapid Prototyping for XR, taught by Mark Billinghurst. This lecture is about Cross Platform Development.
Deep Learning for Image Processing on 16 June 2025 MITS.pptx
Deep Learning for Image Processing on 16 June 2025 MITS.pptxresming1 This covers how image processing or the field of computer vision has advanced with the advent of neural network architectures ranging from LeNet to Vision transformers. It covers how deep neural network architectures have developed step-by-step from the popular CNNs to ViTs. CNNs and its variants along with their features are described. Vision transformers are introduced and compared with CNNs. It also shows how an image is processed to be given as input to the vision transformer. It give the applications of computer vision.
Proposal for folders structure division in projects.pdf
Proposal for folders structure division in projects.pdfMohamed Ahmed Proposal for folders structure division in projects
Complete guidance book of Asp.Net Web API
Complete guidance book of Asp.Net Web APIShabista Imam Unlock the full potential of modern web development with the Complete Guidance Book of ASP.NET Web API—your all-in-one resource for mastering RESTful services using Microsoft’s powerful ASP.NET Core framework. This book takes you on a step-by-step journey from beginner to expert, covering everything from routing and controllers to security, performance optimization, and real-world architecture.
Structured Programming with C++ :: Kjell Backman
Structured Programming with C++ :: Kjell BackmanShabista Imam Step into the world of high-performance programming with the Complete Guidance Book of C++ Programming—a definitive resource for mastering one of the most powerful and versatile languages in computer science.
Whether you're a beginner looking to learn the fundamentals or an intermediate developer aiming to sharpen your skills, this book walks you through C++ from the ground up. You'll start with basics like variables, control structures, and functions, then progress to object-oriented programming (OOP), memory management, file handling, templates, and the Standard Template Library (STL).
MATERIAL SCIENCE LECTURE NOTES FOR DIPLOMA STUDENTS
MATERIAL SCIENCE LECTURE NOTES FOR DIPLOMA STUDENTSSAMEER VISHWAKARMA This PPT contains notes for taking lecture at diploma level.
May 2025: Top 10 Read Articles in Data Mining & Knowledge Management Process
May 2025: Top 10 Read Articles in Data Mining & Knowledge Management ProcessIJDKP Data mining and knowledge discovery in databases have been attracting a significant amount of research, industry, and media attention of late. There is an urgent need for a new generation of computational theories and tools to assist researchers in extracting useful information from the rapidly growing volumes of digital data.
This Journal provides a forum for researchers who address this issue and to present their work in a peer-reviewed open access forum. Authors are solicited to contribute to the Journal by submitting articles that illustrate research results, projects, surveying works and industrial experiences that describe significant advances in the following areas, but are not limited to these topics only.
Improving performance of apriori algorithm using hadoop
- 1. IJRET: International Journal of Research in Engineering and Technology eISSN: 2319-1163 | pISSN: 2321-7308 _______________________________________________________________________________________ Volume: 03 Issue: 12 | Dec-2014, Available @ http://www.ijret.org 277 IMPROVING PERFORMANCE OF APRIORI ALGORITHM USING HADOOP Ravindra Bachate1 , Hyder Ali Hingoliwala2 1 Department of Computer Engineering, JSPM’s JSCOE, Pune, Maharashtra, India 2 Department of Computer Engineering, JSPM’s JSCOE, Pune, Maharashtra, India Abstract Spatial data is a data having a geological information. This paper explores the use of Hadoop framework to improve the performance of Apriori algorithm for spatial data mining. FP growth algorithm is better than Apriori but it fails in certain situations. By applying the Apriori algorithm parallely using Hadoop framework to spatial data, we can perform well as compare to FP growth. This paper includes clustering based on geological location, classification based on mineral resource type and spatial coherence between mineral resources. Spatial data mining find out the different association rules by observing the spatial data by using Apriori algorithm. The result of the paper will indicate the accurate prediction of occurrence of commodity with respect to other commodity of mineral resources. Keywords: Hadoop, data mining, association rules, clustering, spatial coherence --------------------------------------------------------------------***---------------------------------------------------------------------- 1. INTRODUCTION FP growth algorithm is a very popular algorithm used for the association mining due its performance and less storage requirement. With these pros, it has some limitations also which are considered rarely. This paper focuses on the limitations of FP growth association mining algorithm and trying to suggest improvements in Apriori algorithm which can overcome the limitations of FP Growth association mining algorithm. As we aware about the hardware cost, it’s decreasing day by day, so there is no need to concentrate more on storage requirement. FP growth has two limitations – 1. It is difficult to use for interactive mining where the user may change support value as per requirement. 2. It is not suitable where the data has been increasing with time. So to deal with limitations of FP growth, this paper suggest to implement Apriori algorithm using Hadoop for association rule mining. 1.1 Mineral Resources Data System Mineral Resources Data System is a collection of data describing metallic and nonmetallic mineral resources in the world [7]. It includes resource name, location, commodity, geologic characteristics, resource description, production, reserves, and references. As MRDS contains mineral resources data around the world, it is large and complex. If data size goes beyond the Tera Byte, it is difficult to process and mine using FP growth algorithm. The performance of FP growth algorithm hampers when adding the new records and also by changing the support value. As the mineral resources data set is collected from various regions and people, we need to perform ETL (extract, transform and load) operations on the data for processing and mining. 1.2 Hadoop Map Reduce To deal with unstructured and big data like mineral resources data system (MRDS), we need a best technology which can cope with it. There are two options available, parallel DBMS and Hadoop Map Reduce technology. Hadoop has another projects also which can be used for the mining purpose like Hive. But internally again all the projects using Hadoop Map Reduce technique [5]. It gives a better data processing performance with minimum cost and time as compare to parallel DBMS because it works with commodity hardware. Hadoop stores data in the form of blocks on Hadoop Distributed File System i.e. HDFS. The Hadoop framework provides a solution for problems of massive data processing; because it runs applications on large cluster built of commodity hardware with failure tolerance [4].Unstructured data can be processed with Hadoop Map Reduce technique which is not possible with RDBMS. Map Reduce provides flexibility and fault tolerance which is not with parallel DBMS. Map Reduce provides automatic parallelization, data partitioning, task scheduling, handling machine failures and manages inter- machine communication. Hadoop is totally transparent from the end user. The rate of growing an unstructured data is much more as compare to the structured data. The unstructured data includes media files, heavy text files, csv files, log files etc. 2. RELATED WORK Association rule mining algorithm includes to find coverage, support, confidence, lift and interesting [4]. Coverage defines the proportion of case data specified on the Left Haft Hand Side of the rule. Support of an association rule means percentage of task relevant data for which the pattern is true. Confidence gives the trustworthiness associated with the patterns discovered. Lift is a measure of the importance of the association. The term interesting gives the strength of associations between sets of items in the association rule.
- 2. IJRET: International Journal of Research in Engineering and Technology eISSN: 2319-1163 | pISSN: 2321-7308 _______________________________________________________________________________________ Volume: 03 Issue: 12 | Dec-2014, Available @ http://www.ijret.org 278 Hongyong Yu, Deshuai Wang [1] proposed a system for data processing and mining log data of SaaS cloud using Hadoop. This paper focuses on Hadoop’s Map Reduce technique and the algorithm used for data mining by Hongyong Yu, Deshuai Wang. They suggest by applying Apriori algorithm concurrently in the distributed system, performance of Apriori algorithm can be increased in proportional to the number of nodes in the distributed system [1]. Every day, 2.5 quintillion bytes of data are created and 90 percent of the data in the world today were produced within the past two years [3].The commonly used software technology cannot cope with massive data and big challenge is to extract an important information from it. Big data has large volume, heterogeneous format and decentralized data control. The example of big data applications are Facebook, Twitter and Google. It is a big challenge to manage and mining a massive data because of its volume, different file formats and growing rate of the data in the world. There are many challenges with big data such as storage, processing, variety and cost. 3. PROPOSED SYSTEM This paper proposes a system which overcomes the problem faced in FP growth association mining algorithm. To improve the performance of Apriori algorithm, it is implemented parallely using Hadoop map reduce technique. The proposed system has three modules 1. Spatial Clustering 2. Spatial Classification 3. Spatial Coherence Before implementing these modules, we need to perform Extract, Transfer and Load operations on the raw data. Because the data available may be in the various form and having an unnecessary information into to it. So first we need to extract the required data from raw data and then transfer it in to the csv format. To process this data, it should be loaded on to the HDFS. Fig-1 Proposed System 3.1 Spatial Clustering There are various algorithms available for clustering like k means but here in this paper, Hadoop partition technique is used to perform spatial clustering based on the geological location like Alaska, California etc[2]. It runs parallel on different nodes concurrently. Because of this, time required for spatial clustering is less as compared to k-means clustering algorithm. In this paper, U.S. mineral resources dataset is taken. In effect, there are 50 clusters formed with respect to the states in United State. These clusters have all the records which belongs to respective state. Intention behind to form the clusters is to make it specialize and reduce the processing time and cost. 3.2 Spatial Classification Here, the same technique is used for implementing a spatial classification which is used for spatial clustering. In this module, again the data set is clustered according to the type of commodity which is required for the next module. In spatial classification, the records having same commodity put in to the separate cluster to make it more specialize. It means, e.g. Alaska state may have another sub clusters with commodity type like gold, silver, copper etc. 3.3 Spatial Coherence Objective of this paper is to find out the spatial coherence. Spatial coherence means if we are mining for gold then what are the possibilities of getting other mineral resources at the same location. For this, association mining is required. As we have discussed above, FP Growth algorithms fails in two situations. So to overcome this, this paper introduces parallel Apriori algorithm for spatial association mining. In Apriori algorithm, first of all, we have to find out the candidate keys. After this, we have to perform two task, one is to find out the support values and second is to find out the confidence. Formula’s to calculate a support and confidence for gold with copper.
- 3. IJRET: International Journal of Research in Engineering and Technology eISSN: 2319-1163 | pISSN: 2321-7308 _______________________________________________________________________________________ Volume: 03 Issue: 12 | Dec-2014, Available @ http://www.ijret.org 279 1. Support Formula Where N is all the records in a cluster 2. Confidence Formula Here, a formulas for calculating support and confidence of gold is given. Support of (gold => copper) gives the probability of having gold with copper in all the records whereas confidence gives the probability of having of gold with copper with respect to all the gold records in a cluster. The value of support is always less than the confidence. 4. RESULTS To implement this idea, we have taken a dataset of MRDS which is a spatial data set of mineral resources in U.S.[7]. This sample data has around 3.5 lack records. Table -1: Support and Confidence Commodity Support (%) Confidence (%) (Gold -> Copper) 2.89 9.30 (Gold -> Copper, Lead, Silver) 0.72 2.3 (Gold -> Silver) 2.17 6.97 (Gold -> null) 24.63 79.06 Here, the Apriori algorithm is performed on single node and multi node Hadoop to compare the performance. Also we find the association rules for this spatial data for a single cluster. Table -2: Execution on Single and Multinode System No. of Node Execution Time in Sec 1 5.2 2 3.1 3 2.3 The result in Table-1 shows the different association rules with respect to gold.Table-2 shows the performance of Apriori algorithm using single node and multi node system. It is observed that if we implement the Apriori Algorithm parallel on Hadoop, performance is improved. 5. CONCLUSION FP growth algorithm has two limitations, it cannot be used for dynamic data size and where the support needs to change according to situation. By applying the Apriori Algorithm concurrently using Hadoop, we can overcome the problems faced by FP growth association mining algorithm. Also we can improve the speed of association rule mining for spatial data as compare to the FP growth algorithm as Hadoop is a distributed system. ACKNOWLEDGEMENTS I express true sense of gratitude towards my project guide Prof. H.A. Hingoliwala, Associate Professor Computer Department for his invaluable co-operation and guidance that he gave me throughout my project. I specially thank our P.G coordinator Prof. M. D. Ingle for inspiring me and providing me all the lab facilities. I would also like to express my appreciation and thanks to HOD Prof. S.M. Shinde & JSCOE Principal Dr. M.G. Jadhav and all my friends who knowingly or unknowingly have assisted me throughout my hard work REFERENCES [1]. Hongyong Yu, Deshuai Wang, “Mass Log Data Processing and Mining Based on Hadoop and Cloud Computing” .The 7th International Conference on Computer Science & Education (ICCSE 2012)July 14-17, 2012. Melbourne, Australia. [2]. Duck-Ho Bae Coll. of Inf. & Commun., Hanyang Univ., Seoul, South Korea Ji-Haeng Baek ; Hyun-Kyo Oh ; Ju- Won Song ; Sang-Wook Kim, “SD-Miner: A SPATIAL DATA MINING SYSTEM” Network Infrastructure and Digital Content, 2009. [3]. Xindong Wu,Fellow, IEEE,Xingquan Zhu, Senior Member, IEEE, Gong-Qing Wu, and Wei Ding,Senior Member, IEEE, “Data mining with big data,” IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 26, NO. 1, JANUARY 2014. [4]. Binbin He, Ying Cui, Jianhua Chen, Pingjing Xie, “A Spatial Data Mining Method for Mineral Resources Potential Assessment,” IEEE 978-1-4244-8351- 8/11,2011IEEE. [5]. Hadoop: The definitive Guide, 3rd ed., O’Reilly, Tom White, 2012 [6]. Hadoop, http://hadoop.apache.org/ [7]. MRDS, http://tin.er.usgs.gov/mrds/