FiDoop: Parallel Mining of Frequent Itemsets Using MapReduce

FiDoop: Parallel Mining of Frequent Itemsets Using
MapReduce
Dr G Krishna Kishore1
Suresh Babu Dasari2
Computer Science and Engineering Computer Science and Engineering
V. R. Siddhartha Engineering College V. R. Siddhartha Engineering College
Vijayawada, Andhra Pradesh, India Vijayawada, Andhra Pradesh, India
gkk@vrsiddhartha.ac.in dasarisuresh88@gmail.com
S. Ravi Kishan3
Computer Science & Engineering
V.R.Siddhartha Engineering College
Vijayawada, Andhra Pradesh
suraki@vrsiddhartha.ac.in
Abstract: Existing parallel digging calculations for
visit itemsets do not have a component that
empowers programmed parallelization, stack
adjusting, information conveyance, and adaptation
to non-critical failure on substantial bunches. As an
answer for this issue, we outline a parallel incessant
itemsets mining calculation called FiDoop utilizing
the MapReduce programming model. To
accomplish compacted capacity and abstain from
building contingent example bases, FiDoop joins
the incessant things Ultrametric tree, as opposed to
ordinary FP trees. In FiDoop, three MapReduce
occupations are actualized to finish the mining
undertaking. In the essential third MapReduce
work, the mappers autonomously disintegrate
itemsets, the reducers perform mix activities by
building little Ultrametric trees, and the genuine
mining of these trees independently. We actualize
FiDoop on our in-house Hadoop group. We
demonstrate that FiDoop on the group is touchy to
information dissemination and measurements, in
light of the fact that itemsets with various lengths
have diverse decay and development costs. To
enhance FiDoop's execution, we build up a
workload adjust metric to quantify stack adjust
over the group's registering hubs. We create
FiDoop-HD, an augmentation of FiDoop, to
accelerate the digging execution for high-
dimensional information investigation. Broad tests
utilizing genuine heavenly phantom information
exhibit that our proposed arrangement is productive
and versatile.
Keywords - MapReduce, Frequent Itemsets Mining,
Hadoop, Ultrametric, Celestial Spectral Data.
1. Introduction:
Visit Itemsets Mining (FIM) is a center issue in
affiliation run mining (ARM), succession mining,
and so forth. Accelerating the procedure of FIM is
basic and basic, on the grounds that FIM utilization
represents a critical segment of mining time
because of its high calculation and
information/yield (I/O) power. At the point when
datasets in present day information mining
applications turn out to be too much substantial,
successive FIM calculations running on a
singlemachine experience the ill effects of
execution disintegration. To address this issue, we
explore how to perform FIM utilizing MapReduce
a broadly embraced programming model for
handling huge datasets by misusing the parallelism
among registering hubs of a group. We
demonstrate to disseminate an extensive dataset
over the group to adjust stack over all bunch hubs,
in this manner enhancing the execution of parallel
FIM.
2. LITERATURE REVIEW
Data mining faces a lot of challenges in the big
data era. Association rule mining algorithm is not
sufficient to process large data sets. Apriori
algorithm has limitations like the high I/O load and
low performance. The FP-Growth algorithm also
has certain limitations like less internal memory.
Mining the frequent itemset in the dynamic
scenarios is a challenging task. A parallelized
approach using the MapReduce framework is also
used to process large data sets .The most efficient
the recent method is the FiDoop using Ultrametric
tree (FIUT) and MapReduce programming model.
FIUT scans the database only twice. FIUT has four
advantages. First: I reduces the I/O overhead as it
scans the database only twice. Second: only
frequent itemsets in each transaction are inserted as
nodes for compressed storage. Third: FIU is
improved way to partition database, which
International Journal of Computer Science and Information Security (IJCSIS),
Vol. 16, No. 5, May 2018
153 https://sites.google.com/site/ijcsis/
ISSN 1947-5500

significantly reduces the search space. Fourth:
frequent itemsets are generated by checking only
leaves of tree rather than traversing entire tree,
which reduces the computing time. The mining of
frequent itemsets is a basic and essential work in
many data mining applications. Frequent itemsets
extraction with frequent pattern and rules boosts
the applications like Association rule mining, co-
relations also in product sale and marketing. In
extraction process of frequent itemsets there are
number of algorithms used like FP-growth, E-clat
etc. But unfortunately these algorithms are
inefficient in distributing and balancing the load,
when it comes across massive data. Automatic
parallelization is also not possible with these
algorithms. To defeat these issues of existing
algorithms there is need to construct an algorithm
which will support the missing features, such as
automatically parallelization, balancing and good
distribution of data. This paper is focusing on an
efficient methodology to extract frequent itemsets
with the popular MapReduce approach. This new
methodology consist an algorithm which is build
using Modified Apriori algorithm, called as
Frequent Itemset Mining using Modified Apriori
(FIMMA) Technique. This methodology works
with three mappers, independently and
concurrently by using the decompose strategy. The
result of these mappers will be given to the
reducers using the hash table method. Reducer
gives the top most frequent itemsets.
3. Proposed System
In Proposed System a new data partitioning method
to well balance computing load among the cluster
nodes; we develop FiDoop-HD, an extension of
FiDoop, to meet the needs of high dimensional data
processing.
Step 1: Count the occurrence of each item.
Figure 3.1:Frequency of each item
Step 2: We start making pairs out of the
frequent itemsets we got in the above step.
Figure 3.2:Frequent item sets pairs.
Step 3: After getting the frequent Item Pairs, we
start counting the occurrence of these pairs in the
Transaction Set.
Figure 3.3:Frequency of itemset pairs
Step 4: Make combinations of triples using the
frequent Item pairs.
To make triples, the rule is: IF 12 and 13 are
frequent, then the triple would be 123. Similarly, if
24 and 26 then triple would be 246.
So, using the above logic and our Frequent Item
Pairs table, we get the below triples:
Figure 3.4:Frequent itemset triplets.
Vol. 16, No. 5, May 2018
ISSN 1947-5500

Step 5: Get the count of the above triples
(Candidates).
Figure 3.5:Frequency of itemsets triplets.
After, this, if we can find quartets, then we find
those and count their occurrence/frequency.
If we had 123, 124, 134, 135, 234 and we wanted
to generate a quartet then it would be 1234 and
1345. And after finding quartet we would have
again got their count of occurrence /frequency and
repeated the same also, until the Frequent ItemSet
is null.
Thus, the frequent ItemSets are:
- Frequent Itemsets of Size 1: 1, 2, 4, 5, 6
- Frequent Itemsets of Size 2: 14, 24, 25, 45, 46
- Frequent Itemsets of Size 3: 245
3.1 METHODOLOGY
In Proposed System a new data partitioning method
to well balance computing load among the cluster
nodes; we develop FiDoop-HD, an extension of
FiDoop, to meet the needs of high dimensional data
processing. FiDoop is efficient and scalable on
Hadoop clusters.
The proposed system involves the following steps:
 Load the data base into the system.
 Perform mining on all datasets of the
database.
 Calculate the support values and
confidence values of the datasets.
 Sort the elements based on their support
values.
 Set the threshold support value.
 Extract the elements with support values
above threshold.
Approach
1) Finding the Frequent Items: During the
first step, the vertical database is divided
into equally sized blocks (shards) and
distributed to available mappers. Each
mapper extracts the frequent singletons
from its shard. In the reduce phase, all
frequent items are gathered without
further processing.
2) k-FIs Generation: In this second step, Pk,
the set of frequent itemsets of size k, is
generated. First, frequent singletons are
distributed across m mappers. Each of the
mappers finds the frequent k-sized
supersets of the items by running Eclat to
level k. Finally, a reducer assigns Pk to a
new batch of m mappers. Distribution is
done using Round-Robin.
3) Subtree Mining: The last step consists of
mining the prefix tree starting at a prefix
from the assigned batch using Eclat. Each
mapper can complete this step
independently since sub-trees do not
require mutual information.
Figure 3.1.1 Map Reduceprocess
4. IMPLEMENTATION:
Data set: Groceries data set in csv format.
INPUT: Transactions dataset i.e groceries dataset.
OUTPUT: Frequent itemsets
There are three modules in the proposed system.
They are as follows:
MODULE 1:
Vol. 16, No. 5, May 2018
ISSN 1947-5500

The first mapper program would mine the
transaction database by removing infrequent sets.
This output from the map is given to reducer as an
input which would order the frequent itemsets in
descending order and would build a FP tree.
Algorithm:
Input: minsupport, DBi;
Output: FP tree
1. function MAP(key offset, values DBi)
2. //T is the transaction in DBi
3. for all T do
4. items ←split each T;
5. for all item in items do 1. count++ 2. end for
6. output( item, count);
7. end for
8. end function
10. reduce input: (itemset, count )
11. function REDUCE(key item, values count)
12. Items=sort(itemset, count) /*sorts the items in
descending order*/
13. fptree_generation(items); /*generates FP tree */
14. end function
MODULE 2:
The second map - reducer program takes the output
from the second reducer , which would recursively
processes the data and generates a minimum 2 Item
sets using the FiDoopHD algorithm.
Algorithm:
Input: List,
Output:-FP Tree
1. function MAP(List)
2. // M is the size of the List 2. for all (k is from M
to 2) do
3. for all (k-itemset in List) do
4. decompose(k-itemset, k-1, (k-1)-itemsets);
/*Each k-itemset is only decomposed into (k-1)-
itemsets */
5. (k-1)-file ← the decomposed (k-1)-itemsets
6. union the original (k-1)-itemsets in (k-1)-file; 2.
for all (t-itemset in (k-1)-file) do 3. t -FP-tree←t-
FP-tree generation(local-FPtree,t itemset);
8. output(t, t-FP-tree);
9. end for
10. end for
11. end for
12. end function
5. OUTPUT:
The following diagrams shows the implementation
of Fidoop and display of frequent itemsets for the
given datasets.
Figure 5.1 Execution of Fidoop
. Figure 5.2: Generation of Output File and
Success File
Vol. 16, No. 5, May 2018
ISSN 1947-5500

Figure 5.3: Display of Frequent Item Sets
6. CONCLUSION AND FUTURE WORK
To mitigate high communication and reduce
computing cost in MapReduce-based FIM
algorithms, we developed FiDoop-DP, which
exploits correlation among transactions to partition
a large dataset across data nodes in a Hadoop
cluster. FiDoop-DP is able to partition transactions
with high similarity together and group highly
correlated frequent items into a list.
7. REFERENCES
1) Shreedevi C Patil “A Survey on Parallel
Mining of frequent Itemsets in
MapReduce”, International Journal of
Innovative Research in Computer and
Communication Engineering, Volume
4,Issue-6, June,2016.
2) Prajakta G. Kulkarni , S.R.Khonde “An
Improved Technique Of Extracting
Frequent Itemsets From Massive Data
Using MapReduce”, International Journal
of Engineering and Technology ,Volume-
9,July,2017.
3) ShivaniDeshpande,HarshitaPawar,Amruta
Chandras,AmolLanghe “Data Partitioning
in Frequent Itemset Mining on Hadoop
Clusters” , International Research Journal
of Engineering and Technology (IRJET) ,
Volume: 03 Issue: 11 ,November,2016.
4) Divya.M.G,Nandini.K,Priyanka.K.T,Vand
ana.B “Weighted Itemset Mining from Big
Data using Hadoop”, International Journal
of Advanced Networking & Applications
,ISSN: 0975-0282,February,2016.
5) Roger Pressman, titled “Software
Engineering - a practitioner's approach”,
Fifth Edition.
6) Herbert Schildt, titled “The Complete
Reference Java”, Seventh Edition.
7) Tom White, titled “Hadoop: The
Definitive Guide”, Third Edition.
8) Robin Nixon , titled “Learning PHP,
MySQL & JavaScript”.
9) J.des Rivie` res, J.Wiegand “Eclipse: A
platform for integrating development
tools”, IBM SYSTEMS JOURNAL,
Volume: 43, NO 2, 2004.
Vol. 16, No. 5, May 2018
ISSN 1947-5500

FiDoop: Parallel Mining of Frequent Itemsets Using MapReduce

More Related Content

What's hot (19)

Similar to FiDoop: Parallel Mining of Frequent Itemsets Using MapReduce (20)

Recently uploaded (20)

FiDoop: Parallel Mining of Frequent Itemsets Using MapReduce