![What does our Accumulo Iterator
look like?
●
We can re-use Accumulo's Combiner type here:
override def reduce:(key: Key, values:
Iterator[Value]) Value = {
// deserialize and combine all intermediate
// values. This logic should be identical to
// what is in the mr.Combiner and Reducer
}
●
Our function has to be commutative because major
compactions will often pick smaller files to combine,
which means we only see discrete subsets of data in an
iterator invocation](https://image.slidesharecdn.com/xxxtremeaggregation-140926162102-phpapp01/85/Xxx-treme-aggregation-10-320.jpg)
![Counting in practice (pt 2)
With that declaration, we can infer that the user wants their
operation to be summing over each instance of a given
pairing, so we can say the base value is 1 (sounds a bit like
word count, huh?). We need a key for each base value and
partial computation to be reduced with. For this simple
pairing we can have a schema like:
<field_1>0<value_1>0...<field_n>0<value_n> count: “”
[] <serialized long>
I recently traveled from Baltimore to Denver. Here's what that
trip would look like:
origin0bwi0destination0dia count: “” [] x01](https://image.slidesharecdn.com/xxxtremeaggregation-140926162102-phpapp01/85/Xxx-treme-aggregation-12-320.jpg)
![How you like me now?
● Aggregate(
onKeys((“origin”))
producing(TopK(“destination”)))
● <field1>0<value1>0...<fieldN>0<valueN> <op>:
<relation> [] <serialized data structure>
●
Let's use my Baltimore->Denver trip as an
example:
origin0BWI topk: destination [] {“DIA”: 1}](https://image.slidesharecdn.com/xxxtremeaggregation-140926162102-phpapp01/85/Xxx-treme-aggregation-15-320.jpg)
Xxx treme aggregation
- 1. Let's Aggregate XXXXTREME JOE BEDWELL EDITION @BillSlacum Accumulo Meetup, Sep 23, 2014
- 2. Have you heard of... … TSAR, Summingbird? (Twitter) … Mesa? (Google) … Commutative Replicated Data Types? Describe a system that pre-computes aggregations over large datasets using associative and/or commutative functions.
- 3. What's it all about, Alfie? Imagine we have a dataset that describes fights between two cities. We'd at some point run a SQL query similar to SELECT DISTINCT destination FROM flights WHERE origin=”BWI” to see where everyone is feeing from Baltimore. We can pre-compute, in parallel, this answer for systems that have too much data to compute it every time a user queries the DB.
- 4. What do we need to pull this off? We need data structures that can be combined together. Numbers are a trivial example of this, as we can combine two numbers using a function (such as plus and multiply). There are more advanced data structures such as matrices, HyperLogLogPlus, StreamSummary (used for top-k) and Bloom filters that also have this property! val partial: T = op(a, b)
- 5. What do we need to pull this off? We need operations that can be performed in parallel. Associative operations are espoused by Twitter, but for our case operations that are both associative and commutative have the better property that we can get correct results no matter what order we receive the data. Common operations that are associative (summation, set building) are also commutative. op(op(a, b), c) == op(a, op(b, c)) op(a, b) == op(b, a)
- 6. Wait a minute isn't that... You caught me! It's a commutative monoid! From Wolfram: Monoid: A monoid is a set that is closed under an associative binary operation and has an identity element I in S such that for all a in S, Ia=aI=a Commutative Monoid: A monoid that is commutative i.e., a monoid M such that for every two elements a and b in M, ab=ba.
- 7. Put it to work The example we're about to see uses MapReduce and Accumulo. The same can be accomplished using any processing framework that supports map and reduce operations, such as Spark or Storm's Trident interface.
- 8. We need two functions... Map – Takes an input datum and turns into some combinable structure – Like parsing strings to numbers, or creating single element sets for combining Reduce – Combines the merge-able data structures using our associative and commutative function
- 9. Yup, that's all! ● Map will be called on the input data once in a Mapper instance. ● Reduce will be called in a Combiner, Reducer and an Accumulo Iterator! ● The Accumulo Iterator is configured to run on major compactions, minor compactions, and scans ● That's five places the same piece of code gets run-- talk about modularity!
- 10. What does our Accumulo Iterator look like? ● We can re-use Accumulo's Combiner type here: override def reduce:(key: Key, values: Iterator[Value]) Value = { // deserialize and combine all intermediate // values. This logic should be identical to // what is in the mr.Combiner and Reducer } ● Our function has to be commutative because major compactions will often pick smaller files to combine, which means we only see discrete subsets of data in an iterator invocation
- 11. Counting in practice (pt 1) We've seen how to aggregate values together. What's the best way to structure our data and query it? Twitter's TSAR is a good starting point. It allows users to declare what they want to aggregate: Aggregate( onKeys((“origin”, “destination”)) producing(Count)) This describes generating an edge between two cities and calculating a weight for it.
- 12. Counting in practice (pt 2) With that declaration, we can infer that the user wants their operation to be summing over each instance of a given pairing, so we can say the base value is 1 (sounds a bit like word count, huh?). We need a key for each base value and partial computation to be reduced with. For this simple pairing we can have a schema like: <field_1>0<value_1>0...<field_n>0<value_n> count: “” [] <serialized long> I recently traveled from Baltimore to Denver. Here's what that trip would look like: origin0bwi0destination0dia count: “” [] x01
- 13. Counting in practice (pt 3) ● Iterator combines all values that are mapped to the same key ● We encoded the aggregation function into the column family of the key – We can arbitrarily add new aggregate functions by updating a mapping of column family to function and then updating the iterator deployment
- 14. Something more than counting ● Everybody counts, but what about something like top-k? ● The key schema isn't fexible enough to show a relationship between two fields ● We want to know the top-k relationship between origin and destination cities ● That column qualifier was looking awfully blank. It'd be a shame if someone were to put data in it...
- 15. How you like me now? ● Aggregate( onKeys((“origin”)) producing(TopK(“destination”))) ● <field1>0<value1>0...<fieldN>0<valueN> <op>: <relation> [] <serialized data structure> ● Let's use my Baltimore->Denver trip as an example: origin0BWI topk: destination [] {“DIA”: 1}
- 16. But how do I query it? ● This schema is really geared towards point queries ● Users would know exactly which dimensions they were querying across to get an answer – BUENO “What are the top-k destinations for Bill when he leaves BWI?” – NO BUENO “What are all the dimensions and aggregations I have for Bill?”
- 17. Some thoughts to think about it ● Prepare functions – Preparing the input to do things like time bucketing and normalization (Jared Winick's Trendulo) ● Age off – Combining down to a single value means that value represents all historical data. Maybe we don't care about that and would like to age off data after a day/week/month/year. Mesa's batch Ids could be of use here. ● Security labels – Notice how I deftly avoided this topic. We should be able to bucket aggregations based on visibility, but we need a way to express the best way to handle this. Maybe just preserve the input data's security labeling and attach it to the output of our map function?
- 18. FIN (hope this wasn't too hard to read) Comments, suggestions or infammatory messages should be sent to @BillSlacum or [email protected]