Link Analysis in Networks - or - Finding the Terrorists

Link Analysis in
Networks
- or Finding The Terrorists

Friday, 15 November 2013

About James
Mathematician turned
Computer Scientist

lives in London, UK

talks fast
Works for cisco
bad at blogging


Objectives
What is link analysis
history lesson
graph theory basics
network theory concepts
link analysis basics
link analysis in the wild
getting started with link analysis

What is link Analysis (1)
Which nodes are key or central to the network?
Which links can be severed or strengthened to
most effectively impede or enhance the operation
of the network?
Can the existence of undetected links or nodes be
inferred from the known data?
What types of structured groups of entities occur
in the data set?

What is link Analysis (2)
What are the relevant sub-networks within a
much larger network?
Are there similarities in the structure of subparts
of the network that can indicate an underlying
relationship (e.g., modus operandi)?
What data model and level of aggregation best
reveal certain types of links and sub-networks?

Organised Crime
vs
Terrorism


History


G uns, Drugs & Gangs
? - pirates, gangs, bandits and highway-robbers
4BC - Goths and Vandals
1800s - Yakuza, Triad, Maﬁa, Maﬁya
1920s+ - La Cosa Nostra, cartels, ethnocentric
gangs and syndicates, IRA
1970s - ETA
1990s - Al-Qeada
2000s - Anonymous

Japan's three biggest banks
face yakuza links inquiry
Loans to mobsters scandal at
Mizuho prompts wider
investigation into Mitsubishi UFJ
and Sumitomo Mitsui groups

http://www.theguardian.com/world/2013/oct/30/japan-three-biggestbanks-yakuza-links-inquiry

0th Generation


1st Generation
Generally
accepted ﬁrst
formalisation
was in 1975
with the
Anacpapa
Chart of
Harper and
Harris

2nd Generation
GUI software that essentially replicated the
manual and hand-drawn 1st generation tools,
notably:

• i2
• Netmap
• Crimeﬂow
Due to automated computation information could
be updated in real-time
Still often requires a domain expert

2nd Generation


3rd Generation
do not require domain experts for usage
aggregate sources - most data is digitised now
rich meta-data models
improved computational power and algorithms
billions of nodes and relationships


Deduction
vs.
Inference


Graph Theory
the basics


Defn 1: Undirected Graph
an undirected graph, G, is an ordered pair G(V, E)
where
V is a set of objects called vertices
E is the set of 2-element subsets of V called
edges
If E does not contain e(v1, v2) such that v1 = v2
then G is a simple graph

Example
V = { london, paris, amsterdam, madrid }
E = { {london, paris}, {paris, amsterdam},
{paris, madrid} }


Defn 3: Labels
A label is some value, e.g integer, colour,
enumeration
An edge-labelled graph is one where some or all
of the edges have labels
A vertex-labelled graph is one where some or all
of the vertices have labels
A labelled graph maybe edge-labelled, vertexlabelled, or both

Defn 2: Directed Graph
a directed graph, G, is an ordered pair G(V, E)
where
E is the set of ordered 2-element subsets of V
called edges
For a vertex v the in-degree is the number of
edges in E that end at v. The out-degree of v is
the number of edges that start ar v

Example

Credit to sciﬁcat @
deviantart and Sheldon from
the big bang theory

Example
V = { rock, scissors, paper, lizard, spock }
E={
{rock, scissors}, {rock, lizard},
{scissors, paper}, {scissors, lizard},
{paper, rock}, {paper, spock},
{lizard, paper}, {lizard, spock},
{spock, rock}, {spock, scissors}
}

Defn 3: Multigraph
a multigraph, G, is an ordered pair G(V, E) where
E is the multiset of 2-element subsets of V called
edges
if the elements of E are ordered pairs then G is a
directed multigraph


Defn 4: Subgraph
given a graph G(Vg, Eg) a graph H is a subgraph
H(Vh, Eh) iff Vh < Vg and Eh < Eg
if Vh = Vg then H is a spanning subgraph of G


Defn 4: Walks
given a graph G(V, E) a walk W is a sequence of
edges from E s.t. for any adjacent elements
wi = (vr, vs), wi+1 = (vt, vw) then vs = vt

If a walk begins & ends on the same vertex it is
a closed, otherwise it is open

Defn 4: Cycle
A closed walk is called a cycle. A cycle must have
length greater than 0.

Defn 4: Cyclic & Acyclic
a graph g is said to be acyclic iff there is no
subgraph which is a cycle graph

Defn 4: Complete Graph
A graph G(V, E) with |V| = n is a complete graph Kn
if for every vertex vi there exists an edge (vi, vk)
in E for k = 1..n, and i ≠ k

Defn 4: Cliques
Given a graph G(Vg, Eg) and a subgraph H(Vh, Eh),
|Vh| = k, if H is a complete graph then H is a clique
of order k, or a k-clique

Examples


Defn 5: Strongly Connected
A graph G is strongly connected iff for every pair
of vertices {vi, vj} in G there exists a path which
starts at vi and ends at vj
Given a graph G and a subgraph H, if H is
maximally strongly connected we call H a strongly
connected component of G

Network Theory
basic Concepts


Communities
A network is said to have community structure if
the nodes can be grouped into (potentially
overlapping) subgraphs such that each is densely
connected.
Methods for ﬁnding communities:
minimum-cut method
hierarchial clustering
Girvan-newman algorithm
modularity maximisation
clique analysis

Small Worlds
A small-world network is a graph G(V, E) where
the average minimum path length between any
two vertices is L where
L α log |V|
Small-worlds are typically comprised of cliques
and near-cliques

Random Graphs
Erdős and Renyi studied properties of random
graphs in 1959
A random graph G is a graph G(V, E) where the
probability an edge (vi, vj) exists is given by p
=> the average degree k is approx. p * |V|


if k < 1
small isolated clusters
small diameters
short average path lengths
if k = 1
one dominant cluster appears
diameter peaks
high average path lengths
if k > 1
approaches single strongly connected component
diameter decreases
average path lengths decrease

If the relationships between people in the real
world can be modelled by a random graph then
because the average person knows more than 1
other (k >> 1) then the majority of people are
connected by short paths


Alpha Model
Watt (1998) proposed the α-model of networks
The α-model corrects the following in the random
model:
Relationships generally aren’t random
Relationships are often “tit for tat”
Relationships usually form clusters


Beta Model
The α-model is a signiﬁcantly better model of
real world network but it too has limitations
Primary limitation is that the chance of distant or
random connections is unrealistically low
Watts and Strogatz (1999) propsed the β-model
to correct this
For a range of value of β these networks exhibit
“small world” properties

Scale-Free Networks
Discovered in 1965 but little interest until 1999
when realised how accurately they modelled many
real-world networks
Consider a random graph with the following
degree distribution depending on two values α and
β. Suppose there are y vertices of degree x
where x and y satisfy
log y = α - (β log x)

Power Law Distribution


Random vs. Scale-Free

Random
graph

Scale-free
graph

Scale-Free Properties
Scale-free graphs are small-worlds
The number of vertices with higher degree than
the average is very common
Such vertices are called hubs
Primary hubs are supported by secondaries,
tertiary, etc
Thus scale-free networks are fault-tolerant
Vertices tend to form communities with hubs
providing inter-community connection

Link Analysis
an introduction


Link analysis and network theory provide
techniques for analysing structure in a system of
interacting agents, represented as a network
Most well known examples are web search
engines:
HITS (Hypertext Induced Topic Search) - ask.com
PageRank - Google
TrustRank - Yahoo!

Matrices
row vector (1 x n)

square matrix
(2 x 2)


column vector
(n x 1)

rectangular matrix
(2 x 3)

Matrix Operations
addition

scalar multiplication

transpose


Matrix Multiplication
Given an n*m matrix P, and an m*o matrix Q then
PQ is the n*o matrix where each element PQij is
given by
for example:


Noncommutative:
Associative:
Distributive over matrix addition:
Scalar multiplication is associative over matrix
multiplication:
Transpose:

Eigenvectors
given a square n*n matrix A and a non-zero nvector v, v is a (right) eigenvector of A iff

we call λ an eigenvalue.
Eigenvalues & eigenvectors can be real or complex


Example


Incidence Matrix
Rock Scissors Paper Lizard Spock
Rock

0

0

1

0

1

Scissors

1

0

0

0

1

Paper

0

1

0

1

0

Lizard

1

1

0

0

0

Spock

0

0

1

1

0


Other Representations
Adjacency matrix
Incidence list
Adjacency list
Edge lists
Topological distance matrix


Centrality
Centrality is a measure of how luminous a given
vertex in a graph is
In an undirected graph centrality measures
consider all edges
In a directed graph centrality measures consider
only out-edges


For a graph G(V, E), and P the set of all paths in
G we deﬁne:
Degree-centrality

Closeness-centrality

Betweenness-centrality


Prestige
Prestige is a measure of how visible a given
vertex in a graph is. In undirected graphs we
consider all edges, but in directed graphs prestige
considers in-edges only
Similar to centrality metrics, we have degreeprestige, proximity prestige and rank prestige

PageRank
“PageRank works by counting the number
and quality of links to a page to determine
a rough estimate of how important the
website is. The underlying assumption is
that more important websites are likely to
receive more links from other websites”
[Facts about Google and Competition]

Simple PageRank
Given the web is a graph G(V, E) where |V| = n,
i.e. n pages, the PageRank of page vi is

and the initial rank of page vi is


Given the adjacency matrix M of a graph
G(V, E) we construct the hyperlink matrix H
such that

Note that H is normalised, i.e each column
sums to 1, and all entries are non-negative.
H is said to be a stochastic matrix

The PageRank vector R of a graph G(V,E)
with hyperlink matrix H is given by

That is PageRank is the primary eigenvector
of H. We can iteratively calculate R using
the power method


Simple PageRank Issues
Dangling pages
Orphan pages
Cycles
Rank sinks
Sensitivity to initial PageRank vector

Real PageRank
use sparse matrix representations
up to 3 billion rows and columns
if probability of teleportation is > 0.15
PageRank converges in less than 100
iterations
may use alternative to “random surfer”
model

Link Analysis in
the Wild


How The NSA Works
The methods used by the NSA Prism project
include:
Blah blah blah blah blah
Blahblahblah and by the name of
Blah blah blah blah blah blah balh
and trained black-ops dolphins


Knowledge Mash-Ups
Multiple data sources
full text search
social graphs
telephone, email & browsing history
Representations may not be appropriate for
analysis
Data may need to be transformed and managed
using non-relational data structures
Important to remember non-mathematicians
analysts prefer to work with visual

TerroristRank
TerroristRank works by counting the number and
quality of links to a person to determine a rough
estimate of how important the person is. The
underlying assumption is that more important
terrorists are likely to receive more links from
other terrorists


How to Find a Terrorist
Given a graph of actors and their interactions
determine the communities
extract the subgraph of communities containing
the actors of interest
calculate the “terrorist rank” of the subgraph
actors with the highest ranks are “suspects”


Limitations
typically graph algorithms are non-linear in time
and/or space complexity
adding new nodes & edges can have a dramatic
impact
real world networks are often dynamic
metrics like rank must be constantly recalculated


Issues
Information overload
data mining maybe incomplete or ﬁnd false positive
relationships
intentional or subconscious human ﬁltering of data
sources
malicious data (e.g SEO of malware by link sites)
changes in alleigence (“when good goes bad”)

Deduction
Prosecution
vs.
Inference
Prevention

Getting Started
with Graph
Algorithms


Matlab
a mathematical workbench
aimed at scientists & mathematicians not
developers
has graph algorithms “plugin” gaimc
(generally) slower than native code
has Apis for bi-directional integration with Java
good for learning the mathematics behind
algorithms

Cern Colt
http:/
/acs.lbl.gov/software/colt/
high performance data structures and operations
collections (including primitive templated lists & maps)
matrices
linear algebra
mathematics & statistics
random sampling & number generation
documentation a bit hit and miss
api not always natural to java programmers

Jung 2.0
http:/
/jung.sourceforge.net/
pure java graph algorithm library
OO Api
uses cern colt library for matrix representations and
operations
uses in-memory storage
performance limitations are generally related to this
easy to extend
requires good understanding of algorithms and internals to
maximise performance

Neo4j
transactional property graph database (acid compliant)
property graphs are:
labelled directed multigraphs
both vertices and edges can have any number of key/value
properties associated with them.
numerous in-built algorithms
powerful ﬂexible query language
allows implementation of algorithms
Community version limits total number of nodes
excellent spring integration
Mark needham http:/
/www.markhneedham.com/blog/

Alternatively
Hadoop & map-reduce
gremlin - a groovy based graph DSL
scala-graph - young & operator-overloading hell
R programming language


Summary


Objectives Review
What is link analysis
history lesson
graph theory basics
network theory basics
link analysis basics
link analysis in the wild
getting started with link analysis

Link analysis is just one tool in the box for extracting
information from graphs
much of the skill lies in:
ﬁltering the raw graph to prevent information overload
pruning the graph to allow expensive algorithms to
compute an effective answer
work iteratively; start by extracting simple data from a
graph before trying, say, community analysis
understand enough of the underlying mathematics to
choose the right tool for the job

Thank You


Link Analysis in Networks - or - Finding the Terrorists

More Related Content

Viewers also liked (8)

Similar to Link Analysis in Networks - or - Finding the Terrorists (20)

Recently uploaded (20)

Link Analysis in Networks - or - Finding the Terrorists