Representation learning on graphs
3 likes1,466 views
This document discusses representation learning on graphs. It begins by explaining why graph representation learning is important as graphs are pervasive in many scientific disciplines. It then discusses various techniques for graph representation learning including graph embedding methods like DeepWalk and Node2Vec, message passing neural networks, and graph generation methods like variational autoencoders. The document concludes by discussing challenges in graph reasoning to deduce knowledge from graphs in response to queries.
Representation learning on graphs
- 1. Representation learning on graphs 11/05/2019 1 HCMC, May 2019 A/Prof Truyen Tran Deakin University truyentran.github.io [email protected] @truyenoz letdataspeak.blogspot.com goo.gl/3jJ1O0
- 2. 11/05/2019 2 Graph dynamics Graph generation Graph reasoning Message passing Embedding Why bother? Graph representation
- 3. Why learning of graph representation? Graphs are pervasive in many scientific disciplines. The sub-area of graph representation has reached a certain maturity, with multiple reviews, workshops and papers at top AI/ML venues. Deep learning needs to move beyond vector, fixed-size data. Learning representation as a powerful way to discover hidden patterns making learning, inference and planning easier. 11/05/2019 3
- 5. Biology & pharmacy Traditional techniques: Graph kernels (ML) Molecular fingerprints (Chemistry) Modern techniques Molecule as graph: atoms as nodes, chemical bonds as edges 11/05/2019 5 #REF: Penmatsa, Aravind, Kevin H. Wang, and Eric Gouaux. "X- ray structure of dopamine transporter elucidates antidepressant mechanism." Nature 503.7474 (2013): 85-90.
- 6. Chemistry DFT = Density Functional Theory Gilmer, Justin, et al. "Neural message passing for quantum chemistry." arXiv preprint arXiv:1704.01212 (2017). 11/05/2019 6 • Molecular properties • Chemical-chemical interaction • Chemical reaction • Synthesis planning
- 7. Materials science 11/05/2019 7 Xie, Tian, and Jeffrey C. Grossman. "Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties." Physical review letters 120.14 (2018): 145301. • Crystal properties • Exploring/generating solid structures • Inverse design
- 8. Videos as space-time region graphs (Abhinav Gupta et al, ECCV’18)
- 10. 11/05/2019 10 Graph dynamics Graph generation Graph reasoning Message passing Embedding Why bother? Graph representation
- 11. Skip-gram Negative sampling Loss function Predicting neighbors
- 12. DeepWalk (KDD-2014) Considered as #epochs For Hierarchical Softmax Iterate over each epoch Finding neighbours of each node Update embedding of this node Neighbour nodes Window size Embedding matrix
- 13. Node2Vec (KDD-2016) Similar to DeepWalk in using Skip-gram model for unsupervised learning. Only modifies the search for neighboring nodes that balance between BFS and DFS. Defines edge embedding based on node embedding Can solve link prediction problem 2nd order Random Walk Consider random walk that just travelled edge (t, v). The walk will decide which is the next node x that it should go from v by computing p and q are hyper-parameters
- 14. Node2Vec (Cont.) Why it scales over DeepWalk? Use Negative Sampling instead of Hierarchical Softmax Batch learning
- 15. 11/05/2019 15 Graph dynamics Graph generation Graph reasoning Message passing Embedding Why bother? Graph representation
- 16. Message passing 11/05/2019 16 Thin column Relation graph Stacked learning Column nets#REF: Pham, Trang, et al. "Column Networks for Collective Classification." AAAI. 2017.
- 17. Factored message passing 11/05/2019 17 #REF: Do, Kien, et al. "Attentional Multilabel Learning over Graphs-A message passing approach." Machine Learning, 2019.
- 18. Graph attention (Do et al arXiv’s17, Veličković et al ICLR’ 18) 11/05/2019 18 Learning deep matrix representations, K Do, T Tran, S Venkatesh, arXiv preprint arXiv:1703.01454
- 19. 11/05/2019 19 Graph dynamics Graph generation Graph reasoning Message passing Embedding Why bother? Graph representation
- 20. Graph morphism Input: Graph Output: A new graph. Same nodes, different edges. Model: Graph morphism Method: Graph transformation policy network (GTPN) 11/05/2019 20 Kien Do, Truyen Tran, and Svetha Venkatesh. "Graph Transformation Policy Network for Chemical Reaction Prediction." KDD’19.
- 21. Graph recurrence Graphs that represent interaction between entities through time Spatial edges are node interaction at a time step Temporal edges are consistency relationship through time
- 22. Challenges The addition of temporal edges make the graphs bigger, more complex Relying on context specific constraints to reduce the complexity by approximations Through time, structures of the graph may change Hard to solve, most methods model short sequences to avoid this
- 23. Structural RNN CVPR16 best student paper Saxena group, Cornell Example: human microwaving food Middle: s-t graph capturing spatial and temporal interactions between the human and the objects. Top: Schematic representation of structural-RNN architecture
- 24. Structural RNN Node features: human and object poses, Edge features: relative orientation Node labels: human activity and object affordance. affected by both its node and its interactions with other nodes (edges)
- 25. Structural RNN – from s-t graph Each factor is represented as a RNN Form a feed forward bipartite graph from edge factors to node factors Sharing factors between node and edge of same semantics type More compact Support node # changes
- 26. Structural RNN - Applications Activity detection and anticipation Skeleton tracking Driver maneuver prediction
- 27. Message-Passing Encoder-Decoder Recurrent Net 11/05/2019 27 • Prelim version published in (Morais et al, CVPR’19) • Multiple interacting channels • Graphs are dynamics, with attention. • Mixture density network at each time step
- 28. 11/05/2019 28 Graph dynamics Graph generation Graph reasoning Message passing Embedding Why bother? Graph representation
- 29. Technical challenges No regular structures (e.g. grid, sequence,…) Graphs are permutation invariant: #permutations are exponential function of #nodes The probability of a generated graph G need to be marginalized over all possible permutations Generating graphs with variable size Diversity of generated graphs Smoothness of latent space
- 30. Generation methods Classical random graph models, e.g., An exponential family of probability distributions for directed graphs (Holland and Leinhardt, 1981) Deep Generative Model Methods: Variational Graph AutoEncoders Graphite: Iterative Generative Modeling of Graphs GraphVAE: Towards Generation of Small Graph using Variational AutoEncoder Junction Tree Variational AutoEncoder for Molecular Graph Generation Sequence-based & RL method: GraphRNN - A Deep Generative Model for Graphs Multi-Objective De Novo Drug Design with Conditional Graph Generative Model 11/05/2019 30
- 31. Variational Autoencoder (Kingma & Welling, 2013) Two separate processes: generative (hidden visible) versus recognition (visible hidden) http://kvfrans.com/variational-autoencoders-explained/ Gaussian hidden variables Data Generative net Recognising net
- 32. Generative Adversarial Networks (Goodfellow et al,2014) 11/05/2019 32 GAN architecture. Source: DL4J (Goodfellow’s, NIPS 2014)
- 33. Variational methods Minimize the upper bound on the negative log-likelihood, equivalent to maximizing the ELBO:
- 34. Adjacency matrix Edge types Node types k>n The graph size are bounded Latent vector for whole graph GraphVAE (Simonovsky and Komodakis)
- 35. Junction Tree VAE (Jin et. al.)
- 36. Constraint during adding nodes
- 37. GraphRNN A case of graph dynamics: nodes and edges are added sequentially. Solve tractability using BFS 11/05/2019 37 You, Jiaxuan, et al. "GraphRNN: Generating realistic graphs with deep auto-regressive models." ICML (2018).
- 38. Graphs step-wise construction using reinforcement learning 11/05/2019 38 You, Jiaxuan, et al. "Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation." NeurIPS (2018). Graph rep (message passing) | graph validation (RL) | graph faithfulness (GAN)
- 39. 11/05/2019 39 Graph dynamics Graph generation Graph reasoning Message passing Embedding Why bother? Graph representation
- 40. Reasoning Reasoning is to deduce knowledge from previously acquired knowledge in response to a query (or a cue) Early theories of intelligence: focuses solely on reasoning, learning can be added separately and later! (Khardon & Roth, 1997). 11/05/2019 40 Khardon, Roni, and Dan Roth. "Learning to reason." Journal of the ACM (JACM) 44.5 (1997): 697-725. (Dan Roth; ACM Fellow; IJCAI John McCarthy Award)
- 41. Inferring relations 11/05/2019 41 https://www.zdnet.com/article/salesforce-research-knowledge-graphs-and-machine- learning-to-power-einstein/ Do, Kien, Truyen Tran, and Svetha Venkatesh. "Knowledge graph embedding with multiple relation projections." 2018 24th International Conference on Pattern Recognition (ICPR). IEEE, 2018.
- 42. Querying a graph 11/05/2019 42 Controller … Memory Graph Task/query Bioactivities 𝒚𝒚 query #Ref: Pham, Trang, Truyen Tran, and Svetha Venkatesh. "Graph Memory Networks for Molecular Activity Prediction." ICPR’18. Message passing as refining node & query representation
- 43. Querying a graph (2) 11/05/2019 43 https://rylanschaeffer.github.io/content/research/neural_turing_machine/main.html Neural Turing Machine (Grave et al, 2014) Differentiable Neural Computer (Grave et al, 2016)
- 44. Querying multiple graphs 11/05/2019 44 𝑴𝑴1 … 𝑴𝑴𝐶𝐶 𝒓𝒓𝑡𝑡 1 …𝒓𝒓𝑡𝑡 𝐾𝐾 𝒓𝒓𝑡𝑡 ∗ Controller Write𝒉𝒉𝑡𝑡 Memory Graph Query Output Read heads Pham, Trang, Truyen Tran, and Svetha Venkatesh. "Relational dynamic memory networks." arXiv preprint arXiv:1808.04247(2018). Detects high-order patterns from disconnected paths Learns graph similarity Models graph-graph interaction Supports structured queries
- 45. On going and future work Graph translation, graph2graph, graph2seq Graphs in context (e.g., crystals, human activities in scene) Semi-supervised learning Manifolds learning Graph as hidden layers (e.g., see Ying et al, NIPS’19) Generative models of sequence of graphs 11/05/2019 45 Theoretical properties of graphs (e.g., Xu et al, ICLR’19) Graph matching Higher-order graph neural networks (e.g., Morris, AAAI’19) Graphs for logical inference Graphs of multi-agent communication Continuous time graph dynamics Graph density – anomaly detection
- 46. For more refs Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2019). A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.00596. Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., ... & Gulcehre, C. (2018). Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261. Goyal, P., & Ferrara, E. (2018). Graph embedding techniques, applications, and performance: A survey. Knowledge-Based Systems, 151, 78-94. Lee, J. B., Rossi, R. A., Kim, S., Ahmed, N. K., & Koh, E. (2018). Attention models in graphs: A survey. arXiv preprint arXiv:1807.07984. Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., & Vandergheynst, P. (2017). Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4), 18-42. Hamilton, W. L., Ying, R., &Leskovec, J. (2017). Representation learning on graphs: Methods and applications. IEEE Data Engineering Bulletin. Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E. (2016). A review of relational machine learning for knowledge graphs. Proceedings of the IEEE. 104.1, 11-33. 11/05/2019 46
- 47. The graph team @ 11/05/2019 47 A/Prof. Truyen Tran Dr Trang Pham (Now @Google) Dr Vuong Le Mr Kien DoMr Tin Pham Mr Thao Minh Le Dr Thin Nguyen
- 48. 11/05/2019 48 Thank you! http://ahsanqawl.com/2015/10/qa/ We’re hiring PhD & Postdocs [email protected] https://truyentran.github.io/scholarship.html