Networking in Java with NIO and Netty
64 likes12,277 views
The document is a presentation by Kanstantsin Slisenka on Java networking, focusing on non-blocking I/O using NIO and the Netty framework. It discusses the limitations of traditional blocking I/O models, presents live coding examples, and concludes with insights into when to utilize NIO and frameworks like Netty for high-performance networking. The presentation emphasizes the importance of asynchronous programming for handling many connections efficiently.
Networking in Java with NIO and Netty
- 1. 1CONFIDENTIAL NETWORKING IN JAVA WITH NIO AND NETTY KANSTANTSIN SLISENKA JUN 1, 2017
- 2. 2CONFIDENTIAL ABOUT ME Java Backend engineer Speaker at Java Tech Talks, SEC Online, CMCC Tech Talks, IT Week I’m interested in Complex Java backend, SOA, databases High load, fault-tolerant, distributed systems KANSTANTSIN SLISENKA EPAM Systems, Lead Software Engineer
- 3. 3CONFIDENTIAL WHY IS IMPORTANT TO DIG INTO JAVA NETWORKING Classic Java project Tomcat + Spring + Hibernate + … High load, latency-sensitive Custom TCP/UDP solution
- 4. 4CONFIDENTIAL AGENDA Blocking vs non-blocking resources1 Blocking I/O2 Non-blocking I/O3 Netty: high performance networking framework4
- 5. 5CONFIDENTIAL BLOCKING VS NON-BLOCKING RESOURCES client thread read/write response Blocking resources Database transactions, RPC/WS calls blocked
- 6. 6CONFIDENTIAL BLOCKING VS NON-BLOCKING RESOURCES client thread read/write response client thread read/write nope read/write response client thread read/write callback with response read/write nope register callback Blocking resources Non-blocking resources Database transactions, RPC/WS calls Queues, WebSockets, Non-blocking data storage blocked
- 7. 7CONFIDENTIAL NETWORK SOCKETS Socket is blocking API over OS and Network hardware not a physical object PERMANENT KNOWN PORT ANY UNUSED PORT SOCKET SOCKET Max ports = 65535
- 8. 8CONFIDENTIAL JAVA SOCKETS = BLOCKING IO Client Server
- 9. 9CONFIDENTIAL SOCKET SERVER LIVE CODING EXAMPLE
- 10. 10CONFIDENTIAL Server 3 2 Client 2 Client 3 Client 1 SOCKET SERVER ARCHITECTURE no data no data data transfer in progress 1 Socket Socket SocketThread (blocked*) Thread (running) Thread (blocked*) *until keep alive timeout Fixed thread pool Requests queue
- 11. 11CONFIDENTIAL Server 3 2 Client 2 Client 3 Client 1 SOCKET SERVER ARCHITECTURE no data no data data transfer in progress 1 Socket Socket SocketThread (blocked*) Thread (running) Thread (blocked*) Client 4 *until keep alive timeout Fixed thread pool Requests queue new connection
- 12. 12CONFIDENTIAL Server 3 2 Client 2 Client 3 Client 1 SOCKET SERVER ARCHITECTURE no data no data data transfer in progress 1 Socket Socket SocketThread (blocked*) Thread (running) Thread (blocked*) Client 4 *until keep alive timeout Fixed thread pool Requests queue new connection drop by timeout
- 13. 13CONFIDENTIAL SOCKET SERVER CONCLUSION 1. Ineffective thread utilization – many threads in waiting/blocked state – extra memory for stack and thread local resources 2. Poor performance – context switching overhead 3. Limited number of connections – when thread pool is fully consumed Benefits Drawbacks 1. Simple development well known Input/OutputStream API 2. No check for partially received messages thread is blocked until fully reads message
- 14. 14CONFIDENTIAL IO VS NIO IO (blocking) Thread Thread Thread socket socket socket BLOCKING READ/WRITE WAIT WAIT WAITWAIT
- 15. 15CONFIDENTIAL IO VS NIO IO (blocking) NIO (non-blocking) NON BLOCKING READ/WRITE Thread buffer buffer buffer socket channel socket channel socket channel Thread Thread Thread socket socket socket BLOCKING READ/WRITE WAIT WAIT WAITWAIT ALWAYS RUNNING NOTIFICATIONS selector
- 16. 16CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector socket channel
- 17. 17CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector socket channel 1 WRITE READY
- 18. 18CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector PUT DATA2 socket channel 1 WRITE READY
- 19. 19CONFIDENTIAL NIO DATA SENDING Thread buffer buffer buffer socket channel socket channel selector PUT DATA2 SEND DATA3 socket channel 1 WRITE READY
- 20. 20CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector socket channel
- 21. 21CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector RECEIVE DATA1 socket channel
- 22. 22CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector RECEIVE DATA1 socket channel 2 READ READY
- 23. 23CONFIDENTIAL NIO DATA RECEIVING Thread buffer buffer buffer socket channel socket channel selector GET DATA3 RECEIVE DATA1 socket channel 2 READ READY
- 24. 24CONFIDENTIAL NIO BYTE BUFFER POSITION LIMIT CAPACITY 0 1 2 3 4 5 6 7 8 9 Empty buffer BYTE ARRAY
- 25. 25CONFIDENTIAL NIO BYTE BUFFER POSITION LIMIT CAPACITY 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 LIMIT CAPACITY Empty buffer Writing mode BYTE ARRAY BYTE ARRAY POSITION 0 1 2 3 4 5 6 7 8 9
- 26. 26CONFIDENTIAL NIO BYTE BUFFER POSITION LIMIT CAPACITY 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 LIMIT CAPACITY 1 0 1 1 0 POSITION LIMIT CAPACITY Empty buffer Writing mode Reading mode BYTE ARRAY BYTE ARRAY BYTE ARRAY buffer.flip()buffer.clear() POSITION 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
- 27. 27CONFIDENTIAL DIRECT BUFFERS 0 1 2 3 4 5 6 7 8 JVM Network Interface Card 0 1 2 3 4 5 0 1 2 3 4 5 Physical memory 9 Shared memory address space Shared memory address space Memory should be continuous because NICs are block- oriented devices
- 28. 28CONFIDENTIAL NIO SERVER LIVE CODING EXAMPLE
- 29. 29CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress buffer socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer Business logic
- 30. 30CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress buffer socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer Business logic
- 31. 31CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress buffer get ready channels socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer Business logic
- 32. 32CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE buffer Client #2 Client #1 no data data transfer in progress socket channel #3 Client #3 data transfer in progress get buffer put read write get ready channels socket channel #2 socket channel #1 Business logic WORKER THREAD SELECTOR SELECTOR WORKER THREAD buffer
- 33. 33CONFIDENTIAL Server BASIC NIO SERVER ARCHITECTURE new connection buffer Client #2 Client #4 Client #1 no data data transfer in progress new connection socket channel #3 Client #3 data transfer in progress get buffer put register in selector read write get ready channels socket channel #2 socket channel #1 WORKER THREAD SELECTOR SELECTOR WORKER THREAD Business logic buffer
- 34. 34CONFIDENTIAL CAN WE GO TO PRODUCTION WITH A SINGLE THREAD? https://www.pinterest.com/banjolele/one-man-band/
- 36. 36CONFIDENTIAL SELECTOR PRODUCTION-READY NIO SERVER ServerSocket Channel acceptor thread new connection Acceptor threads Separate thread to not lose incoming connections when selector threads are overloaded
- 37. 37CONFIDENTIAL SELECTOR SELECTOR PRODUCTION-READY NIO SERVER ServerSocket Channel selector thread … acceptor thread Socket Channel SELECTOR selector threadSocket Channel SELECTOR selector threadSocket Channel new connection Selector threads Multiple processing threads to get benefit from multi core CPU Acceptor threads Separate thread to not lose incoming connections when selector threads are overloaded
- 38. 38CONFIDENTIAL SELECTOR SELECTOR Application thread pools (long running tasks) PRODUCTION-READY NIO SERVER ServerSocket Channel selector thread … worker thread worker thread worker thread application thread acceptor thread Socket Channel SELECTOR selector threadSocket Channel SELECTOR selector threadSocket Channel new connection Selector threads Multiple processing threads to get benefit from multi core CPU Acceptor threads Separate thread to not lose incoming connections when selector threads are overloaded Application threads For long tasks (back-end, DB, computation), to avoid blocking of selector threads
- 39. 39CONFIDENTIAL PARTIAL MESSAGE RECEIVING PROBLEM Message reader splits/joins data blocks into messages Data block Message Message Message Message Message Message message reader Data block Data block MessageMessage Data block
- 40. 40CONFIDENTIAL NIO CONCLUSION Complexity • complex code • learning curve for developers • handling partially received messages • combining async with sync code (need different thread groups) Benefits Drawbacks Single shared thread for many connections • less memory • less context switching overhead
- 41. 41CONFIDENTIAL NETTY ONE FRAMEWORK TO RULE THEM ALL
- 42. 42CONFIDENTIAL NETTY: JAVA NETWORK FRAMEWORK Some projects using Netty
- 44. 44CONFIDENTIAL transport Boss EventLoopGroup NIO NETTY SERVER ARCHITECTURE Network Socket Native JNI Thread pool for new connections
- 45. 45CONFIDENTIAL transport Boss EventLoopGroup channel channel pipeline Worker EventLoopGroup EventExecutorGroup NIO NETTY SERVER ARCHITECTURE Inbound Handler Inbound Handler Inbound Handler Outbound Handler Outbound Handler Outbound Handler Network Socket Native JNI Separate thread pool for blocking logic I/O thread pool For non-blocking logic Thread pool for new connections
- 49. 49CONFIDENTIAL channel channel pipeline NETTY HANDLERS HTTP Request Decoder HTTP Object Aggregator SSL Handler encrypted encrypted bytes http messages Your Business Logic Handler full http messages buffer
- 50. 50CONFIDENTIAL channel channel pipeline NETTY HANDLERS HTTP Request Decoder HTTP Object Aggregator HTTP Request Encoder SSL Handler encrypted encrypted bytes http messages bytes Your Business Logic Handler full http messages buffer full http messages
- 51. 51CONFIDENTIAL NETTY SERVER LIVE CODING EXAMPLE
- 52. 52CONFIDENTIAL WHAT IS FASTER: IO OR NIO?
- 53. 53CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * ? ? ? ? NIO is faster Context switching is very slow Threads take up too much memory Thread synchronization will kill you
- 54. 54CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf ? ? ? Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster
- 55. 55CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf ? ? Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster New POSIX library in Linux > 2.6 IDLE thread costs almost zero Context switching is much much faster
- 56. 56CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf ? Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster New POSIX library in Linux > 2.6 IDLE thread costs almost zero Context switching is much much faster Don’t overuse thread-local objects!
- 57. 57CONFIDENTIAL WHAT IS FASTER: IO OR NIO? May be yes, may be not * * Always point to holy-wars, see www.mailinator.com/tymaPaulMultithreaded.pdf Context switching is very slow Threads take up too much memory Thread synchronization will kill you IO is 25-30% faster (see benchmark*) NIO is faster New POSIX library in Linux > 2.6 IDLE thread costs almost zero Context switching is much much faster Don’t overuse thread-local objects! Use non-blocking data structures
- 58. 58CONFIDENTIAL CONCLUSION • Huge number of connections • Medium or slow data rate per connection • Asynchronous world (event-based) • Non blocking database • Queues • Web-sockets • Callbacks, listeners • Data streaming • Don’t use standard NIO, use frameworks like Netty • Highly skilled team because of complex code Use sockets* Use NIO* *Not a strict rule, always point to holy-wars • Small number of connections • Huge data rate per connection • Synchronous world (procedure-based) • Classic enterprise applications • Request-response protocols • Blocking database • Blocking RPC or web-service calls • Bottleneck is not server, but blocking resource • Make servers stateless and scale horizontally
- 59. 59CONFIDENTIAL Java Network Programming, 4’th edition O’REILLY Elliotte Rusty Harold BUY A BOOK, MAKE THEM RICH Netty in Action Manning Norman Maurer Java NIO O’REILLY Ron Hitchens • Code examples: github.com/kslisenko/java-networking • NIO performance: www.mailinator.com/tymaPaulMultithreaded.pdf • NIO tutorial: tutorials.jenkov.com/java-nio/index.html • ITT 2015 - Heinz Kabutz - The Multi-threading, Non Blocking IO: youtube.com/watch?v=uKc0Gx_lPsg • Netty - One Framework to rule them all by Norman Maurer: youtube.com/watch?v=DKJ0w30M0vg • Scalable IO in Java: http://gee.cs.oswego.edu/dl/cpjslides/nio.pdf