Presentation about Overthere for J-Fall 2011
- 1. Overthere Design and implementation of a Java remote file and execution framework xebialabs.com Vincent Partington CTO, XebiaLabs
- 2. Speaker • Been working with Java since 1996. Built first production Servlet in 1997. • At Xebia since 2003, CTO of XebiaLabs since 2008. • Interests: middleware, deployment automation, performance, security. Basically anything Java and hard! xebialabs.com
- 3. Agenda • Why Overthere? • Exploring the API. • Designing, extending and testing Overthere. xebialabs.com
- 4. The problem • For an internet language and an enterprise language, Java has pretty poor support for manipulating remote resources. • Apache’s HttpClient is way better than JDK’s HTTP support. • There is no facility to manipulate remote files (FTP, WebDAV, SFTP, SCP, CIFS, etc.) • There is no facility to run commands on remote machines. xebialabs.com
- 5. The solution • Interfaces: java.io.File and java.lang.Process should’ve been interfaces! • Factory: And there should be a factory to create instances of them!! • SPI: And there should be an SPI to hook in new implementations!!! xebialabs.com
- 6. Overthere • OverthereFile - equivalent of java.io.File. • OverthereProcess - equivalent of java.lang.Process. • Overthere - the factory. • OverthereConnectionBuilder - the SPI xebialabs.com
- 7. Create a connection (Unix) ‣ Examine ExecuteOnUnix in the examples project. xebialabs.com
- 8. Create a connection (Windows) ‣ Examine ExecuteOnWindows in the examples project. xebialabs.com
- 9. Execute a command ‣ Run ExecuteOnUnix in the examples project. xebialabs.com
- 10. Start a process ‣ Run StartProcess in the examples project. xebialabs.com
- 11. Read a file ‣ Run ReadFile in the examples project. xebialabs.com
- 12. Write a file ‣ Run WriteFile in the examples project. xebialabs.com
- 13. Get info about a file ‣ Run GetFileInfo in the examples project. xebialabs.com
- 14. Manipulate a file ‣ Run ManipulateFile in the examples project. xebialabs.com
- 15. Manipulate a directory ‣ Run ManipulateDirectory in the examples project. xebialabs.com
- 16. Supported technologies • SSH - for Unix and Windows: • SFTP - using SFTP for file transfer. • SCP - using SCP for file transfer. • SUDO - using SUDO for command execution (and SCP for file transfer). • INTERACTIVE_SUDO - as SUDO but tries to answer sudo password prompts. • SFTP_CYGWIN - using SFTP for file transfer, to a Windows host running OpenSSH on Cygwin. • SFTP_WINSSHD - using SFTP for file transfer, to a Windows host running WinSSHD. • CIFS - for Windows: • TELNET - using Telnet for command execution. • WINRM - using WinRM for command execution (Java implementation). • LOCAL - Layer on top of java.io.File and java.lang.Process. xebialabs.com
- 17. Advanced topics • Designing Overthere. • Extending Overthere. • Testing Overthere. xebialabs.com
- 18. Designing Overthere (1/4) • Make OverthereFile an interface or a subclass of java.io.File? xebialabs.com
- 19. Designing Overthere (2/4) • Exceptions or booleans for delete(), mkdir(), mkdirs() and renameTo()? • Use checked or unchecked exceptions? xebialabs.com
- 20. Designing Overthere (3/4) • Use JSch or SSH/J to handle SSH connections? xebialabs.com
- 21. Designing Overthere (4/4) • Telnet or WinRM for remote execution on Windows? xebialabs.com
- 22. Extending Overthere - SPI • At startup, all classes implementing OverthereConnectionBuilder are found. • When a connection is created, all builders are invoked in turn to see if they can build a connection for that protocol and with those connection options. ‣ Examine LocalConnection and SshConnectionBuilder. xebialabs.com
- 23. Testing Overthere • How to test? • Against target operating systems - Unix and Windows. • Portable - no VMware images. • From IDE (adhoc) and from build systems. xebialabs.com
- 24. Subproject: itest-support • Framework to allow for integration testing with virtual images. • An ItestHost is a host against which to run tests. • Created by invoking the ItestHostFactory. • Depending on settings in itest.properties and system properties one of two things happens: • A pointer to an existing host (VMware image or manually launched EC2 instance) is returned. • An AMI is launched and when it has started up, a pointer to the EC2 is returned. • At the end of the itest, any EC2 instances launched on-the-fly are terminated. xebialabs.com
- 25. Some examples • Parametrized tests to run multiple connection types on one platform: • SCP, SFTP, etc. on Unix or • TELNET, WINRM_HTTP, etc. on Windows. ‣ Examine OverthereOnUnixItest and its parent classes. xebialabs.com
- 26. SSH tunneling • Port 445 blocked by Amazon and many internet providers. • Solution: create SSH tunnels when starting itests. And don’t forget to use the new host and port in your test code! ‣ Examine OverthereOnWindowsItest.setTypeAndOptions() xebialabs.com
- 27. Ad-hoc usage • Provisioning an EC2 AMI can take a while, especially provisioning a Windows AMI. • Might even want to run just one test while developing a new feature. ‣ Examine overthere/itest.properties and overthere/src/test/resources/itest.properties. xebialabs.com
- 28. Open source • Overthere 1.0 is part of Deployit 3.5 and up. • Overthere has been released as open source, under the ASLv2 license. • Check the Github repository: https://github.com/xebialabs/overthere • Have fun and don’t forget to mention any issues! • Even better: send us a pull request with a fix. :-) xebialabs.com
- 29. More information • Overthere: https://github.com/xebialabs/overthere • XebiaLabs: http://www.xebialabs.com/ • Integration testing: http://bit.ly/ess5Xh xebialabs.com