DevOOPS: Attacks and Defenses for DevOps Toolchains
12 likes4,680 views
The document presents a detailed discussion on security challenges and defenses related to DevOps toolchains, emphasizing the importance of prioritizing security over speed. It outlines common mistakes in devops infrastructure, highlights monitoring and protection strategies for employees and their workstations, and addresses vulnerabilities associated with various technologies like Jenkins, Redis, and AWS. Overall, it aims to increase awareness and provide solutions to improve security practices within DevOps environments.
DevOOPS: Attacks and Defenses for DevOps Toolchains
- 1. SESSION ID:SESSION ID: #RSAC Chris Gates DevOOPS: Attacks And Defenses For DevOps Toolchains HTA-W02 Sr. Security Engineer Uber @carnal0wnage Ken Johnson CTO nVisium @cktricky
- 2. #RSAC SOMETHING AWESOME TO GET US STARTED Link to slides and URLs in this presentation: http://bit.ly/RSA-Devoops
- 10. #RSAC Ken Johnson (@cktricky) CTO (@nVisium) Railsgoat Co-Author Prior US Navy Spoke a ton about (In)Security of: Rails DevOps Web Frameworks AWS Who Ken
- 11. #RSAC Who Chris Chris Gates (CG) @carnal0wnage Sr. Security Engineer (Uber) NoVA Hackers Co-Founder US Army, Army Red Team, Applied Security, Rapid7, Lares, Facebook http://carnal0wnage.attackresearch.com
- 12. #RSAC TL;DR Don’t prioritize speed over security Understand devops tools’ auth model...or lack of it Get pwned real bad, then get a real auth model – hello mongodb Out of date or insecure implementation can lead to pwnage Dev/Ops building infrastructure can be dangerous without thought and training around security. It’s ok to teach them :-)
- 13. #RSAC Why This Talk Increase awareness around DevOps Infrastructure Security Provide Solutions Show common mistakes/misconfigurations with DevOps testing Sections are broken up between Human, Host, and Infrastructure
- 14. #RSAC Employee Intelligence (Human) Making it difficult (for employees) to allow attackers to walk into our environment
- 15. #RSAC Monitoring External Services Numerous ways for employees to accidently release data Pastebin-like sites GitHub — Gists — Code Repositories BitBucket, CodeCommit, etc — https://en.wikipedia.org/wiki/Comparison_of_source_code_hosting_facilities Examples Slack tokens in GitHub AWS creds in .dotfiles Tokens in logs/dumps/configs/code snippets
- 16. #RSAC Examples
- 17. #RSAC Examples
- 18. #RSAC Examples
- 19. #RSAC Examples
- 20. #RSAC Monitoring Slack (Solutions) 20 Slack Team Access logs (For Paid Slack Only) https://api.slack.com/methods/team.accessLogs https://github.com/maus-/slack-auditor code to pull these logs
- 21. #RSAC Monitoring GitHub (Solutions) Solutions to move away from public GitHub Gitlab, Gitolite, GitHub Enterprise, Phabricator Enable 2 Factor on anything that has 2 Factor! Audit who has access to your repos Have a process to remove ex-employees Audit their personal repos for leaks Regularly search your repos for sensitive data Create work github accounts instead of joining personal ones to org
- 25. #RSAC Monitoring GitHub (Solutions) GitMonitor (for pay service) https://gitmonitor.com/
- 27. #RSAC Monitoring Pastebin* (Solutions) 27 Host internal Pastebin Plugins for stash Phabricator Stikked Multiple Open Source Tools for monitoring pastebin* https://github.com/jordan-wright/dumpmon https://github.com/xme/pastemon https://github.com/cvandeplas/pystemon
- 29. #RSAC Monitoring Pastebin* (Solutions) 29 For Pay Services
- 30. #RSAC Workstation Protection (Host) Protecting and monitoring employees on their development workstations (and servers too)
- 31. #RSAC Why Developer Laptop Hardening Sensitive information stored on their systems Almost always admin on their systems Sloppy code/key/token hygiene can lead to loss of keys to the kingdom One key to rule them all Want to identify badness as soon as possible
- 32. #RSAC Host Protections Developer Laptop Hardening Osquery (OSX/Linux/Windows*) Doorman Block Block Little Snitch Carbon Black / Sysmon Splunk / ELK Simian Munki
- 33. #RSAC Host Protections osquery (https://osquery.io/) “osquery is an operating system instrumentation framework for OS X, Linux, and FreeBSD. The tools make low-level operating system analytics and monitoring both performant and intuitive.” “osquery exposes an operating system as a high-performance relational database. This allows you to write SQL queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.”
- 35. #RSAC Host Protections osquery File Integrity Monitor Yara Rules Query Packs
- 36. #RSAC Host Protections Doorman (https://github.com/mwielgoszewski/doorman) “Doorman is an osquery fleet manager that allows administrators to remotely manage the osquery configurations retrieved by nodes.”
- 38. #RSAC Host Protections BlockBlock (https://objective-see.com/products/blockblock.html) Kernel hook to identify any time software wants to persist Prompt to allow or deny Little Snitch (https://www.obdev.at/products/littlesnitch/index.html) “Little Snitch intercepts these unwanted connection attempts, and lets you decide how to proceed.”
- 39. #RSAC Host Protections (Block Block)
- 40. #RSAC Host Protections (Little Snitch)
- 41. #RSAC Host Protections CarbonBlack (https://www.carbonblack.com/) Host based agent Monitor process create, writes, registry queries, net connections Create rules/watchlist for known bad behavior Mimikatz --> company_name:*gentilkiwi* FileVault Encryption Disabled --> process_name:fdesetup cmdline:disable Unsigned JAR exe c--> process_name:*.jar digsig_result: (digsig_result:"Unsigned") OSX dump user hashes --> process_name:dscl cmdline:ShadowHashData
- 47. #RSAC Host Protections (Patch Management) Why do we bring this up? Some people aren’t aware you can perform free OSX patch management There are a lot of OSX developer shops without an “enterprise budget” Patch management is a no-brainer and security 101 Solved for Windows, more difficult for OSX / Linux
- 48. #RSAC Host Protections (Patch Management) OSX Patch Management – Simian “Simian is an enterprise-class Mac OS X software deployment solution.” Allows you to push munki updates Free / OSS Runs on Google cloud Project: https://github.com/google/simian
- 49. #RSAC Host Protections (Patch Management) OSX Patch Management – Simian
- 50. #RSAC Host Protections (Patch Management) OSX Software Management – Munki “Munki is a set of tools that, used together with a webserver-based repository of packages and package metadata, can be used by OS X administrators to manage software installs (and in many cases removals) on OS X client machines.” https://www.munki.org/munki/
- 51. #RSAC Host Protections (Patch Management) OSX Software Management – Munki
- 52. #RSAC Production Protection (Infra) Jenkins, Redis, Memcache, Docker, Hadoop, AWS
- 54. #RSAC Hudson/Jenkins “Hudson is a continuous integration (CI) tool written in Java, which runs in a servlet container, such as Apache Tomcat or the GlassFish application server” Very popular If you can’t pwn Jenkins then try GlassFish or Tomcat :-)
- 55. #RSAC Shodan search for X-Hudson Hudson/Jenkins
- 56. #RSAC Hudson/Jenkins Jenkins Issues Multiple Remote Code Execution (RCE) vulnerabilities over the years https://wiki.jenkins-ci.org/display/SECURITY/Home Advisories are not well publicized Ex: CVE-2015-1814 Ex: CVE-2016-9299 Weak coverage with Vulnerability Scanners API token same access as password Jenkins builds and deploys code
- 57. #RSAC Hudson/Jenkins If no authentication required Trivial to gain remote code execution via script console Metasploit Module exploit/multi/http/jenkins_script_console Exploit module will also use credentials https://www.pentestgeek.com/2014/06/13/hacking-jenkins-servers-with-no-password/ http://www.labofapenetrationtester.com/2014/06/hacking-jenkins-servers.html http://zeroknock.blogspot.com/search/label/Hacking%20Jenkins
- 59. #RSAC Hudson/Jenkins Metasploit exploit module for script console
- 60. #RSAC Hudson/Jenkins You can lock down script console access by turning on authentication However, if it’s set to local auth, you can register as a regular user :-) ...then get access to the /script
- 61. #RSAC Hudson/Jenkins Can you browse a workspace?
- 64. #RSAC Hudson/Jenkins (Solutions) If possible, require authentication for everything on Hudson/Jenkins Monitor for security issues and updates Challenging b/c full impact of issues can be watered down in the advisory Segment Hudson/Jenkins from Corp Logical separation by groups Either on single instance or multiple servers Monitor Jenkins slave activity/net connections osquery
- 66. #RSAC Redis Defaults No encrypted communication No credentials by default Doesn’t have to be root, but usually is Port 6379 (TCP) Binds to all interfaces Moral of the story? Keep off the interwebs! Update redis.conf to bind to 127.0.0.1 https://redis.io/topics/security READ
- 67. #RSAC Redis How prevalent is this?
- 68. #RSAC Redis You can navigate the DB with the redis-cli
- 69. #RSAC Redis Or use the Redis Desktop Manager
- 71. #RSAC Redis Remote Code Execution (RCE) on Redis http://antirez.com/news/96 http://benmmurphy.github.io/blog/2015/06/04/redis-eval-lua- sandbox-escape/ https://gist.github.com/lokielse/d4e62ae1bb2d5da50ec04aadccc6edf1 Writable redis running as root? Get shell
- 72. #RSAC Redis Wanted to see how prevalent…what is that?!?!
- 73. #RSAC Redis Wanted to see how prevalent…what is that?!?! Altcoin miner!
- 74. #RSAC Redis How are they doing? $$$
- 75. #RSAC Redis How are they doing? $$$
- 76. #RSAC Redis Open Redis? Get shells
- 77. #RSAC memcache Free & open source, high-performance, distributed memory object caching system No code exec, but fun things get put into memcache Examples
- 78. #RSAC memcache
- 79. #RSAC memcache
- 80. #RSAC memcache
- 81. #RSAC memcache
- 82. #RSAC In-Memory Database (Solutions) Apply authentication (strong passwords!) AUTH for redis Bind to localhost if possible If possible, enable SSL/TLS Segment In-Memory Databases from Corp (and the public in general) Be aware of the data you put in these databases Don’t store keys, passwords, etc Logs Logs Logs
- 83. #RSAC Big Data
- 84. #RSAC Hadoop The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
- 85. #RSAC Hadoop Common Attack Points No authentication by default (Kerberos possible) Front Ends (Hue, Ranger, etc) https://hadoopecosystemtable.github.io/ Hadoop WebUI RCE via Hadoop Streaming Utility Great Resource on Hadoop Hacking http://archive.hack.lu/2016/Wavestone%20-%20Hack.lu%202016%20- %20Hadoop%20safari%20-%20Hunting%20for%20vulnerabilities%20- %20v1.0.pdf
- 87. #RSAC Hadoop Access gives you full HDFS access via the GUI
- 89. #RSAC Hadoop Defenses Use Kerberos Limit Exposed Hadoop Ports and Services Change default passwords Logs Logs Logs osquery
- 91. #RSAC Docker Common Docker Security Issues Protect Docker registry Vulnerable/Backdoored Docker Images (Lack of) Isolation of Containers Secrets in code Docker daemon == root
- 92. #RSAC Shipyard Shipyard (https://github.com/shipyard/shipyard) Shipyard enables multi-host, Docker cluster management. It uses Docker Swarm for cluster resourcing and scheduling. Default Creds: admin/shipyard
- 93. #RSAC Shipyard Default Creds: admin/shipyard Command exec if you can gain access
- 94. #RSAC Cloud Security - AWS Common AWS flaws
- 95. #RSAC AWS – Attack Exposed Credentials Vulnerable Applications/Systems Misconfiguration
- 98. #RSAC Exposed Credentials Stolen or lost machine Commit of dotfiles to a repo, gist, pastebin, etc. Commit source with keys in it Compromised developer/ops/etc. machine
- 99. #RSAC Exposed Credentials Keys are often stored on developer or ops machines Typically can be found under 1. ~/.aws/config 2. ~/.bashrc 3. ~/.zshrc 4. ~/.elasticbeanstalk/aws_credential_file
- 102. #RSAC Exposed Credentials More examples of AWS keys on GitHub
- 104. #RSAC Vulnerable Applications/Systems Once you have keys, utilize the interrogate tool to verify AWS permissions https://github.com/carnal0wnage/aws-interrogate The tool requests various functionality in order to determine authorization
- 105. #RSAC Vulnerable Applications/Systems Example of the tool in action
- 108. #RSAC Vulnerable Applications/Systems Machine is compromised Attacker grabs metadata info Uses these credentials to pivot
- 109. #RSAC Vulnerable Applications/Systems Browse to this address from compromised machine http://169.254.169.254/latest/meta-data/iam/security-credentials/ Obtain credentials here and pivot
- 110. #RSAC Vulnerable Applications/Systems Talk/tool to help with this process https://www.blackhat.com/docs/us-14/materials/us-14-Riancho-Pivoting-In- Amazon-Clouds-WP.pdf https://andresriancho.github.io/nimbostratus/
- 112. #RSAC Misconfiguration Insecurely Configured Services Lack of Monitoring Lack of IAM Hardening
- 114. #RSAC Misconfiguration - Insecurely Configured Services We’re going to provide examples of two services S3 – Insecure Bucket Policies RDS – Default Credentials
- 115. #RSAC Misconfiguration – Insecurely Configured Services Open S3 buckets is a very popular way to bring pain to your company Bucket permissions can be confusing and easy to mess up
- 116. #RSAC Misconfiguration – Insecurely Configured Services S3 has an interesting misconfiguration where buckets aren’t public but they are accessible to *any* AWS key.
- 117. #RSAC Misconfiguration – Insecurely Configured Services S3 has an interesting misconfiguration where buckets aren’t public but they are accessible to *any* AWS key.
- 118. #RSAC Misconfiguration – Insecurely Configured Services The misconfiguration appears to be “Any Authenticated AWS User” permission
- 119. #RSAC Misconfiguration – Insecurely Configured Services Review S3 buckets to determine security policy https://gist.github.com/cktricky/faf0f40116e535a055b7412458136917
- 120. #RSAC Misconfiguration – Insecurely Configured Services Rdsadmin = Default account created by AWS “To provide management services for each DB instance, the rdsadmin user is created when the DB instance is created.” Have found rdsadmin with blank or weak passwords
- 121. #RSAC Misconfiguration – Insecurely Configured Services
- 123. #RSAC Misconfiguration - Lack of Monitoring 127 AWS comes pre-packaged with services to do this Services CloudTrail = Logs CloudWatch = Alarms and Events Config = Change Management VPC Flow Logs = Network Activity Logs
- 124. #RSAC Misconfiguration - Lack of Monitoring CloudTrail is primarily used for log collection Other services like CloudWatch, for example, use those logs to filter relevant data
- 125. #RSAC Misconfiguration - Lack of Monitoring 129
- 126. #RSAC Misconfiguration - Lack of Monitoring 130 CloudTrail Config CloudWatch
- 127. #RSAC Misconfiguration - Lack of Monitoring An earlier talk on AWS security, dedicated to using these services: https://www.youtube.com/watch?v=g-wy9NdATtA&feature=youtu.be The gist is that it is very easy and yet often overlooked
- 128. #RSAC Misconfiguration - Lack of Monitoring Tool to list the monitoring services configuration: CloudWatch CloudTrail Config https://gist.github.com/cktricky/f19e8d55ea5dcb1fdade6ede588c6576
- 129. #RSAC Misconfiguration - Lack of Monitoring Output from an AWS environment we had keys for
- 130. #RSAC Misconfiguration - Lack of Monitoring We see a lack of monitoring time and time again Impact If the environment changes, nobody knows If your bill is being blown up, again, nobody knows Won’t detect malicious activity Won’t be able to perform incident response FINANCIALLY LIABLE TO AWS
- 131. #RSAC Misconfiguration - Lack of Monitoring 135 An example of creating an alert, that counteracts our interrogate tool shown earlier Creates an alert for Unauthorized Activity Event on our AWS account Is FREE and uses built-in AWS technology Reports specific details to Slack
- 132. #RSAC Misconfiguration - Lack of Monitoring http://www.slideshare.net/KenJohnson61/aws-surival-guide Shows you have to trigger for interesting AWS events and alert in Slack, etc.
- 133. #RSAC Misconfiguration - Lack of Monitoring Monitoring Takeaways There are MANY things you can do with AWS technology to alert yourself to issues – this was one example Review “Well Architected Framework” from AWS which discuss monitoring and other controls: — http://d0.awsstatic.com/whitepapers/architecture/AWS_Well- Architected_Framework.pdf
- 134. #RSAC Lack of IAM Hardening
- 135. #RSAC Misconfiguration - Lack of IAM Hardening IAM = User, Group, Roles, Access Policies, etc. – Management You CAN take steps to make it harder to use compromised credentials You CAN take steps to limit access to only required AWS assets You CAN replace the need to hardcode AWS keys in source code …. Its just that *very often*, people don’t
- 136. #RSAC Misconfiguration - Lack of IAM Hardening 140 IAM Hardening Checklist: 1. Don’t Use The Root Account! 2. Audit IAM user policies 3. Multi-Factor Authentication 4. Use Roles
- 137. #RSAC Misconfiguration - Lack of IAM Hardening Don’t Use the Root Account! Disable or delete the access keys Setup CloudWatch Alarm (shown in “previous talk” links)
- 138. #RSAC Misconfiguration - Lack of IAM Hardening Audit IAM Permissions Tool to inspect each user’s permissions: https://gist.github.com/cktricky/257990df2f36aa3a01a8809777d49f5d Will create a CSV file Provides you with — Usernames — Inline Policies — Managed Policies — Groups
- 139. #RSAC Misconfiguration - Lack of IAM Hardening Why this is important If you house sensitive data, you need to know who has access Permissions should be a need-to-have/know situation in order to limit damage should creds get stolen AWS is a flexible environment that changes – your permission model might need to change with it (inventory it)
- 140. #RSAC Misconfiguration - Lack of IAM Hardening Tool output
- 141. #RSAC Misconfiguration - Lack of IAM Hardening Multi-Factor Authentication (MFA) = 2 Factor Authentication Not just for the Web, place on the API as well
- 142. #RSAC Misconfiguration - Lack of IAM Hardening Use Roles Is *like* a user but is not an IAM user Replaces the need for hardcoded Access Key ID & Secret The extent of what a role can do is heavily controlled by you, the administrator Credentials automatically rotate via STS — Available here on an EC2 instance: http://169.254.169.254/latest/meta-data/iam/security-credentials/ If you’re using the AWS-SDK gem/egg/etc – credential handling is built-in If you’re using something like Paperclip + Rails, try Fog to leverage Roles — https://github.com/thoughtbot/paperclip/issues/1591
- 143. #RSAC Misconfiguration - Lack of IAM Hardening Example attaching Role to ElasticBeanstalk instance
- 144. #RSAC Conclusion Don’t prioritize speed over security Vulnerabilities are the same (what was old is new again) Developers now deploy and manage the full stack for their application(s) Equip & Educate them with ways to do this securely Developers possibly have the keys to the whole kingdom on their laptop. Protect and monitor those assets One token to rule them all
- 145. #RSAC Thanks and Contact Chris Gates Sr. Security Engineer Uber @carnal0wnage Ken Johnson CTO nVisium @cktricky For slides and URLs in this presentation: http://bit.ly/RSA-Devoops