![10
Finding Instances
• Usually port 8080, also check 8443,80,443,
other alternates like 9090,8888
• Nmap Script Engine (NSE) (see right)
• HTTP screenshot tools like Aquatone or
WitnessMe
Script Name Info
http-title Look for titles containing
“[Jenkins]”, e.g., “Sign In
[Jenkins]” or “Dashboard
[Jenkins]” (← anonymous
read!)
http-server-header Jenkins will usually use
Jetty
http-enum In case of context path
issues
http-headers Look for X-Hudson* and X-
Jenkins* headers
http-devframework Includes fingerprints for
Jenkins (based on headers)
broadcast-jenkins-discover See “Auto-discovering
Jenkins on the network”](https://image.slidesharecdn.com/jenkins-newtemplate-200207044932/85/Intro-to-Pentesting-Jenkins-10-320.jpg)
![18
• Exploit URL:
/descriptorByName/org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript/checkScript?sandbox
=True&value=[groovy code]
– (Jenkins implements DescriptorByNameOwner) ServletContext.getAttribute("app").
– (SecureGroovyScript)
getDescriptorByName("org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScr
ipt").
– (FormValidation) doCheckScript("[groovy code]", true)
• Part of Script Security plugin: not core Jenkins, but very common
• Prerequisites: vulnerable plugin & Jenkins versions, read access (including anonymous)
• Brian’s super-duper top-secret protip: public class x{public
x(){org.kohsuke.stapler.Stapler.getCurrentResponse().getWriter().println("hello world")}}
CVE-2019-1003029/CVE-2019-1003005 (doCheckScript)
Dynamic Routing](https://image.slidesharecdn.com/jenkins-newtemplate-200207044932/85/Intro-to-Pentesting-Jenkins-18-320.jpg)
![24
Lateral Scripting
• Script console (when you have permission) &
build steps can be run both on Jenkins
instance itself and on build nodes/agents
• Usually connected through either:
– Master → agent SSH connection
– Agent → master TCP connection
• Agents may be different:
– OS (e.g., Linux master with [potentially domain-
joined] Windows agent for C# projects)
– Environment/segment (e.g., master in dev,
agent in / ACL’d to prod [CDE?!] for
deployment)
If you have Overall/RunScripts access, the “Script Console” menu item will show up for a specific machine when you
go to its page (i.e., click it)](https://image.slidesharecdn.com/jenkins-newtemplate-200207044932/85/Intro-to-Pentesting-Jenkins-24-320.jpg)
![30
Password Cracking
• Potentially helpful for lateral movement to other Jenkins instances, other dev infrastructure
(Artifactory, Nexus Repository Manager, GitHub Enterprise, JIRA, etc.)
• Hashes stored in $JENKINS_HOME/users/[username]/config.xml
• Modern versions of Jenkins use bcrypt
– Potentially still worth it to try with a targeted wordlist + rules: jenkins, [company name], scm,
password, etc.
• Older versions & users that have not logged in since upgrade (I think?) use salted SHA256
– Format: salt:sha256("password{salt}") (from https://www.philipp.haussleiter.de/2011/11/jenkins-
password-hash-format/)
– Crack by replacing all salts with {salt} and using password.salt as format
– echo "salt:hash" | awk -F: '{print $2":{"$1"}"}'](https://image.slidesharecdn.com/jenkins-newtemplate-200207044932/85/Intro-to-Pentesting-Jenkins-30-320.jpg)
Intro to Pentesting Jenkins
- 1. 1 Intro to Pentesting Jenkins February 6th, 2020 Brian D. Hysell
- 2. 2 • Consultant @ Synopsys Software Integrity Group – Application Pentesting, Network Pentesting, Secure Code Review / Guidelines / Training, etc. – Mostly just grep for passwords, rack up hotel status, write reports • Previously: Consultant @ Accenture Security Threat & Vulnerability Management • Like TV, travel, baking About Me
- 3. 3 Intro to Jenkins • About Jenkins • Visual Tour • Why Care about Jenkins? • Finding Instances • Initial Access
- 4. 4 About Jenkins • Most popular CI/CD system – probably 50- 70% marketshare – Open source – Rich plugin ecosystem – Long history of security vulnerabilities • Forked(-ish) from project called Hudson • Used for all sorts of common dev (& often ops) tasks: – Running builds, tests – Publishing built artifacts (NRM, Artifactory, etc.) – Deploying software – Checking uptime/health – Stopping/starting/restarting servers – Applying patches
- 9. 9 Securing Jenkins (internal or external) should be a major concern across red, purple, and blue teams: • Supply-Chain Integrity: Compromise could lead to backdoored apps deployed to production or customer environments • Intellectual Property Protection: Access to Jenkins can give an (possibly malicious) user source code, binaries, and knowledge of infrastructure/architecture • Privileged Access Management: Instances often contain credentials for source & artifact repositories, test & production systems, etc. • Segmentation: Jenkins supports distributed executors which may span different environments (dev/prod, PCI/non-PCI, different AD domains, etc.) Why Care about Jenkins?
- 10. 10 Finding Instances • Usually port 8080, also check 8443,80,443, other alternates like 9090,8888 • Nmap Script Engine (NSE) (see right) • HTTP screenshot tools like Aquatone or WitnessMe Script Name Info http-title Look for titles containing “[Jenkins]”, e.g., “Sign In [Jenkins]” or “Dashboard [Jenkins]” (← anonymous read!) http-server-header Jenkins will usually use Jetty http-enum In case of context path issues http-headers Look for X-Hudson* and X- Jenkins* headers http-devframework Includes fingerprints for Jenkins (based on headers) broadcast-jenkins-discover See “Auto-discovering Jenkins on the network”
- 11. 11 Initial Access • Some level of anonymous access is common, especially internally – Typically read-only – Sometimes administrative! • Check if sign-ups are enabled • Auth bypass: discussed in next section • Conventional techniques: password spraying, brute force, credential stuffing • Get creative; I have found Jenkins creds in: – Build logs of another Jenkins instance – Job env vars on another Jenkins instance – Python scripts in company Artifactory install – Entry point of a Docker container running on a compromised box • Look for both passwords and API keys From http://tuyenvuong.me/programming/deploy-web-application-jenkins-capistrano/ & https://commons.wikimedia.org/wiki/File:Emojione_1F630.svg
- 12. 12 Scripting & Exploitation • Selected RCE Attack Vectors • Script Console • Build Steps • Dynamic Routing
- 13. 13 Selected RCE Attack Vectors Vector CVE Access Level Script Console N/A • Overall/RunScripts enabled for anonymous; or • Compromised credentials for user with Overall/RunScripts Build Step N/A • Compromised credentials for user with Job/Build and Job/Configure permissions Script Security Plugin doCheckScript method CVE-2019-1003029, CVE-2019-1003005 (POC) • Overall/Read access enabled for anonymous; or • Compromised credentials; or • Authentication bypass CVE-2018-1000861 Jenkins Remoting (deserialization) CVE-2015-8103 (POC1) • Jenkins CLI port open 1 Protips: https://github.com/NickstaDB/ysoserial/tree/templatesimpl-reverse-shell & https://github.com/BrianOfTheCosmos/broodwich/blob/master/deserialization/src/main/java/party/itistimeto/broodwich/deserialization/CommonsCollectionsThreePayload.java
- 14. 14 Script Console • Script console – Run arbitrary Groovy code: read/modify Jenkins internals, read/write files, run external commands – Requires Overall/RunScripts permission – should be restricted to admins, but could be applied to lower-privileged users, even anonymous (!) • REST API is convenient: https://support.cloudbees.com/hc/en- us/articles/217509228-execute-groovy-with-a- rest-call – On older versions (<2.96), need Jenkins-Crumb header from /crumbIssuer/api/json: https://wiki.jenkins.io/display/JENKINS/Remote+a ccess+API#RemoteaccessAPI-CSRFProtection • Jenkins CLI can also be nice (useful for other scripting / enumeration tasks as well) • Generic Java RCE tip: make sure to use Process.waitFor() From https://2019.pass-the-salt.org/files/slides/12-Hacking_Jenkins.pdf From https://issues.jenkins-ci.org/browse/JENKINS-46526
- 15. 15 Build Steps • See if you can create or edit (and then run -- separate permissions!) projects/builds • Builds can contain “Execute shell” or “Execute Windows batch command” steps with code of your choice • Can choose which node to run build on, including master: https://stackoverflow.com/questions/9841156/stick- jobs-to-the-master • Be careful: – If you create a project, delete it after building (if able) – If you have to edit an existing project’s build steps: – Back up config (can do with Jenkins CLI) – Disable email, Slack, etc. notifications – Don’t deploy anything – Check relationships to other projects, triggered builds, etc. – Etc. Overview From https://docs.bitnami.com/installer/apps/jenkins/configuration/use-custom-executable/
- 16. 16 Build Steps • Many plugins allow execution of user-defined Groovy code, usually as part of build – E.g., Groovy plugin, workflow-cps (“Pipeline: Groovy”) plugin – Other examples: https://wiki.jenkins.io/display/JENKINS/Script+Security+Support+i n+Plugins • Most integrate with Script Security plugin: scripts must be approved by admin or run in sandbox – E.g., Groovy plugin’s “Execute system Groovy script” build step, “Use Groovy Sandbox” option • Sandbox blacklists various classes/methods; see https://github.com/jenkinsci/script-security- plugin/blob/master/src/main/resources/org/jenkinsci/plugins/script security/sandbox/whitelists/ • Sandbox escapes frequently identified; check regression tests in script-security repo for POC: – SECURITY-1579: https://github.com/jenkinsci/script-security- plugin/commit/415b6e2f3fa0c2e4bd2f9c4a589a9e1fc9cbac8b – SECURITY-1658: https://github.com/jenkinsci/script-security- plugin/commit/0e7da14171ed1d03ff72f6910392e630b40a8590 • Sandbox escapes occasionally applicable with only Overall/Read (!), see https://jenkins.io/security/advisory/2017-04-10/ Sandbox Escape Hint: Cause an exception to print output, e.g., java.nio.file.Paths.get(expression + "0") → InvalidPathException
- 17. 17 Dynamic Routing • Jenkins uses a web framework called Stapler – Same original author as Jenkins (née Hudson) • Stapler parses URLs & calls methods in an EL-like manner against a root object fetched with ServletContext.getAttribute("app") • hudson.WebAppMain.contextInitialized (): “Creates the sole instance of jenkins.model.Jenkins and register it to the ServletContext” • TLDR: URLs in Jenkins are chains of getter and doer method calls against a Jenkins singleton or objects returned higher in the chain Introduction From https://jenkins.io/doc/developer/handling-requests/routing/ Note: if, like I did, you’re wondering about the inconsistency between Jenkins#getLog() and Hudson#getJob() (i.e., which is actually root?): Hudson is a subclass of Jenkins, and it’s an object of that type that hudson.WebAppMain.contextInitialized() registers as root
- 18. 18 • Exploit URL: /descriptorByName/org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript/checkScript?sandbox =True&value=[groovy code] – (Jenkins implements DescriptorByNameOwner) ServletContext.getAttribute("app"). – (SecureGroovyScript) getDescriptorByName("org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScr ipt"). – (FormValidation) doCheckScript("[groovy code]", true) • Part of Script Security plugin: not core Jenkins, but very common • Prerequisites: vulnerable plugin & Jenkins versions, read access (including anonymous) • Brian’s super-duper top-secret protip: public class x{public x(){org.kohsuke.stapler.Stapler.getCurrentResponse().getWriter().println("hello world")}} CVE-2019-1003029/CVE-2019-1003005 (doCheckScript) Dynamic Routing
- 19. 19 • Exploit URL: /securityRealm/user/admin/descriptorByName/… – (Jenkins) ServletContext.getAttribute("app"). – (HudsonPrivateSecurityRealm) getSecurityRealm(). – (User implements DescriptorByNameOwner) getUser("admin"). – (Descriptor) getDescriptorByName("...") • https://devco.re/blog/2019/01/16/hacking-Jenkins-part1-play-with-dynamic-routing-en/ • Poorly-documented aspect of bypass: getUser(String) is method of HudsonPrivateSecurityRealm (default implementation), not of SecurityRealm abstract class CVE-2018-1000861 (Authentication Bypass Exploit) Dynamic Routing
- 20. 20 Unauth RCE with CVE-2019-1003029/CVE-2019-1003005 (doCheckScript) Dynamic Routing Diagram from: https://2019.pass-the-salt.org/files/slides/12-Hacking_Jenkins.pdf See also: https://github.com/orangetw/awesome-jenkins-rce-2019, https://github.com/orangetw/awesome- jenkins-rce-2019/pull/2
- 21. 21 Post-Exploitation & Lateral Movement • Embracing Groovy • Lateral Scripting • Scraping Jenkins • Raiding Stored Credentials • “Persistence” with API Keys • Password Cracking • Miscellaneous Post-Exploitation
- 22. 22 Embracing Groovy Operation Code Spawn reverse shell https://gist.github.com/frohoff/fed1ffaab9b9beeb1c76 List directory contents (non- recursive) new File('.').eachFile { println(it) } Print file contents println new File('/path/to/file').getText('UTF- 8') Nslookup println InetAddress.getByName('google.com').getHostAddr ess() • Groovy is a JVM-based scripting language • Essentially a superset of Java: supports Java syntax and APIs, plus its own syntax extensions and standard library • Some things you might normally do by dropping into an OS shell you can do straight in Groovy Basics
- 23. 23 Embracing Groovy Jenkins Internals Programmatically add API token From https://support.cloudbees.com/hc/en-us/articles/115003090592-How-to-re- generate-my-Jenkins-user-token List all admin users From https://stackoverflow.com/a/34274738 If only these could be combined somehow… 😉 More script samples: • https://github.com/cloudbees/jenkins-scripts/ • https://github.com/samrocketman/jenkins-script-console-scripts
- 24. 24 Lateral Scripting • Script console (when you have permission) & build steps can be run both on Jenkins instance itself and on build nodes/agents • Usually connected through either: – Master → agent SSH connection – Agent → master TCP connection • Agents may be different: – OS (e.g., Linux master with [potentially domain- joined] Windows agent for C# projects) – Environment/segment (e.g., master in dev, agent in / ACL’d to prod [CDE?!] for deployment) If you have Overall/RunScripts access, the “Script Console” menu item will show up for a specific machine when you go to its page (i.e., click it)
- 25. 25 Scraping Jenkins • Internal Jenkins instances very often have Overall/Read permission set for anonymous users – Compromised credentials, sign-ups apply here too, of course • “Just” read access to Jenkins can be a goldmine – Build logs – Environment variables – Project workspace (source code) – Build artifacts • REST API is easiest, no need to parse HTML • Tools: – https://github.com/DolosGroup/Jenkins-Pillage: build logs, env vars; only lists URLs – https://github.com/lc/jenkinz: build logs – https://github.com/gquere/pwn_jenkins/blob/master/dum p_builds/jenkins_dump_builds.py: build logs, env vars – Jenkins CLI client + Bash Overview & Tools From https://medium.com/@aseem.shrey/mind-your-logs-how-a-build-log-from-a-jenkins-leaked-everything- 603cf07fa85
- 26. 26 • No magic, just search/grep for things like: – “-p”, “--password” – “Authorization: Basic” – “net use” and similar (Windows creds) – “jenkins-cli.jar” (Jenkins API keys) – eyJ… (JWT) • Have found lots of creds this way: domain users, Jenkins API keys w/ Overall/RunScripts, SSH creds, creds to other parts of deployment infra • Cannot emphasize enough how useful searching through this info can be for lateral movement • Don’t forget to check workspaces too – Artifacts if you’re desperate Finding Credentials Scraping Jenkins
- 27. 27 Raiding Stored Credentials • Jenkins has a built-in mechanism to store credentials used by build pipelines • Symmetrically encrypted with key stored on disk • With script access: run a Groovy script like https://github.com/DolosGroup/Jenkins- Pillage/blob/master/groovy/decrypt-credentials.groovy • With file system access: grab secrets/master.key, secrets/hudson.util.Secret, and credentials.xml (or other files containing encrypted data) – https://github.com/gquere/pwn_jenkins#files-to-copy-after-compromission – https://github.com/gquere/pwn_jenkins/blob/master/offline_decryption/jenkins_offline_decrypt.py – https://gist.githubusercontent.com/menski/8f9980999ed43246b9b2/raw/ebac1b809ac4c10d4aa2c49b 8bb2a6c37dc4808a/jenkins-decrypt.py – https://github.com/tweksteen/jenkins-decrypt/blob/master/decrypt.py • N.B.: many plugins erroneously store passwords in plain-text config files and/or print them into the value attributes of password inputs on the config page Built-in Storage
- 28. 28 Raiding Stored Credentials Plugins From https://plugins.jenkins.io/mask-passwords
- 29. 29 “Persistence” with API Keys • Jenkins <= 2.128: API keys are stored in users/<USERNAME>/config.xml under same symmetric encryption scheme as credentials (may also apply to newer versions with legacy tokens carried over in upgrades) – Decrypt offline with aforementioned scripts – Or read value and then decrypt with println(hudson.util.Secret.decrypt("{...}")) – Or do 100% in-memory with script & Jenkins internal classes • Newer versions are hashed, but support multiple tokens per user – See previous slide for adding new tokens
- 30. 30 Password Cracking • Potentially helpful for lateral movement to other Jenkins instances, other dev infrastructure (Artifactory, Nexus Repository Manager, GitHub Enterprise, JIRA, etc.) • Hashes stored in $JENKINS_HOME/users/[username]/config.xml • Modern versions of Jenkins use bcrypt – Potentially still worth it to try with a targeted wordlist + rules: jenkins, [company name], scm, password, etc. • Older versions & users that have not logged in since upgrade (I think?) use salted SHA256 – Format: salt:sha256("password{salt}") (from https://www.philipp.haussleiter.de/2011/11/jenkins- password-hash-format/) – Crack by replacing all salts with {salt} and using password.salt as format – echo "salt:hash" | awk -F: '{print $2":{"$1"}"}'
- 31. 31 Miscellaneous Post-Exploitation • In theory, even when Jenkins does not creds to directly deploy artifacts, you may be able to move laterally by backdooring them – E.g., push backdoored JAR to private Maven repo & wait for it to be downloaded / deployed by another person / system – Probably too risky & too time-consuming for most pentests; difficult to control scope – This & similar attacks should still be kept in mind while threat modeling, though • Jenkins can serve arbitrary static files out of $JENKINS_HOME/userContent – may be good for exfiltration • Regular user session or remember me cookies and API keys can (Usually? Always?) be used interchangeably – Regular user session can access API – API key can access UI
- 32. 32 Conclusion • Real-World Attack Paths • Recommendations for Security & Ops Engineers
- 33. 33 • Searched for “jenkins” in an artifact store • Found several Python scripts containing Jenkins usernames and API keys • Hostnames in script incorrect/down, so sprayed against known Jenkins instances • Discovered one valid instance-username-key combination • Enumerated projects with Jenkins CLI; had configure access to one • Backed up config; replaced with system Groovy build step: sandbox escape, add & print API key for all admin users • Used admin key to access “Configure System”; Mask Passwords Plugin config contained password for master deployment service account in clear text Example #1 Real-World Attack Paths
- 34. 34 • Found Jenkins instance A with anonymous read enabled • Scraped build logs & found API creds for Jenkins instance B • Used script console (over REST API) to grab SSH private key from agent X (connected to B) • SSH’d into Jenkins instance C, then from instance C into agent Y (on PCI network) • Using key on agent Y, SSH’d into prod PCI hosts Example #2 Real-World Attack Paths
- 35. 35 • Inventory Jenkins systems and patch, patch, patch! • Apply least privilege via matrix-based authorization (preferably project-based matrix) – Never have anonymous script console access – Even “Logged-in users can do anything” is too risky – Anonymous read access may not be terribly risky per se – but disabling is an easy win • Don’t leak creds in build logs, environment variables, workspaces*, artifacts*, etc. – *Dependent on your org’s maturity WRT credential management • Pay requisite attention to Jenkins instances with privileged access: – Access to prod, CDE, publicly-released builds, etc. – Access to master deployment keys/passwords/accounts Recommendations for Security & Ops Engineers
Editor's Notes
- #24: https://support.cloudbees.com/hc/en-us/articles/115003090592-How-to-re-generate-my-Jenkins-user-token https://stackoverflow.com/questions/28953687/how-to-identify-admins-on-jenkins-using-groovy-scripts