Devops - Accelerating the Pace and Securing Along the Way - Thaddeus Walsh
- 1. DevOps - Pushing the Pace and Securing Along the Way DevOps Columbia Meetup, Thurs, 9/27/2019 Thaddeus Walsh
- 2. Who the Heck is This? • Solution Architect for ZeroNorth (Risk Orchestration platform, HQ in Boston, MA) • Formerly a Customer Success Manager at Tenable Inc. • Decade in Enterprise IT Solutions Field • Background includes ITIL/Storage Infrastructure/Systems of Record/Asset Management/Vulnerability Management/AppSec • UMBC Alum (Class of ‘08) – don’t ask about my major • Live in Columbia, MD with my wife and 2 yr. old son
- 4. The Problem Defined: The Security Processes/Practices/Tools that were developed to secure Agile/Waterfall/Monolithic development methodologies don’t work in the context of a DevOps ecosystem
- 6. Who Here? • Has Negotiated or Witnessed a Negotiation with Security/AppSec to exempt an application from mandatory steps in Security Validation Processes? • Has seen a build get blocked due to the presence of a Security Vulnerability? • Has had Security come to them about embedding SAST/SCA tools into build pipelines?
- 7. The “Proactive” Security Lifecycle SCA SAST Container DAST TVM Assessing: Open Source Code Libraries & Packages Source Code or Compiled Code, Binaries, Build Artifacts Container Images Running Web Application Running Host, OS, Network Endpoint Very Fast & Easy Slow, Source of most False- Positives, Dumpster-Fire Basically SCA for a Container Image Overcoming Authentication is the Key to good findings – (Risks) Scanner as close to target as possible – (Risks)
- 8. How did Security Manage pre-DevOps (pdo.) • Review Meetings • Roundtable discussions regarding changes to the application. Traditional Change Risk Analysis (Likelihood the change will cause an incident/Impact of prospective incident) • Security Gates • Policy states: Thou Shalt Not … X • Test for X, if no X, then release! • Tool Based Vulnerability Detection (*this one’s important*) • Remediation Assistance & Developer Training • Code Remediation, RCA/Vuln Correlation, CBT, IBT, etc.
- 10. UUTF Happened – this doesn’t work now
- 11. What used to take 1-2 weeks… Now happens 3-7 times a day While Developers were accelerating delivery, Security was fruitlessly trying to pump the breaks
- 12. The Root of the Impasse • Security’s PDO processes aren’t viable in the timeframe afforded by an aggressive release schedule • This was never a problem in the past because delivery/deployment was laboriously slow (compared to today) • Rather than acknowledge the world was changing, many Security teams doubled down on their Status Quo processes • Many Security teams became ex communicated from development touchpoints because they were seen as barriers to execution and delivery
- 13. Blame it on the Vendor! • Security Tool Vendors were already competing on speed of assessment, meaning to get much faster, they would have to rebuild their assessment engines from the ground up • Security Tool Vendors were all 1-2 years behind on building in support for pipeline-initiated scans (customer driven tools react only after market demand for the feature justifies the investment, Scanning Tool Vendors *still* aren’t DevOps shops themselves) • Tool-sourced findings were/are still mostly useless to dev • False Positives, Duplicative/Overlapping Results, Unclear path to Resolution • Ticket Overhead for days
- 14. But, But, But… Security is still really freaking important
- 15. Ok, so we can’t completely ignore Security… But How Does Security Fit Into This Hot Mess?
- 16. Security is just another measure
- 17. Step 1 – Security has to Change Perspective • Vulns you know about aren’t more risky than vulns you don’t know about • All your apps have unknown vulns – and that’s OK • There is some level of Risk that is acceptable in Production • We can fix issues as we identify them • Co-opt the expedience of DevOps to quickly fix Security issues • Just because the Vuln wasn’t fixed first doesn’t mean it’s not being worked on
- 18. Step 2 – So now that’s out of the way… • Rules for Security: • No ”Hard Gates” • No disrupting builds or build systems • Asynchronous Scanning is the primary mode of operation • Human interventions should be the non-default behavior • Security Defects must be communicated in the same vehicle/format as functional defects • No ticket bloat – the fewest tickets necessary to solve the problem, with as much info as necessary to remediate, and no more • Compensating Controls are valid solution (though, WAF rules are not a forever solution) • Rollback is the undesirable, but acceptable lever for unacceptable Prod Risk • Most of the threat surface area is humans (doing dumb things) and unpatched systems (as a result of humans doing dumb things)
- 19. Step 3 – On to the Dev Side of the Table • Rules for Dev: • An app that leaks customer/patient/company data is as bad as an app that does no work • Security issues are equally important as functional or performance issues • You can build great applications securely and quickly • If you don’t understand a Security issue, AppSec is there to help • False-Positives must be proven (operate in good faith) • You will need to understand required Inputs for Security Tools • You will eventually have a build rolled back for security reasons • Security needs to be aware of the new hotness you’re using (Golang, etc.)
- 20. What does this look like?
- 21. Commit Build Test Evaluate Deploy Static Assessment Software Composition Analysis Container Scan Dynamic Assessment TVM Assessment
- 22. Another Example
- 23. 23
- 24. Parting Thoughts • Not every security check must come from a tool – (custom linters) • Knowing what you have and where it is = Half the Battle • CVSS Sucks (ask me why!) • Bookmark NVD (https://nvd.nist.gov/), Follow @CVEnew on Twitter • Keep CI/CD Platforms Updated! No Excuses! (https://jenkins.io/security/advisory/2019-09-25/) • If this is a problem you’re looking to solve, I have business cards Thaddeus Walsh – Solution Architect, ZeroNorth Inc. – Columbia, MD
Editor's Notes
- #24: Let me separate this into two threads. The first is that these guys are automating from top to bottom and putting their business processes in the cloud. They are replacing people, going digital, creating sensors, and everything else. Netflix says they have software-defined the continually-delivered and constantly failing processes that run our business. They’ve defined the business process and the underlying technology components for resilience. With their DevOps, the human plugs into the machine-driven process as the exception, just like automation began as individual complements to the manual process. Herein are two levels of Spinnaker processes exemplified. Note two things: There’s a multi-level abstraction at play here. This isn’t just the simple mechanics of a SW build process (CI) that’s been taken over by software. Instead, see the break-out and representation of business processes as they cross departmental boundaries. Even this ‘Simple’ example describes what we’d consider to be an “alternative flow,” in user stories – behavioral areas often left uncovered. Here we can see a commitment to “Design for failure” built into the fore of the software, its processes and controls, and in the examples/tutorial for the product (Spinnaker’s) most basic use. As a bonus, you can also see that the human is included in this process, basically, as a “value add” as opposed to an essential element that would resist replacement. Humans, in essence, are an optimization designed to enhance the speed of conclusion rather than a hold-out – some value that resists automation. Bob has to keep up to us. We’ve augmented Bob with canaries and if Bob is too slow, we’ll just bypass him.