How to create and manage a rock-solid DevSecOps framework

DevSecOps basics

DevSecOps is the addition of security considerations and practices to an organization's CI/CD workflow. DevSecOps does not replace existing development paradigms. Rather, it expands and complements those paradigms by adding a comprehensive layer of security throughout the development cycle.

DevSecOps is a way to solve the problem of developers reserving security checks and testing for the later stages of a project -- often as it nears completion and deployment.

Late-stage security checks lead to two problems. Security flaws can go undetected and make it into the released software, and security issues detected at the end of the software development lifecycle demand considerably more time, resources and money to remediate than those identified early on.

Together, ever-shorter development cycles and the need to manage project costs and mitigate business risks have resulted in a shift-left approach to software security. Shift left signifies moving security checks closer to the beginning of a project, imagining a timeline laid out left to right. The farther left security and other operational concerns move, the more incorporated they are into the design and creation of the product. The goal of DevSecOps is to design, build, test and deploy software in which developers assign as much importance and consideration to security as any other major software feature or functionality.

When implemented well, DevSecOps can yield the following benefits:

• reduced security risk to organizations, users and customers;
• detection of software flaws and vulnerabilities early in the product lifecycle;
• faster remediation of security issues;
• improved communication and collaboration across development, security and operations teams; and
• better overall code quality.

However, incorporating security into an existing Agile development process can pose several drawbacks:

• introduction of changes -- and possible bottlenecks -- to existing workflows;
• additional training and expertise necessary for both the development and security teams; and
• disrupted development workflows due to poor communication or initiatives that aim for zero vulnerabilities.

Create a modern DevSecOps framework

Since DevSecOps is an expansion or enhancement of existing software development practices, it's easiest to consider a DevSecOps framework in terms of the Agile approach to planning, coding, testing, deployment and ongoing operation.

Planning and design

DevSecOps starts early in the development lifecycle, typically in design or planning. For example, security concerns affect goals set during sprint planning. A DevSecOps framework brings security goals into the planning phase in the following ways:

• Create coding standards and conduct peer reviews. Security flaws can enter a product when developers write various sections of code in different ways. A team that establishes and enforces a consistent set of coding standards -- and evaluates a codebase against it -- can prevent mistakes that introduce vulnerabilities. Peer code reviews help ensure that software meets coding standards and that developers catch common programming errors.
• Use security plugins for integrated development environments (IDEs). IDEs are extensible platforms that often accept security plugins that check code for potential vulnerabilities, in the same way that the IDE would point out missing punctuation or syntax errors. IDE-based security checks offer developers static code analysis before they make any commits to a repository.
• Conduct threat modeling. Threat modeling encourages developers to approach an application as a hacker would. They should consider potential abuses of the application, ways to prevent those abuses and how to prioritize those preventative actions in the application design goals. Common threat modeling methodologies include STRIDE, DREAD and OWASP. Multiple threat models can be combined to strengthen design security.

Coding and code management

DevSecOps doesn't stop at security-minded coding standards and peer reviews. Put security at the center of codebase management and maintenance to avoid introducing vulnerabilities. Common techniques to ensure security in code commits and management include the following:

• Manage security in dependencies. External libraries, open source code and reused modules are common in software development. However, external code might not include the same security standards and precautions as internal work. Developers should review the security of all dependencies, verify that they are authentic and delivered securely, and ensure that the latest versions are used for each project.
• Scan code and repositories. Repository scanning tools can perform static analysis of code committed to repositories before build execution, checking for vulnerabilities, hardcoded credentials and other common oversights. Vulnerability testing and other static testing are vital for code security. Repository scanning adds safety for larger teams in which many developers access the same repository.
• Secure the development pipeline. Upon detecting a security issue, a comprehensive DevSecOps framework might adjust the development workflow to prevent code commits to default or trunk branches until the issue is remediated. An attacker could also compromise the pipeline itself by introducing malicious code or stealing credentials. Consequently, organizations should implement and review security controls within their development pipelines.

Testing

Testing should detect application flaws and security issues. Automation and orchestration in build, test and release pipelines should include running security tools when code is deployed for testing -- for example, checking for vulnerabilities during unit testing. Common test considerations for a DevSecOps framework include the following:

• Integrate dynamic application security testing (DAST). DAST and penetration testing have historically been the last step in the development cycle. DevSecOps adherents should add DAST, pen testing and other types of dynamic vulnerability testing to the build's test regimen within the pipeline. Full DAST and other dynamic vulnerability testing, such as security acceptance testing, can be time-consuming, but lighter test regimens are an option that can yield faster results, while identifying issues missed in static testing.
• Secure the infrastructure. DevSecOps security considerations should go beyond the application itself to include the deployment environment, whether local or cloud infrastructure. Consider implementing policy-driven VMs, containers and Kubernetes clusters. Tools such as Microsoft Azure Policy and AWS Organizations enforce security-driven policies on cloud infrastructures. Infrastructure as code is one way to build standard and known-good application environments.

Deployment and operation

Security concerns do not end with a successful build. DevSecOps practices carry forward into deployment and operational environments with the following:

• Configuration management. The infrastructure environment created to host the application must be stable. Any attempt to change an established infrastructure configuration could be malicious activity. Tools that monitor and enforce an infrastructure configuration should be a central element of an organization's DevSecOps framework. Cloud providers offer these tools as well, such as Microsoft Defender for Cloud and Microsoft Sentinel.
• Intrusion detection and behavioral analytics. Analytics tools, including intrusion detection and prevention systems, establish baselines in traffic patterns and performance and then look for anomalies indicative of suspicious or malicious activity in the workload or network. Such tools are well established and should be embraced by organizations adopting DevSecOps.
• Ongoing security testing. DAST, pen testing and other types of security testing should not end with the testing phase of the application's lifecycle. Conduct testing, such as port scanning and fuzz testing, routinely and whenever the team suspects the code has new weaknesses. For example, if a newly discovered bug appears in a processor's command set, the operations or security support administrators should act. When testing confirms potential vulnerabilities, they can develop and deploy new patches.
• Alerting and reporting. Security tools and policies must be paired with comprehensive alerting and reporting. Developers and project stakeholders should receive actionable intelligence from operations in a way that supports prompt issue identification and remediation.
• Post-mortems. Even with the best security efforts, an organization is likely to eventually experience security issues in its application or infrastructure. When a security incident occurs, it's important to conduct blameless post-mortems. Teams should work to identify and remediate the issue and then use the experience to tune future development and operational efforts to prevent subsequent issues.

DevSecOps tools

Organizations shifting to DevSecOps should evaluate their toolchain -- and each tool within it -- for security tasks. Some organizations already use DevSecOps-ready tools, while others will need to update or replace tools. While the list of potential DevSecOps tools is too large to include in full here, the following are some common examples of suitable tools for each major phase of the DevSecOps pipeline:

Planning and design. Team discussions and collaboration on security considerations highlight potential oversights and risks for the project and workflow. Common tools used for planning include issue-tracking and management tools, like Atlassian Jira, and communication tools, like Slack.

Threat modeling is a major planning issue under DevSecOps. Specialized tools, like IriusRisk, CAIRIS, Kenna.VM, Microsoft Threat Modeling Tool, SD Elements, securiCAD, Tutamantic, Threagile, ThreatModeler and OWASP Threat Dragon, provide threat modeling capabilities.

Coding and code management. Developers prepare secure code using static code analysis, IDE plugins like pre-commit hooks and repository scans. Code review tools include PMD, Checkstyle, Gerrit, Phabricator, SpotBugs and Find Security Bugs. When choosing a code review tool, select one designed for the project's programming language and IDE or toolchain interoperability.

DevSecOps security practices in the build phase include software component analysis, static application software testing and unit testing that analyzes the new code, as well as any dependencies. Common tools for build analysis include SonarQube, SourceClear, OWASP Dependency-Check, Retire.js, Snyk and Checkmarx.

Testing. The test phase is focused on using DAST to identify vulnerabilities associated with application operations, such as user authentication, authorization, SQL injection and API endpoints. DAST tools include Invicti (formerly Netsparker), Astra Pentest, Acunetix, PortSwigger, Detectify, Rapid7, Mister Scanner, AppScan and AppCheck.

General DevSecOps testing tools that address attack proxies and network protocol fuzzing include Boofuzz, OWASP Zed Attack Proxy, BDD-Security, IBM AppScan, Gauntlt, JBroFuzz and Arachni. Choose tools based on their suitability for the application type and tests needed and their interoperability with other tools in use.

Deployment and operation. A DevSecOps release candidate should be securely coded, build-checked and thoroughly tested. Next, DevSecOps teams establish a hardened operating environment for the release. They set up access control, network firewall access and secrets management. Change and configuration management tools are central to a DevSecOps model at the deployment stage. Common configuration management tools include Red Hat Ansible, Chef, Puppet, Salt, HashiCorp Terraform and Docker.

DevSecOps teams then focus on situations in the live runtime environment. Differences between testing and production environments should be identified and studied carefully, as they are often a sign of security issues. Runtime performance tools include Osquery, Falco and Tripwire.

In addition, teams use chaos engineering tools, like Chaos Monkey and Gremlin, to evaluate a deployment for -- perhaps untested -- faults, such as server crashes, drive failures and network connectivity issues. The aim is for the deployment to either survive the disruption or fail gracefully.

Feedback loops are core to DevSecOps success. Tools are useless unless the results they produce are cycled back into the development process. Take advantage of reporting and analytics across the toolchain to evaluate the security status of the current release, and use that insight to improve the next development cycle.