The 15th day’s post of Mercari Advent Calendar 2020 is brought to you by Gloria Chow from the Mercari Product Security Team.

Hello, this is @gloria from the Product Security Team. Previously, I wrote about Testdeck, a tool developed and used at Mercari for microservices QA and security testing that was recently open-sourced.

This time, as a part of our Advent Calendar event for 2020, I decided to write about DevSecOps— a term that you may have already heard as it has been gaining attention in the industry in recent years. In this article I will introduce the basic concepts of DevSecOps, why it is important, some challenges that are commonly faced, and how we apply it to our work here at Mercari.

What is DevSecOps?

First, let us start with a quick recap of its predecessor, DevOps. DevOps is an approach to software development that combines Development (coding and testing) with Operations (release and monitoring). It consists of four key pillars: Continuous Integration, Continuous Testing, Continuous Delivery, and Continuous Monitoring.

Traditionally, a developer’s job was to simply code a feature, pass it to QA to test, and then to the Operations team to handle the rest. However, in DevOps teams, developers not only code, but they are also actively involved in the testing, deployment, and monitoring of the very software that they create.

One key trait of DevOps teams is what is known as “Shift-Left Testing”— since developers are heavily involved in the testing of their own code through unit testing and other forms of automated testing, the bulk of testing effectively shifts from the Testing phase of the software development lifecycle (SDLC) into the Development phase. This shift allows bugs to be found and fixed earlier rather than waiting until immediately before release to begin testing.

Now going back to DevSecOps. As its name suggests, it combines a third key party into the DevOps process, Security, and adds a fifth key pillar— Continuous Security. This means that security becomes an integral part present in every part of software development— instead of Security Engineers stepping in only after development of a feature is complete, security becomes present in every stage of the SDLC and is involved in every medium, from the application itself to the infrastructure (containers, servers, network, etc.) and technology stack.

As with DevOps, developers in DevSecOps teams take on responsibility for the security of their code by applying practices such as secure coding and security testing in their daily work. The practice of integrating security tools into the DevOps process is known as Security as Code (SaC) because its purpose is to add automated security checks, tests, and scans to the development flow without causing any negative impact or delay. Thus, security becomes constant and effortless— something that is always present rather than being an afterthought.

Why Is It Important?

Below are three reasons for why many companies today are interested in applying DevSecOps to their development processes.

Shorter Release Cycles and Quicker Releases

In DevOps, testing is shifted left to begin in the Development phase, bugs are found earlier and testing is conducted in a gradual manner rather than as one big chunk right before release. The same applies to security— traditionally, security activities such as penetration testing came in only at the end of the lifecycle but under DevSecOps, they begin in the early design stages and continue to be present in every stage of development.

DevSecOps practices effectively remove the famous QA and Security bottleneck problem in two ways:

1. It reduces the chances of serious bugs and security vulnerabilities being found (and needing to be fixed) at the last minute immediately before release. At traditional Waterfall and Agile organizations, teams often find QA and Security becoming blockers to their release as they uncover major issues at the last minute, but under DevSecOps, the chances of this occuring are greatly reduced as problems are usually found and fixed earlier.

2. Developers no longer have to sit around waiting for QA and Security to do their thing! Since developers are the ones who know their code best, it is more efficient for them to write tests as they are coding (or even before coding, a well-known practice known as Test-Driven Development) as opposed to handing off the work to QA afterwards. As for security, it is simply more efficient to build security into the software from the beginning instead of trying to make an insecure software into a secure one as that typically involves intrusive changes such as changes to the architecture or feature logic itself.

Another defining characteristic of DevSecOps teams is to rely heavily on automation. QA and security tests are run frequently, as often as on every code commit, to ensure consistency of software quality and security. This allows DevSecOps teams to release much quicker than teams in Waterfall or Agile organizations that don’t make use of automation.

Increased Software Quality and Security

DevSecOps prioritizes software quality and security by making sure that testing and security activities start early and have influence on the very design and architecture of the software. This allows major issues, especially fundamental ones in logic and architecture, to be caught and fixed earlier, therefore reducing the number of serious bugs and security vulnerabilities that make it into Production.

Practices such as Security as Code (SaC) and Infrastructure as Code (IaC) also work to reduce the risk of human error that can seriously impact the Production environment. Under SaC and IaC, security activities, deployment configuration, and other operations are automated as much as possible to ensure consistency with every release.

Better Collaboration and Growing Talent

DevSecOps nurtures a collaborative environment and encourages knowledge sharing across the four key groups involved in the development process. This can be a rewarding experience for everyone involved because it helps employees grow their skills and potentially move into other roles as future career options. As I wrote about in a previous blog article, modern developers should at least have what is called “T-shaped skill” — general knowledge in many domains, while specializing in one (or more if they wish). Developers working in DevSecOps environments naturally grow into T-shaped talent because they are able to acquire skills in testing, deployment, monitoring, security, and more.

As for non-developer roles, they can benefit from being able to work more closely with developers to ensure that their concerns and needs are heard early on in the development process. In fact, a common problem with traditional approaches to development is that very little collaboration occurs as each group’s work is focused on their own respective role in the lifecycle. This causes teams to become siloed and work completely in isolation despite working on the same product.

Shift-Left Security at Mercari

Shift-Left Testing is to Continuous Testing, as Shift-Left Security is to Continuous Security. Under DevSecOps, security practices are also shifted left to start in the very beginning of the software development lifecycle, and they continue throughout every stage.

At Mercari, we have also applied Shift-Left Security to our development processes. Below are some examples of security activities that you can also apply to your organization’s SDLC:

Stages 1-3: Planning, Analysis, and Design

• Design and requirements reviews: At Mercari, Product Security is involved in the initial planning of new features and conducts security reviews of the proposed user flow, use cases, business requirements, etc. in order to identify potential areas for abuse. By doing so, we can find and fix security flaws in the design before feature development has even started, ensuring that the feature is built with security in mind.

• Threat Modeling: This process involves brainstorming all possible threats and vulnerabilities in the system, evaluating risk and impact, and listing up countermeasures to reduce risk. By putting on our hacker hats and brainstorming everything that can possibly go wrong, we can prepare for the worst and uncover issues before real hackers do.

• Architecture review: At Mercari, Security Engineering reviews infrastructure and the flow of data through proposed systems to make suggestions that help minimize the risk and impact of potential attacks.

Stage 4: Implementation

• Security code review: Code reviews of sensitive areas of code such as functions dealing with Personal Identifiable Information (PII) or payment are conducted to ensure that handling of sensitive data is secure, and that there is no strange logic that can potentially be abused. Security code reviews are also done on non-sensitive parts of code to check that secure coding best practices are followed (e.g. no hardcoded secret keys or credentials, strong encryption algorithms are used, etc.).

Stage 5: Testing and Integration

• Security testing: Once a feature is testable, Product Security conducts security testing to find vulnerabilities or loopholes that can potentially be abused by malicious users.

• Static application security testing (SAST): SAST tools check code for common issues such as the usage of vulnerable libraries, coding mistakes that can lead to XSS, SQLi, etc. At Mercari, we use SAST tools such as WhiteSource and MobSF in our CI/CD pipeline to ensure that builds fail when code changes don’t meet our security requirements.

• Dynamic application security testing (DAST): DAST tools attempt to execute common attacks (XSS, SQLi, etc.) against the software in order to identify areas that can successfully be exploited. At Mercari, we also run DAST tools such as Netsparker against our applications to find vulnerabilities.

Stage 6: Maintenance

• Automated tools and monitoring: Security does not end with release! Regularly running vulnerability scans and security regression tests, as well as having systems for monitoring, alerting, and auditing is necessary in today’s age of ever-increasing cyber attacks. At Mercari, we use web application firewall (WAF) and monitoring tools such as Signal Sciences and our own in-house SOAR (Security Orchestration, Automation and Response) in detecting and responding to unusual activity, as well as tools like Sysdig in securing infrastructure.

Challenges

Although DevSecOps brings many benefits, it can also be a challenge to implement correctly. As with any software development methodology, transitioning may initially slow down development as the team struggles to adapt to changing roles and responsibilities, as well as a new working style and culture. Any organization attempting to transition should be fully prepared for and willing to accept that productivity may initially drop before benefits can be seen.

Another challenge is that DevSecOps requires heavy collaboration between teams. Developers, QA, Security, and Operations (SRE and Infrastructure teams) must work closely together because the boundaries between where responsibilities begin and end become blurred as their work is spread across all stages of the software development lifecycle. While traditional Waterfall and Agile models allowed each of those groups to conduct their work independently within their respective stages in the lifecycle, DevSecOps teams should be prepared for active collaboration and communication across these groups throughout the entire lifecycle. A common misconception of DevSecOps is that it moves all responsibilities of testing, deployment, monitoring, and security solely onto the developers, fully eliminating the need for QA, Security, and Operations staff but this is not true. The goal is not to eliminate these groups, but to instead have them take on a more supportive role, acting as enablers to developers and working collaboratively alongside them.

Finally, the most obvious challenge is the amount of technical skill and knowledge required of developers in order to build a successful DevSecOps team. Since developers hold responsibilities in testing, deployment, monitoring, and security in addition to coding, they must learn new skills such as test automation and the security mindset for example. Organizations that wish to transition should ensure that their developers have adequate access to training and education, as well as continued support and knowledge sharing from QA, Security, and Operations engineers. At Mercari, we have created our own internal training program, Security Champions, precisely to help improve security awareness and share knowledge of secure development practices across teams.

Conclusion

DevSecOps is no doubt, one of the most trending topics in the industry recently. Although it can be a challenge for some organizations to apply, it brings many benefits such as quicker releases, fewer bugs and security vulnerabilities, and a higher degree of collaboration and knowledge sharing across teams. With the increasing number of cyber attacks in recent years, it is more important now than ever that software be built with security in mind rather than as an afterthought, and this can only be accomplished through embracing Shift-Left Security and other DevSecOps practices.

I hope that you found this article helpful in understanding DevSecOps and why you should consider adopting its practices. Thank you for reading and please come back tomorrow for the next article in Mercari’s 2020 Advent Calendar!

Mercari is looking for software engineers who share our mission and values. https://careers.mercari.com/job-categories/engineering/

Tomorrow’s blog post —the 16th in the Advent Calendar— will be written by afroscript. Hope you are looking forward to it!