IT operations practices continue to evolve as companies transition from on-premises IT to public cloud services, monoliths to microservices, and Waterfall to Agile and DevOps.
Those changes have radically altered what's required of IT ops pros at mobile video on-demand firm Vuclip Inc., which began its three-year transition from a monolithic Java-based app to microservices in public cloud in 2016. The company now expects to have more than a hundred microservices in Google Kubernetes Engine and AWS Elastic Compute Cloud by September 2019.
"The number of people we needed to support the application we have and the skills we needed have all undergone dramatic change," said Jishnu Kinwar, vice president of technical operations at Vuclip, based in Singapore, with U.S. offices in San Jose, Calif.
During its migration project, Kinwar's team built a CI/CD pipeline to allow app development teams to be independent, and it did away with central deployment and infrastructure management teams. The team has also built an infrastructure capable of service discovery, application performance monitoring through OpenTracing, and all application services now run on Kubernetes.
Vuclip bakes security into its infrastructure deployments at multiple levels. It spins up cloud VMs with all ports closed, and app development teams must justify an exception to open a port on a firewall, especially to traffic coming in from the internet. At the application server level, a highly distributed, automatically deployed app calls for web application firewall (WAF) security in kind.
Distributed app demands distributed security
Vuclip works with service provider partners for video distribution over content delivery networks in 22 different countries. In addition to blocking bad actors on the web, the firm wants to incentivize its mobile subscribers with zero-rating apps, which incur no data charges when used. This means it must both blacklist malicious IPs and whitelist those it grants free access to streaming bandwidth, without confusing the two.
The number of people we needed to support the application we have and the skills we needed have all undergone dramatic change.
Jishnu KinwarVice president of technical operations, Vuclip Inc.
The company compared WAF products, including Imperva's Incapsula for IP blacklisting and whitelisting, and chose Wallarm because it could do that with the least amount of false positives and also scale with Vuclip's NGINX farm, Kinwar said.
Wallarm's WAF consists of two parts: a C module that runs on each server, called the Wallarm filter node, and the Wallarm cloud, which processes filter nodes' data with machine learning techniques to detect traffic patterns. Vuclip uses filter nodes with Chef during its CI/CD process, which deploys them as NGINX Plus servers spin up and scales them back when the servers shut down.
DevSecOps proponents advocate a secure-by-design approach to application development that removes the need for trusted infrastructure, but Vuclip hasn't reached that state yet.
"There are companies which are way ahead, but we're not so far ahead in that game," Kinwar said. PortSwigger's Burp code scanning tool can tell developers if their code contains OWASP vulnerabilities, but that's the extent of security by design at Vuclip so far. "A lot of that falls on infrastructure today," he said.
DevSecOps future calls for service mesh, monitoring update
Vuclip's IT operations staff has shrunk 30% since 2016, and all of its IT pros have application coding skills in addition to systems knowledge. The company has compressed its app deployment cycles from two weeks to multiple deployments per day, and teams deploy separately at will.
Vuclip wants to move from NGINX Plus to the Envoy proxy and Istio service mesh, primarily to add fault injection, a type of chaos engineering process that tests the effect of changes on complex distributed systems.
"With so many people making so many changes, it's difficult to know if engineering makes a mistake which, if all the stars line up, could bring the system down," Kinwar said. "How can we inject fault and see how different services behave?"
Wallarm doesn't yet support Envoy deployment, but plans to add that in 2019, the company said.
Kinwar said he also wants to see advances in dependency-based alerting as Vuclip's microservices network mesh gets more complex. Many DevOps monitoring tools capture logs and traces and allow admins to log in and check for anomalies, but he said he would appreciate proactive alerts that pinpoint the root cause of problems.
"I don't have the luxury of adding more people," Kinwar said.
Vuclip has reached a NoOps state for application deployment, but it will always need at least some IT ops expertise, Kinwar said.
"You do need people who understand networks and systems, who would know how memory is being allocated if something goes wrong," he said. "Those skills are not going away, and not a lot of engineering students are taught those things."
Dig Deeper on Systems automation and orchestration
