Let's find out what is testing in DevOps.
In the fields of software mobile app development and testing, there has been no greater trend in the past few years than the emergence of the merger of development and testing, the two becoming one: namely, DevOps.
DevOps is a multi-headed animal: it’s a philosophy, it’s a practice, it’s a culture, it’s an ethic, it’s a framework, it’s a continuous feedback loop.
It has become so well adopted and received by the dev community that DevOps and the DevOps testing suites that are currently being deployed and used seem to be unstoppable and evolving at neck-breaking paces.
The Goals of DevOps & DevOps Testing
DevOps’ aims and goals cover just about every aspect of the entire app delivery pipeline:
- Higher deployment frequency
- Quicker to market rates;
- Lower failure rate of releases
- Less downtime for updates
A fusion of cultural philosophies, practices, and tools that increases a dev team’s ability to deliver applications and services at high velocities: improving products at a faster pace, enabling organizations to better serve their customers and compete more effectively in the market
“DevOps has come a long way and has created much authority in the field of software development. Every organization today wants to embrace this and move forward in the industry.
There are many tools and platforms when it comes to DevOps, BTW, DevOps is not about tools or just automation, it is more than that.” analyst Pavan Belagatti writes here on his blog.
And as the chart below details, the adoption of DevOps will continue.
According to Statista, many business organizations are adopting DevOps and there is an increase up to 17% in 2018 than what was about 10% in the year 2017.
What Is Testing In DevOps?
Practically speaking, testing in DevOps is a delicate balance within the overlap of goals, skills, and communication between developers, QA testers, and operation teams.
Built on similar frameworks and pooling together a vast amount of resources in the unlimited terabytes in the online clouds, it is safe to say that testing in DevOps has already been built into the development process, merging skill sets as well as toolboxes and kits for the most efficient software app development.
DevOps puts a strong emphasis on the collaboration and cooperation and communication between all teams as the graphic below suggests.
The DevOp Process: The 6 C’s
The heart of DevOps philosophies has been dubbed the 6 C’s and each term shares the word Continuous and we will take a deeper look at each:
- Continuous Planning
- Continuous Integration
- Continuous Testing
- Continuous Monitoring
- Continuous Delivery
- Continuous Development
Fusing together the operations team, testers, developers, and business analysis to one platform, continuous planning aims to realize a clear release plan for deployment as well as updating.
In software development, developers regularly merge their code changes into central repositories, after which automated builds and tests can be executed.
“The key goals of continuous integration are to find and address bugs quicker, improve software quality, and reduce the time it takes to validate and release new software updates,” according to Amazon Web Services (AWS), a leader in the DevOps movement.
Continuous Testing: Manual & Automated
Designed to ensure top-notch product quality, continuous testing employees the use of a microservices architecture with each service runs in its own process and communicates with other services through typically an API.
Automated testing in DevOps leverages the power of several different frameworks or programming languages and implement them independently.
As with Continuous Testing, monitoring the myriad of changes and updates and releases to ensure continued and seamless functionality while scanning all systems constantly for bugs and new errors that might crop up.
In the DevOps world, code changes are automatically built, tested, and prepared for a release to production bestowing upon developers a deployment-ready build that has already passed through all rigorous QA testing processes.
Here the aim is for DevOps to allow the release of all changes in as much an automated and pre-tested manner to be automatically delivered to production.
DevOps: Increased Configurations, Communication and Collaboration
In DevOps, code is used to automate standardized and repeatable OS and host configurations, freeing developers and system administrators to focus not on basic IT administration but to the actual improvements of their mobile apps.
The use of DevOps is essentially a collaboration physically bringing together the workflows and responsibilities of development and operations, allowing them to better share information by the use of chat, support/report ticket systems and knowledge base systems.
DevOps Feedback Loop
As the infographic above illustrates, DevOps creates a continuous feedback loop where the planning, coding, building, and testing of an app are automatically fed into the production cycle of releasing, deploying, operating, and monitoring an app for its optimal performance.
This is not just testing. But at the third C of the 6Cs suggest, this is continuous testing as in always ongoing, never stopping, never an end to the testing. In short, DevOps won’t stop, can’t stop making more tests.
Top DevOps Tools For Developers and Testers
The following is a rundown of the top DevOps tools available and widely used today, according to Mayank Tripathi at hackr.io
Slack leads the pack of the top communication tools used by teams for effective collaboration on projects, allowing “developers to collaborate using toolchains in the same environment they are communicating with other maintenance and service members.”
As a continuous integration server, Jenkins automates the complete build cycle of an app. This open-source tool features a Pipeline “which can be utilized by the developers to automatically commit code into the repository, run test cases, as well as fetch reports obtained after testing.”
Employing containerization, Docker allows for the secure packaging, deploying, and running of applications no matter which running environment. Sporting the Docker Engine, “the containers can be accessed which in turn can execute applications in a remote environment. According to a report, 2 out of 3 companies who have tried this application have adopted it within 30 days of using it.”
However, according to other online tech reports, a DevOps tool named Kubernetes is on its way to becoming the fastest growing container technology with a fully-fledged open source community around it.
How To Do Testing in DevOps
But let’s dive a little deeper into how to do actual tests in the DevOps ecosystem. First thing to note, DevOps is the evolution of the Agile model which issued shockwaves throughout the development and testing environments as well as significantly altering app release frequency.
Before, apps were developed byte by byte, step by step. The Agile model attempts to bridge that gap between developing and testing and eventually releasing.
DevOps takes, what some have described as, “a step further” with actual agile teams evolving to handle every facet and nuance of the whole app creation cycle: development, testing and deployment.
So what does this actually look like? Dan Ashby sum up testing in DevOps nicely.
There are what he calls: code reviews, exploratory testing, unit checks and there are pairs of developers stationed alongside their corresponding companion testers, who together at every step are there to do their jobs, supporting each other along the DevOp expressway.
“And if you are a developer writing some code,” Ashby excitedly reports “then you can even pair up with the tester who will share some really cool test ideas about different risks, different interactions, different perspectives, different variables and different combinations of all these things to help you right there while you code the software.”
However, what is Uber cool about Ashby’s perspective on how integral testing is to the entire DevOps life cycle. Check out his graphic below.
Testing DevOps life cycle
What is important here is how the graphic demonstrates that at basically every point in the cycle, every move forward to the next stage, must be preceded by a test.
In other words, only positive test results allow for the team to move forward, only perfect results give them the green light on the DevOps infinity shaped pathway.
Testing everything: DevOps tests ideas, how to branch them, the coding of them, the merger and building of them, as well as on the other side the releasing, deploying, operating, and monitoring of them.
How To Automate Testing in DevOps
Of course, without the recent advances in testing automation, none of the continuous testing scenarios could ever exist.
Leveraging the power of faster and more powerful processing machines plus the unlimited access to Big Data stored across several clouds on the net, automated testing in DevOps is a flowering field in the testing industry.
Just as we saw above with how DevOps requires testing to be constant, testing everything, DevOps also always favors automating … you guessed it … everything.
And these automated tests really cover the rainbow gamut in functionality tests, performance tests, resilience tests, and of course, security tests, one test growing ever more importance in the age where the Firefox browser wants to monitor your log in to certain sites and check those sites for their past security breaches or invasion of privacy issues.
Essentially, as Ham Vocke advises in his ebook “The Practical Testing Pyramid”, automating testing in the DevOps multiverse is all about designing a solid strategy.
He helpfully relates it to building a testing pyramid where higher and higher levels of testing are built upon a solid foundation of already stable testing loops comprised primarily of Unit Tests at the bottom, Service Tests in the middle, and UI Tests top.
This illustrates the following DevOps Testing Principle: the bottom of the pyramid requires quicker, more isolated tests and while the top represents slower, more integrated tests.
The Test Pyramid
“Stick to the pyramid shape to come up with a healthy, fast, and maintainable test suite: Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end.” Vocke writes.
The ability to write solid Unit Tests and thereby design the unit architecture that sets up test data, calls your method under each test, and finally “asserts that the expected results are returned.”
Similarly, with Integration Testing and Contracting Testing, a basic structure is needed to be built on the simplest methods of interacting with databases and APIs.
With UI Testing, Vocke advises “Selenium-based tests will be your best choice” as they offer screenshots of your app functioning across the different flavors of OS and devices out there and can report on various usability issues as well as give design teams of “how good it looks”.
Call it what you want: a test suite, a test portfolio. But it is clear automated testing the DevOps world requires a pre-designed set of toolchains (similar to Bitcoin’s blockchains) a chain of tools, none with more hierarchical importance than the other, but that together form an invaluable safety net of 100% passed, 100% green-lighted automated testing.
Vocke also advises: “Yes, it would be really nice if our profession could settle on some well-defined terms and all stick to it. Unfortunately, this hasn't happened yet.
And since there are many nuances when it comes to writing tests it's really more of a spectrum … which makes consistent naming even harder.”
Testing Challenges In DevOps
Nobody disagrees: a web app company adopting DevOps will surely result in some amazing benefits, but as with all things related to the Internet of Things, implementation and execution the process can also result in some challenging moments for the teams.
The following are three some of the most common challenges of DevOps adoption:
- Highly fragmented platforms including multiple OS/Android versions, across a wide range of devices and hardware such as phones, tablets and wearable technologies.
- App Store difference: With Google Play Store for Android and App Store for iOS, mobile apps must undergo a submission and reviewed process.
- And from the user’s perspective, each one still has a right to not go with the flow and can refuse to download and install the latest update, throwing a monkey wrench into the whole push deployment model .
Testing Metrics In DevOps
Monitoring and Logging
As with all things DevOps, the ability for teams to constantly monitor and scan metrics and logs to see how performance factors into every move made by an end-user.
Data must be captured, categorized, and then analyzed with similarly new fresh logs generated and will offer insights on how better an app’s performance and stability may be further tweaked in the future.
Proactive monitoring apps can also generate alerts while performing real-time analysis of an app’s services.
The following is a shortlist of DevOps metrics which must be actively monitored and tracked 24/7:
- Deployment frequency
- Change volume
- Deployment time
- Lead time
- Customer tickets
- Automated test pass percentage
- Defect escape rate
- Service level agreements
- Failed deployments
- Error rates
- Application usage, traffic and performance
DevOps In Testing: Conclusion
DevOps is all about doing everything, all at the time, with never a second to waste.
In the world of millions of deployments and releases and users, it only takes a 1-second app crash to lose a user’s patronage forever.
The only way to make it perfect, to get it right, not the hundredth time but the first time, is by adopting a DevOps strategy which elevates the importance of app testing to the infinitely co-creative app building process.