Agile CRM software implementations configure and create more software in shorter cycles. DevOps responds to this increased pace and frequency by automating the build, test, integration and deployment processes. As stated by Forrester analyst Kurt Bittner, "If agile was the opening act, continuous delivery is the headliner."
Continuing to follow application lifecycle management (ALM) processes built for waterfall deployments, and built upon labor-intensive tests, low-level migration scripts, manual interventions and as much time as you need, will fail to keep up with agile implementation methods and negate business benefits such as rapid and flexible response to change, the frequent delivery of incremental software releases, outcomes that better align with business objectives and faster time to market.
This is why Continuous Integration, Continuous Test and Continuous Delivery tools and processes are required. To work effectively, these steps must be dynamically linked and automated with a DevOps playbook and workflow automation.
A Gartner report advised that, "DevOps is not a market, but a tool-centric philosophy that supports a Continuous Delivery value chain." In this article I'm going to focus on a DevOps toolchain used for Microsoft CRM software implementations. I'm not going to try to identify all the potential toolchain tools, but will share many of the tools that have contributed to successful Dynamics 365 CRM DevOps programs.
DevOps Toolchain for Microsoft Dynamics 365
CRM DevOps toolchains are built upon technology stacks that align with the CRM software. For example, Microsoft DevOps is the most used initial tool for Microsoft Dynamics 365 CRM. When acquiring additional tools, you will lower IT cost and make better selections if you group your tools with DevOps processes and needed capabilities, such as the following.
Agile Development is the start of the continuous build process. Agile or Scrum sprint backlogs define software increments to be built over short but high velocity iterations. The sprint increments become the output which upon check-in to version control kick off the Continuous Build process.
- Visual Studio Team Explorer or Team Web Access are the initial access and project visibility tools. While the DevOps or VSTS Web Access portal can provide end to end management for all agile and DevOps team members, developers will most likely instead use VS Team Explorer to connect with Team Services.
- Team Rooms are online chat rooms that provide central forums to be shared by non-technical and technical team members. Team rooms facilitate a central messaging and document repository, a place to record daily scrums or other meetings, and a real-time view of user story events and progress. For example, when a dev team member accepts or advances a user story, the user story status is automatically updated and a message is inserted in the team room. Because of the tight integration between Dynamics CRM and Teams, agile teams sometimes prefer to use Teams for online collaboration. While Teams may offer advantages such as better support for a broader audience, the lack of integration and support for agile process events reduces any benefits.
- Team Foundation Version Control (TFVC) can be used for centralized source code version control. This method uses a centralized file server to sync software snapshots with dev clients.
- Git is an alternative to TFVC to be used for distributed version control. This method does not require a central server as each dev client clones the latest software repository with history.
- Check-In Policy Controls are rules configured in DevOps or VSTS that enforce dev check-in and similar policies. These can be helpful controls to ensure dev staff compile, comment and run tests (such as Code Analysis) that verify code quality before check-in.
- Build alerts can be user-configured notifications directed to SOAP endpoints instead of email addresses. These alert types can be configured to monitor the build and trigger an action if the build fails.
This phase automates the frequent integration of new and updated code with the central code repository. The goal is to automate error free builds that are then picked up for deployment in designated environments.
Developers upload their code to a build server which merges their code with the code base, compiles the latest version, initiates automated quality control processes (which generally include unit tests, integration tests, static and dynamic tests, performance tests and other checks) and delivers a package for release.
Continuous integration solves the long-time problem of 'integration hell.' The longer a programmer holds onto his copy of the central code base, the more likely he will incur merge conflicts when his branch is checked back in to the source code repository. Continuous integration mitigates this problem by promoting frequent integration, often multiple times per day. Software quality is improved and the DevOps deployment process is accelerated when merged code quality control measures are performed on a continuous basis instead of the more common practice of applying quality control near the end of the process. Continuous integration is one step in continuous delivery.
The continuous integration process and tools will flag code defects and failed builds more frequently in order to fix and eliminate these issues at the source. However, without a change in culture which makes those errors highly visible and encourages team members to take responsibility for their quick resolution, the tools alone will not deliver the needed impact.
- In Microsoft DevOps or VSTS you can create a build definition to support your continuous integration build. The build definition can specify the build steps and follow-on quality control tests; run on a build server or the hosted build agent; and initiate on demand, on a schedule or upon every check-in.
- Feature toggles are an alternative to branching that can aid DevOps programs by accelerating continuous integration. Branching is used for feature isolation, bugs or hotfixes. It's also used when developers need to work on multiple things at the same time, or check in incomplete code. Distributed version control systems can accommodate branching, but the process incurs challenges such as merge conflicts. Feature toggles use simple if statements to hide or display code. When the feature is complete, the if statement is removed.
- SonarQube is a continuous inspection code quality tool. This open source product works well with C# code and examines architecture, code rules, duplication, unit tests, defects, comments and complexity. Where most continuous testing tools operate at points in time on each programmers' computer, SonarQube initiates upon check-in on the build server. Over time agile dev projects accumulate technical debt. Failing to measure and remediate that debt decreases programmer productivity and ultimately leads to significant refactoring. SonarQube is particularly well suited to identifying technical debt. Its SonarQube Quality Model measures technical debt and suggests steps to resolve.
- Continuous integration testing generally includes QA staff using the Test Hub in the Web Access portal. Tests are organized in hierarchies with test plans at the top, and then cascading down to test suites which group test cases. Requirements-based Test Suites link back to user stories, and query-based Test Suites can group test cases in user defined ways. Test plans can be copied and modified as needed for each successive sprint.
- Microsoft Test Manager has overlap with Test Hub but is more commonly used by DevOps teams and QA professionals. Like Test Hub, MTM comes with Visual Studio and manages test plans, test suites and test cases. However, unlike Test Hub, it's a desktop app that better integrates with the local operating system as its running on premise and not from the cloud.
- The Coded UI test builder and editor are built upon the Microsoft UI Automation API and support automated GUI testing. The tool records your actions, permits defined values in specified fields and generates the code for playback and testing.
- Load Test Hub is a Visual Studio service that runs URL-based load tests. Team Services can record and replay cloud-based load tests for performance and scalability analysis. Because the test is normally performed from a single agent and IP address, issues such as caching, latency, hops and jitter will impact test results. Microsoft also offers Dynamics CRM Online performance and scalability testing for a fee. While the Microsoft test offers greater assurance, the Load Test Hub is a good early analysis step.
DevOps demands an uninterrupted pace and doesn't allow long stabilization periods. Continuous test demands more testing more often. It automates code analysis and either returns code to developers for rework or advances the code to the next step in the DevOps process.
It also helps shift the all too common process of inserting automated testing tools run by people that didn't create the code and initiated just ahead of the deployment process to instead imposing testing rigor closer the source and with the people that own the code. Some DevOps teams refer to this as a 'Shift Left' testing approach as testing is expanded to every step prior to late stage testing.
A reality of software development is that undisciplined development programs will result in programmers cutting corners in areas such as documentation, unit testing, technical debt and other post-coding tasks. A DevOps program must bring visibility and rigor to the software delivery pipeline and throughout the end to end software development lifecycle. Here are several DevOps test tools that can help.
- Test-driven development (TDD) isn't really a tool but a test-first programming concept that links user stories to software requirements and supporting test cases. Each new CRM software feature starts by creating a succinct test that anticipates the user story requirements and exception conditions. As opposed to unit tests which are most often written after the code is developed, TDD requires the programmer to first consider outcomes and errors before writing any code. TDD tests are small, insulated units often defined as classes, or groups of related functions called modules. TDD tests can flow into larger code sets and be triggered automatically during code base reviews which occur before the build process. Errors are typically identified and resolved near the source as TDD unit tests notify the programmer of the defect before the code is forwarded to QA or downstream testers. Test driven development aids short development iterations, which makes it ideal for high velocity sprints and DevOps.
- There are many Unit Tests tools that can be applied to Microsoft CRM custom code, including IntelliTest, Test Explorer, Microsoft Fakes isolation framework, the Microsoft unit test framework for managed code, and code coverage tools. Determining the best tool(s) should be a part of your Software Development Lifecycle (SDLC). It's equally important that your methods test for the right outcomes. Unit tests should isolate and test individual functions or procedures that fulfil product backlog items. Good unit testing requires the developer to consider and verify inputs, outputs, and error conditions before code is passed to Quality Assurance (QA).
- IntelliTest is one of my favorite Visual Studio tools for Dynamics CRM code testing. This tool interrogates .NET code and generates test cases, unit tests and test data. It also delivers alerts when other programmers or code subsequently breaks one of the tests. Compared to manual methods, this tool increases code coverage testing and decreases the effort for dev staff.
- Code Analysis in Microsoft DevOps or VSTS improves code quality early in the development cycle by examining managed assemblies, applying rules to test code styles, and detecting violations of the .NET Framework Design Guidelines. My experience is that issues detected by Code Analysis result in fewer downstream errors and fewer break-fix iterations.
- Code Metrics are software measures that bring objective code scoring to custom software development. A code metric that I have found to be influential in DevOps for Dynamics CRM software programs is the Maintainability Index. Cloud CRM systems offer many advantages, but when custom code veers outside the standards or constructs of the CRM software, companies incur greater complexity, time and cost for semi-annual upgrades. The Maintainability Index displays color coded scores between 1 and 100. A green score between 20 and 100 will result in the fewest maintainability problems. A yellow score between 10 and 19 suggests significant manual efforts for CRM upgrades will be required. A red score of 1 and 9 predicts troublesome software updates and upgrades that begin to resemble the fork lift upgrade process of on-premise systems, which cloud CRM systems were to replace. Code metrics also score critical factors such as security, performance and scalability. Detecting these types of issues at the source and time of initial development saves significant time and cost as compared to detecting these issues later in the DevOps cycle, when code refactoring will be required.
- SonarQube is a complimentary code quality detection tool. This open source software supports C# and other languages commonly used to modify Microsoft CRM and detects issues with projects, methods and classes. It scores coding standards, code coverage, code complexity, support for architectural guidelines and many other factors which improve CRM software quality and aid a CRM DevOps program.
- Layer Validation applies layer diagrams to validate code design. Layer diagrams illustrate the intended CRM application architecture in Visual Studio. They include major components and functional units of the design and their interdependencies. Unlike traditional architecture diagrams, layer diagrams test and validate that custom code and its dependencies conform to the architectural design. For example, architecture design requires that the Microsoft CRM user interface (UI) modifications integrate to the API or application layer. Any custom code that results in the UI integrating to the data layer would be shown as a violation in the layer diagrams. This tool can also be used to predictively model architectural changes in the layer diagram and show the impact to code and affected dependencies. Including this step in your CRM DevOps process will reduce lengthy refactoring of technical debt that would otherwise accumulate and be detected much later in the cycle.
- Roslyn is the .NET compiler that supports dynamic compilation and execution as well as code refactoring. Roslyn is written in C# and vb.net which exposes the compiler and makes it extensible for custom code analysis (i.e. test for custom rules in your code) or refactoring with API services.
- The most comprehensive QA test tool I've found is Rainforest. It's a crowdsourced QA-as-a-Service platform for on demand testing. QA leads author functional and regression tests in concert with an array of variables. Rainforest then coordinates the testing among their 50,000+ qualified testers and delivers results in less than an hour. It's a powerful tool for going well beyond happy path scenarios and achieving higher quality testing results that are essential for mission critical software and consumer or customer facing applications. Applause is another crowd sourced QA tool that can be leveraged to apply more testing than any single QA team could do manually.
- User Acceptance Testing (UAT) is the one testing method that is seldom automated. The benefit of users manually testing the product against user stories and real-world conditions is that they not only verify product readiness, they also verify user preparedness. Steps to improve UAT include designing good test cases, using test data indicative of real data, testing beyond happy-path scenarios, testing in a pre-prod environment (i.e. an exact replica of the production environment), testing performed by actual users, and recognizing omissions are more likely new user stories than defects.
Continuous testing is symbiotic with continuous integration. The cost of identifying and resolving defects at the source is exponentially lower than the cost of detecting defects in downstream testing and then routing those bugs back to the source. Early testing detection and resolution is pre-requisite to implementing DevOps.
This phase pushes code increments to non-production test or stage environments where they can be approved and pushed to production. It automatically builds, tests, and creates deployment-ready software increments available for any endpoint or environment. Depending upon your DevOps framework, it either includes or extends continuous build and continuous integration so that updates and new versions can be released to production on demand.
Continuous deployment is sometimes confused with continuous delivery. Continuous deployment automatically pushes updates to production, while continuous delivery permits updates to be deployed to production upon manual confirmation. As DevOps shops mature, they may evolve from continuous delivery to continuous deployment. However, this progression is generally not a two step process, and facilitated with application software such as Automic.
Release Management directs the delivery pipeline from build packages to the production environment.
- Azure DevOps, TFS or VSTS Release Management automates software deployment and testing. It can be configured to fully automate the delivery pipeline all the way to production, or deploy in multiple increments which require approvals before proceeding. Once the build server generates its package, the Release Definition defines the steps, variables, approvers and permissions that advance the code from version control through each successive environment – such as from Dev to QA to UAT to Pre-production to Production. The build deployment to each environment can be automated based upon completion of the build, a schedule, or a successful deployment to a previous environment.
In achieving DevOps automation, it's critical that the delivery pipeline always follow the prescribed steps, and shortcuts to skip code movement from one environment to another are prohibited. It shouldn't be lost that this piece of the DevOps toolchain must also support collaboration among what are traditionally siloed QA and IT operations roles.
- Kubisys creates on demand production based lab environments that can replicate your real network. These staging environments can clone your network operating systems, integrated applications and even data to provide a more production-accurate test environment ahead of releases.
- Microsoft has been using Lifecycle Services (LCS) for Dynamics 365 Finance and Operations ERP software environment provisioning and change management for a few years. It's a simple but effective tool for traceability and logs, change management, and revert to stable state if required. It's not yet available for Dynamics 365 CRM software, but is on the horizon and will be a helpful insert to the Microsoft CRM toolchain when available.
Monitoring and Management
DevOps Monitoring and Management tools for Dynamics 365 CRM should include infrastructure monitoring, application measurement and log analysis. Some DevOps shops include security monitoring while others assign this critical function to dedicated infosec groups. The tools must work together to observe end-to-end CRM software flow, deliver IT and user metrics, and provide traces that include code level details. DevOps teams use this information to provide specific and actionable feedback to developers and measure user engagement. Tools that can further acquire feedback from users and customers will bolster CRM user adoption and user experience goals.
Because monitoring tools often include overlapping data, which then must be compared and parsed during incident response or discovery exploration, having fewer integrated tools saves time, reduces conflicting messages and simplifies the alert investigation and response process.
- Application Insights is an Azure-based API service that can be used to monitor Dynamics 365 CRM software utilization, performance, availability, dependencies and failures. It allows you to instrument your code, send telemetry data to the cloud, configure alerts, and view dashboards from the Azure portal or within your VSTS team rooms.
This tool is still young, and in my opinion does not offer the breadth or depth of several market leading DevOps monitoring tools. However, it's rapidly evolving, integrated with Dynamics 365 CRM Online, extensible and can aid important goals such as maintaining a reliable CRM service and continuously improving performance and usability. Power BI can be used as an intermediary when consolidating data from Application Insights and other monitoring tools.
DevOps More Than a Toolchain
While this blog post focused on a Microsoft CRM DevOps toolchain, don't lose sight that DevOps is a framework embedded in agile principals and consisting of a culture movement that emphasizes collaboration and communication among developers, QA staff and IT operations.
Key to success is assigning ownership throughout the DevOps cycle, reaching agreement on how infractions will be recognized and resolved, and using tools to bring visibility to these responsibilities, violations and resolutions. While not called out in the DevOps toolchain, a critical tool is the dashboard that brings program visibility to all DevOps team members and stakeholders, and also displays on a large flat screen monitor in a public viewing area.