Why is Azure DevOps Slow and How to Speed It Up

Author

Reads 650

Black Laptop Computer Turned on Showing Computer Codes
Credit: pexels.com, Black Laptop Computer Turned on Showing Computer Codes

Azure DevOps can be frustratingly slow, especially when you're working on tight deadlines. One reason for this slowness is the large number of projects and repositories that can slow down the system.

Lack of proper configuration is a common culprit behind slow Azure DevOps performance. This can include things like not optimizing build pipelines or not using caching effectively.

Azure DevOps can handle a large volume of users and projects, but if not properly scaled, it can lead to performance issues. In one instance, a team of 50 users experienced a significant slowdown due to inadequate scaling.

A slow Azure DevOps can be caused by a variety of factors, including network issues, poor code quality, and inadequate resource allocation.

Worth a look: Azure Devops Issues

Troubleshooting Azure DevOps

Troubleshooting Azure DevOps can be a daunting task, but there are some common issues that can slow down your workflow.

One of the most common causes of slowness in Azure DevOps is a high number of pending changes. If you have a large number of changes pending, it can lead to performance issues.

Credit: youtube.com, Azure - Troubleshooting Slow requests in Windows Azure Web Sites

To resolve this, try to reduce the number of pending changes by committing or shelving them regularly. This will help keep your repository up to date and improve performance.

Another issue that can cause slowness is a large number of build agents. If you have too many build agents running at the same time, it can lead to contention and slow down your builds.

By limiting the number of build agents and optimizing your build process, you can improve performance and reduce the likelihood of slowness.

Consider reading: Watch Divergent

Do Not Run Pipeline on All Commits

Running a pipeline on every single commit can be a recipe for disaster. This can lead to a bottleneck in your build agents, where a branch "steals" the agents from a branch that needs them to complete a pull request.

We've all been there - limited resources and too many demands. To prevent this, we enabled batching, which means only the newest commit on a branch is built and older commits are not.

Credit: youtube.com, How to troubleshoot azure pipeline issue step by step #devopstutorial #azuredevops #azurepipeline

Running a pipeline on all commits also means wasted resources and unnecessary work. We set the pipeline to only build on our main branches, which has greatly reduced the workload.

We've also configured the pipeline to pass before merging in a pull request, ensuring that only high-quality code makes it to the main branch. This has streamlined our development process and reduced the risk of errors.

A Missing Part

A missing part in Azure DevOps is the inability to authenticate permanently using Personal Access Tokens (PAT). This is because PATs have an expiration time, making them unsuitable for a permanent automation solution.

You can't deploy Azure DevOps agents from your DSC configuration due to this limitation. Unfortunately, I had to exclude the deployment of the agent from my DSC configuration for this reason.

Azure DevOps agents automatically update themselves when they run a task that requires a newer version of the agent. This means you don't need to worry too much about the maintenance of Azure DevOps agents once they're installed.

Optimizing Pipeline Performance

Credit: youtube.com, Faster Azure DevOps Pipelines (#259)

Optimizing pipeline performance is crucial to ensure your Azure DevOps pipeline runs smoothly and efficiently. We can achieve this by optimizing the pipeline triggers to only run on main branches and the tip of every pull request.

Enabling batching in the pipeline means that only the newest commit on a branch is built, and older commits are not. This helps prevent a branch from "stealing" build agents from another branch that needs them to complete a pull request.

By configuring the pipeline to only build on main branches and the tip of every pull request, we can free up build agents to do the most important work. This change alone can significantly improve pipeline performance.

Taking a closer look at the checkout process, we can see that it takes a long time due to checking out all git LFS files in each pipeline. By moving the required files out of git LFS and into the regular git repository, we can save a chunk of time, bringing the checkout time down from 2 minutes to 55 seconds.

Furthermore, we can set the fetch depth to 0 in the checkout step, which takes the checkout time further down from 55 seconds to 7 seconds.

For another approach, see: Azure Devops down

Cache NuGet Packages

Credit: youtube.com, How to REDUCE Build Time Using Caching in Azure Pipelines (Cache)

Caching NuGet packages is a common approach to speed up pipeline performance. However, it may not always yield the expected results.

Restoring a cache of external dependencies can take around 45 seconds, which is slower than running a "dotnet restore" command that happens in under 30 seconds.

In one instance, trying to implement caching resulted in a slower pipeline overall, with the caching process adding around 30 seconds to the total time.

Configure Checkout Options

Configuring checkout options can significantly impact pipeline performance.

Two minutes is an eternity for a checkout, and in this case, it was due to checking out all git LFS files in each pipeline.

We took the required files out of git LFS and put them into the regular git repository, saving us a chunk of time.

This change reduced the checkout time from 2 minutes to 55 seconds.

Setting the fetch depth to 0 in the checkout step can take the checkout time further down from 55 seconds to 7 seconds.

The full checkout step looks like this.

You might like: Azure Pipeline Checkout

Common Issues and Solutions

Credit: youtube.com, Day 16/16 Azure DevOps Troubleshooting & Solutions for Common Issues

Azure DevOps slow? Don't worry, I've got you covered. Here are some common issues and their solutions.

One common issue is a slow Azure DevOps server due to a large number of users. This can be resolved by implementing user throttling to prevent excessive usage.

Azure DevOps can be slow if there are too many builds running concurrently. To fix this, you can limit the number of parallel builds or implement a build queue.

A slow Azure DevOps server can also be caused by a large number of work items. To resolve this, you can implement work item filtering and optimization.

Azure DevOps can be slow if the database is not optimized. To fix this, you can run regular database maintenance tasks, such as indexing and statistics updates.

Another issue is a slow Azure DevOps server due to a large number of test runs. To resolve this, you can implement test filtering and optimization.

Azure DevOps can be slow if there are issues with the network or infrastructure. To fix this, you can troubleshoot and resolve any network or infrastructure issues.

A fresh viewpoint: Azure Devops Server Free

Result and Conclusion

Credit: youtube.com, Master CI-CD Optimization in Azure DevOps: What You Must Know!

By implementing smart checks in our Azure DevOps pipeline, we've managed to shorten the feedback loop and make our development process more efficient. This has resulted in a significant reduction in build times, from 10 minutes to just 2 minutes on average.

A commit can now affect multiple projects, making the deploy stage run in 1 minute on average. This technique also consumes fewer resources, which has several advantages: other pipelines don't need to wait until a build or release agent is free, it's cheaper to implement, and deployments are more reliable.

This approach has repaid itself after just a couple of runs, making the time invested into making our pipeline smarter well worth it.

Result

Taking a pipeline from 7 minutes to 3 minutes is a significant improvement. This was achieved by tweaking idiosyncrasies specific to the project.

The author notes that not all of this advice is broadly applicable, but similar tweaks can result in faster pipelines for other projects.

Build times on Microsoft hosted agents can vary by up to 30-50%, so it's essential to take the times with a grain of salt.

The final YAML file is the result of these optimizations, which can serve as a reference for others looking to improve their pipeline performance.

Conclusion

Diverse team collaborating on a software project in a contemporary office setting.
Credit: pexels.com, Diverse team collaborating on a software project in a contemporary office setting.

In the end, the key to a faster CI/CD pipeline is to minimize the amount of work it needs to do. By only building and deploying modified projects, we save a significant amount of time.

The worst-case scenario still takes 10 minutes during the build stage, but this only happens sporadically. When the slow project isn't included, the build stage takes just 2 minutes on average, or sometimes even just a few seconds.

A commit usually affects 2 projects, making the deploy stage run in 1 minute on average. This technique also consumes fewer resources, which has several advantages, including:

  • other pipelines don't need to wait until a build or release agent is free
  • it's cheaper (and more effective) in comparison to upgrading servers to speed up your builds
  • a deploy is more reliable and causes fewer errors

While we still have to wait an hour occasionally, 90% of the time this isn't the case. By making our CI/CD pipeline smarter, we've shortened the feedback loop from a merged Pull Request to a new deploy to our QA environment.

Service Health and Monitoring

The Azure DevOps Services status portal is a valuable tool for monitoring service health. It displays indicators that reflect the severity of a service health event, based on the number of customers affected by the issue.

A fresh viewpoint: Azure Devops Health

Credit: youtube.com, Setting up Service Health Alerts

The indicators displayed on the portal are divided into four categories: Healthy, Degraded, Unhealthy, and Advisory. The highest severity events affect a large percentage of our customers and render some parts of the product unusable.

If you notice an unhealthy indicator, it means that a significant number of customers are affected by the issue, and you may experience difficulties using certain parts of the product. This is a critical situation that requires immediate attention.

The Azure DevOps Services team works hard to resolve unhealthy events as quickly as possible, but in the meantime, you can check the status portal for updates and follow any recommended workarounds.

Angular Slot Swap Issues

The Azure App Service slot swap can be slow due to a default warmup request that tries to access your app by issuing a GET request to the root of the app on port 80 or 8080.

This warmup request can fail if your app isn't listening on these ports, which is often the case with Angular applications.

For another approach, see: Azure Devops Parallelism Request

Credit: youtube.com, How to use Deployment Slots in Azure App Service | Azure Tips & Tricks

By default, the "dotnet new angular" template configures the WebApi app to run on port 5000 for HTTP and 5001 for HTTPS, which can cause the slot swap warmup request to fail.

Deploying an Angular/WebApi app using Visual Studio's Publish option can also result in a blank screen if the container port mapping is incorrect.

Try deploying a 'Release' build of your Angular app to resolve issues with service calls failing due to the Angular service proxy.

For more insights, see: Azure Devops App

Angular Slot Swap Failing

The problem of Angular Slot Swap failing is often more about misconfiguration than a problem with the Azure App Service Slot Swap itself.

Incorrect port mappings can cause issues, as seen in the "dotnet new angular" template, which configures the WebApi app to run on port 5000 for HTTP and 5001 for HTTPS, while the App Service slot warmup tries to listen for the app on port 80 or 8080.

Deploying an Angular app using Visual Studio's Publish option can also lead to the same problem, as it published fine but the App Service ran with a blank screen due to incorrect container port mapping.

Explore further: Azure Service Connection

Woman in focus working on software development remotely on laptop indoors.
Credit: pexels.com, Woman in focus working on software development remotely on laptop indoors.

During development, the Angular app's service calls get pushed through a proxy server running on yet another port, which can cause issues if not taken into account, especially when deploying a 'Debug' build of the Angular app.

Deploying a 'Release' build of the Angular app can help resolve service call failures caused by the Angular service proxy.

Slot Swap Explanation

The slot swap process in Azure App Service is more complex than you might think. It's not just a simple swap of code, but rather a process that ensures your app is warmed up and ready to accept requests.

To start the swap, the App Service issues a GET request to the root of your app on port 80 or 8080. This is because the App Service treats your application as a container and handles the SSL/HTTPS implementation internally.

If your app isn't listening on port 80 or 8080, the swap warmup request will fail. This can cause delays and issues with the slot swap process.

The App Service will try to access your app on port 80 or 8080 by default, so make sure your app is configured to listen on one of these ports.

Patricia Dach

Junior Copy Editor

Patricia Dach is a meticulous and detail-oriented Copy Editor with a passion for refining written content. With a keen eye for grammar and syntax, she ensures that articles are polished and error-free. Her expertise spans a range of topics, from technology to lifestyle, and she is well-versed in various style guides.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.