When the cloud runs out of VMs

Consider reserving capacity if you’re counting on Azure

One of my posts on Azure Security.

Free Content on Jobs in Cybersecurity | Sign up for the Email List

I’ve been teaching a revamped version of my Azure class over the past few weeks. It’s been very challenging as I opened a new account a while back and have been trying out some features in preview. I paid for a support plan to try to resolve some issues, most of which turned out to be Azure bugs. I understand about a service in preview having bugs, but it took a month to get to the right person to test what I was explaining to support with detailed steps and screen shots to resolve the problem. I haven’t had a chance to go back and test it again to confirm the issue is resolved.

On the other hand, when you just want to launch a virtual machine, that’s incredibly basic and should just work. I’ve seen strange things on Azure before, such as when I taught the class in Australia the platform defaulted to a crazy expensive VM size, but in general launching VMs was not a problem.

When I went to revamp my slides related to VMs and disk encryption, that’s when I realized there’s something else going on with Azure. It appears there is a capacity issue. In the past I used to have my students create free accounts and run through labs. Thankfully, I’m not doing that in this class. Free accounts have always had strange issues and limitations but generally you could at least create a VM. Now I can’t even do that — and I’m in a paid account. I’m actually paying $100 for support as noted previously and my account has been open for a while but under a year probably. I set up a new account to use for research and testing.

To date I have not been able to create a virtual machine with availability zones in any region in the US that I’ve tried, with any operating system or machine size. I’ve verified multiple times with support that the quota exists. I’m using 0 of 25,000 VMs and 0 of 10 of many sizes of VMs in specific regions.

I eventually was able to find another region in the world where I could get something running. I’m only creating ONE VM. Any VM. It’s not a huge VM. It doesn’t need to be a small VM. I have no real specific requirements other than I wanted to demonstrate one thing in Windows and one thing in Linux.

I started asking on Twitter if other people are having this problem and they are. One person mentioned it seems to be with newer accounts. Another person has research credits. I have been going back and forth with Azure support on this for about two weeks now. I started searching around on Twitter and in support and Q & A forums and finding many examples where people just flat-out cannot create a virtual machine of any type in any region.

This is a really long post documenting issues as I dealt with them on Azure which you probably don’t have time to read, but you can find similar accounts of capacity issues currently at the bottom of the post.

The problem seems to be mainly with student accounts, free accounts, and potentially with new accounts. However, my account is not super new, it’s not free, and it’s not a student account. Azure support told me in my case it has something do with multiple subscriptions. However, I am trying to create VMS in any tenant or subscription in my account including the default with the global administrator or a user assigned the appropriate permission. I submitted quota increase requests even though I already have quota showing in my account.

I was, for my purposes, able to create a VM to show what I need to show in Azure. I’m doing something very simple. However, it seems like there’s a bigger capacity issue here that people should be aware of when using Azure. I read reports of people in Europe not able to create resources in their production accounts. If you are dependent on Azure you might want to think about reserving the capacity you need in advance.

Of course this kind of defeats the pay-as-you go aspect of the cloud. Now we’re back to reserving servers in data centers. It’s going to cost more. To avoid an outage, however, you may need to take this step in your Azure account. And of course, that will exacerbate the problem for smaller accounts like mine. But I’m not using Azure for production so for me, at least, it’s not a big problem. I’m more concerned for my clients using Azure if this trend continues and starts to affect larger accounts. Also with larger accounts they generally have dedicated support teams and don’t report things like this online — so who knows how many accounts are really facing this issue.

~~~~~~~~

Update: Finally got VMs working. I submitted multiple support requests before got it fixed but ultimately I got it fixed through a limit increase request where I explained the problem was not actually limits but this other issue. I requested to allow all the B and D series instances in three regions I wanted to use. I didn’t have time to check if it was working or not until I was in the middle of my next Azure class so I tested it out in front of the students and that seemed to do the trick. You might have to make multiple requests but that approach eventually seemed to solve the problem.

~~~~~~~

Update: I thought this was fixed. Well kind of. I could select from different VM sizes finally, but then when I went to deploy while prepping for the next day of an Azure class, I got capacity errors again. I was prepping for class so didn’t have time to get back to Azure support about it. During class I used a different user to try to deploy VMs, a non-global admin (the global admin worked in the prior class) and once again I was able to deploy a VM during class. This random error may be affecting a lot more clients than just myself. There’s no way I could count on this platform for production deployments with these random capacity issues. I’m not sure if it’s only in US East or other regions. I don’t know if it’s a glitch or a true capacity issue. I sent Azure support some more information but I need to move on to other things. This is not my full-time job to monitor and resolve this problem…though it feels like it. :) Hope it is addressed so people are not affected by it going forward.

~~~~~~~~~

Update: Came back from vacation to teach my last day of Azure class. I left one VM in a stopped state figuring I’d be able to start it later after having so many problems starting one in the first place. Nope.

After clicking the start button I got this error:

Failed to start virtual machine.

Failed to start virtual machine ‘x’. Error: Operation ‘Start VM’ is not allowed on VM ‘x’ since the VM is generalized.

I’ve seen ways to fix this such as making a backup and starting a new VM from that backup but that’s beside the point. I’m not doing anything extensive in this account I’m using. It’s all really quite simple. I cannot even create or reuse a VM consistently. There’s no way I’d ever use Azure for my cloud pentesting services for this reason. I’d be in the middle of a penetration test and unable to restart a VM I had configured and was using for a test. I have only had a problem on AWS on one penetration test for a short period of time in a particular region and I’ve been using AWS for years. I use Azure only for testing, research, and training and this has been my experience, along with the issues in my running log of problems with Azure support (which I will be canceling shorty, I only got it to go through any updates since my last class . I’ve essentially paid about $300 to report bugs for three months.)

Azure has a lot of really great features. I’m impressed with some of them. I just can’t use the platform for my purposes and business size on a consistent basis due to these issues. I hope they can fix it.

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2022

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab