Azure OpenAI Limitations and Optimization Strategies

Azure OpenAI can be a game-changer for businesses, but like any powerful tool, it has its limitations. One major limitation is the cost, which can add up quickly if not managed properly.

To give you a better idea, the Azure OpenAI pricing model is based on the number of tokens processed, with a minimum of 10,000 tokens per month. This can be a challenge for small businesses or developers who need to test the waters without breaking the bank.

The good news is that there are strategies to optimize your usage and reduce costs. For instance, using the Azure OpenAI free tier can help you get started without incurring costs.

Quotas in Azure OpenAI Service are a way to manage the rate at which your deployments can consume Azure OpenAI resources. Quotas are assigned to your subscription on a per-region, per-model basis in units of Tokens-per-Minute (TPM).

The default quotas and limits for Azure OpenAI are set to prevent excessive resource consumption. For example, the default DALL-E 2 quota limits are 2 concurrent requests, while the default DALL-E 3 quota limits are 2 capacity units (6 requests per minute).

On a similar theme: Azure Blob Storage Limits

You can use quota to limit the rate at which your deployments can generate text, translate languages, or answer questions. Quota can also help prevent your deployments from consuming too many resources, which could lead to unexpected costs.

Here are some key quota limits to keep in mind:

To request an increase to the default quotas and limits, you can submit a request from the Quotas page of Azure AI Foundry. Due to high demand, quota increase requests are being accepted and will be filled in the order they're received.

You might like: How to Get Access to Azure Openai Service

Azure OpenAI has a rate limit of 6 requests per minute (RPM) per 1000 tokens per minute (TPM). This means that if you make a call to the Azure OpenAI API, you'll be processed approximately every 10 seconds, and roughly 6 calls will be successfully completed per minute.

You can choose to assign portions of your TPM quota to each of your model deployments. For example, if your quota is 150,000 TPM, you can have 1 deployment with a TPM of 150,000 or 2 deployments with a TPM of 75,000, as long as the sum of your TPM across all deployments is no more than 150,000.

If you try to breach the TPM limit by making a call with max_tokens set to 1000, you'll only be able to process one API call per minute, and all other calls will result in 429 errors.

Here's a rough estimate of the RPM and TPM limits for different model deployments:

If you encounter 429 responses from OpenAI or Azure OpenAI, you can try reducing the number of requests you're making to the API or increasing your rate limits by contacting OpenAI or Microsoft.

Optimizing usage is key to staying within your quota limits in Azure OpenAI Service. You can optimize your prompts and token usage by setting the max_tokens parameter as close as possible to your expected response size.

This parameter determines the maximum number of tokens generated in the response, and setting it too high can lead to unnecessary token usage. Consider the amount of input and output tokens getting generated to make informed decisions.

To stay within your quota limits, stick to the recommended guidelines: set your max_tokens parameter carefully to avoid using up more tokens than necessary.

Worth a look: Azure Storage Account Limitations

Optimizing your prompts and token usage is crucial to efficient AI usage.

A token is a segment of a word, and when you send a request to an OpenAI API, the input is sliced up into tokens and the response is generated as tokens.

To limit the number of tokens generated in the response, use the max_tokens parameter.

Setting this parameter too high will result in using up more of your TPM per request than necessary.

Always set max_tokens as close as possible to your expected response size to avoid unnecessary token usage.

Azure OpenAI Service offers different usage tiers to help you manage your usage and costs. These tiers only apply to standard, data zone standard, and global standard deployment types.

The usage tier determines the level of usage above which you might see larger variability in response latency. This can happen for customers with high sustained levels of usage.

A customer's usage is defined per model and is the total tokens consumed across all deployments in all subscriptions in all regions for a given tenant.

To give you a better idea, here are some example usage tiers per month:

To stay within your quota limits, stick to the recommended guidelines, such as avoiding high sustained levels of usage. This can help you maintain consistent latency for your customers.

You can have up to 30 OpenAI resources per region per Azure subscription.

The default DALL-E 2 quota limits are 2 concurrent requests.

Default DALL-E 3 quota limits are 2 capacity units, which translates to 6 requests per minute.

The default Whisper quota limits are 3 requests per minute.

The maximum prompt tokens per request vary per model, so be sure to check the Azure OpenAI Service models for more information.

To avoid HTTP 431 errors, consider reducing the number of custom headers in your API requests, as the current limit is 10 custom headers.

If you exceed the TPM (Tokens-per-Minute) limit, only one API call will be processed per minute, and all other calls will result in 429 errors.

Take a look at this: Azure Openai Api Key

You can request increases to the default quotas and limits from the Quotas page of Azure AI Foundry.

Quota in Azure OpenAI Service is a feature that enables you to manage the rate at which your deployments can consume Azure OpenAI resources, with a quota assigned to your subscription on a per-region, per-model basis in units of TPM.

Here are some examples of how you can use quota to manage your Azure OpenAI resources:

To send a request to increase the quota limit, select the model for which you want to increase the Quota limit in Azure OpenAI Service, and enter the required details, including your subscription ID, region, and model, as well as a description of your scenario to justify the quota increase.

Curious to learn more? Check out: Azure Devops Attachment Size Limit

The maximum number of custom headers in API requests is 10, and you can use up to 100,000 Provisioned throughput units per deployment.

The maximum file size for Assistants and fine-tuning is 512 MB, and the maximum size for all uploaded files for Assistants is 100 GB.

The maximum number of training jobs per resource is 100, and the maximum simultaneous running training jobs per resource is 1.

The total size of all files per resource (fine-tuning) is 1 GB, and the maximum training job time is 720 hours.

The maximum training job size is 2 Billion tokens in the training file multiplied by the number of epochs.

Recommended read: Azure Openai Assistant Api

To stay within your quota limits in Azure OpenAI Service, stick to the recommended guidelines.

Staying within your quota limits is crucial to avoid unnecessary costs.

The recommended guidelines to remain in limits are outlined in the Azure OpenAI Service documentation.

Sticking to these guidelines will help you avoid going over your quota limits.

Staying within your limits can save you money and reduce frustration.

Explore further: Azure Subscription Limits