Add Resilience to Your Flows with Power Automate Work Queues

Work queues are a new Power Automate tool built on top of your Dataverse instance with an out-of-the-box user interface similar to Approvals. They give you a way to store data and assign it a status to ensure items have been processed. Work Queues can be configured to have an expiration time, which is how long an item is held in the queue before being non-destructively flagged as expired, where they can only go from Queued to On Hold and back. Queues can optionally have JSON or XML schema enforcement for the items they hold. This ensures that only the correct items are placed in a queue for processing.

Queues were originally implemented for Power Automate Desktop and may receive further improvements, but today we'll be using them to ensure records are processed, giving us a safety net if a Power Automate flow were to fail.

In this example, we'll make a Power Automate flow that processes payments coming from a QuickBooks webhook when a new payment is received. To begin, we'll create a new Work Queue by navigating to the Work Queues page at make.powerautomate.com (above) and selecting New.

Note: Work Queues can be added to Solutions, but at the time of writing this can not be created from a solution.

Give your queue a name and any other optional metadata you'd like. Add a schema now (optional) as you are not able to add or edit one later. Click Create when done.

We'll now create an HTTP-triggered Power Automate flow that writes payload data to this queue. To do this, you'll need your Queue's ID which you can get from the Work Queue Dataverse table.

We'll keep this intake flow simple, however in a real-world scenario you would want to implement some best practices such as retries and failure notifications as well as an action to validate the payload and route it to a specific Work Queue if you are going to use a single intake for multiple Work Queues.

In this example we're simply creating an open POST endpoint with the Dataverse write operation to write the trigger body (JSON payload from the webhook) into a new queue item record. Keep in mind that this trigger could be anything - all Power Automate triggers have a trigger body that can be stored as a work queue item.

Once saved, we'll be assigned an HTTP endpoint that we can configure as our webhook endpoint in our QuickBooks developer dashboard.

Now I can test it by creating a new payment in my QuickBooks test environment.

And we have a queue item!

Now that we have an intake flow, we'll need to create a processing flow. For this example, we'll read the queue every hour to check for queued items and process them.

For this, we'll create a scheduled cloud flow that runs every hour.

In the trigger settings, we'll ensure only one instance of this flow is running at the same time by turning on Concurrency Controls and setting the degree of parallelism to 1. This will ensure the flow doesn't trip over itself as it tries to pick up queued items and set their status.

We'll now do a Dataverse list rows operation with an OData filter so that only Queued or Error work queue items from my payment queue that have been requeued 3 times or fewer are retrieved. This can be as simple or as complex as you'd like but at the very least you need to make sure you're only getting queued items from the correct work queue.

_workqueueid_value eq 'YOUR_WORKQUEUE_ID' and (statuscode eq 0 or statuscode eq 4) and requeuecount le 3

For reference, you can find a list of the work queue item status codes in the table Status column, it is not a global choice. Same for the Status Reasons, which are more granular statuses for each status. Here is a table of labels with their status, verify your values are the same before using this table.

Status	statecode Value	Status Reason	statuscode Value
Queued	0	Queued	0
Processing	1	Processing	1
Processed	2	Processed	2
On Hold	3	Paused	3
Error	4	GenericException	4
Error	4	ITException	5
Error	4	BusinessException	6
Error	4	DeadLetter	7
Error	4	ProcessingTimeout	8

Note: This example is a batch processor, meaning we're processing all outstanding items in any order. If you'd like to process one work queue item at a time in a first-in-first-out or priority-based manner, you can perform a Dequeue bound action on the Work Queues table and pass in the Work Queue ID to get the next queued item in line. This will automatically set the Work Queue Item's status to Processing.

From here, back to our batch processor, I have to set the queue item's status to Processing so that I'm able to later change it to Processed, Error, or any other status. A queue item can not go from queued to another status without first being set to Processing.

After that, I can do a Parse JSON data operation so that I can work with the data within the queue item's input and process the payment in Dataverse (which I would recommend creating as a child flow). Finally, I need to update the queue item's status again to indicate the outcome.

Queue item statuses can not be set back to Queued when they are in a Processing state, they can only go into a Processed or Error state.

In this example, we'll set them to Error so they'll be reprocessed and increment their Requeue Count field so we can eventually cut off reprocessing to track how many times they're been processed. We could also change our original OData filter to pick up errors to try to re-process them.

add(items('Apply_to_each')?['requeuecount'], 1)

There are other fields to track the process such as Processing Start Time, Completed On which are set automatically when the status changes. There's also Processor Type (Cloud flow in this case), and a Delay Until field that you can reference in your OData query to delay processing.

After running the flow manually as a test, we have a processed queue item!

So there you have it! Creating a Power Automate Work Queue gives you a central repository for outstanding items that need to be processed and allows you to process them in order or batches, audit processing steps through their history, and keep a record of any items that failed to process and reprocess them automatically or with human intervention.

Here are some advantages to using Work Queues over traditional triggering and error handling in a single Power Automate flow:

Separation of Concerns - With Work Queues, the intake, processing, and error handling of data can be separated into individual flows. Updates or changes to one part of the system are unlikely to impact other areas, reducing the risk of introducing errors. This makes the system more maintainable and simplifies the process of upgrading or modifying one flow.

Genuine Flow Errors - When using a Work Queue, data errors are handled by flagging the Work Queue Item as an error and the flow succeeds. This means when a flow does fail, it's a genuine error with the process itself that needs to be addressed immediately.

Prioritization - In cases where certain items need to be processed before others, Work Queue Items have a low, normal, and high priority status that can drive which records are processed first.

Process Visibility - Since Work Queues and Work Queue Items are Dataverse tables, you can easily create PowerBI reports that show item status, process start/end/duration times, requeue counts, expired items, and which items are queued.

Manual Intervention Interface - If there is an item that ultimately errors out and needs to be manually processed, the Work Queue gives an interface for users to go in and see a history of the process, failure details, and gives them a way to flag the item as processed once they've handled it and document outliers that should be addressed in the flow.

Audit Trail - Processing records through a Work Queue gives you an audit trail of when items were received and processed as well as processing time. This can be crucial for upholding SLAs or when your company needs to report on data processing for internal or regulatory purposes.

Enhanced Retry Mechanisms - Since Work Queue Items have granular statuses and requeue counts, you can tailor retries in different flows that handle a variety of data processing errors such as approval timeouts, maximum retries exceeded, or dependency failures.

See a list of Work Queue known limitations here.