Implementing Parallel Asynchronous Loops in C#: From Parallel.ForEach to ForEachAsync Evolution

Problem Background and Challenges

In C# parallel programming, developers often need to handle parallel processing of collections, especially when involving asynchronous operations. While the traditional Parallel.ForEach method effectively handles synchronous operations, it encounters significant issues when dealing with asynchronous lambda expressions.

Consider this typical scenario: a developer wants to process a collection in parallel and call asynchronous methods within each iteration. An initial implementation might look like:

var bag = new ConcurrentBag<object>();
Parallel.ForEach(myCollection, async item =>
{
  // some pre-processing logic
  var response = await GetData(item);
  bag.Add(response);
  // some post-processing logic
});
var count = bag.Count;

The main issue with this implementation is that Parallel.ForEach does not wait for asynchronous operations to complete. Since the asynchronous lambda actually returns Task objects while Parallel.ForEach expects an Action<T> delegate, all asynchronous operations execute on background threads while the main thread continues with subsequent code, ultimately causing bag.Count to return 0.

Traditional Solutions and Their Limitations

A common workaround involves removing the async keyword and manually waiting for task completion:

var bag = new ConcurrentBag<object>();
Parallel.ForEach(myCollection, item =>
{
  // some pre-processing logic
  var responseTask = GetData(item);
  responseTask.Wait();
  var response = responseTask.Result;
  bag.Add(response);
  // some post-processing logic
});
var count = bag.Count;

While this approach works, it suffers from several serious drawbacks: First, it completely bypasses the intelligent scheduling mechanism of async/await, potentially causing thread blocking and performance degradation; Second, it requires manual exception handling, increasing code complexity; Finally, this synchronous waiting pattern may trigger deadlock risks, particularly in UI threads or ASP.NET contexts.

Modern Solution Using Task.WhenAll

A more elegant solution leverages Task.WhenAll combined with LINQ's Select method:

var bag = new ConcurrentBag<object>();
var tasks = myCollection.Select(async item =>
{
  // some pre-processing logic
  var response = await GetData(item);
  bag.Add(response);
  // some post-processing logic
});
await Task.WhenAll(tasks);
var count = bag.Count;

The advantages of this approach include:

• Full utilization of the non-blocking特性 of async/await
• Automatic handling of task scheduling and exception propagation
• Clean, maintainable code
• Avoidance of thread blocking and deadlock risks

The implementation principle involves using the Select method to create an asynchronous task for each element in the collection, then using Task.WhenAll to await all task completions. This method achieves true parallel asynchronous processing rather than simple multi-threaded synchronous waiting.

Parallel.ForEachAsync in .NET 6

With the release of .NET 6, Microsoft officially introduced the Parallel.ForEachAsync method specifically designed for parallel asynchronous scenarios:

var urls = new [] 
{
    "https://dotnet.microsoft.com",
    "https://www.microsoft.com",
    "https://stackoverflow.com"
};

var client = new HttpClient();

var options = new ParallelOptions { MaxDegreeOfParallelism = 2 };
await Parallel.ForEachAsync(urls, options, async (url, token) =>
{
    var targetPath = Path.Combine(Path.GetTempPath(), "http_cache", url);

    var response = await client.GetAsync(url);

    if (response.IsSuccessStatusCode)
    {
        using var target = File.OpenWrite(targetPath);
        await response.Content.CopyToAsync(target);
    }
});

Parallel.ForEachAsync provides the following important features:

• Native support for asynchronous lambda expressions
• Configurable maximum degree of parallelism control
• Built-in cancellation token support
• Optimized task scheduling mechanism
• Consistency with existing Parallel class APIs

Performance Considerations and Best Practices

When selecting a parallel asynchronous solution, consider the following key factors:

Parallelism Control: Excessive parallelism may cause resource contention and performance degradation. It's recommended to adjust the maximum degree of parallelism based on specific scenarios, typically setting it to 2-4 times the number of CPU cores.

Resource Management: In asynchronous parallel operations, pay attention to thread safety of shared resources (such as HttpClient, database connections, etc.). Using thread-safe collections or appropriate synchronization mechanisms is recommended.

Exception Handling: Exception handling in parallel asynchronous operations requires special attention. When using Task.WhenAll, exceptions are wrapped in AggregateException and need proper handling.

Memory Usage: Large-scale parallel operations may consume significant memory, especially when processing large collections. Consider batch processing or streaming patterns.

Practical Application Scenarios

Parallel asynchronous loops are particularly useful in the following scenarios:

Batch Web Request Processing: When needing to send requests to multiple API endpoints and process responses, parallel asynchronous processing can significantly improve throughput.

File Processing: When batch reading, processing, or writing files, parallel asynchronous operations can fully utilize I/O bandwidth.

Database Operations: When executing multiple independent database queries or update operations, parallel processing can reduce overall response time.

Compute-Intensive Tasks: Although async primarily targets I/O-intensive tasks, combining with parallel processing can also bring performance improvements in certain computational scenarios.

Summary and Outlook

Parallel asynchronous programming in C# has evolved from initial workarounds to official standard support. Developers can now choose the most appropriate solution based on project requirements and target framework versions:

• For .NET 6 and above, using native Parallel.ForEachAsync is recommended
• For older versions, Task.WhenAll combined with Select is a reliable choice
• Avoid using traditional blocking patterns like Wait() and Result

As asynchronous programming patterns continue to mature and optimize, future C# versions may provide more efficient parallel asynchronous processing tools, further simplifying developers' work.