1

C#: Read lines from a text file in batches and process in parallel

C#: Read lines from a text file in batches and process in parallel

 

I have had numerous requests from readers if I could provide some sample C# code which will read lines from a text file in batches and process in parallel, so here it is!

 

As bench-marked in my article, “The fastest way to read and process text files”, here is some sample code which does the following:

  1. reads an array of CSV files line by line
  2. when a “batch” threshold (eg, a set number of lines) is hit, it will then process all the lines that have been read in parallel
  3. will repeat until all the entirety of the file(s) has been read.

 

This code is not meant to be production ready, but to simply serve as a quick guide and starting point to help you with your own production ready code.

 

Also included are the database and datatable snippets you’ll need if you’re uploading into a database. Obviously if you don’t want/need this functionality, just remove or comment out.

 

This sample code, as it stands, assumes:

  • the text files are of type CSV and SimpleDelimited
  • the processed data is uploaded into a database
  • the column names in the files and database table match

 

Hope this helps! If you have any questions, feel free to ask in the comments!