1

C#: Read lines from a text file in batches and process in parallel

C#: Read lines from a text file in batches and process in parallel

 

I have had numerous requests from readers if I could provide some sample C# code which will read lines from a text file in batches and process in parallel, so here it is!

 

As bench-marked in my article, “The fastest way to read and process text files”, here is some sample code which does the following:

  1. reads an array of CSV files line by line
  2. when a “batch” threshold (eg, a set number of lines) is hit, it will then process all the lines that have been read in parallel
  3. will repeat until all the entirety of the file(s) has been read.

 

This code is not meant to be production ready, but to simply serve as a quick guide and starting point to help you with your own production ready code.

 

Also included are the database and datatable snippets you’ll need if you’re uploading into a database. Obviously if you don’t want/need this functionality, just remove or comment out.

 

This sample code, as it stands, assumes:

  • the text files are of type CSV and SimpleDelimited
  • the processed data is uploaded into a database
  • the column names in the files and database table match

 

Hope this helps! If you have any questions, feel free to ask in the comments!

 

 

  • StackOverflow41

    This is awesome! Thanks so much for posting this “starter” code! It’s exactly what I was trying to do because we have to read a lot of big text files (many lines) and I was wanting to process batches of lines at once instead of line by line!