Introduction
Always wanted to build a file importing function? I recently had to build such functionality. And, as any other developer, I wanted to build the importing functionality as generic as possible. Off-course the pitfall you are facing is the one that you are so busy creating generic functionality that you are way of target and find yourself building software that has so many features and just a few of them used. The “You Aren’t Gonna Need It” syndrome (YAGNI). But let’s get to the point. The implementation described in this article shows one way to create a relative simple and generic enough solution. The main trick will be to use the data from the file and persist it in an object in the application. We start with reading a file stream from any where in the application. The file will be loaded from an ASP.NET web form. The sample code is written in the C#.NET programming language. This article includes subjects like reading data from a comma seperated text file, creating a generic method, using the generic .NET list object, persisting data read from the text file by using .NET reflection principles.
Reading a stream
Reading a stream is not so hard. Today ASP.NET provides a standard FileUpload control to do the job. It provides a browse button to select a text file and delivers the contents of the file selected to the application in stream or byte array. Just create an event handler and implement code to put the stream into.
protected void btnStartImport_Click(object sender, EventArgs e)
{
if (FileUploadControl1.HasFile)
{
Stream selectedStream = FileUploadControl1.FileContent;
…
}
}
This is all that you need to do to get the file stream from the FileUpload control.
Ways to read a stream
The FileUpload control is also capable of delivering the uploaded file as a byte array. This off-course is a valid way to handle things, but things get more complicated further on while working with the file content. Byte arrays are at a lower technical level, so you have to do more to accomplish the same as with a stream. .NET provides nice components that work well with streams. They handle all the stuff that has to be done when working with byte arrays. The .NET Stream class is a good option if you do not want to do some die hard byte array surfing.
Generic method
Let’s get to the real job. Importing the stream and produce a list of usable rows the file contains. During this sequence several steps have to be taken. But first the method has to get a name and has to be nicely embedded into a class. The class I put the import functionality in is called FileImporter. The import method I have created is called Import, how nifty….
public static List<T> Import<T>(Stream importStream){}
The method call looks like this.
List<Customer> customers = FileImporter.Import<Customer>(selectedStream);
The implementation path
Now that the class has been constructed the following steps have to be implemented:
- Create a container to drop the rows found in the imported file.
- Process the stream and create a string array out of it.
- Loop through all elements in the string array just constructed.
- Create an instance of the type provided at the method call.
- Split each element by the field seperator, usually a comma or semicolon, and dump the result in a second string array.
- Use the second string array to get it data from and put in an instance of type T, the type provided at the method call.
- Add the business object to the container created earlier.
- Return the container.
The steps above are a rough implementation path. I usually use such a path to guide myself when creating the implementation.
Create a container
The easiest container to use is a generic list. Use the provided type from the method call, T.
…
List<T> container = new List<T>();
…
The container can now be used to store objects of type T. For instance, when the method call is like FileImporter.Import
, the type of T would be Customer. When creating a generic list with T, this means that the generic list can only containt object of type Customer. The container.ToArray produces a real array of Customer objects.
Process the stream
The next step is to create a list of rows. My basic file importer uses a string array for easy access to the row data. To create such a row list I have created a private method within the FileImporter class called ProcessFile.
private static string[] ProcessFile(Stream file)
{
List<string> rows = new List<string>();
StreamReader reader = new StreamReader(file);
String line;
// Remove the header line from the file.
reader = RemoveHeader(reader);
while ((line = reader.ReadLine()) != null)
{
rows.Add(line);
}
return rows.ToArray();
}
This is realy simple code. Create another generic list with type string. Use a StreamReader to get an entry to the file. Loop through all the lines in the stream. I have choosen to skip the first line (Through a private method RemoveHeader()), because it is common practice to use the first line of a text file for definition of the fields and their order. This is also known as the Meta data of the text file.
Next thing to do is to check wheter the line index nmber is greater than 0, if so add the row to the list of strings, if not then continue with the next line. Finally return the created list.
Loop through all the elements
Now that a readable list of string has been created, you can start reading the data from it by constructing a simple for each loop.
foreach (string row in rows)
{
string[] fields = row.Split(separator.ToCharArray());
object instance = Activator.CreateInstance(typeof(T));
…
}
At this point you can split the row as you like. In my case I have choosen to create a seperator private field, read a seperator from my web.config file and store it in the seperator field. By using this technique you create a felxible way of configuring the type of seperator used in the files that have to be imported. Working with a config file in your application is out of the scope of this article, but when surfing around you will find tons of information about using configuration files.
So use the Split method to create a string array with all the fields in it. Besides creating this string array of fields, you need another object. This object is of the type provided at the method call to the Import method. This object has to be instanciated before we can store data in it. Line 4 of the sample above, shows how to create an instance of the type provided to the method.
Persisting the data
The most important and main goal for this file import functionality is to get the data out of the file and persist it to an object within the application. To start this process I need an object. Line 4 of the former sample has created such an object. I have to loop through the objects properties and get the data from the row to persist it to the property. This technique is called reflection. In short, with reflection you are able to get information, such as a list of the properties or methods, from a .NET type. Here is the code. The type used to get info from a property is called PropertyInfo.
foreach (PropertyInfo pi in typeof(T).GetProperties())
{
string name = typeof (T).Name + pi.Name;
string value = ConfigurationManager.AppSettings[name].ToString();
int pos;
if (!int.TryParse(value, out pos)) continue;
pi.SetValue(instance, pi, fields[pos]);
}
Again I use the web.config file to get the position of the required fields in the text file. I construct the key to get the value for the position from the application settings in the config file by using the name of the type T, provided at the method call, and the name of the property info. This position is the index of the field that holds the value for the property. Whit the code in lines 6 and 7 in the above code sample I get the value from the config file and try to parse it to int. This is for consistency reasons only. If the int.TryParse fails it means that the key is not found in the config file or something is wrong with the value of it. If any problem occurs the loop continues to the next PropertyInfo in the list of properties. If the TryParse succeeds a valid position has been found in the config file and the process can proceed. Now persist the value from the fields list to the object by using the PropertyInfo.SetValue method as shown at line 8 in the code sample above.
Add the persisted object to the list
The last job is to add the persisted object to the list created earlier in the Import method.
list.Add((T) instance);
Make sure that you cast the instance to the type T provided at the method call and add it to the list of the same type. I you forget the cast you will get an invalid cast error at compile time because you cannot implicitly cast an instance of type object to type T. The final stage of the Import method will be returning the list.
Conclusion
With this article I have shown you how you can have generics work for you and build simple file importing functionality. Because the implementation is on a generic level it can be used in several different scenarios. I know that more implementations lead to the same results so take the presented solution into consideration and use it to your advantage.
Wouter Goedvriend
Senior Consultant at Capgemini the Netherlands
Personal weblog - http://blog.goedvriend.com