There's recently been a bit of banter (here and here) regarding the appropriateness of applying the MVC pattern to ASP.Net applications (and probably any application). I'm quite opinionated in this area so I felt I should chime in, from the "over-engineered vs. under-engineered" design perspectives.

I find that the more experience you get with application design, the more you uncover this inherent conflict that exists in every architecture. It's a balance between what you want to see today, and what you want to do tomorrow. There's really no avoiding the need for expansion in any program, and to ignore that is just going to mean a massive rewrite later on, and yet you don't want to bog down your first version with an insanely complex n-tier codebase from hell either.

I've seen it happen time and time again. There's really two scenarios that occur: under-engineered and over-engineered: Either the system is written with complete disregard for future expansion (interweaving domain and presentation logic, for example), or the system is subject to the mind of a naïve genius who single-handedly develops an uber-design based on what they think will happen to the business and the application and ultimately produces a system that is way over-engineered, and then (surprise surprise) the requirements change and the entire codebase has to be re-developed from scratch.

Off the top of my head, here's a few key principles I've derived from this formula:

For example, rather than validating an order in the OKButton_Click event procedure in a form class, if the validation is quite complex then just put it into a separate function and pass the order details in as a parameter, with an intention of making it independent from the presentation layer code. It's a common sense and simple step to take, but it makes the code far more expandable and manageable if this approach permeates your entire application. It also opens the code up to refactoring, if you need to move the code out to another project later on.

It's important that you define the signature or interface on to your service function with a bit of thought. For example, think about the name of the service, what comes in and what goes out. Identify your services with the intention of re-use, and of centralizing the unit of business knowledge into one easily accessible method. Apply the DRY principle

If you set your code up appropriately, refactoring can be a very effective way of having your application evolve. Anything that you can do to make your code easier to refactor will make your life much easier when the time comes, and help you maintain maximum agility. Keeping your code simple, not interdependent, and maximizing your re-use of code are all fundamental to achieving a codebase that is open to refactoring.

To achieve this, to the extent possible - write your service functions as if they were global functions. This way you're restricting yourself to just what is passed in as parameters, and reducing your dependency on any state that might be maintained in the class and the rest of the application.

Keep your data in your typed datasets, and your code in service functions. In any company and in any system, the data is the most important asset. Most of the services, if they're written as discreet units, can be replaced at a later date - but the data cannot. Having easily replaceable services is what makes the system agile, easier to build upon and more adaptable to changes in the business and application requirements.

Always treat the data like it's a guest to your application. Wherever possible, without making any significant compromises to performance, reduce where you're caching the data. Disconnected and semi-permanent caching of data often reduces your ability to expand the application, because now any additional services that use that data must get it from your cache. If you must cache data, find a way to keep that cache live - maybe not in real-time but at least give the cache an expiry so it can periodically refresh.

Above all, keep your code simple. And if you're looking for a principle, this has been coined the KISS or "Keep It Simple, Stupid!" Principle.

Keeping code simple is often a matter of weighing up the cost of changing something later on, compared to putting up with unnecessary complexity now. If you really think about it, most of the time the opportunity to refactor code at a later date is a real possibility.

For example, don't create separate projects for each class, on the basis that you might expand that class at some point and it might be too big to share with another project, or that you might want to share that code with other applications one day so it deserves to be put into another project. Neither of those reasons justify creating a new project! It's simply a matter of keeping your code agile enough that it can be refactored into separate projects as and when it's needed.

Another couple of classic over-complications: why create an interface for your class just because it might need an interface one day? As long as the class isn't used outside of your project, the day that you need to expose it through an interface will be a simple matter of refactoring and rebuilding your class. The overhead of maintaining both the class and interface until then just isn't justified. The same goes for public properties. If it's an internal class, why not just declare a public field instead? When you do need to control access to the field, then you can easily add a property to encapsulate that field.

So I think the bottom line is that you don't need any more layers than a semi-formal distinction between your domain logic (services), your data (model) and the glue that keeps it all together (the view and controller). If the level of separation is just a matter of different functions, then that's fine and can be refactored to something more structured when the need arises.

And above all, KISS!

These days few of us have to worry about backing up our work, because our companies typically have a pretty good system in place that backs files up transparently to us. However, if you do any private work or research at home, then backing up can often be forgotten completely.

Usually I back up on to my keyring memory stick. These days those things come standard with 128 or even 256Mb... and some friends of mine have 1Gb sticks.

The trick to backing up quickly and efficiently is in filtering the files that you copy. My memory stick is just a 64Mb one and assuming you also have limited space, you also probably wouldn't want to copy all binaries, program databases and support files onto your backup device either.

In fact, my requirements are pretty precise:

1. I want to backup only the things that aren't generated during a build (eg. mostly just manually written source files)
2. I want to keep all the history of the files.
3. Each time I back up, I only want to update files that have changed, not the entire thing.

Like any sensible developer, I keep all my source in a source control system. In fact this is pretty much a requirement if you want to backup a history of each file.

Now I'm done praising the DataSet and loosely coupled programming, to balance it out a bit I'm going to highlight some of my 'pet peeves' relating to my experiences using ADO.Net, and specifically the implementation of the DataSet (in no particular order):

To be fair, DataSets do support the concept of a relationship being either one-to-one or one-to-many, but only indirectly through the ParentConstraint and ChildConstraints properties. However, while this is kind-of supported at the DataSet level, it is seemingly ignored by the typed-dataset generator that creates our type-safe DataSets. Instead it treats all relationships as one-to-many.

This makes coding against the typed-dataset in C# a little ugly -- something that should be as simple as stock.quote.price becomes stock.quote[0].price, introducing magic numbers that really make your code look more cryptic than necessary.

The Data Adapter model for populating a dataset involves running a Fill method on the data adapter, and passing it a reference of the DataSet instance to populate. While this works fine if both the data adapter and dataset are local, when the adapter is accessed through remoting you no longer have the option of passing in the DataSet as a parameter, because the DataSet does not extend MarshalByRefObject and is always passed by value across remoting boundaries.

This means that when we run the command, a new copy of the DataSet is passed to the adapter, and any changes it might make to that structure will be lost when the function returns.

Given that DataSets were really designed for this service-oriented environment, it seems to me that it would have made sense to design the adapter pattern to work better in a distributed environment.

The way around this is to wrap the Fill method in a new service that instead returns a new DataSet. At the receiving end, this DataSet can then be used to access the data, or in the case where a pre-existing DataSet is defined in a form and bound to a number of objects, this DataSet can then be merged into the main DataSet. Which leads me on to...

ADO.Net is very much oriented around the concept of DataSets, holding multiple tables that are related to one another. However, things aren't always as complex as that and more-often-than-not you end up with a DataSet containing just one DataTable. Such a situation is common, and overly complicated thanks to the seemingly needless DataSet required to use the DataTable in many common environments.

One such environment is merging changes or new records received from a service. This is only supported through the DataSet, a merge on a table alone is not supported.

Another environment where there is a lack of alignment between the typed DataTable and typed DataSet is when you come to remoting or persistence. While the typed DataSet supports serialization so that it can be passed by value in remoting calls, the typed DataTable does not.

However, I should add that support for serialization in the DataTable has been fixed in the next version of .Net (version 2.0). Also, it is possible to work-around this by editing the generated code and adding the required serialization constructor to the typed-datatables (although this is not recommended as it will be lost in the next generation).

Of course, they don't provide serialization of DataRows, but that's a whole other can of worms...

Maybe because it's designed to work in a disconnected environment, but given that so many systems work in an environment where data is constantly updating it seems surprising that DataSets don't have better support for realtime updates. The main problem is that there is no way to differentiate between data updated by the user (through in-place editing, for example) and data updated from the source, and even if there was a way to differentiate there certainly aren't any merge conflict resolution features built in to the DataSet.

The workaround is basically to disable user-editing of the DataSet completely, or to implement your own realtime data store that exposes the required ADO.Net interfaces (IBindingList and ITypedList).

Another issue is to do with the granularity of the updates managed by the DataSet.

In a highly concurrent environment, you may be in a situation where two or more users are updating different fields of the same row, that a field representing a small fraction of the size of the row is being updated multiple times a second, or that a user only has permission to update certain fields.

In each of these cases you may run into problems because the DataSet does not record which fields have been updated in a particular row. As such, the adapter / command builder will try to commit the entire row even though one field has changed, causing problems such as permission exceptions and others as listed above.

A possible workaround is to compare old values for new values and determine yourself which values should be sent as part of the update. However, the work involved in doing this is not trivial, and you may still have to handle merge conflict resolution for cases where the same field is being updated.

When the data comes back from a service call to populate the DataSet the typical pattern is to put the details of any failures into the DataSet's rows themselves, supposedly so you can display them automatically in a grid. This makes the assumption that your DataSet is only really used for binding to user-interface forms, which is a dangerous assumption.

It's dangerous because it's not true, and in many cases you will not code to check for the presence of those error flags in the row and will continue processing under the impression that everything is OK.

It's also bad because the error is reported as a simple string. No exception, no error structure, not even an error code to go on.

The workaround is to expose your errors by throwing an exception, guaranteeing that the error will be noticed - if not by the calling code then by terminating the application with an unhandled exception error. It guarantees that processing will not continue with the impression everything is OK - because things certainly are not OK if errors have occurred.

If you wish to retain the ability to have errors associated with rows, I suggest you create a custom exception that contains a copy of the DataSet containing the rows with errors flagged.

Relationships are a big step forward in making relational data more manageable in an application and from user-interface components. I can now provide a DataSet to a grid and, using the relationship data held within that DataSet, the grid will then display that data hierarchically, automatically processing the relationships to sub-tables when the rows are expanded.

The problem here is the assumption that the DataSet already contains all the data it needs. In many cases, the data is being expanded into a large sub-table of related rows, and it's simply not manageable or scalable to pre-fetch all the related data when the DataSet is first populated.

The most common workaround is to delay the population of the sub-table fetch until the data is required. However this is more difficult than you may think. The real problem here is that the DataSet raises no events and has no callbacks that allow you to intercept a request for the related data.

Instead, people rely on grid events and intrinsic code knowledge in order to intercept interests in the relationship and delay the fetch till that time. A better solution would be to have the DataSet raise an event whenever the child rows are requested, populating the required rows on-demand.

The result of processing a relationship on a DataSet is a list of child rows, represented as a DataRow array. It would have made more sense to me if this were returned as a type of DataView or table, so I can then further filter the rows in which I'm interested, make changes to that list that I can later commit and so on...

If you're a UI component programmer, the DataView class is a wonderful thing: It provides most of the functionality you need to display data in a DataTable, including filtering and knowing when to update rows that have been edited by the user or outside source.

However, while the filtering is great - offering SQL like statements that are processed in-memory, and keeping the results of that query up-to-date in realtime - if only lets you process the data in that single table.

Now if it just let you join between more than one table, then that would save a considerable amount of coding, especially when it comes to processing hierarchical views and updates to data involved in complex joins.

The DataSet is a wonderful piece of architecture, and has provided a great bridge from the connected to disconnected data processing world. However there are shortcomings when using the DataSet in real world applications and I hope I've highlighted some of them here.

I'm sure I'm missing some, and I'm open to the possibility that some of my statements may be misunderstood or misunderstandings on my behalf, so please comment if you feel so inclined.

ADO.Net is really so radically different to traditional ADO that it's somewhat misleading to give the technology the same name.

Traditional ADO is largely oriented around recordsets that let us navigate over data retrieved from a connected data source (typically ODBC or OLE/DB drivers) without having to know where the data orignated (at least, that's the theory).

ADO.Net largely replaces the old recordset concept with a 'DataSet'. The concept of DataSets dramatically changes the way we program databases by separating the code that accesses the database from the data itself. In old ADO the focus was on separating the interface onto the database from the database itself, through interfaces that can have implementation 'plugged in', often relying on ODBC drivers to provide the adapter to the database engine. However, the data and the implementation are still tightly bound to one another, making access to the data in a disconnected environment largely impossible.

DataSets achieve separation of data and database by providing what is really a disconnected cache of the data: a recordset onto an in-memory copy of the data.

Although largely sold as a data access technology optimized for the Web, the disconnected nature of ADO.Net runs deeper than that, having its roots in the movement towards something known as service orientation, an architecture based on loosely coupled systems.

For years a fundamental conflict existed between the requirements for enterprise applications as being distributed, autonomous components running over a potentially unreliable network, and the intrinsicly connected and not particularly forgiving technology commonly referred to as 'distributed objects', such as DCOM and CORBA. It wasn't really until the Internet gained sufficient popularity and web services standards were established that the industry started to catch on that having control of an object living on another machine really wasn't that useful in a highly dynamic and stateless environment such as the Web.

A web service becomes something you can request once-off, without any previously established connections or state required, making it far more suitable for use over stateless protocols such as HTTP. .Net also provides us with the ability to access these web services through its built-in support for standards such as SOAP and single-call .Net remoting.

However we need more than simply the ability to run services. Loosely coupled programming requires a whole new paradigm shift and a whole new set of tools. No longer can you rely on the server being there the whole time, and instead you shoot off service requests, get the result and disconnect. And as such traditional ADO's connected recordset model really doesn't fit in, because it intrinsicly requires the server to be available while you use the data.

And so DataSets provide the missing link in this puzzle: They provide a way to store the result of a communication with a server in a disconnected environment so you can freely read, search, and even modify the data without requiring a persistent connection to the server. In fact, given the fact that disconnected programming is really the central focus of ADO.Net, I think it probably would have made more sense for them to call the technology DDO - Disconnected Data Objects - rather than Active Data Objects.

DataSets are really just the tip of the iceberg in ADO.Net, and I hope to cover other aspects of this disconnected data toolset in future entries.

Welcome to my work blog.

I'm an independent software consultant specializing in enterprise development using .Net tools, and so you'll find that most of my posts are about either .Net or enterprise development. I hope you find something useful in here from time to time.