>Does anybody have any opinions on the subject of data warehousing?.
Yes, I've worked on a few of these projects over the last few years.
Many are now tied into architectural notions of "business objects" or
"three layer architectures" as well.
Quote:>Do you think it is another fad that will stay fashionable for a while
>and then become redundant
Of course. They always do. It doesn't mean that it's any more or less
of a useful technique.
Quote:> or do you think it is a necessary part of
>corporate integration within the knowldege society.
Nothing is _necessary_. You make anything work if you're prepared to
foot the bills.
On an omniscient scale, it's a nice idea. It's also technically
feasible to do it. The biggest problem is usually one of office
politics; "data warehousing" is basically a shorthand for "the return
of the faceless IT department" and this will suffer considerable user
resistance from those who've become fond of the freedom that
in-department development can bring.
An ideal solution would build well-managed corporate data resources
that were still flexible to user demands.
There would be a presentation layer (OLE servers, business objects, or
a well-defined SQL Server interface of published queries and stored
procedures). These business objects would offer data services in a
business-oriented manner, hiding the underlying structure and
implementation details in textbook manner.
On top of this presentation layer would be built applications,
possibly built by small teams within departments. These would use RAD
techniques to produce end products quickly and flexibly. RAD is a
particularly appropriate technology, because the amount of processing
done within the business objects reduces the implementation complexity
of the application layer. This is particularly useful with tools like
VB, which offer good development speed but have poor implemnetation
features for sophisticated data manipulation.
There are many reasons why we don't see this ideal:
Conflict of interest between teams. Each team is responsible for
producing one layer of the solution and it's in their interests to
produce the simplest and most expedient solution to their own layer,
not to produce the best overall system. Current management techniques
are adversarial and empire-building, with no-one taking an overall
interest in the _project_.
Poor quality product. Building a project end-to-end is simpler than
building an architectural project, as you can keep any corners cut
within the team. If you're writing layers OTOH, your published
interfaces must be good, and they must be implemented well. You can't
take a "short cut" between the layers without comproising the
architecture, and thus the work of several teams.
RAD. No-one knows how to do it. RAD (in practice) means "just bang it
together - we don't have to do it right any more". This is a
particular soapbox of mine, but there are some pretty awfully managed
projects out there.
Inflexible business objects. A good business object requires some
particularly careful business analysis beforehand. The prevailing
atmosphere of "We can now build huge projects overnight" means that
any analysis and design phase is impossibly squeezed for time and
effort. This leads to poorly thought out objects which are
inappropriate (bad analysis) and inflexible (bad design)
Slow business objects. There is a big temptation to make objects do
all things for all men, simply because no-one takes the time and
trouble to work out what really needs doing. This leads to over-blown
objects that are too slow in operation. Although there are many
techniques (e.g. caching & access-on-demand) that could be
implemented inside the objects to alleviate this, it's not in the
object team's short-term interests to implement this, and higher level
management is too blinkered to see this as an advantage to the overall
project, beyond being yet another non-essential feature that can be
cut to meet deadlines.
Quote:> Does anyone have
>any examples of companies carrying out this process and the resultant
>advantages/ disadvantages that it has brought.
Clean machine rooms. The first one I worked on was for a 10 year old
business that had (for political reasons) collected a ridiculous
number of disparate and uncommunicative mini systems. The IT rework
allowed all of these systems to be replaced by SQL Servers on a number
of commodity PC platforms. We killed off the expensive mini, we killed
off the _variety_ of minis, and we killed off the need to maintain a
raised-floor machine room environment.
Centralisation of business practices and rules, eliminating
inconsistencies between different parts of the business. Although
these incidents are often publically embarassing when they arise,
they're rarely serious problems for the business (maybe a posting to
comp.risks about airline duble-booking errors).
Sharing of corporate data, encouraging trendy management practices of
"Open Business" and data sharing.
Tight security to data that does need it, as the access is constricted
to one easily managed bottleneck.
Done well, it combines good housekeeping practices for the core data,
with the flexibility that ad hoc small-scale development projects can
Quick, easy and reliable (i.e. unlikely to go over-budget) development
of front-end applications.
Good facilities for data mining. Mind you, when did you ever see
someone so high up that they could usefully benefit from smart data
mining, yet had the development effort available to implement them.
On the last three advantages, these are all theoretical. I've not yet
seen _one_ project that pulled them off.
This is a large business-wide project and much of it needs to be
implemented simultaneously. The size of this project often exceeds the
management's capacity to cope and the entire project crashes and
All the company's IT eggs are in one basket, so any problems in
implementing it are likely to have repercussions at board level.
There's a tendency to hide problems for as long as possible, then
over-react when they become too big to hide.
Particular implementation techniques (particularly remote OLE) may be
tied into one vendor's product line.
The whole notion of "RAD" and its kin has been used (wrongly) as an
excuse for shoddy workmanship, bad management and incompetent
development work. This whole industry deserves a good kicking.
Combining three layer & two layer architectural models in the same
environment is an absolute recipe for disaster.