Data warehousing – Data warehousing technology

Throughout the history of systems development, the primary emphasis had been given to the operational systems (simple OLTP -Online transact ion processing systems) and the data they process. Little effort was put on building up an integrated information management system that is capable of capturing, storing & handling the distributed operational data (current & historic of the enterprise i massive amounts in a structured & centralized data repository and at the same time supporting the routine transaction processing tasks so that any required data is available at the centralized data repository in a structured manner, most tuned for high performance intelligent data analysis.

Since enterprise data is an asset and resource of the enterprise, a system that satisfies the above requirement has a great relevance in today’s enterprise information management. As the enterprise goes on working over the years, bulk of enterprise data is also going to be accumulated in its routine OLTP systems. If there is no efficient means to capture, archive & handle this valuable resource in a structured manner both in current & historic terms in a centralised data repository, it would be a tedious task for the enterprise to search and analyse its bulk of enterprise data for intelligent decision making. Eventhough the routine OLTP systems provides a means for data analysis and low level decision making, they are not designed in a way that supports archiving & handling massive & distributed data for supporting centralized data analysis and decision making.

The downside of today’s varied enterprise wide desktop applications is that i leaves the data fragmented and oriented towards very specific areas in various islands of information. For eg. a company having various branches at different locations will have its enterprise data scattered at various locations and bulk of data accumulates in each of these locations daily. This creates of information for islands the enterprise to deal with, while centrally managing its bulk of distributed enterprise data. Since the enterprise data resides in various islands of information that are not well structured and integrated, the effort spend on organizing, standardizing, integrating and extracting it to the decision maker in required form with minimum amount of time is great.

Now, even though the analysis systems such as DSS & EIS enabled business users to manage their business data, still there was problem of not having a standard set of procedures and architecture to store, structure and manage massive, historical and distributed data for intelligent and high performance queries and trends analysis.

Towards this goal of setting up such an integrated and centralized Data analysis system, that is capable of storing, structuring and managing massive historical and distributed enterprise data for intelligent and high performance queries and trends analysis, we go for Data warehousing, in simple words we go for creating a warehousing system for handling enterprise data. It was at this point when ITIS departments thought of building such an efficient and integrated Data warehousing system, that the evolution of the new technology -Data warehousing -started. W.H Inmon was one of the early proponents who put forward the concepts of Data warehousing and he has stated the standard concepts, techniques & guidelines for building up a Data warehousing system.

Supporting factors for the evolution of Data warehousing technology

The most important factor that has supported the evolution of Data- warehousing is the forward movement in the hardware & software technologies, sharply decreasing prices and increasing power of computer hardware coupled with the ease of use of today’s software especially the database softwares. This has made possible the quick development of Data warehouses and quicker analysis of hundreds of GBs of information & business knowledge.

The upcoming of the new Relational database technology and the object- oriented technology supported by powerful database engines such as Oracle8+ server, MS SQL Server etc and a plethora of database utilities has now made Data warehouse application development efficient and easier.

Nowadays processor speeds has risen to the order of speeds that has made complex computations and database queries executed within microseconds. sophisticated processor hardware architectures such as the symmetric multiprocessor (SMP) and Massively parallel processed (MPP) have come to mainstream computing with inexpensive machines. Higher capacity memory chips, a key component influencing the performance of a Data warehouse, are available at very low prices. Now it is possible to have a moderately priced machine with terabytes of HD space with high speed RAM. Computer bus such as PCI controller, the Universal serial bus (USB) and interfaces such as Ultra SCSI has made the I/O incredibly fast. Just 2 decades ago, it would have taken a roomful of disk-drives to store information that can now be easily stored on a single 1 inch hard disk.

Server operating systems like Windows NT and Unix have brought mission critical stability and powerful features to the distributed computing environment. The powerful OS concepts such as virtual memory, multi-tasking and symmetric multiprocessing are now available on inexpensive operating platforms. Operating systems such as Win NT have made these powerful systems very easy to setup and operate reducing the total cost of ownership of these powerful servers.