An improved approach in data warehousing ETLM process for detection of changes in source data

Rosana L. de B. A. RochaLeonardo Figueiredo CardosoJano Moreira de Souza

The ETLM process (extraction, transformation, load, and materialization) comprises the basis for data acquisition and organization of a data warehouse (DW) environment. In the DW area, a well-known issue consists of the detection of changes in source data. This problem arises when the data source does not possess or does not provide the information about the changes that have occurred. The main scenarios in which this problem is found may be characterized by: (i) the impossibility to instrument the DBMS (triggers, transaction log, stored procedures, replication, materialized views, old and new versions of data, etc), due to security policies, data property or performance issues; (ii) the lack of instrumentation resources on the DBMS; (iii) the use of legacy technologies such as file systems or semi-structured data; (iv) the existence of application proprietary databases and ERP systems. In order to deal with this issue, controls have to be implemented to enable the detection of these changes and to reflect them into the DW environment. In this paper, a framework is proposed for the organization of the DW environment, and some of the main issues involved in the ETLM process are commented on. Having this framework as a basis, the main approaches to solve the detection of changes in source data are described. Our technique, using signatures to mark off and detect changes is presented. This technique was derived from the comparison of database snapshots approach: the technique is simple and can be applied to all four scenarios above. The paper also presents a case study in the DW project developed for Rio Sul Airlines, a regional aviation company owned by the Brasilian Varig group.

Caso o link acima esteja inválido, faça uma busca pelo texto completo na Web: Buscar na Web

Biblioteca Digital Brasileira de Computação - Contato:
     Mantida por: