Thursday 4 September 2008

Data Integrator Get Started Note (1)

DI Architect Overview

Overview of Standard Data Integrator


  • Data Integrator Designer : to create applications containing work flows
  • Data Integrator repository: a set of tables that hold user-created and predefined system objects, source and target metadata, and transformation rules, 2 types: local repository & central repository.
  • Data Integrator Job Server :starts the data movement engine that integrates data from multiple heterogeneous sources, performs complex data transformations, and manages extractions and transactions from ERP systems and other sources. 2 modes batch or real-time mode.
  • Data Integrator engine :processes to perform data extraction, transformation, and movement.
  • Data Integrator Access Server: real-time, request-reply message broker that collects
    message requests, routes them to a real-time service, and delivers a message reply within a user-specified time frame.
  • Data Integrator Administrator : provides browser-based administration of Data Integrator. Resources including:
    • Scheduling, monitoring, and executing batch jobs
    • Configuring, starting, and stopping real-time services
    • Configuring Job Server, Access Server, and repository usage
    • Configuring and managing adapters
    • Managing users
    • Publishing batch jobs and real-time services via Web services
  • Data Integrator Metadata Reports applications : provide four applications for exploring your metadata:
    • Impact and lineage analysis
    • Operational dashboards
    • Auto documentation
    • Data validation
  • Data Integrator Metadata Integrator :seamlessly share metadata with Business Objects Intelligence products.

  • Data Integrator Web Server
  • Data Integrator Service
  • Data Integrator SNMP Agent: Simple network management protocol (SNMP) support.
  • Data Integrator Adapter SDK
    • Reading, writing, and request-reply from Data Integrator to other systems
    • Request-reply from other systems to Data Integrator
Data Integrator distributed architecture



Data Integrator System Configuration

Data Integrator Bin/config investigation

昨天不小心非法了DI,然后就起不来了。:(折腾一上午终于搞定,排障过程如下:
1.Investigate Log files and notice log\Errorlog.txt提示说找不到DICodePages.txt.
2.为了确认报错的确是由client启动未遂造成的,将其备份,清空 reproduce,成功。
3.比较bin\下面的file,发现所有的*.txt均不见了。
List 如下:
  • DICodepages.txt
  • DILanguages.txt
  • DITerritories.txt
  • DSConfig.txt (Configuration information! Important!)
    • Job Server Configuration
    • SNMP Configuration
    • Repository Configuration.
  • errormsg.txt
  • jsConfig.txt
  • MLMessages.txt
  • tracemsg.txt
之后,Client, Server 都可以work了,真是奇怪。这些文件,貌似除了config是可以修改的,其他都应该是只读的呀。

启动了Client开始编辑以后,发现这些file已经被改名为数字文件(1.txt,2.txt,3.txt etc....),难道一定要用这种方式进行读写么?

然后,就发现,其实,是我在文件名称的时候,不小心把他们都改了,一定是因为他们在root目录下进行的,So,之后就演变成为绝对路径和相对路径的问题了。哈哈。真是诡异呀。

后记:最终的原因是我的脚本有一个循环修改file名字的步骤。但是它又没有找到制定的文件夹,所以,就在bin文件夹下面的(*.txt)进行了循环修改,导致了上述错误。fix了DI job以后,一切正常。不过,由此发现一件事情,就是DI 难道不支持多语言版本么?Error info居然就这样在Bin文件夹下面裸奔......实在有点无语。

Data Integrator Log Investigation

Root of Log
  • $InstallFolder\Log
Folder Structure
ROOT
  • \BulkLoader
  • \Mataexchange
  • \$PCache: Data Cache from Server, also configure the place on Server Manager.
  • \$JobServer1
  • \$JobServer2...
  • WebServices.log
  • inslog.txt(for install information)
  • errorlog.txt
  • monitor.txt
  • Service_eventlog.txt
  • AL_RWJobLauncherLog.txt (Job server launcher)
  • Tracelog.txt (trace info?)
  • Server_configlog.txt (Server configuration update log via Server Manager)
  • Statistics log — A list of each step in the job, the number of rows processed by that step, and the time required to complete the operation.


More Details for logs.
  • errorlog.txt: A list of any errors produced by the RDBMS, Data Integrator, or the computer operating system during the job execution. e.g. Error Number <80101>...Error message
  • service_eventlog.txt: tracks server starts events. e.g.[Wed Jul 30 16:03:05 2008 ] Job Server on DataIntegratorJobServer1 Port 3500 is started.[Wed Jul 30 16:03:05 2008 ] SNMP Agent on Broker Port 4001 is started.[Sat Aug 09 12:27:58 2008 ] Quit Job Service...
  • Monitor.txt : A list of the job steps in the order they started. +di_job_al_mach_info/di_df_al_mach_info_1/Query, READY, 0, 0.000, 7.559-di_job_al_mach_info/di_df_al_mach_info_1/$di_machFileName, READY, 0, 0.000, N di_job_al_mach_info|DATAFLOW di_df_al_mach_info
  • AL_RWJobLauncherLog.txt: (Job server launcher) :09_03_2008 23:00:00 CRWJobLauncherApp::InitInstance called.09_03_2008 23:00:00 BODI-1250134: Launching Job (no wait, no status). INET ADDR , GUID .