IBM Information Server 8.X (DataStage): Architecture and Its Components

What is DataStage?

  • An ETL tool to Extract, Transform and Load the data into data mart or data warehousing
  • Used for data integration projects such as data warehousing, ODS (Operational Data Store) and can connect to major databases like Teradata, Oracle, DB2, SQL Server etc.
  • Designed ETL jobs can migrate in different environments such as Dev, UAT and Prod by importing and exporting DataStage components
  • Can manage metadata in the jobs
  • Can schedule, execute and monitor the jobs in DataStage

DataStage Architecture:

DataStage allow us to develop the jobs in Server or Parallel editions. Parallel edition uses the parallel processing capabilities for processing the data and is ideal for large volumes of data.

Components:

  • Designer
  • Director
  • Administrator

Administrator:

The following tasks performed using the administrator.

  • Add, delete, and move projects
  • Set up user permissions for projects
  • Purge job log files
  • Set the timeout interval on the engine
  • Trace the engine activity
  • Set job parameter defaults
  • Issue WebSphere DataStage Engine commands from the Administration client
  • Configure parallel processing jobs settings.
  • Create/set environmental variables.

Enabling job administration in the Director client:

These features let WebSphere DataStage operators release the resources of a job that has aborted or hung, and so return the job to a state where it can run.

This procedure enables two commands in the Director menu.

  • Cleanup Resources
  • Clear Status File

Designer:

  • Design and develop using the graphical design tool
  • Various stages like General, Database, File, Processing stages used while developing jobs
  • Table definitions can be imported directly from the data source or data warehousing tables
  • Jobs are compiled using the designer and it checks for any compilation errors in primary inputs, reference outputs, key expressions, transforms etc.
  • Import and/or export projects from different environments
  • Server, mainframe and parallel jobs can be created using the designer
  • Define parameters in parameters page under the properties and will be used accordingly in development phase
  • Can created custom routines
  • Multiple jobs can be selected for compilation and provide the report after the compilation is finished

Director:

  • Validate, schedule, run, and monitor jobs run by the DataStage Server
  • Job status displays the current status like running, compiled, finished, aborted and not compiled
  • Job log displayes the log file for the selected job
  • Reset the job if the status is aborted or stopped before running it again.
  • Provides the execution times of the jobs
  • Ability to clean up the resources (if administrator has enabled this option)

Along with these jobs, DataStage provides containers (local containers and shared containers) and sequence jobs allow to specify a sequence of server or parallel jobs to run.

Next Post

Shopping Smart For Hip-Hop Clothing and Urban Clothing

In recent years, because of bad news about young people getting too wrapped up in image and forgetting what really matters, some heavyweights in the hip-hop world have given up wearing expensive jewelry on-stage. The problem is that some youngsters have been willing to commit crimes so they have the […]