Top Menu

Jump to content
  • Momentum Documentation
    • View all projects
Home
    • Projects
    • News
    • Getting started
    • Introduction video

      Welcome to OpenProject

      Get an overview

      Get a quick overview of project management and team collaboration with OpenProject.
      You can restart this video from the help menu
    • Help and support
    • Upgrade to Enterprise Edition
    • User guides
    • Videos
    • Shortcuts
    • Community forum
    • Professional support
    • Additional resources
    • Data privacy and security policy
    • OpenProject website
    • Security alerts / Newsletter
    • OpenProject blog
    • Release notes
    • Report a bug
    • Development roadmap
    • Add and edit translations
    • API documentation
  • Sign in
      Forgot your password?
      Create a new account

Side Menu

  • Overview
  • News
  • Forums
  • Inset BI
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Impulse DW
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Momentum User Guide
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Wiki
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Documents
You are here:
  • Impulse DW
  • 2. Warehouse Management
  • 2.4 Ingesting Data Into Tables or Datasources
  • 2.4.3 Ingesting From External File/Storage System

Content

2.4.3 Ingesting From External File/Storage System

  • More
    • Table of contents

Impulse supports ingesting data from the following external file systems:

  1. Amazon S3: Ingest a file of files stored in S3 bucket. This is the default storage system if Impulse is running on Amazon EC2 or you purchased the impulse license from the AWS Marketplace.
  2. HDFS: Ingest file or files stored in Hadoop Distributed File System (HDFS)
  3. Momentum: Ingest data from Momentum storage. Momentum provides a highly scalable ETL, including data ingestion from a wide variety of sources, transformation, cleaning, blending, and merging with multiple sources. It also allows ingesting data in automated fashion and creating indexes in Impulse.
  4. Google Cloud Storage: This is the default storage system if Impulse is running on Google Cloud.

To ingest data from the external system, follow these steps:

  1. From the main navigation menu, click "Load Data" (See Figure 2.4.3a below as an example)
  2. Fill out the form:
    1. Warehouse: Select the warehouse from the drop down options
    2. Datasource: Enter the table or datasource name.
    3. Input Source: Select the external system to ingest data from.
    4. Input Path: Provide the fully qualified path to the data directory or a single file. For example:
      1. Momentum: fully qualified component name, e.g. accure.tr.sampledata
      2. S3: absolute path of the file or directory, e.g. s3://mybucket/mydir
      3. GCS: absolute path of the file or directory, e.g. gs://mybucket/mydir
      4. HDFS: absolute path of the file or directory, e.g. hdfs://ip-address:port/directory/path
    5. File Format: Select the input file format
    6. Input Header: Enter a comma separated list of header columns if the input format is CSV, TSV or PSV and the input files do not contain the header in the first line.
    7. Click Next and follow the Step 2 as described in the previous section Uploading File Using Impulse UI

Figure 2.4.3a: Screen showing the form fields for ingesting data from external systems

Loading...