Top Menu

Jump to content
  • Momentum Documentation
    • View all projects
Home
    • Projects
    • News
    • Getting started
    • Introduction video

      Welcome to OpenProject

      Get an overview

      Get a quick overview of project management and team collaboration with OpenProject.
      You can restart this video from the help menu

    • Help and support
    • Upgrade to Enterprise Edition
    • User guides
    • Videos
    • Shortcuts
    • Community forum
    • Professional support

    • Additional resources
    • Data privacy and security policy
    • OpenProject website
    • Security alerts / Newsletter
    • OpenProject blog
    • Release notes
    • Report a bug
    • Development roadmap
    • Add and edit translations
    • API documentation
  • Sign in
      Forgot your password?
      Create a new account

Side Menu

  • Overview
  • News
  • Forums
  • Inset BI
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Impulse DW
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Momentum User Guide
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Wiki
    • Table of contents
      • Expanded. Click to collapseCollapsed. Click to showImpulse DW
        • Expanded. Click to collapseCollapsed. Click to show1. Registration and Account Management
          • Hierarchy leaf1.1 Signup
          • Hierarchy leaf1.2 Password Change
          • Hierarchy leaf1.3 Profile Management
          • Hierarchy leaf1.4 Forgot Password
        • Expanded. Click to collapseCollapsed. Click to show2. Warehouse Management
          • Hierarchy leaf2.1 Create a Warehouse
          • Hierarchy leaf2.2 Edit Warehouse
          • Hierarchy leaf2.3 Datasources In Warehouse
          • Expanded. Click to collapseCollapsed. Click to show2.4 Ingesting Data Into Tables or Datasources
            • Hierarchy leaf2.4.1 Ingesting From Momentum Data Pipeline
            • Hierarchy leaf2.4.2 Uploading File Using Impulse UI
            • Hierarchy leaf2.4.3 Ingesting From External File/Storage System
          • Expanded. Click to collapseCollapsed. Click to show2.5 Add Data to Existing Tables
            • Hierarchy leaf2.5.1 Update Existing Index
          • Hierarchy leaf2.6 Delete Table Records (Rows)
          • Hierarchy leaf2.7 Delete Tables or Datasources
          • Hierarchy leaf2.8 Monitoring Indexing Tasks
          • Hierarchy leaf2.9 View Datasource Stats
        • Expanded. Click to collapseCollapsed. Click to show3. BI Integration
          • Hierarchy leaf3.1 MVInsight Integration
          • Hierarchy leaf3.2 Tableau Integration with Impulse
        • Expanded. Click to collapseCollapsed. Click to show3. Security, Roles and Privilege Management
          • Hierarchy leaf3.1 Enable SSL
          • Hierarchy leaf3.2 Securing Backend SQL Engine
          • Hierarchy leaf3.3 Sharing and Access Control
          • Hierarchy leaf3.4 Add User
          • Hierarchy leaf3.5 Edit User
          • Hierarchy leaf3.6 Create Role
          • Hierarchy leaf3.7 Delete Role
          • Hierarchy leaf3.8 Assign Role to User
          • Hierarchy leaf3.9 Edit User Privilege
        • Expanded. Click to collapseCollapsed. Click to show4. System Administration
          • Hierarchy leaf4.1 System Configuration
          • Hierarchy leaf4.2 Managing System Services
        • Expanded. Click to collapseCollapsed. Click to show5. Impulse DW Restful API
          • Hierarchy leaf5.1 API Token
          • Hierarchy leaf5.2 API Reference
        • Expanded. Click to collapseCollapsed. Click to show9. Release Notes
          • Hierarchy leaf9.1 Open Source Software Components and Libraries
      • Expanded. Click to collapseCollapsed. Click to showInset BI
        • Hierarchy leaf1. Getting Started with Inset BI
        • Hierarchy leaf2. Connecting to Databases
        • Hierarchy leaf3. Visualizing Data
        • Hierarchy leaf4. SQL Lab
        • Hierarchy leaf5. User Management and Access Control
        • Hierarchy leaf6. Working with Semantic Layer
        • Hierarchy leaf7. Alerts and Reports
        • Hierarchy leaf8. Release Notes
      • Hierarchy leafInset BI
      • Expanded. Click to collapseCollapsed. Click to showMLOPs
        • Hierarchy leaf1. Getting Started with MLOps
        • Hierarchy leaf2. Deploying ML Models
        • Hierarchy leaf3. Predicting
        • Hierarchy leaf4. Monitoring
        • Hierarchy leaf5. Access Control
        • Hierarchy leafRelease Notes
      • Expanded. Click to collapseCollapsed. Click to showMomentum User Guide
        • Hierarchy leaf1. Getting Started With Momentum
        • Expanded. Click to collapseCollapsed. Click to show2. ETL
          • Hierarchy leaf1.1 Ingester
          • Hierarchy leaf1.2 Transformer
          • Hierarchy leaf1.3 Custom Processor
          • Hierarchy leaf1.4 Emitter
          • Hierarchy leaf1.5 Data Pipeline
        • Expanded. Click to collapseCollapsed. Click to show3. Machine Learning
          • Hierarchy leaf3.1 Model Training
          • Hierarchy leaf3.2 ML Prediction
          • Hierarchy leaf3.3 Computer Vision
          • Hierarchy leaf3.4 NLP
        • Hierarchy leaf4. Streaming
        • Hierarchy leaf5. Process Automation
        • Hierarchy leaf6. Visualization
  • Documents
You are here:
  • Impulse DW
  • 2. Warehouse Management
  • 2.4 Ingesting Data Into Tables or Datasources
  • 2.4.2 Uploading File Using Impulse UI

Content

2.4.2 Uploading File Using Impulse UI

  • More
    • Table of contents

Impulse provides a convenient way to create a table and upload data to it. Data uploaded to impulse is partitioned and indexed for efficient query. This section describes how to upload data into a table using Impulse's file upload mechanism.

Step 1: Upload Data

  1. From the main navigation menu, click "Load Data"
  2. Drag and drop as many files as you want to upload to a table. You may browse and upload files as well. See Figure 2.4.2a below.
  3. Fill out the form (See Figure 2.4.2b below):
    1. Warehouse: from the drop down, select the data warehouse in which you wish to create the table.
    2. Datasource: Give a meaningful name to your datasource. The datasource is analogous to a table in RDBMS paradigm. If the table name within the selected warehouse exists, the  data will  be uploaded in the existing table, else a new table will be created.
    3. Input Source: Since we are uploading, leave the default selection as "Browse & Upload". For other types of input source, see the appropriate sections of this document.
    4. File Format: Select the appropriate file format of the data file you are uploading. The supported file formats are:
      1. Parquet
      2. CSV or comma separated values
      3. TSV or tab separated values
      4. PSV or pipe separated values
      5. JSON (line delimited) meaning each line in the file represents a single row
    5. Input Header: This field is optional. If your input file is delimited (csv, psv, tsv) and does not contain the field header as the first line in each uploaded file, provide a comma separated list of header. Leave this field empty if your input file contains the header, otherwise, the ingestion engine will try to ingest the first line as data and not as header.
    6. Click Next button to configure the indexing and partitioning of data for efficient query execution.

Step 2: Column Mapping and Partition Parameters

After clicking the "Next" button in step 1, the next page will shows the parameters for the step 2 (see Figure 2.4.2c for example). These parameters control how the data indexing and partitions will be created. The description of each field within this step is as follows:

**For the best result, use a date or time based column as the primary partition column. If none of the column can be parsed as a date/time, do not use any partition.**

  1. Datasource: the table name (as set in step 1 above)
  2. Secondary Partition Strategy: This defines the column or columns that will be used to create the secondary partition. Impulse supports two types of secondary partition strategies:
    1. Dynamic: This is the best partition strategy and does the most efficient partitioning based on the data. In most cases, you will leave this as the default secondary partition strategy.
    2. Single Column: If your data will have only one column in the group by or where clause, this single column based strategy will likely to work the best. However, this is highly discouraged to use a single column based partitioning.
  3. Primary Partition Granularity: If you have a date/time based primary column, this parameter specifies how your data will be split into partitions. For example, if you select a "day" for the granularity level, the entire data will be grouped by day and split into partitions.
  4. Missing Datetime Placeholder: If you select a date/time based column as the primary partition column and if any of the rows contains invalid/missing/null values for the primary partition column, it will fill the missing value with this placeholder datetime.
  5. Max Parallel Upload Tasks: This parameter defines how many threads the system will create to upload the data in parallel. For a single node deployment, this should be set at maximum of 60% of number of available CPU cores in your server. For example, if you have 32-core CPU, set the max parallel tasks as 20 or less. For a distributed cluster nodes, this value should be  60% of the sum of cores of all worker nodes.
  6. Upload Mode: Specify whether you want to append rows to and existing table or overwrite existing partition.

Field Mapping: System will try to guess the datatypes of each column. In case of incorrect interpretation, you should edit the datatypes of every column that were incorrectly interpreted. Only the "STRING" "LONG" and "DOUBLE" datatypes are supported. Dates are represented as a STRING datatype.

From the field mapping section, select the primary partition column, preferably a datetime column.

you must specify the datetime format of the primary partition column. ISO date format and joda-time datetime (  https://www.joda.org/joda-time/key_format.html ) format are supported.

If your secondary partition strategy is "Single Column" based, you must select the secondary partition column.

At the bottom of the page, the system displays a few lines of actual data to help you to see the datatype, format and sample values of the actual dataset.

To start indexing, click the "Load and Index" button.

This will open the "Tasks" page that shows a list of all active or completed indexing tasks. See Figure 2.4.2d as an example.

Figure 2.4.2a: Browse or drag-and-drop to upload files to a datasource

Figure 2.4.2b: Screen showing file upload options, data warehouse name, and datasource name

Figure 2.4.2c: Screen showing data ingestion, field mapping, and partitioning parameters

Figure 2.4.2d: Screen showing task status after indexing is triggered

Loading...