Primeur Online Docs
Data Shaper
Data Shaper
  • 🚀GETTING STARTED
    • What is Primeur Data Shaper
      • What is the Data Shaper Designer
      • What is the Data Shaper Server
      • What is the Data Shaper Cluster
    • How does the Data Shaper Designer work
      • Designer Views and Graphs
      • Data Shaper Graphs
      • Designer Projects and Sandboxes
      • Data Shaper Designer Reference
    • How do the Data Shaper Server and Cluster work
      • Data Shaper Server and Cluster
      • Data Shaper Server Reference
    • VFS Graph Components
      • DataOneFileDescriptor (DOFD) metadata
      • Passing parameters from Data One Contract to Data Shaper graph
      • Inheriting Data One context attributes in Data Shaper graph
  • DATA SHAPER DESIGNER
    • Configuration
      • Runtime configuration
        • Logging
        • Master Password
        • User Classpath
      • Data Shaper Server Integration
      • Execution monitoring
      • Java configuration
      • Engine configuration
      • Refresh Operation
    • Designer User Interface
      • Graph Editor with Palette of Components
      • Project Explorer Pane
      • Outline Pane
      • Tabs Pane
      • Execution Tab
      • Keyboard Shortcuts
    • Projects
      • Creating Data Shaper projects
      • Converting Data Shaper projects
      • Structure of Data Shaper projects
      • Versioning of server project content
      • Working with Data Shaper Server Projects
      • Project configuration
    • Graphs
      • Creating an empty graph
      • Creating a simple graph
        • Placing Components
        • Placing Components from Palette
        • Connecting Components with Edges
    • Execution
      • Successful Graph Execution
      • Run configuration
      • Connecting to a running job
      • Graph states
    • Common dialogs
      • URL file dialog
      • Edit Value dialog
      • Open Type dialog
    • Import
      • Import Data Shaper projects
      • Import from Data Shaper server sandbox
      • Import graphs
      • Import metadata
    • Export
      • Export graphs to HTML
      • Export to Data Shaper Server sandbox
      • Export image
      • Export Project as Library
    • Graph tracking
      • Changing record count font size
    • Search functionality
    • Working with Data Shaper server
      • Data Shaper server project basic principles
      • Connecting via HTTP
      • Connecting via HTTPS
      • Connecting via Proxy Server
    • Graph components
      • Adding components
      • Finding components
      • Edit component dialog box
      • Enable/disable component
      • Passing data through disabled component
      • Common properties of components
      • Specific attribute types
      • Metadata templates
    • Edges
      • Connecting Components with Edges
      • Types of Edges
      • Assigning Metadata to Edges
      • Colors of Edges
      • Debugging Edges
      • Edge Memory Allocation
    • Metadata
      • Records and Fields
        • Record Types
        • Data Types in Metadata
        • Data Formats
        • Locale and Locale Sensitivity
        • Time Zone
        • Autofilling Functions
      • Metadata Types
        • Internal Metadata
        • External (Shared) Metadata
        • SQL Query Metadata
        • Reading Metadata from Special Sources
      • Auto-propagated Metadata
        • Sources of Auto-Propagated Metadata
        • Explicitly Propagated Metadata
        • Priorities of Metadata
        • Propagation of SQL Query Metadata
      • Creating Metadata
        • Extracting Metadata from a Flat File
        • Extracting Metadata from an XLS(X) File
        • Extracting Metadata from a Database
        • Extracting Metadata from a DBase File
        • Extracting Metadata from Salesforce
        • SQL Query Metadata
        • User Defined Metadata
      • Merging Existing Metadata
      • Creating Database Table from Metadata and Database Connection
      • Metadata Editor
        • Opening Metadata Editor
        • Basics of Metadata Editor
        • Record Pane
        • Field Name vs. Label vs. Description
        • Details Pane
      • Changing and Defining Delimiters
      • Editing Metadata in the Source Code
      • Multi-value Fields
        • Lists and Maps Support in Components
        • Joining on multivalue fields (Comparison Rules)
    • Connections
      • Database Connections
        • Internal Database Connections
        • External (Shared) Database Connections
        • Database Connections Properties
        • Encryption of Access Password
        • Browsing Database and Extracting Metadata from Database Tables
        • Windows Authentication on Microsoft SQL Server
        • Snowflake Connection
        • Hive Connection
        • Troubleshooting
      • JMS Connections
      • QuickBase Connections
      • Hadoop Connections
      • Kafka Connections
      • OAuth2 Connections
      • MongoDB Connections
      • Salesforce Connections
    • Lookup Tables
      • Lookup Tables in Cluster Environment
      • Internal Lookup Tables
      • External (Shared) Lookup Tables
      • Types of Lookup Tables
    • Sequences
      • Persistent Sequences
      • Non Persistent Sequences
      • Internal Sequences
      • External (Shared) Sequences
      • Editing a Sequence
      • Sequences in Cluster Environment
    • Parameters
      • Internal Parameters
      • External (Shared) Parameters
      • Secure Graph Parameters
      • Graph Parameter Editor
      • Secure Graph Parameters
      • Parameters with CTL2 Expressions (Dynamic Parameters)
      • Environment Variables
      • Canonicalizing File Paths
      • Using Parameters
    • Internal/External Graph Elements
    • Dictionary
      • Creating a Dictionary
      • Using a Dictionary in Graphs
    • Execution Properties
    • Notes in Graphs
      • Placing Notes into Graph
      • Resizing Notes
      • Editing Notes
      • Formatted Text
      • Links from Notes
      • Folding Notes
      • Notes Properties
    • Transformations
      • Defining Transformations
      • Transform Editor
      • Common Java Interfaces
    • Data Partitioning (Parallel Running)
    • Data Partitioning in Cluster
      • High Availability
      • Scalability
      • Graph Allocation Examples
      • Example of Distributed Execution
      • Remote Edges
    • Readers
      • Common Properties of Readers
      • ComplexDataReader
      • DatabaseReader
      • DataGenerator
      • DataOneVFSReader
      • EDIFACTReader
      • FlatFileReader
      • JSONExtract
      • JSONReader
      • LDAPReader
      • MultiLevelReader
      • SpreadsheetDataReader
      • UniversalDataReader
      • X12Reader
      • XMLExtract
      • XMLReader
      • XMLXPathReader
    • Writers
      • Common Properties of Writers
      • DatabaseWriter
      • DataOneVFSWriter
      • EDIFACTWriter
      • FlatFileWriter
      • JSONWriter
      • LDAPWriter
      • SpreadsheetDataWriter
      • HIDDEN StructuredDataWriter
      • HIDDEN TableauWriter
      • Trash
      • UniversalDataWriter
      • X12Writer
      • XMLWriter
    • Transformers
      • Common Properties of Transformers
      • Aggregate
      • Concatenate
      • DataIntersection
      • DataSampler
      • Dedup
      • Denormalizer
      • ExtSort
      • FastSort
      • Filter
      • Map
      • Merge
      • MetaPivot
      • Normalizer
      • Partition
      • Pivot
      • Rollup
      • SimpleCopy
      • SimpleGather
      • SortWithinGroups
      • XSLTransformer
    • Joiners
      • Common Properties of Joiners
      • Combine
      • CrossJoin
      • DBJoin
      • ExtHashJoin
      • ExtMergeJoin
      • LookupJoin
      • RelationalJoin
    • Others
      • Common Properties of Others
      • CheckForeignKey
      • DBExecute
      • HTTPConnector
      • LookupTableReaderWriter
      • WebServiceClient
    • CTL2 - Data Shaper Transformation Language
    • Language Reference
      • Program Structure
      • Comments
      • Import
      • Data Types in CTL2
      • Literals
      • Variables
      • Dictionary in CTL2
      • Operators
      • Simple Statement and Block of Statements
      • Control Statements
      • Error Handling
      • Functions
      • Conditional Fail Expression
      • Accessing Data Records and Fields
      • Mapping
      • Parameters
      • Regular Expressions
    • CTL Debugging
      • Debug Perspective
      • Importing and Exporting Breakpoints
      • Inspecting Variables and Expressions
      • Examples
    • Functions Reference
      • Conversion Functions
      • Date Functions
      • Mathematical Functions
      • String Functions
      • Mapping Functions
      • Container Functions
      • Record Functions (Dynamic Field Access)
      • Miscellaneous Functions
      • Lookup Table Functions
      • Sequence Functions
      • Data Service HTTP Library Functions
      • Custom CTL Functions
      • CTL2 Appendix - List of National-specific Characters
      • HIDDEN Subgraph Functions
    • Tutorial
      • Creating a Transformation Graph
      • Filtering the records
      • Sorting the Records
      • Processing Speed-up with Parallelization
      • Debugging the Java Transformation
  • DATA SHAPER SERVER
    • Introduction
    • Administration
      • Monitoring
    • Using Graphs
      • Job Queue
      • Execution History
      • Job Inspector
    • Cluster
      • Sandboxes in Cluster
      • Troubleshooting
  • Install Data Shaper
    • Install Data Shaper
      • Introduction to Data Shaper installation process
      • Planning Data Shaper installation
      • Data Shaper System Requirements
      • Data Shaper Domain Master Configuration reference
      • Performing Data Shaper initial installation and master configuration
        • Creating database objects for PostgreSQL
        • Creating database objects for Oracle
        • Executing Data Shaper installer
        • Configuring additional firewall rules for Data Shaper
Powered by GitBook
On this page
  • Initial Check-Out of Project from Repository
  • GIT
  • SVN
  • Adding Server Project to Version Control
  • GIT
  • SVN
  • Connecting Server Project to Existing Repository
  • GIT
  • SVN
  • Getting Changes from Repository
  • GIT
  • SVN
  • Committing into Repository
  • GIT
  • SVN
  1. DATA SHAPER DESIGNER
  2. Projects

Versioning of server project content

PreviousStructure of Data Shaper projectsNextWorking with Data Shaper Server Projects

Last updated 1 month ago

This section describes the most basic use cases of versioning server projects with two popular version control systems. It describes the way to use the particular version system and Data Shaper Designer together. It does not serve to replace the documentation of any particular version control system.

We encourage you to read the documentation for version control system tool of your choice as well.

  • Getting Changes from Repository

Server projects store files remotely and locally. This allows you to use version control system of your choice. You can use an Eclipse plugin as well as an external tool. The basic workflow of usage of Git and SVN versioning systems in Data Shaper Designer is described below.

Version Control System Files and Synchronization

Version control system metadata files (e.g .svn directory) must not be synchronized between Designer and Server. These files avoided from synchronization are listed in Preferences under section. If you use Bazaar, Git, Mercurial, or SVN, you do not have to care about this, as it is already configured. If you use any other version control system, you should add its files and directories to this list of ignored files.

Versioning Tools

The following text describes the basic work flows with Git and SVN. For both, an approach with an Eclipse plugin (EGit, Subclipse), external GUI tool (TortoiseGit, TortoiseSVN) and command line utilities is described.

EGit is an Eclipse plugin for versioning with Git. It is included in Data Shaper Designer.

Subclipse is an Eclipse plugin for versioning with SVN. Since it is not part of Data Shaper Designer, you need to install it, e.g from Eclipse Marketplace.

TortoiseGit and TortoiseSVN are external GUI tools.

Initial Check-Out of Project from Repository

There is an existing project in the repository. Your task is to create a new server project with content from the repository.

GIT

To version files with Git, you can use Eclipse plugin (EGit), external GUI or command line tool.

Eclipse Plugin - EGit

  1. Clone the remote Git repository: Switch to Git perspective. In the Git Repositories tab, click Clone a Git Repository and add the clone to this view. Enter the remote repository location and password. Choose branches to be tracked. Enter the path for the local repository. Switch back to Data Shaper perspective.

  2. Import the project. From the File menu, choose Import. Choose Git > Projects from Git and click Next. Choose Existing local repository. Choose the repository. Select the Import existing projects option. Select projects that should be imported.

  3. Convert the project to Server Project. Right click the project in Project Explorer and choose Convert to server project. Enter Data Shaper Server URL, User name and Password. Select Create new sandbox. Enter the sandbox Name. Select type of merge. As you created a new sandbox, you can use the Use local content only (sandbox will be cleaned) option.

You have a new server project. The project has content of a master branch of local repository.

EGit allows you to have more projects within the save repository.

External GUI Tool - TortoiseGit

  1. Clone the remote repository. In File Explorer, right click and choose Git Clone. In Git clone - TortoiseGit dialog, enter the path to remote repository and the path to local repository. Click OK. Enter password. Close the cloning log window.

  2. Import the project.

  3. Convert project to Server Project.

External Command Line Tool

  1. Import the project.

  2. Convert project to Server Project.

SVN

Eclipse Plugin - Subclipse

  1. Convert the project to Server Project.

External GUI Tools - TortoiseSVN

You can use an external graphical tool to check out an svn project. The following steps describe checking out of an existing project with help of TortoiseSVN.

  1. Check-out the content of projects from SVN: Right click the project directory in File Explorer and choose SVN Checkout…​ from context menu. Delete the last entry from the Checkout directory text field. Click OK. TortoiseSVN complains that the directory is not empty. Choose Yes.

  2. Import the project.

  3. Convert the project to Server Project.

Command Line Tools

  1. Check out the project from svn: Move to the directory with the Project in workspace (on the computer with Designer). Type svn checkout --force /path/to/repository. The --force forces svn to overwrite the content created during the project creation. Otherwise, you would have to resolve conflicts after checkout.

  2. Import the project.

  3. Convert the project to Server Project.

Adding Server Project to Version Control

This section describes a way to create a brand new versioned server project and to commit it to a new remote repository. It describes the way to add an existing unversioned server project to the repository as well.

Generally, create a new server project, then add it under a version control system.

GIT

EGit does not place the .git directory into the project directory. It creates the local Git repository outside the project, creates a project directory within the repository, and links the project directory into the workspace.

Eclipse Plugin - Egit

  1. Create a new local repository. Open a new perspective. Choose Git. In Git Repositories tab, click the Create a new Git Repository and add it to this view icon. In Create a Git Repository dialog, enter the path to the local Git repository. Switch back to Data Shaper Perspective.

  2. Create a new Server Project.

  3. Share the project with Team → Share wizard: Right click the project name in the Project Explorer and choose Team > Share Project…​. In the Share Project wizard, choose Git. Create a new Git repository: click the Create…​ button in the upper right corner and choose new repository location. If you intend to store more projects within one repository, enter Path within repository. You usually do not check the Use or create repository in parent folder of project checkbox. Click Finish. The Git share wizard will move you project to the local Git repository and link the project to your workspace.

  4. Optionally, ignore the files or directories that you would not like to commit into repository: right click the file in Project Explorer and choose Team > Ignore.

  5. Commit the changes into the local repository: Right click the project in **Project Explorer **and choose Team > Commit. Choose files, type the commit message, and click Commit.

  6. Push the changes into to the remote repository: Switch to the Git perspective. Unfold the Project. Right click Remotes and choose Create Remotes…​. Enter the name for remote repository. Usually, it is called origin. In the second step of the wizard, click Change to specify the URI of remote repository. Enter URI and password and click Finish. Click Advanced to specify tracking of remote branches. Choose branches below the Source ref and Destination ref titles. Click Add spec to add the tuple to the list of tracked branches. Tick Save specifications in 'remote' configuration. Click Finish to close Configure Push dialog. Click Save and Push to save the branch tracking and to push the changes from local repository to remote one.

External GUI Tool - TortoiseGit

  1. Create a new local repository. Create a new directory for repository. Right click this directory in the File Explorer and choose **Git Create repository here …​ **from the context menu. Do not make it bare.

  2. Create a new Server Project. In the last step of the wizard, uncheck Use default location and enter the path to the local repository.

  3. Make initial commit. Right click the project directory and choose Git commit → "master" …​ from the context menu. In the commit dialog, right click the files that you do not want to version and choose Add to ignore list > "the file name". In ignore dialog, use defaults. Select all files in repository, type commit message and click commit.

  4. Configure the tracking of remote repository and push the local commit. In File Explorer, right click the project and choose TortoiseGit > Push. In TortoiseGit - Push dialog, click Manage. In Settings - TortoiseGit, enter URL or remote repository and click OK. Click OK to push the changes to remote repository. Type the password and close the log.

External Tool - Command Line

  1. Create a new local Git repository. Type git init /path/to/repository

  2. Create a new Server Project. In the last step of the wizard, uncheck Use default location and enter the path to the local repository.

  3. Optionally, add a list of files or directories that should not be versioned to .gitignore echo file_name >> .gitignore

  4. Commit the changes to local repository. git commit -a -m "Initial commit"

  5. Push the commits to remote repository. git push --set-upstream origin master

SVN

Eclipse Plugin - Subclipse

  1. Share the project with Team → Share wizard. Right click in Project Explorer and choose Team > Share Project…​ from the context menu. In the Share project wizard, choose SVN. Choose Use existing repository location. Enter folder name. Click Next. Type the Commit message. Click Finish. The root directory of your project has been committed.

  2. Optionally, add a list of files or directories that should not be versioned to svn:ignore: In Project Explorer in Data Shaper Perspective, right click the file or directory and select Team > Add to svn:ignore…​.

The project has been created and the root directory has been committed into the repository.

External Tool - TortoiseSVN

TortoiseSVN does not let you choose particular files during the import into repository. Therefore you should create a project directory and import repository first. Then you can set up list of files or directories that should not be committed.

  1. Create a project directory.

  2. Commit the directory to repository: Right click the directory and choose TortoiseSVN > Import. In the dialog, enter the URL of repository and the commit message. Generally, the URL has a format: somePath/MyProjectName.

  3. Check out the repository. In FileExplorer choose the project directory and choose SVN Checkout…​

  4. Avoid committing of files or directories that should not be committed. In File Explorer, right click the file or directory within the project and choose TortoiseSVN > Add to ignore list > "the file name"

  5. Commit the project: In File Explorer, right click the project name and choose SVN Commit…​

External Tool - Command Line

  1. Create a project directory. mkdir MyProject

  2. Import the directory to repository. svn import MyProject url_to_repository -m "Initial import"

  3. Check out the committed directory. It converts the project directory into a working copy. svn co url_to_repository/MyProject MyProject

  4. Create a new server project.

  5. Optionally, add files or directories to svn:ignore. Move to the project and type svn propset svn:ignore file_name.

  6. Add files and commit the changes. svn add * svn commit -m "Initial import"

Connecting Server Project to Existing Repository

There is an existing Server sandbox with some data, graphs, etc. The content of the sandbox is in repository as well.

This section describes a way to create a new Server project corresponding to this sandbox and to attach the project with versioning system.

GIT

The difference is that in the second step of the New Server Project wizard you should choose an existing sandbox.

Eclipse Plugin - EGit

  1. Clone the repository. Switch to Git perspective. In Git Repositories tab, click the Clone a Git Repository and add the clone to this view icon. Enter the remote repository location and password. Choose branches to be tracked. Enter the location at which the local repository will be created. Switch back to Data Shaper perspective.

  2. Import the project from the local Git repository.

  3. Convert the project to Server Project. In the second step of wizard, choose an existing server sandbox. In the last step of the wizard, choose the type of merge, e.g. Merge content - prefer remote files.

External Tool - TortoiseGit

  1. Clone the git repository.

  2. Import the project.

  3. Convert project to Server Project.

External Tool - Command Line

1.Clone the Git repository. git clone /path/to/remote/repo local_repo

  1. Import the project.

  2. Convert the project to Server Project.

SVN

Eclipse Plugin - Subclipse

1.Checkout the project from SVN.

  1. Import the project.

  2. Convert the project to Server Project.

External Tool - TortoiseSVN

  1. Check out the repository to directory with project. Right click the project directory in File Explorer and choose SVN Checkout…​ Specify the checkout directory as current directory and click OK. TortoiseSVN complains that the directory is not empty.

  2. Import the project.

  3. Convert project to Server project.

External Tool - Command Line

  1. Check out the project from SVN. svn checkout --force /path/to/repository

  2. Import the project.

  3. Convert the project to Server Project.

Getting Changes from Repository

You have a server project connected to a repository. This section describes a way to get changes made by your colleagues from the repository.

GIT

In Git terminology, this process is known as git pull. Instead of pull, you can do fetch and merge, or fetch and rebase.

Eclipse Plugin - EGit

In Project Explorer, right click the project and choose Team > Pull

External Tool - TortoiseGit

  1. In File Explorer, right click the project and choose TortoiseGit > Pull.

  2. Choose origin and branch and click OK.

  3. Close the log.

External Tool - Command Line

Switch to the local repository and type git pull.

SVN

In SVN terminology, this process is known as svn update.

Eclipse Plugin - Subclipse

Right click the project name in Project Explorer, and choose Team > Update to HEAD from the context menu.

External Tool - TortoiseSVN

In File Explorer, right click the project directory and choose SVN Update from the context menu.

External tool - Command Line

Move to the project directory and type svn update.

Committing into Repository

GIT

Eclipse Plugin - EGit

  1. Right click the project in Project Explorer, and choose Team > Commit.

  2. Choose files, enter a commit message, and click Commit and Push.

  3. Enter credentials.

  4. View the changes and click OK.

External tool - TortoiseGit

  1. In File Explorer, right click the project and choose Git commit → "master".

  2. Type the commit message, choose files to be committed and click Commit. You can change Commit to Commit & Push.

External Tool - Command Line

  1. Add changes to staging area. Within the repository, type git add path/to/file(s).

  2. Type git commit -m "The commit message" to commit the changes.

  3. Type git push to push the changes to the remote repository.

SVN

Eclipse Plugin - Subclipse

  1. In the Project Explorer view, right click the project name and choose Team > Commit…​

  2. Type the commit message, choose files to be committed, and click OK.

External Tool - TortoiseSVN

  1. In File Explorer, right click the project directory and choose SVN Commit…​

  2. Choose the files to be committed, type a commit message and click OK.

External Tool - Command Line

  1. Add files that you added with svn add path/to/file.

  2. Type svn commit -m "The commit message" to commit the changes.

Clone the git repository Type the command git clone path_to_remote path_to_local e. g. git clone ssh://:30022/home/git/project1 project1

Import project from SVN. Choose File > Import. Choose SVN > Checkout Projects from SVN. In Select/Create Location step of the wizard, choose Create new repository location. Specify location of SVN repository. The URL for remote projects can be, e.g. . The URL for local projects can be, e.g. file:///Users/clover/repositories/svn/repo1. Select the folder (project) to be imported. In Check Out As step, choose **Check out as a project in the workspace **and change ProjectName if necessary. Optionally, change the location to which the project will be checked or add the project to working sets.

Add remote repository. git remote add origin ssh://:30022/home/git/repos/Project.git

Create a new (Synchronized) Server Project. See .

You might need to commit the files and directories as well. See .

Create a new (Synchronized) Server Project. See .

Cloning the remote repository and binding the content of an existing sandbox with the project from this repository is almost same as .

To commit changes into repository, you should have a versioned server project. You can either create a new one (see ) or check out an existing one (see ).

git@127.0.0.1
https://svn.example.org/svn
git@127.0.0.1
Data Shaper Server Project
Data Shaper Server Project
Committing into Repository
Initial Check-Out of Project from Repository
Adding Server Project to Version Control
Initial Check-Out of Project from Repository
Initial Check-Out of Project from Repository
Adding Server Project to Version Control
Connecting Server Project to Existing Repository
Committing into Repository
Ignored Files