Debugging Edges
Last updated
Last updated
Debugging edges means recording of data flowing through the edge for further inspection.
Debugging is useful if you obtain incorrect or unexpected results after running a graph, as it helps you locate and identify the errors in the graph.
By default, debugging is enabled on all edges.
There are several debugging options for each edge. Right-click on the edge and select the Debug option from the context menu:
First 1000 records going through the debugged edge are saved, the rest is ignored. When selected, the debug file size is limited to 1 MB.
By default, the first 1000 records going through debugged edges are saved to debug files. There is also the second limit 1 MB for default viewing of records on Edges in Data Inspector. The exception are graphs running on the Server via automated process (e.g. scheduling, listening) where debugging on edges is disabled by default.
You can restrict the data records that will be saved to debug files. You can set it up in the Properties tab of any debug edge, or you can right-click the debugged edge and set it up in Debug properties accessible from the context menu.
To avoid saving all data records, you can set any of the following four edge attributes either in the Properties tab or in the Debug properties dialog: Debug filter expression, Debug last records, Debug max. records and Debug sample data.
If you specify a filter expression for an edge, data records that satisfy the specified filter expression will be saved to the debug file. Those that do not satisfy the expression will be ignored.
If a filter expression is defined, either all records that satisfy the expression (Debug sample data set to false) or only a sample of them (Debug sample data set to true) will be saved.
The filter expression is defined with the help of Filter Editor.
Filter Editor
The Filter Editor consists of three panes.
The left pane displays a list of record fields, their names and data types. You can select any record field by double-clicking or dragging and dropping. Then a field name appears in the bottom area. The field name consists of a dollar sign, followed by a type of the port (in or out), port number and the name itself. (For example, $in.0.street.)
The right pane displays a list of available CTL functions. Below this pane, there are both comparison signs and logical connections. You can select any of the names, functions, signs and connections by double-clicking. After that, they appear in the bottom area.
You can work with functions, operators and fields in the bottom area and complete the creation of the filter expression. The filter expression is validated on the fly.
Example 5. Debug filter expression
//#CTL2
isInteger($in.0.field1)
If you set the Debug last records property to false, data records from the beginning will be saved to the debug file. By default, the records from the end are saved to debug files. The default value of Debug last records is true.
Remember that if you set the Debug last records attribute to false, data records will be selected from the beginning with a greater frequency than from the end. Alternatively, if you set the Debug last records attribute to true or leave it unchanged, they will be selected more frequently from the end than from the beginning.
You can also set up a limit on the number of data records that will be saved into a debug file. These data records will be taken either from the beginning (Debug last records is set to false) or from the end (Debug last records has the default value or it is set to true explicitly).
If you set the Debug sample data attribute to true, the Debug max. records attribute value is only the threshold that limits how many data records could be saved to a debug file. Data records will be saved at random, some of them will be omitted, others will be saved to the debug file. In this case, the number of data records saved to a debug file will be less than or equal to this limit.
If you do not set any value of Debug sample data, or if you set it to false explicitly, the number of records saved to the debug file will be equal to the Debug max. records attribute value (if more records than Debug max. records go through the debug edge).
The same properties can also be defined using the context menu by selecting the Debug properties option. After that, the following dialog will open:
Let us show how to view the records that have passed through an edge, have met the filter expression and have been saved.
You can view data on edges with debug enabled.
Click an edge and Data Inspector tab in the bottom will display the debugged data. If you click another edge, you will see the data of another edge.
If you intend to see the data of more edges at once, use a new Data Inspector tab: open the context menu with right-click and select the Inspect data.
Data Inspector tab displays debug data of an edge. It lets you see data on readers and writers as well.
If Data Inspector opens, you can see data on edges without using context menu: just click an edge, and Data Inspector displays data of the edge. The displayed Data Inspector view is refreshed after a graph run.
You can sort the records according to any column: click the column’s header. Records can be sorted in ascending or descending order.
You can view data on more edges at the same time. Records from each debug edge are displayed in a separate tab. Feel free to displace the tabs as you need. Each Data Inspector’s title bar contains a reference to viewed edge in the format Edge [Component name → Component name] Run ID: number.
View Modes
Data Inspector is capable of displaying data in four view modes: Table View, Single Record View, Text View and Hexadecimal View. You can switch between the view modes by clicking on the View Mode icon in Data Inspector’s toolbar. A list of available view modes is based on the inspected element: data records for edges can be viewed in Table View and Single Record View; supported view modes for a component are based on the component’s type.
Table View
A default view mode. Displays data in a table, one record per line.
Single Record View
Displays details for a single record only, one field per line. Can also be accessed from a Table View, by choosing Show as Single Record item in a record’s context menu or by pressing the Enter key. When in Single Record View, Show in Table View item in context menu or pressing the Backspace key returns back to Table View.
Text View
Displays the content of an input or output file as a plain text.
Hexadecimal View
Displays the content of an input or output file in a hexadecimal mode.
Actions on Data Inspector
Following actions are available from Data Inspector toolbar.
Refresh
Data is refreshed automatically when a graph run finishes and after performing actions that require refresh (applying a filter, switching the truncate option). Manual refresh might be useful, for example, when source file of inspected component has changed.
Keyboard shortcut: F5
Pin Data
If you open Data Inspector from the context menu and at least one unpinned Data Inspector already exists, the Pin Data option will be applied.
Lock Data
Quote Strings in Lists
Displays items of the lists quoted. It makes it easy to see which comma is a delimiter and which one is a part of the list item.
Show View When Content Changes
This option makes the Data Inspector tab active when its content has changed.
Truncate Long Values
Show Unprintable Characters
Unprintable characters (line breaks, space characters, etc.) are displayed as a proxy character.
View Mode
Switches between view modes, see View Modes.
Open New Data Inspector View
This action opens a new view with the same content.
Tip: open a new data view and lock the old one. You will be able to see differences between two graph runs.
Additional actions are available from Data Inspector’s menu or from context inside the view. The Data Inspector’s menu can be accessed by clicking on the arrow in right side of Data Inspector’s toolbar.
Copy
In Table View and Single Record View modes, you can copy either a whole table row (or more rows) or a value of a single cell. In Text View and Hexadecimal View, it is possible to copy a selected text.
Whole records can be copied by using Ctrl+C keyboard shortcut or using Copy item in the context menu. Fields in copied records are delimited by tabulators. If pasted into a spreadsheet (e.g. Microsoft Excel), they fill spreadsheet cells.
A single cell value can be copied by choosing Copy Cell from a context menu of the particular cell.
Hide/Show Columns
Actions for hiding columns are available only in Table View mode. They allow to select which columns will be displayed and which not. They are available from Data Inspector’s menu under Hide/Show Columns or from a context menu of a column header.
Hide Column - available from header’s context menu, hides the particular column
Hide Other Columns - available from header’s context menu, hides the other columns
Show All and Hide All - available from Data Inspector’s menu, shows/hides all columns
Show Selected…​ - opens a dialog that allows to configure visible columns
Columns - allows to hide or show a column by checking or unchecking it in the menu
Go to Line…​/Go to Record…​
Opens a dialog that requests a line number. After confirming the dialog, the requested record or line will be highlighted. If necessary, the dialog scrolls to display the record.
Keyboard shortcut: Ctrl+L
Filtering Records
In Table View and Single Record View modes, it is possible to apply a filter to the displayed records.
In the right bottom corner of Data Inspector, there is a filter widget that shows the state of the filter. It also serves to modify the filter, clear the filter or disable it temporarily.
New Filter Expression
When you have created a filter, the text Filter is not set changes to Filter is active.
Disabling the Filter
You can disable the filter by clicking the Filter is active text. The filter is not applied and the text changes to Filter is not active.
An alternative way to disable the filter is to click the arrow next to the filter widget and choose Active from the menu. Tick before the menu item disappears.
Enabling the Filter
You can enable the filter by clicking the text again.
Another way to enable the filter is to click the arrow next to the filter widget and choose Activate from the context menu.
Search Data
The Search Data allows you to look up a text in the records.
You can open the Search Data panel using the Ctrl+F shortcut, or by choosing **Search Data…**​ from the Data Inspector’s menu.
The search panel contains a text area where you can type an expression.
Next to the text area, there are the Mark all found matches and Case sensitivity buttons. If Mark all found matches button is checked, all found matches for the search expression are highlighted. If you enable the Case sensitivity option, the search will be case sensitive.
The Options…​ button gives you access to Search Options.
Search Options
If Entire cells option is checked, the searched text must match the cell entirely.
Direction lets you choose a search order - you can search in direction of rows, or columns.
You can also select which column will be searched in: all, only visible or one column from the list.
The Bulb icon on the left side of the text area indicates that Content Assist is available by pressing Ctrl+Space.
The OK button searches for the first occurrence and closes the Find dialog.
The Cancel button closes the dialog.
Export Data to CSV
You can export the debug data to CSV without a clipboard.
To export records to CSV, click the arrow in the upper right corner and choose Export to CSV. You can use Ctrl+E as well. The CSV files can be subsequently loaded into a spreadsheet editor or processed by another graph.
When the directory specified by File URL does not exist, it is created before export of the file itself.
Export can run in the background and the user can do another work meanwhile. Progress is reported in Progress view in the bottom right corner of designer.
Data Inspector Preferences
Data Inspector lets you change its default configuration. You can set up a preferred view mode, show/hide unprintable characters, truncation of strings and byte arrays and number of loaded lines/records.
Truncate long values
This option truncates data shown in the Data Inspector and in the detail shown after you double click a record.
When checked, the number of characters shown is limited to 254 (including the […​] string indicating truncation) both in the Data Inspector and the detail. When unchecked, the limit for Data Inspector is 300 characters, while the detail shows the whole value of the field.
Changing the setting requires restart of the Data Shaper Designer.
If you want to disable debugging on multiple edges simultaneously, select the edges by left-clicking while holding down the Ctrl key first.
If you want to turn off debugging, you can click the Graph editor in any place outside the components and the edges, switch to the Properties tab and set the Debug mode attribute to false. This way you can turn off all debugging at a time.
Bug icons do not disappear, but edge debugging is not performed. If you disable debugging this way, it can be enabled back keeping the original configuration.
Alternatively, you can select Run > Debug Configurations…​ on the menu bar and check the Disable edge debugging option.
All records going through a debugged edge are saved. When selected, the option is indicated by the icon on the selected edge.
Debugging on the selected edge is disabled. When selected, the option is indicated by the icon on the selected edge.
Allows you to set several edge attributes, see . When selected, the option is indicated by the icon on the selected edge.
After you run the graph, one debug file is created for each debugged edge. You can analyze the data records from the debug files (.dbg extension), see .
Debugging on edges can be disabled, see .
The Filter Editor is described in the documentation on .
Data Inspector loads only first 100 records by default. To load more records, scroll down the view and new records will load automatically. Alternatively, you can click on the Load More button at the bottom of the view. There is also the Load All button which loads all available records. Use this button only when the number of available records is small.
You can drag the tab into a new window. Use the Load More button when observing records while your graph is still running - they are loaded on your click as they are produced by graph’s transformations.
The Refresh button lets you perform manual refresh of debug data.
Pin Data binds Data Inspector to a specific edge or a component. If you pin data , and another edge or component is selected, Data Inspector’s content will not change. But it will still be automatically refreshed after a graph run is finished or when performing data inspection on the same edge or component.
Lock Data locks the content of Data Inspector so that it is not refreshed automatically (e.g. after a graph run). Locked state also disables manual refresh, so data cannot be refreshed by accident.
This action is available from the drop-down menu list in the Data Inspector toolbar.
When Truncate Long Values is active, values of loaded data fields are truncated to the first 253 characters or array elements to improve performance when loading huge data records. Disable this option to show entire field values.
In a new Data Inspector, the widget in the right bottom corner shows the Filter is not set text. Click the text to open the Filter Editor and define a filter expression. For information on how to create filter expression, see .
If you check the Regular expression checkbox, the expression you have typed into the text area will be used as a .
To disable debugging, right-click on the edge and select Debug > No records from the context menu. Disabled debugging is indicated by the icon
