Simple Searches

Text Search

In the edit box, enter the text, e.g. a filename. The names of the Saved Queries described here will appear at the top of the search hint list. Then, click the Search button on the right to run your query.
You can activate the Autorefresh button to have the Flow Board updated automatically.

Search by Date

Entering a date, the system will propose you to search since that day to the current day or up to the specified day:

• gg/mm/yyyy[since]: this query will search for flows from the selected date until today. For example, entering 18/12/2023 [since] will retrieve all results from December 18th up to the current date.
• gg/mm/yyyy[until]: this query will search for flows up to the selected date. For example, entering 20/12/2023 [until] will retrieve all results up to December 20th.

These options can be used together to search for flows in the interval specified, e.g. from December 18th 2023 to December 20th 2023 as shown in the figure below:

Search by Status

• successfully[status]: this query will search for all flows having Integrity Status OK.
• error[status]: this query will search for all flows having Integrity Status ERROR.
• running[status]: this query will search for all flows having Integrity Status RUNNING.

Search by Business Status

The Business Status depends on the status of cut-offs applied to the flow events:

• onschedule[business]: this query will search for flows in status completed, successful and on schedule
• pending[business]: this query will search for flows in status to be completed
• warning[business]: this query will search for flows nearly to the deadline
• error[business]: this query will search for flows uncompleted or completed after the scheduled time

For more details refer to sections Dataflow Inquiry and Setting Cut-offs.

The autocomplete function will provide suggestions while you type into the Search field. As shown in the following figure, the successfully[status] value is automatically proposed.

Search by Flow Source

Click the Flow Source link, to drill down results.
The Flow Source will be added automatically to the Search field.

Search by Attribute Value

Clicking an Attribute Value, it will be added automatically to the Search field. Attributes are configured by the Administrator, as described in Dataflow Attributes.
For example, in the figure below the PDF value has been added to the Search bar.

As an alternative, you can start typing an attribute in the Search field and the autocomplete function will propose the value starting with the typed letters.

Multiple search keys can be combined, separated by a space:

For further information and advanced query features, read the following pages.

Advanced Searches: the Data Watcher Query Language

Data Watcher Query Language (DWQL) can be used in the Flow Inquiry section search bar to run a search query string on a product managed entity.
This section covers the basics of DWQL as applicable to dataflow instance entity.

Language reference

Literals

Literals are constant values that a user specifies in a DWQL query, for instance a filename "myfile.txt" is literal.
Generally, literals containing blanks/spaces must be enclosed in double quotation marks, otherwise each space-separated substring is considered a distinct literal on its own.
DWQL supports typed literals with specific semantics as shown in the table below:

Clauses

Free text search clause
A free text search clause is simply a stemStringLiteral or wcstringLiteral that will be used to match any indexed field of dataflow instances.

Semantic search clauses
Semantic search clauses match specific attributes of dataflow instances and are expressed in the form of literal[searchScope]. supported semantic search clauses are listed in the table below:

Query option clauses
DWQL language allows for query options, which influence how the query is executed, rather than what the query returns.
From a syntax point of view, a query option is a special search clause having a semantic search scope prefixed by "opt-" that appears top level (i.e. not nested) in a query in implicit or explicit AND with other top level clauses.

Logical operators and expression nesting

A DWQL query is an expression made of "AND" / "OR" logical operators, case sensitive, wired together with "(" / ")" balanced parentheses having DWQL clauses as operands.
On top of the explicit "AND" operator, also the implicit "AND" operator is supported. This means that if a sequence of two or more clauses appears in a DWQQL query without any logical operator in between, an "AND" operator is assumed.

Considerations about performance

The execution time of a query is mainly influenced by two factors:

• Size of the underlying database, i.e. the quantity of dataflow instances stored in the database
• The computational complexity of the query itself

The complexity of the query is mainly influenced by two factors:

• The number of documents that need to be scanned and returned
  • DWQL queries are designed to take advantage of indexes, but on very large databases even the scanning of an index can be costly
• The filter clauses applied to string values
  • As a general rule, a full string match (i.e. stringLiteral and stemstringLiteral) or prefix match (i.e. stemprefixstringLiteral) can be much more performant than its counterpart allowing wildcards based pattern matching (i.e. wcstemstringLiteral and wcstemprefixstringLiteral)
  • Moreover, not all wildcard patterns have the same weight, as a general rule a pattern including multiple wildcards and/or including a wildcard at the beginning of the string can be very costly

Performance optimization tips:

• Set a sensible retention policy for the data in your database
  • You can regularly run Data Watcher data cleaner command (mongodb-cleaner.sh) to implement your policy
• Limit the number of returned documents by filtering them up-front
  • In most common Data Watcher scenarios, an effective way of restricting the number documents/indexes entries to be analyzed is to add sensible time based clauses to the query (e.g. "last 3 hours"[time])
• Express your query with semantic search clauses instead of free text search clauses, when possible
  • As a general rule, semantic search clauses are underpinned by more selective indexes
• Express filters on string values in the simplest form possible
  • Don't use wildcards if you don't need to
  • Express your wildcard filter in a simple form

Finally, if you really do need to run a complex query but the default time allowed for a query to complete it is not enough, you can allow for more time using the previously discussed [opt-ttlsecs] query options.

Examples

DWQL query: invoice
Description: this query returns all dataflow instances having at least one value in its indexed field starting with "invoice".

DWQL query: invoice AND payroll
Description: this query returns all dataflow instances having both "invoice" and "payroll" as value of an indexed field.
This example gives the same results as the following one, which uses the AND operator implicitly.

DWQL query: invoice payroll
Description: this query returns all dataflow instances having both "invoice" and "payroll" as value of an indexed field.
Compared to the previous example, this example uses the AND operator implicitly.

DWQL query: 'MonthlySales-2021-??.txt'[entity] 180[opt-ttlsecs]
Description: this query returns all dataflow instances that contain a file name conforming to the pattern 'MontlySales-2021-??.txt', and allow 3 minutes of time for the query to complete before aborting.

DWQL query: myfile1.txt AND mymodel1
Description: this query returns all dataflow instances having both myfile1.txt and mymodel1 as value of an indexed field.

DWQL query: "SFO Invoices"[model] AND (warning[business] OR error[business]) AND "last 45 minutes"[time]
Description: this query returns all dataflow instances associated to "SFO Invoices" model and having experienced a cut-off warning or a cut-off error in the last 45 minutes.

DWQL query: myfile1.txt AND error[status] AND "last 2 hours"[time]
Description: this query returns all dataflow instances having myfile1.txt as value of an indexed field that have experienced an error in the last two hours.

DWQL query: (ACMESERVER[system] OR ACMESERVER[source] OR ACMESERVER[destination] ) AND "last 2 hours"[time]
Description: this query matches all dataflow instances terminated in the last 2 hours and involving "ACMESERVER" system in each possible role (monitored, non-monitored, source or destination).