Simple Searches

Text Search

Enter text, such as a filename, in the Search edit box at the top of the page and click the lens button on the right to run your query.

The autocomplete function will provide suggestions while you type into the Search field. If a saved query matches your characters, it will appear as well.

If no results match your criteria, No results will appear in the drop-down list.

Search by Date

Entering a date, the system will propose you search from that day to the current day or up to the specified day:

• gg/mm/yyyy[since]: this query will search for flows from the selected date until today. For example, entering 18/12/2024 [since] will retrieve all results from December 18th up to the current date.
• gg/mm/yyyy[until]: this query will search for flows up to the selected date. For example, entering 20/12/2024 [until] will retrieve all results up to December 20th.

These options can be used together to search for flows in the interval specified, e.g. from December 18th, 2024 to December 20th, 2024 as shown below:

Search by Integrity Status

• successfully[status]: this query will search for all flows having Integrity Status Completed.
• error[status]: this query will search for all flows having Integrity Status Error.
• running[status]: this query will search for all flows having Integrity Status Running.

Search by DataFlow Status

The Business Status depends on the DataFlow status of cut-offs applied to the flow events:

• onschedule[business]: this query will search for all flows having DataFlow Status On Schedule. They have been completed successfully and are on schedule.
• pending[business]: this query will search for all flows having DataFlow Status Pending. They will be completed in the future.
• warning[business]: this query will search for all flows having DataFlow Status Warning. They are still pending and near the deadline.
• error[business]: this query will search for all flows having DataFlow Status Error. They are uncompleted or completed after the scheduled time.

Search by Instance Status

The Business Instance Status depends on the Instance status of cut-offs applied to the flow events:

• onschedule[businessInstance]: this query will search for all flows having Instance Status On Schedule. They have been completed successfully and are on schedule.
• pending[businessInstance]: this query will search for all flows having Instance Status Pending. They will be completed in the future.
• warning[businessInstance]: this query will search for all flows having Instance Status Warning. They are still pending and near the deadline.
• error[businessInstance]: this query will search for all flows having Instance Status Error. They are uncompleted or completed after the scheduled time.

For more details refer to sections Dataflow Inquiry and Setting Cut-offs.

Search by Source

Click the Source link, to drill down results.

The Source will be added automatically to the Search field.

Search by Attribute

Clicking an Attribute will be added automatically to the Search field. Attributes are configured by the Administrator, as described in Attributes.
For example, in the figure below the CEO attribute has been added to the Search bar.

As an alternative, you can start typing an attribute in the Search field and the autocomplete function will propose the value starting with the typed letters.

Multiple search keys can be combined, separated by a space:

Advanced Searches: the Data Watcher Query Language

Data Watcher Query Language (DWQL) can be used in the Inquiry section search bar to run a search query string on a product-managed entity.
This section covers the basics of DWQL as applicable to dataflow instance entities.

Language reference

Literals

Literals are constant values that a user specifies in a DWQL query, for instance, a filename "myfile.txt" is literal.
Generally, literals containing blanks/spaces must be enclosed in double quotation marks, otherwise, each space-separated substring is considered a distinct literal on its own.
DWQL supports typed literals with specific semantics as shown in the table below:

Clauses

Free text search clause
A free text search clause is simply a stemStringLiteral or wcstringLiteral that will be used to match any indexed field of dataflow instances.

Semantic search clauses
Semantic search clauses match specific attributes of dataflow instances and are expressed in the form of literal[searchScope]. supported semantic search clauses are listed in the table below:

Query option clauses
DWQL language allows for query options, which influence how the query is executed, rather than what the query returns.
From a syntax point of view, a query option is a special search clause having a semantic search scope prefixed by "opt-" that appears top level (i.e. not nested) in a query in implicit or explicit AND with other top level clauses.

Logical operators and expression nesting

A DWQL query is an expression made of "AND" / "OR" logical operators, case sensitive, wired together with "(" / ")" balanced parentheses having DWQL clauses as operands.
On top of the explicit "AND" operator, also the implicit "AND" operator is supported. This means that if a sequence of two or more clauses appears in a DWQQL query without any logical operator in between, an "AND" operator is assumed.

Considerations about performance

The execution time of a query is mainly influenced by two factors:

• Size of the underlying database, i.e. the quantity of dataflow instances stored in the database
• The computational complexity of the query itself

The complexity of the query is mainly influenced by two factors:

• The number of documents that need to be scanned and returned
  • DWQL queries are designed to take advantage of indexes, but on very large databases even the scanning of an index can be costly
• The filter clauses applied to string values
  • As a general rule, a full string match (i.e. stringLiteral and stemstringLiteral) or prefix match (i.e. stemprefixstringLiteral) can be much more performant than its counterpart allowing wildcards based pattern matching (i.e. wcstemstringLiteral and wcstemprefixstringLiteral)
  • Moreover, not all wildcard patterns have the same weight, as a general rule a pattern including multiple wildcards and/or including a wildcard at the beginning of the string can be very costly

Performance optimization tips:

• Set a sensible retention policy for the data in your database
  • You can regularly run Data Watcher data cleaner command (mongodb-cleaner.sh) to implement your policy
• Limit the number of returned documents by filtering them up-front
  • In most common Data Watcher scenarios, an effective way of restricting the number documents/indexes entries to be analyzed is to add sensible time based clauses to the query (e.g. "last 3 hours"[time])
• Express your query with semantic search clauses instead of free text search clauses, when possible
  • As a general rule, semantic search clauses are underpinned by more selective indexes
• Express filters on string values in the simplest form possible
  • Don't use wildcards if you don't need to
  • Express your wildcard filter in a simple form

Finally, if you really do need to run a complex query but the default time allowed for a query to complete it is not enough, you can allow for more time using the previously discussed [opt-ttlsecs] query options.

Examples

DWQL query: invoice
Description: this query returns all dataflow instances having at least one value in its indexed field starting with "invoice".

DWQL query: invoice AND payroll
Description: this query returns all dataflow instances having both "invoice" and "payroll" as value of an indexed field.
This example gives the same results as the following one, which uses the AND operator implicitly.

DWQL query: invoice payroll
Description: this query returns all dataflow instances having both "invoice" and "payroll" as value of an indexed field.
Compared to the previous example, this example uses the AND operator implicitly.

DWQL query: 'MonthlySales-2021-??.txt'[entity] 180[opt-ttlsecs]
Description: this query returns all dataflow instances that contain a file name conforming to the pattern 'MontlySales-2021-??.txt', and allows 3 minutes for the query to complete before aborting.

DWQL query: myfile1.txt AND mymodel1
Description: this query returns all dataflow instances having both myfile1.txt and mymodel1 as the value of an indexed field.

DWQL query: "SFO Invoices"[model] AND (warning[business] OR error[business]) AND "last 45 minutes"[time]
Description: this query returns all dataflow instances associated with "SFO Invoices" model and having experienced a cut-off warning or a cut-off error in the last 45 minutes.

DWQL query: myfile1.txt AND error[status] AND "last 2 hours"[time]
Description: this query returns all dataflow instances having myfile1.txt as the value of an indexed field that has experienced an error in the last two hours.

DWQL query: (ACMESERVER[system] OR ACMESERVER[source] OR ACMESERVER[destination] ) AND "last 2 hours"[time]
Description: this query matches all dataflow instances terminated in the last 2 hours and involving "ACMESERVER" system in each possible role (monitored, non-monitored, source or destination).