Filter Expression Language
If your filter cannot be described using the graphical filter editor, you can define a manual filter using WebSpy's filter expression language.
To access this option, click the Manual Filter Expression radio button on any filter options dialog.
A filter expression consists of one or more Boolean expressions, connected with either an AND, OR or XOR (exclusive or).
Filter expression example: [IPProtocol] == 6 AND [Size] > 100M
A filter expression can include the following types of values:
Field Names:
Field names are enclosed in square brackets, e.g. [Username]. The list of field names depends on the schema you are using.
DateTimes and TimeSpans:
DateTimes and TimeSpans are wrapped in braces e.g. {20/11/17 09:00:00}.
DateTime Fields contain two properties, .Date and .TimeOfDay.
The .Date property returns the current date and adheres to the format dd/mm/yyyy. e.g. [DateTime.Date] = {20/11/17}.
The .TimeOfDay property is a TimeSpan and adheres to the format d.hh:mm:ss. e.g (([DateTime.TimeOfDay] >= {09:00:00} and [DateTime.TimeOfDay] <= {17:00:00})).
You can use the Manual Filter Expression to remove the .Date or .TimeOfDay properties to return the full timestamp e.g [DateTime] = {20/11/17 09:00:00}.
Strings:
String literals are wrapped in quotes, and can include a quote mark by prefacing it with a backslash, e.g. "Hello", or "Say \"Hello\"".
Numbers:
Numbers do not need to be wrapped in quotes, braces or brackets. They can also be followed by K, M or G to represent Kilobytes, Megabytes or Gigabytes. Numbers with one of these size modifiers can include a decimal point, e.g. 1.5K.
IP Addresses and Subnets expressions:
Like field names, IP addresses need to be surrounded by square brackets. e.g. ([Src.Address] IN [10.1.2.3]). You can also filter on subnets using CIDR notation. e.g. ([Src.Address] IN [10.1.2.3/24]).
Operators
Math and Comparison operators are used when defining functions.
The following operators are available:
Math Operators
| Operator | Description | Strings | Numbers | DateTimes | IP Addresses |
|---|---|---|---|---|---|
| + | Add | Concatenates two strings: "ab" + "cd" = "abcd" |
Sums two numbers: 1 + 2 = 3. |
Adds a Timespan to a DateTime: {16/06/2005 10:35:01} + {30}. Adds a Timespan to a Timespan: {10:35:01} + {01:00:00} |
NA |
| - | Subtract | NA | Subtracts a number from another number: 3 - 1 = 2 |
Subtracts a Timespan from a DateTime: {16/06/2005 10:35:01} - {30}. Subtracts a Timespan from a Timespan: {10:35:01} - 01:00:00} |
NA |
| / | Divide | NA | Divides a number by another number: 10 / 5 = 2 |
NA | NA |
| * | Multiply | NA | Multiplies two numbers: 2 * 2 = 4 |
NA | NA |
| % | Mod | NA | Divides two numbers (floating points rounded to integers) and returns only the remainder: 19 % 6.7 = 5 |
NA | NA |
Comparison Operators
| Operator | Description | Strings | Numbers | DateTimes | IP Addresses |
|---|---|---|---|---|---|
| == or = | Equals | Returns true if two strings are equal: "abcd" == "abcd" |
Returns true if two numbers are equal: 4.1K == 4.1K |
Returns true if two DateTimes are equal: {16/06/2005 10:35:01} == {16/06/2005 10:35:01} |
Returns true if two IP Addresses are equal 192.168.1.63 == 192.168.1.63 |
| != or <> | Not Equal | Returns true if two strings are not equal: "abcd" != "bcda" |
Returns true if two numbers are not equal: 4.1K != 3.9G |
Returns true if two DateTimes are not equal: {16/06/2005 10:35:01} != {16/06/2006 10:35:01} |
Returns true if two IP Addresses are not equal: 192.168.1.63 != 192.168.1.30 |
| < | Less Than | NA | Returns true if the first number is less than the second number: 5 < 7 |
Returns true if the first DateTime is before the second DateTime. {16/06/2005 10:35:01} < {16/06/2006 10:35:01} |
NA |
| <= | Less Than or Equal To | NA | Returns true if the first number is less than or equal to the second number: 5 <= 5 |
Returns true if the first DateTime is before or equal to the second DateTime. {16/06/2005 10:35:01} <= {16/06/2005 10:35:01} |
NA |
| > | Greater Than | NA | Returns true if the first number is greater than the second number: 7 > 5 |
Returns true if the first DateTime is later than the second DateTime. {16/06/2005 10:35:01} > {16/06/2003 10:35:01} |
NA |
| >= | Greater Than or Equal To | NA | Returns true if the first number is greater than or equal to the second number: 7 >= 7 |
Returns true if the first DateTime is later than or equal to the second DateTime. {16/06/2005 10:35:01} >= {16/06/2005 10:35:01} |
NA |
| in | In | Returns true if a substring is present in another string: "bc" in "abcd". Returns true if a string is in a list: "bc" in ["ab", "bc", "cd"] |
Returns true if a number is in a list: 6 in [1, 2, 5..9] |
NA | Returns true if an IP address is in a range: [Source IP] in [192.168.0.0 .. 192.168.0.255] or [Source IP] in 192.168.0.0/24 |
| like | Like | Returns true if a string is similar to another string: "abcd" like "ab*" or "abcd" like [a-z] |
NA | NA | NA |
Add brackets and spaces to taste, e.g. (1 == 1) and 3-(2-1) == 2
Functions
You can also define certain functions using the WebSpy expression language such as extracting only the year from a DateTime field, or aliasing a string.
The following functions are available:
| Functions | Syntax | Example |
|---|---|---|
| Alias | Alias("alias", [field]) |
Alias("Subnets", [SourceIP]) Returns the aliased values for the SourceIP field as defined by the Subnets alias |
| Profile | Profile("string") |
Profile("http://www.webspy.com") Returns the Profile for the url http://www.webspy.com. Profile([Site.Keywords]) Returns the profile for Search Terms entered into search engines. |
| Length | Length("string") | Length("abcd") Returns 4 |
| Date | Date(DateTime) |
Date(16/06/2005 10:55:36) Returns 16/06/2005 |
| Year | Year(DateTime) |
Year(16/06/2005 10:55:36) Returns 2005 |
| Month | Month(DateTime) |
Month(16/06/2005 10:55:36) Returns 6 |
| Day Of Month | DayOfMonth(DateTime) |
DayOfMonth(16/06/2005 10:55:36) Returns 16 |
| Day Of Week | DayOfWeek(DateTime) |
Returns a number ranging from 0 to 6 representing Sunday to Saturday. DayOfWeek(16/06/2005 10:55:36) Returns 4. |
| Day Of Year | DayOfYear(DateTime) |
DayOfYear(16/06/2005 10:55:36) Returns 167 |
| Week Of Year | WeekOfYear(DateTime) |
WeekOfYear(16/06/2005 10:55:36) Returns 24 |
| Time | Time(DateTime) |
Time(16/06/2005 10:55:36) Returns 10:55:36 |
| Hour | Hour(TimeSpan) or Hour(DateTime) |
Hour(16/06/2005 10:55:36) or Hour(10:55:36) Returns 10 |
| substring | substring([Field], start, count] |
substring([Username], 0, 6). e.g. substring("domain\john", 0, 6) Returns domain\ |
| Md5 | Md5([Field]) or Md5([Field], "salt") |
md5([Username]) Returns an md5 string in place of the Username. This is useful for anonymising data in reports such as usernames or IP addresses. Md5([Field], "SecretSalt") Returns an md5 string with increased security. Someone could potentially run a list of known user names through an md5 function to obtain the mappings of usernames to md5 strings. You can therefore also pass a 'salt' string to this function. Unless the user also knows the salt that was used when generating a report, they will not be able to obtain the same md5 string for a given user. |
| Minute Of Day | MinuteOfDay(TimeSpan) or MinuteOfDay(DateTime) |
MinuteOfDay(16/06/2005 10:55:36) or MinuteOfDay(10:55:36) Returns 655 |
| Minute Of Hour | MinuteOfHour(TimeSpan) or MinuteOfHour(DateTime) |
MinuteOfHour(16/06/2005 10:55:36) or MinuteOfHour(10:55:36) Returns 55 |
| Now | Now() |
Now() Returns the current DateTime. |
| CombineArray | combineArray([field]) |
combineArray([Categories]) Returns a comma separated list of Categories. This function can only be applied to fields that contain and array of values (where a single record has multiple values, such as URL categories or email recipients). |
Filter Expression Examples
| Description | Syntax |
|---|---|
| All values in the year 2005 | Year([DateTime]) == 2005 |
| All values between the first of May and the First of June 2005. | [DateTime] >= {01/05/2005} and [DateTime] <= {01/06/2005} |
| All values from 1st May 2005 until today. | [DateTime] >= {01/05/2005} and [DateTime] <= Now() |
| All values in the past 30 days | [DateTime] > (Date(NOW()) - {30}) and [DateTime] <= Date(Now()) |
| All values in the subnet alias group California. | Alias("Subnets", [SourceIP]) == "California" |
| All values in the Subnet alias groups California and Washington. | (Alias("Subnets", [Source IP]) in ['California', 'Washington']) |
| All values in the Subnet 192.168.0.0/24 | [SourceIP] in 192.168.0.0/24 |
| All emails sent to @gmail.com and @hotmail.com addresses | CombineArray[RecipientDomain] = "gmail.com, hotmail.com" or CombineArray[RecipientDomain] = "hotmail.com, gmail.com" |
| All traffic leaving or entering the network (Source = Internal AND Destination = External) OR (Source = External AND Destination = Internal) | ((Alias("Subnets", [Src.Address]) IN ["Internal"]) and (Alias("Subnets", [Dst.Address]) IN ["External"])) or ((Alias("Subnets", [Src.Address]) IN ["External"]) and (Alias("Subnets", [Dst.Address]) IN ["Internal"])) |
Creating Custom Summaries
You can also using the WebSpy Vantage Expression Language to create your own Summaries that are not defined in your Schema. These options are available when adding a Key column in an Analysis Templates, or when running an Analysis using the Add Summary button on the Summaries Page.
For example, you can concatenate two fields together such as Site URLs and Site Queries to rebuild full URLs with
[Site.GetBaseUri] + [Site.Query]. Or display a comma separated list of Categories withCombineArray([Category]).