Pipes

Queries can include pipes for post-processing of events, and can be used for enrichment, aggregations, statistics and filtering.

count

The count pipe will return only statistics. If no arguments are passed, then it returns the total number of events. Otherwise, it returns the number of occurrences for each unique value. Stats are returned in the form

Count the total number of events
process where true | count

// results look like
// {"count": 100, "key": totals"}
Count the number of times each value occurs
process where true | count process_name

// results look like
// {"count": 100, "key": "cmd.exe", "percent": .4}
// {"count": 50, "key": "powershell.exe", "percent": .2}
Count the number of times a set of values occur
process where true | count parent_process_name, process_name

// results look like
// {"count": 100, "key": ["explorer.exe", "cmd.exe", "percent": .4}
// {"count": 100, "key": ["cmd.exe", "cmd.exe", "percent": .4}

unique

The unique pipe will only return the first matching result through the pipe. Unless a sort pipe exists before it, events will be ordered chronologically.

Get the first matching process for each unique name
process where true | unique process_name
Get the first result for multiple of values
process where true | unique process_name, command_line

filter

The filter pipe will only output events that match the criteria. With simple queries, this can be accomplished by adding and to the search criteria. It’s most commonly used to filter sequences or with other pipes.

Find network destinations that were first seen after May 5, 2018
network where true
| unique destination_address, destination_port
| filter timestamp_utc >= "2018-05-01"

unique_count

The unique_count pipe combines the filtering of unique with the stats from count. For unique_count, the original event is returned but with the fields count and percent added.

Get the first result per unique value(s), with added count information
process where true | unique_count process_name | filter count < 5

tail

The tail pipe is similar to the UNIX tail command and will output the latest events coming through the pipe.

Get the most recent ten logon events
security where event_id == 4624
| tail 10

sort

The sort pipe will reorder events coming through the pipe. Sorting can be done with one or multiple values.

Warning

In general, sort will buffer all events coming into the pipe, and will sort them all at once. It’s often good practice to bound the number of events into the pipe.

For instance, the following query could be slow and require significant memory usage on a busy system.

file where true | sort file_name
Get the top five network connections that transmitted the most data
network where total_out_bytes > 100000000
| sort total_out_bytes
| tail 5