Pattern finding is a method of finding one or multiple streams and patterns of data to correlate a particular event. For example: five failed logins, followed by a successful login. It can be performed on the basis of the count and the time of occurrence of the stream. Use the Pattern Finding rules to detect complex event patterns in a large number of logs.
Correlation is the ability to track multiple types of logs and deduce meanings from them. It lets you look for a collection of events that make up a suspicious behavior and investigate further.
A stream consists of a count or occurrence of a query. The query can be a simple search query or an aggregating query. The stream can consist of a having same or a within expression. Stream has notion of time.
Syntax |
Description |
|---|---|
[ ] |
For single streams, square brackets contain a stream of events. |
within |
Keyword to denote the notion of time frame |
having same |
Keyword |
Following are the working examples for pattern finding using single stream:
To find 5 login attempts:
[5 action = "logged on"]
[5 login]
To find 5 login attempts within a timeframe of 2 minutes:
[5 action = "logged on" within 2 minutes]
[5 login within 2 minutes]
To find 5 login attempts by the same user:
[5 action = "logged on" having same user]
[5 login having same user]
To find 10 login attempts by the same user from the same source_address (multiple fields) within 5 minutes:
[10 action = "logged on" having same user, source_address within 5 minutes]
The time format for specifying timeframe are: second(s), minute(s), hour(s) and day(s).
[error] as E
This query finds the logs with errors. It then aliases the result as E and displays the fields prefixed with E such as E.severity, and E.device_ip. You can then use the aliased fields as shown below:
[error] as E | rename E.device_ip as DIP | search DIP = "127.0.0.1"
Pattern finding queries for different conditions:
10 login to localhost (source_address) by the same user for the last 15 minutes.
[10 login source_address = 127.0.0.1 having same user_name within 15 minutes]
The field of a log file with a norm command .
[2 login | norm <username:word> login successful having same username within 10 seconds]
You can join multiple patterns by using Pattern Finding by Joining Streams and Pattern Finding by Following Streams.
You can use a left join to return all the values from the table or stream on the left, and only the common values from the table or stream on the right.
Example:
[table event_prob] as s1
left join [event = * | chart count() by event] as s2
on s1.event = s2.event
You can use a right join to return all the values from the table or stream on the right and only the common values from the table or stream on the left.
Example:
[5 transaction error having same user within 30 seconds] as s1
right join [transaction successful] as s2
on s1.user=s2.user
Join queries are used to link the results from different sources. The link between two streams must have an on condition. The link between two lookup sources or any of the lookup and stream does not require a time-range. Join as a part of a search string, can link one data-set to another based on one or more common fields. For instance, two completely different data-sets can be linked together based on a username or event ID field present in both the data-sets.
The syntax for joining multiple patterns is as follows:
[stream 1] <aliased as s1> <JOIN> [stream 2] <aliased as s2> on <Join_conditions> | additional filter query.
[action = locked] as locked
join
[action = unlocked] as unlocked
on
locked.target_user = unlocked.target_user
| chart count() by locked.target_user, locked.caller_computer,
unlocked.caller_user
[login] as l join [table User] as u on l.user = u.user
To find the events where a reserved port of an Operating System (inside the PORT_MACHINE table) is equal to the blocked port (inside the BLOCKED_PORT table):
[table PORT_MACHINE port<1024] as s1 join [table BLOCKED_PORT] as s2
on s1.port=s2.port
To find 5 login attempts by the same user within 1 minute followed by 5 failed login attempts by the same user within 1 minute
[5 login having same user within 1 minute] as s1
followed by
[5 failed having same user within 1 minute]
To find 5 login attempts by the same user within 1 minute followed by 5 failed attempts by the same user within 1 minute and users from both result are same
[5 login having same user within 1 minute] as s1
followed by
[5 failed having same username within 1 minute] as s2
on
s1.username = s2.username
Pattern Finding by followed by is useful when two sequential streams are connected to an action.
For example:
[2 login success having same user] AS stream1
followed by
[login failure] as stream2
ON
stream1.user = stream2.user
Here,
Syntax |
Description |
|---|---|
[ ] AS stream1 |
A simple pattern finding query aliased as stream1 |
followed by |
Keyword |
[ ] AS stream2 |
A simple search aliased as stream2 |
ON |
Keyword |
stream1.user = stream2.user |
Matching field from the 2 streams |
The syntax for joining multiple patterns is as follows:
[stream 1] <aliased as s1> <followed by> [stream 2] <aliased as s2> <within time limit> on <Join_conditions>| additional filter query.
[stream 1] as s1 followed by [stream2] as s2 within time_interval on s1.field = s2.field
[stream 1] as s1 followed by [stream2] as s2 on s1.field = s2.field
[stream 1] as s1 followed by [stream2] as s2 within time_interval
The inference derived from the above queries:
Streams can be labeled using alias. Here, the first stream is labeled as s1. This labeling is useful while setting the join conditions in the join query.
The operation between multiple streams is carried out using “followed by” or “join”.
Use the followed by keyword to connect two sequential streams anticipating an action, e.g., multiple login attempts followed by successful login.
Use the join keyword to view additional information in the final search. The join syntax is mostly used with tables for enriching the data.
Time limit for occurrence can also be specified.
If you use the join keyword, then specify the on condition.
Join conditions are simple mathematical operations between the data-sets of two streams.
Use additional filter query to mitigate false positives which are generally created while joining a stream and a table. Searching the query with a distinct key from the table displays an error-less result.
[| chart count() by device_ip] AS lookup
JOIN
[device_ip=*] AS log ON lookup.device_ip = log.device_ip
This query does not display histogram but displays the log table.
[device_ip=*] as log join [| chart count() by device_ip] as lookup on
log.device_ip=lookup.device_ip
This query displays both the histogram and the log table.
Note
The Latest command is supported in pattern finding queries.
All the reserved keywords such as on, join, as, and chart are not case-sensitive.
If you want to use reserved keywords in simple search or some other contexts, put them in quotes.
login | chart count() by device_ip | search "count()" > 5