Signatures are rules to extract important field values from the raw logs. The values are then indexed to simplify the search. You can also use them to compare, aggregate, correlate, and create reports for the logs.
For example:
user Bob logged in from source 192.168.2.10
Applying signature against the raw log above gives you concise information.
user <user:word> <action:all> from source <source_address:ip>
This signature extracts the following field-value pairs from the log.
user: Bob
action: logged in
source_address: 192.168.2.10
Logpoint provides normalization policies for different systems and applications. You can create your own signatures inside a normalization package, build them into normalization policies, and use them with your log collector.
Fields are names given to normalized values. Field values can be used in searches to filter specific terms.
For example,
<174>Oct 22 10:52:58 10.141.15.121 user Bob logged in from source 192.168.2.10
The important fields from the above message are user, action, and source_address.
user: Bob
action: logged in
source_address: 192.168.2.10
Values are the words (or terms) extracted from log messages.
In the above example, the field user has the value Bob, the field action has the value Logged in, and the field source_address has the value 192.168.2.10.
Definers are built-in customized regex used to extract information from log messages during normalization.
User<user:word><action:'logged'>in from<source_address:ip>:<source_port:port>using service<service_name:word>
In this expression:
<user:word> matches a word pattern and extracts it as the “user” value.
<action:'logged'> matches the literal string “logged” and extracts it as the “action” value.
<source_address:ip> matches an IP address pattern and extracts it as the “source_address” value.
<source_port:port> matches a port number pattern and extracts it as the “source_port” value.
<service_name:word> matches a word pattern and extracts it as the “service_name” value.
The definers word, ip, and port provide readability in the signatures and help with troubleshooting.
Go to the List of Definers for a detailed list of definers.
Regex patterns are always inside < and > symbols. Always use Fields and Definers (or custom re2 patterns) between these symbols.
Use the definers word(s) to extract a single word or several respectively.
Log message example:
Detailed Tracking A process has exited: Process ID: 17728
Syntax example:
<status:words>A process has <action:word>
Extracted parameters:
Fields
Values
status
Detailed Tracking
action
exited
It is possible to extract integers and floating values with appropriate definers. Use the int and the float keywords to extract integers and floating values respectively. If you are not sure whether the value is an integer or float, you can use the int_float keyword to extract both types of data.
Log message example:
592 Security SYSTEM User Success Audit Logpoint Detailed Tracking A new process has been created: New Process ID: 16668
Syntax example:
<event_id:int><:all>A new<object:word>has been<action:'created'>: New Process ID:<process_id:int>
Extracted parameters:
Fields |
Values |
|---|---|
event_id |
592 |
object |
process |
action |
created |
process_id |
16668 |
Use definer ip to extract any type of IP addresses (IPV4 or IPV6).
Log message example:
%PIX-4-403109: Rec'd packet not an PPTP packet. (ip) dest_address= 192.168.2.14, src_addr= 192.168.2.78, data: string
Syntax example:
<event_id:'403109'><:all>dest_address=<destination_address:ip>, src_addr=<source_address:ip>
Extracted parameters:
Fields |
Values |
|---|---|
event_id |
403109 |
source_adddress |
192.168.2.78 |
destination_address |
192.168.2.14 |
Any sign such as =, :, and - can separate the values. Use backslash ( \ ) to escape separators key for separators as “:”, “,” “-” etc.
Log messages example:
Benchmarker; reporting speed; service: normalizer_pfsense_and_openvpn; actual_mps=0; doable_mps=878;
Syntax example:
reporting speed;<word:\::all>;<word:=:int>;<word:=:int>;
Extracted parameters:
Fields |
Values |
|---|---|
service |
normalizer_pfsense_and_openvpn |
actual_mps |
0 |
doable_mps |
878 |
It is possible to use dynamic mapping to extract similarly placed field-value pairs.
Log message example:
INFO: search; performed; type=audit_log; username='Bob'; query='sig_id=1012 | timechart count()'; time_range=[1326245880, 1326267480]; repos=['127.0.0.1:5504/default', '127.0.0.1:5504/_logpoint']
Syntax example:
<severity:word>:<object:all>; <action:all>;<<word:=:all>;>
Extracted parameters:
Fields |
Values |
|---|---|
severity |
INFO |
object |
search |
action |
performed |
type |
audit_log |
username |
Bob |
query |
sig_id=1012 | timechart count() |
time_range |
1326245880, 1326267480 |
It is possible to extract the values from the log messages using the regex patterns. However, using fields and definers to write signatures is more efficient.
Log message example:
ASA-2-106006: Deny inbound UDP from 192.168.2.222/514 to 202.70.91.12/26 on interface interface_name
Syntax example:
<event_id:'106006'>:<action:'\S+'>inbound<object:'\S+'><:'.*?'> <source_address:'\d+.\d+.\d+.\d+'><:'/'><source_port:'\d+'>to <destination_address:'\d+.\d+.\d+.\d+'><:'/'><destination_port:'\d+'>
Extracted parameters:
Fields |
Values |
|---|---|
event_id |
106006 |
action |
Deny |
object |
UDP |
destination_address |
202.70.91.12 |
destination_port |
26 |
source_address |
192.168.2.222 |
source_port |
512 |
You can use the OR operand with the definers if you are uncertain of the type of data associated with the fields. Switch between the possible definers with the OR operand.
Log message example:
Metrics; Physical Memory; total=2009 MB; use=59.1%; used=1913 MB; free=95 MB
Syntax example:
Physical Memory; <word:=:int>MB; <word:=:int OR float>%; <word:=:int>MB; <word:=:int OR float>MB
Extracted parameters:
Fields |
Values |
|---|---|
total |
2009 |
use |
59.1 |
used |
1913 |
free |
95 |