Parsers extract individual logs from the incoming data and forward them for further processing in the log collection pipeline.
Collectors and fetchers use parsers to process the collected log. In addition to built-in, default parsers you can create your own. The provided regex pattern in the parser splits the message into individual log entries.
Since the Line parser splits logs greater than 12 KB into individual logs of 12 KB, the disk space can fill up quickly while receiving larger log files. You can monitor disk usage from System Monitor.
Parsers¶
To sort the columns in ascending or descending order, move your cursor to the column you want to sort. Click the Down Arrow (
) for ascending order and the Up Arrow (
) for descending order.
Sorting Columns¶
Go to Settings >> Configuration from the navigation bar and click Parsers.
Parsers¶
Click Add Parser.
Addition of a Parser¶
Enter Name, Pattern, and Example.
Click Check to verify if the pattern matches the examples.
Pattern and Examples Check¶
Click Submit.
Now, apply the parser to the collection devices, i.e., collectors and fetchers.
Go to Settings >> Configuration from the navigation bar and click Parsers.
Click the Name of the required parser and update the information. You cannot edit the name of a parser.
Parsers¶
Click the Check button to verify if the pattern matches the examples.
Click Submit.
Click the ? symbol near the top-right corner for context-sensitive help.
Go to Settings >> Configuration from the navigation bar and click Parsers.
Click the Delete (
) icon under Actions.
To delete multiple parsers, select the parsers, click More and choose Delete Selected.
To delete all the parsers, click More and choose Delete All.
Deleting Parsers¶
Click Yes to confirm deletion.
Logpoint provides the following parsers you can use to parse some standarized log formats.
Line Parser splits each line in the log file into individual logs. If the size of the log file is larger than 12 KB, the log file is split into individual logs.
Example:
Line parser splits the following type of log entries into two seperate logs:
Apr 28 08:58:18 EventCode=5156 EventType=0 Keywords=Audit Success Message=A connection permitted. Application Name: App Direction: Inbound Source Address: 21.21.3.133 Source Port: 80 Destination Address: 21.21.3.132 Destination Port:444
Apr 28 08:58:18 EventCode=5156 EventType=0 Keywords=Audit Success Message=A connection permitted. Application Name: App Direction: Inbound Source Address: 21.21.3.133 Source Port: 80 Destination Address: 21.21.3.132 Destination Port:6161
Syslog Parser splits syslog-formatted logs into individual messages using either the newline character (n) or the octet-counting method. The parser splits logs based on the message length specified in the log. If the size of the log exceeds the defined Message Length set from Syslog, log is split into segments of that length. For example, if the message length is set at 12 KB, logs larger than 12 KB size are divided into 12 KB segments. Use Syslog Parser only if the syslog message is formatted in one of the supported syslog formats.
Example:
Syslog parser splits the following type of log entries into two seperate logs:
<135>Apr 28 08:58:18 LogName=Security SourceName=Security audit. EventCode=5156 EventType=0 TaskCategory=Connection Keywords=Audit Success Message=A connection permitted. Process ID: 41 Application Name: App Direction: Inbound Source Address: 21.21.3.133 Source Port: 80 Destination Address: 21.21.3.132 Destination Port:444 \n <165>Apr 28 08:58:18 LogName=Security12 SourceName=Security audit1. EventCode=5156 EventType=0 TaskCategory=Connection Keywords=Audit Success Message=A connection permitted. Process ID: 41 Application Name: App Direction: Inbound Source Address: 21.21.3.133 Source Port: 80 Destination Address: 21.21.3.132 Destination Port:6161
Multi Line Syslog Parser splits multiple syslog messages written in multiple lines into individual logs. It uses Priority Value, or PRI, a numerical value enclosed in angle brackets “<>”, to split the message.
Example:
Multi Line Syslog parser splits the following type of log entries into three seperate logs:
<135>Apr 28 08:58:18
LogName=Security
SourceName=Security audit.
EventCode=5156
EventType=0
TaskCategory=Connection
Keywords=Audit Success
Message=A connection permitted.
Process ID: 41
Application Name: App
Direction: Inbound
Source Address: 21.21.3.133
Source Port: 80
Destination Address: 21.21.3.132
Destination Port:444
<165>Apr 28 08:58:18
LogName=Security12
SourceName=Security audit1.
EventCode=5156
EventType=0
TaskCategory=Connection
Keywords=Audit Success
Message=A connection permitted.
Process ID: 41
Application Name: App
Direction: Inbound
Source Address: 21.21.3.133
Source Port: 80
Destination Address: 21.21.3.132
Destination Port:6161
<161>Mar 19 11:38:18
LogName=Security123
SourceName=Security audit 123.
EventCode=5156
EventType=0
TaskCategory=Connection
Keywords=Audit Success
Message=A connection permitted.
Process ID: 4
Application Name: App
Direction: Inbound
Source Address: 21.21.3.133
Source Port: 80
Destination Address: 21.21.3.132
Destination Port:6161
Email Parser aggregates logs with the same message ID into a single log. It supports logs from the following email services: Exim, Qmail, Cisco IronPort, Sendmail, and Postfix MTA. After aggregation, a compiled normalizer specific to each email service is required in the normalization policy to extract key-value pairs. The table below lists the appropriate compiled normalizer for each service.
Email Service |
Compiled Normalizer |
|---|---|
Exim |
EximMTACompiledNormalizer |
Qmail |
QmailCompiledNormalizer |
Cisco IronPort |
CiscoIronPortESGCompiledNormalizer |
Sendmail |
SendMailCompiledNormalizer |
Postfix MTA |
PostFixCompiledNormalizer |
This parser only works with the Syslog Collector.
Proofpoint Email Protection Parser splits logs coming from Proofpoint’s Email Protection service.
DB2 Parser splits logs from IBM DB2 servers.
RACF Parser splits logs from Resource Access Control Facility (RACF) devices.
CSVParser processes comma-separated values from a file. CSVParser can only be used with file-based collectors and fetchers.
Processes JSON lines from a file. JSONLineParser can only be used with file-based collectors and fetchers.
In addition to these parsers, Logpoint has default parsers specific to integrations. For more details, search for specific parsers in the ServiceDesk.