Additional Notes¶

Process or Count¶

Count and process are keywords and must be enclosed within double quotes.

MsWinEventLog product=* | chart count() as "Count" by product
order by count() desc limit 10

Similarly,

MsWinEventLog product=* "process"=* action=*
| fields product, "process", action, object

Conditional Expression¶

Conditional expression within parenthesis () must be separated explicitly by or.

| chart count(label = delete or label = remove) as remove

Forward Slash Expression¶

Any expression after the forward slash must be enclosed within double quotes.

source_name = "/opt/immune/var/log/audit/webserver.log"
| chart count() by source_address

norm¶

| norm doable_mps=<dmps:'['0-9']'+>

| norm <:'\['><my_field:word><:'\]'> | chart count() by my_field

timechart¶

Limit does not work with timechart.

| timechart count() by col_type

Capturing normalized field values¶

Use norm on command to capture normalized field value in log search result.

Suppose the log search result consists of a log value pair

source_name = /opt/immune/var/log/benchmarker

Now, if you want to capture the first two words of the path,
you can write the query as follows:

| norm on source_name <capture:'\/opt\/immune'>

This feature works well with rex command too.

user=* | rex on user:\s+(?P<account>\S+)@(?P<domain>\S+)
| chart count() by account, domain | search account=*

In the example above, the rex command is used on a field which captures email addresses. The email address is then broken into account and domain using the corresponding regex.

Grok Patterns¶

Logpoint search recognizes the following Grok patterns.

General Patterns

Pattern name	Regular expression
USERNAME	[a-zA-Z0-9._-]+
USER	%{USERNAME}
INT	(?:[+-]?(?:[0-9]+))
BASE10NUM	(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:.[0-9]+)?)\|(?:.[0-9]+)))
NUMBER	(?:%{BASE10NUM})
BASE16NUM	(?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
BASE16FLOAT	\b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:.[0-9A-Fa-f]*)?)\|(?:.[0-9A-Fa-f]+)))\b
POSINT	\b(?:[1-9][0-9]*)\b
NONNEGINT	\b(?:[0-9]+)\b
WORD	\b\w+\b
NOTSPACE	\S+
SPACE	\s*
DATA	.*?
GREEDYDATA	.*
QUOTEDSTRING	(?>(?<!\)(?>”(?>.\|[^"]+)+”\|”“\|(?>'(?>\\.\|[^\\']+)+')\|’‘\|(?>`(?>.\|[^`]+)+`)\|``))
UUID	[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
DOMAINTLD	[a-zA-Z]+
EMAIL	%{NOTSPACE}@%{WORD}.%{DOMAINTLD}
QS	%{QUOTEDSTRING}

Networking-related Patterns

Pattern name	Regular expression
MAC	(?:%{CISCOMAC}\|%{WINDOWSMAC}\|%{COMMONMAC})
CISCOMAC	(?:(?:[A-Fa-f0-9]{4}.){2}[A-Fa-f0-9]{4})
WINDOWSMAC	(?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC	(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6	((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}\|:))\|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}\|((25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)(.(25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)){3})\|:))\|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})\|:((25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)(.(25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)){3})\|:))\|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})\|((:[0-9A-Fa-f]{1,4})?:((25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)(.(25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)){3}))\|:))\|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})\|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)(.(25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)){3}))\|:))\|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})\|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)(.(25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)){3}))\|:))\|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})\|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)(.(25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)){3}))\|:))\|(:(((:[0-9A-Fa-f]{1,4}){1,7})\|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)(.(25[0-5]\|2[0-4]d\|1dd\|[1-9]?d)){3}))\|:)))(%.+)?
IPV4	(?<![0-9])(?:(?:25[0-5]\|2[0-4][0-9]\|[0-1]?[0-9]{1,2})[.](?:25[0-5]\|2[0-4][0-9]\|[0-1]?[0-9]{1,2})[.](?:25[0-5]\|2[0-4][0-9]\|[0-1]?[0-9]{1,2})[.](?:25[0-5]\|2[0-4][0-9]\|[0-1]?[0-9]{1,2}))(?![0-9])
IP	(?:%{IPV6}\|%{IPV4})
HOSTNAME	b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(.?\|b)
HOST	%{HOSTNAME}
IPORHOST	(?:%{HOSTNAME}\|%{IP})
HOSTPORT	%{IPORHOST}:%{POSINT}

Path-related patterns

Pattern name	Regular expression
PATH	(?:%{UNIXPATH}\|%{WINPATH})
UNIXPATH	(?>/(?>[w_%!$@:.,-]+\|.)*)+
TTY	(?:/dev/(pts\|tty([pq])?)(w+)?/?(?:[0-9]+))
WINPATH	(?>[A-Za-z]+:\|\)(?:\[^\?])+
URIPROTO	[A-Za-z]+(+[A-Za-z+]+)?
URIHOST	%{IPORHOST}(?::%{POSINT:port})?
URIPATH	(?:/[A-Za-z0-9$.+!’(){},~:;=@#%_-])+
URIPARAM	?[A-Za-z0-9$.+!’\|(){},~@#%&/=:;_?-[]]
URIPATHPARAM	%{URIPATH}(?:%{URIPARAM})?
URI	%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})? (?:%{URIPATHPARAM})?

Date and time patterns

Pattern name	Regular expression
MONTH	b(?:Jan(?:uary)?\|Feb(?:ruary)?\|Mar(?:ch)?\|Apr(?:il)?\|May\|Jun(?:e)?\|Jul(?:y)? \|Aug(?:ust)?\|Sep(?:tember)?\|Oct(?:ober)?\|Nov(?:ember)?\|Dec(?:ember)?)b
MONTHNUM	(?:0?[1-9]\|1[0-2])
MONTHNUM2	(?:0[1-9]\|1[0-2])
MONTHDAY	(?:(?:0[1-9])\|(?:[12][0-9])\|(?:3[01])\|[1-9])
DAY	(?:Mon(?:day)?\|Tue(?:sday)?\|Wed(?:nesday)?\|Thu(?:rsday)?\|Fri(?:day)? \|Sat(?:urday)?\|Sun(?:day)?)
YEAR	(?>dd){1,2}
HOUR	(?:2[0123]\|[01]?[0-9])
MINUTE	(?:[0-5][0-9])
SECOND	(?:(?:[0-5]?[0-9]\|60)(?:[:.,][0-9]+)?)
TIME	(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
DATE_US	%{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU	%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE	(?:Z\|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND	(?:%{SECOND}\|60)
TIMESTAMP_ISO8601	%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE} (?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE	%{DATE_US}\|%{DATE_EU}
DATESTAMP	%{DATE}[- ]%{TIME}
TZ	(?:[PMCE][SD]T\|UTC)
DATESTAMP_RFC822	%{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_RFC2822	%{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
DATESTAMP_OTHER	%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
DATESTAMP_EVENTLOG	%{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}

Syslog patterns

Pattern name	Regular expression
SYSLOGTIMESTAMP	%{MONTH} +%{MONTHDAY} %{TIME}
PROG	(?:[w._/%-]+)
SYSLOGPROG	%{PROG:program}(?:[%{POSINT:pid}])?
SYSLOGFACILITY	<%{NONNEGINT:facility}.%{NONNEGINT:priority}>
HTTPDATE	%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
SYSLOGHOST	%{IPORHOST}

Log formats

Pattern name	Regular expression
SYSLOGBASE	%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG	%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] “(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?\|%{DATA:rawrequest})” %{NUMBER:response} (?:%{NUMBER:bytes}\|-)
COMBINEDAPACHELOG	%{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

Helpful?