Forwarding TKG logs to vRealize Log Insight Cloud using fluent-bit package
This is a walkthrough of fluent-bit v1.8.15 package configuration with vRealize Log Insight Cloud on a TKG cluster. The configuration in this post is taken from Sam McGeown's detailed post with fluent-bit package v1.7.5
Environment Details¶
TKG 1.6.0¶
- One management cluster (3 CP and 1 worker node)
- One workload cluster
- wld (3 CP and 3 worker nodes)
Version Details¶
tanzu version
version: v0.25.0
buildDate: 2022-08-25
sha: 6288c751-dirty
tanzu cluster list --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN TKR
wld default running 3/3 3/3 v1.23.8+vmware.2 <none> dev v1.23.8---vmware.2-tkg.1
mgmt-avi tkg-system running 3/3 1/1 v1.23.8+vmware.2 management dev v1.23.8---vmware.2-tkg.1
Available packages¶
tanzu package available list
NAME DISPLAY-NAME SHORT-DESCRIPTION LATEST-VERSION
<------ Truncated ------>
fluent-bit.tanzu.vmware.com fluent-bit Fluent Bit is a fast Log Processor and Forwarder 1.8.15+vmware.1-tkg.1
fluxcd-helm-controller.tanzu.vmware.com Flux Helm Controller Helm controller is one of the components in FluxCD GitOps toolkit. 0.21.0+vmware.1-tkg.1
<------ Truncated ------>
Generate a vRealize Log Insight Cloud API Key¶
- Generate a vRealize Log Insight Cloud API Key from
- API Key details
- Name -
tanzu-sup-api-key - URL - https://data.mgmt.cloud.vmware.com/le-mans/v1/streams/ingestion-pipeline-stream
- Key -
API Key
fluent-bit setup and configuration¶
Get management and workload cluster kubeconfig¶
tanzu management-cluster kubeconfig get --admin --export-file $HOME/mgmt-avi-kubeconfig
tanzu cluster kubeconfig get wld --admin --export-file $HOME/wld-kubeconfig
Setup environment variables and kubeconfig¶
export VRLIC_API_KEY=`Add your API Key here`
export ENV_PREFIX="tanzusup"
export MANAGEMENT_CLUSTER_NAME=$(tanzu cluster list --include-management-cluster -o json | jq '.[] | select(.roles[] | contains("management"))| .name' | tr -d '"')
export WORKLOAD_CLUSTER_NAME=$(tanzu cluster list -o json | jq .[].name | tr -d '"')
export TANZU_PACKAGE_NAME="fluent-bit.tanzu.vmware.com"
TANZU_PACKAGE_VERSION=$(tanzu package available list "${TANZU_PACKAGE_NAME}" -o json | jq .[].version | tail -n 1 | tr -d '"' )
export TANZU_PACKAGE_VERSION
export PACKAGE_USER_DEFINED_NAME="fluent-bit"
export TANZU_PACKAGE_NAMESPACE="tanzu-system-logging"
export MANAGEMENT_TANZU_KUBECONFIG="$HOME/$MANAGEMENT_CLUSTER_NAME-kubeconfig"
export WORKLOAD_TANZU_KUBECONFIG="$HOME/$WORKLOAD_CLUSTER_NAME-kubeconfig"
echo $ENV_PREFIX
echo $MANAGEMENT_CLUSTER_NAME
echo $WORKLOAD_CLUSTER_NAME
echo $TANZU_PACKAGE_NAME
echo $TANZU_PACKAGE_VERSION
echo $PACKAGE_USER_DEFINED_NAME
echo $TANZU_PACKAGE_NAMESPACE
echo $MANAGEMENT_TANZU_KUBECONFIG
echo $WORKLOAD_TANZU_KUBECONFIG
````
#### Output
```bash
echo $ENV_PREFIX
tanzusup
echo $MANAGEMENT_CLUSTER_NAME
mgmt-avi
echo $WORKLOAD_CLUSTER_NAME
wld
echo $TANZU_PACKAGE_NAME
fluent-bit.tanzu.vmware.com
echo $TANZU_PACKAGE_VERSION
1.8.15+vmware.1-tkg.1
echo $PACKAGE_USER_DEFINED_NAME
fluent-bit
echo $TANZU_PACKAGE_NAMESPACE
tanzu-system-logging
echo $MANAGEMENT_TANZU_KUBECONFIG
/root/mgmt-avi-kubeconfig
echo $WORKLOAD_TANZU_KUBECONFIG
/root/wld-kubeconfig
Generate fluent-bit default data values file¶
cat << EOF > $HOME/$PACKAGE_USER_DEFINED_NAME-default-values.yaml
namespace: "$TANZU_PACKAGE_NAMESPACE"
fluent_bit:
config:
service: |
[Service]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
outputs: |
[OUTPUT]
Name http
Match *
Host data.mgmt.cloud.vmware.com
Port 443
URI /le-mans/v1/streams/ingestion-pipeline-stream
Header Authorization Bearer $VRLIC_API_KEY
Format json
tls On
tls.verify On
inputs: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser cri
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[INPUT]
Name systemd
Tag kube_systemd.*
Path /var/log/journal
DB /var/log/flb_kube_systemd.db
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Read_From_Tail On
Strip_Underscores On
[INPUT]
Name tail
Tag apiserver_audit.*
Path /var/log/kubernetes/audit.log
Parser json
DB /var/log/flb_kube_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines On
[INPUT]
Name tail
Tag audit.*
Path /var/log/audit/audit.log
Parser logfmt
DB /var/log/flb_system_audit.db
Mem_Buf_Limit 50MB
Refresh_Interval 10
Skip_Long_Lines On
filters: |
[FILTER]
Name record_modifier
Match *
Record environment $ENV_PREFIX-tkgm
Record log_type kubernetes
Record tkg_cluster $ENV_PREFIX-$MANAGEMENT_CLUSTER_NAME
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
[FILTER]
Name modify
Match kube.*
Copy kubernetes k8s
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_Under kubernetes
parsers: |
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name docker-daemon
Format regex
Regex time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name logfmt
Format logfmt
[PARSER]
Name syslog-rfc5424
Format regex
Regex ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name syslog-rfc3164-local
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
Time_Keep On
[PARSER]
Name syslog-rfc3164
Format regex
Regex /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
Time_Key time
Time_Format %b %d %H:%M:%S
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name kube-custom
Format regex
Regex (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
streams: ""
plugins: ""
daemonset:
resources: { }
podAnnotations: { }
podLabels: { }
EOF
Install fluent-bit package¶
tanzu package install "${PACKAGE_USER_DEFINED_NAME}" \
--package-name "${TANZU_PACKAGE_NAME}" \
--version "${TANZU_PACKAGE_VERSION}" \
--values-file $HOME/$PACKAGE_USER_DEFINED_NAME-default-values.yaml \
--namespace "${TANZU_PACKAGE_NAMESPACE}" \
--kubeconfig "${MANAGEMENT_TANZU_KUBECONFIG}" \
--create-namespace
Package Installation Successful¶
kubectl get pkgi,apps -n tanzu-system-logging
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
packageinstall.packaging.carvel.dev/fluent-bit fluent-bit.tanzu.vmware.com 1.8.15+vmware.1-tkg.1 Reconcile succeeded 3m56s
NAME DESCRIPTION SINCE-DEPLOY AGE
app.kappctrl.k14s.io/fluent-bit Reconcile succeeded 3m53s 3m55s
kubectl get all -n tanzu-system-logging
NAME READY STATUS RESTARTS AGE
pod/fluent-bit-2ltgl 1/1 Running 0 4m34s
pod/fluent-bit-46gmw 1/1 Running 0 4m34s
pod/fluent-bit-mw9l7 1/1 Running 0 4m34s
pod/fluent-bit-x2ckt 1/1 Running 0 4m34s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/fluent-bit 4 4 4 4 4 <none> 4m34s
VRLIC Log view¶
- Example - Filter logs by cluster and
kube-systemnamespace
