Logging, Monitoring, and Debugging

On This Page

Overview

There are a variety of ways in which you can monitor, log, and debug the execution of platform application services, tools, and APIs.

For further troubleshooting assistance, visit Iguazio Support.

Logging Application Services

The platform has a default tenant-wide log-forwarder application service named “Log Forwarder” for forwarding application-service logs. The logs are forwarded to an instance of the Elasticsearch open-source search and analytics engine by using the open-source Filebeat log-shipper utility. The log-forwarder service is disabled by default. To enable it, in the Services dashboard page, select to edit the Log Forwarder service; in the Custom Parameters tab, set the Elasticsearch URL parameter to an instance of Elasticsearch that will be used to store and index the logs; then, save and apply your changes to deploy the service.

Typically, the log-forwarder service should be configured to work with your own remote off-cluster instance of Elasticsearch.

Note
  • The default transfer protocol, which is used when the URL doesn’t begin with “http://” or “https://”, is HTTPS.
  • The log-forwarder service doesn’t support off-cluster Elasticsearch user authentication, regardless of whether you use an HTTP or HTTPS URL.
  • The default port, which is used when the URL doesn’t end with “:<port number>”, is port 80 for HTTP and port 443 for HTTPS.

In cloud platform environments you can also configure the log-forwarder service to work with a tenant-wide Elasticsearch service that uses a pre-deployed on-cluster vanilla installation of Elasticsearch. To do so, you first need to create such an Elasticsearch service: from the Services dashboard page, select to create a new service of type “Elasticsearch”, optionally edit the default configuration, and apply your changes to deploy the service. You can then configure the log-forwarder service to forward the application-service logs to this service by selecting your Elasticsearch service from the drop-down list of the Elasticsearch parameter.

In cloud platform environments, after enabling the log-forwarder service you can view the forwarded logs in the Logs page of the platform dashboard (provided you have the Service Admin management policy). Note that this page displays all logs from the configured log-forwarder Elasticsearch instance, regardless of their origin. It’s therefore recommended that you use a dedicated Elasticsearch instance for each platform cluster that enables log forwarding.

Warning
Note that you cannot delete an Elasticsearch service while it’s being used by the log-forwarder service.

Monitoring Application Service

The platform has a default tenant-wide monitoring application service named “Monitoring” for monitoring Nuclio serverless functions and gathering performance statistics. The monitoring service is disabled by default. The gathered statistics are displayed within a Nuclio dashboard that’s automatically added to all Grafana platform services that are created or restarted after the monitoring service is enabled. It’s recommended that before you enable the monitoring service, you create a shared Grafana service for viewing the monitoring service logs.

Kubernetes Tools

You can use the Kubernetes kubectl CLI from a platform web-shell or Jupyter Notebook application service to monitor, log, and debug your Kubernetes application cluster:

  • Use the get pods command to display information about the cluster’s pods:

    kubectl get pods
  • Use the logs command to view the logs for a specific pod; replace POD with the name of one of the pods returned by the get command:

    kubectl logs POD
Note
To run these commands from a web-shell service, the service must be configured with the “Log Reader” service account or higher. For more information about the web-shell service accounts, see the web-shell introduction.

For more information about the kubectl CLI, see the Kubernetes documentation.

Event Logs

The Events page of the dashboard displays different platform event logs:

  • The Event Log tab displays system event logs.
  • The Alerts tab displays system alerts.
  • The Audit tab displays a subset of the system events for audit purposes — security events (such as a failed login) and user actions (such as creation and deletion of a container).

The Events page is visible to users with the IT Admin management policy — who can view all event logs — or to users with the Security Admin management policy — who can view only the Audit tab.

Cluster Support Logs

Users with the IT Admin management policy can collect and download support logs for the platform clusters from the dashboard. Log collection is triggered for a data cluster, but the logs are collected from both the data and application cluster nodes.

You can trigger collection of cluster support-logs from the dashboard in one of two ways; (note that you cannot run multiple collection jobs concurrently):

  • On the Clusters page, select open the action menu () for a data cluster in the clusters table (Type = “Data”); then, select the Collect logs menu option.
  • On the Clusters page, select to display the Support Logs tab for a specific data cluster — either by selecting the Support logs option from the cluster’s action menu () or by selecting the data cluster and then selecting the Support Logs tab; then, select Collect Logs from the action toolbar.

You can view the status of all collection jobs and download archive files of the collected logs from the data-cluster’s Support Logs dashboard tab.

API Error Information

The platform APIs return error codes and error and warning messages to help you debug problems with your application. See, for example, the Error Information documentation in the Data-Service Web-API General Structure reference documentation.

See Also