Guide to GCP’s Logging Query Language

Introduction

Google Cloud Platform’s Logging Query Language (LQL) is a powerful tool for fetching real-time data from both Google Cloud and non-GCP resources. This guide aims to demystify LQL, making it accessible for both beginners and experienced users.

Accessing Logs Explorer

Begin by navigating to the Logs Explorer in your GCP console. This is your starting point for building queries, either manually or using filters.

Basic Operations in Logs Explorer

The Logs Explorer UI supports:

• Text string searches across fields
• Filter menu toggling
• Advanced query composition
• Various query execution operations (view, run, edit, save, etc.)

Moving Beyond UI Constraints

For specific queries, you may need to go beyond the UI. Google’s guide on LQL is a great resource for understanding the syntax, which includes SQL-like operations.

Understanding Queries and Comparisons

• A query includes any line of code with an expression.
• A comparison involves values or Boolean comparisons.

Example String-Based Comparison

resource.type = "gce_instance" AND resource.labels.project_id = "project_one"

This resembles a SQL query, providing a familiar framework for those experienced in SQL.

LQL Operators

LQL supports several operators similar to SQL, facilitating diverse and complex queries:

• = (equal)
• != (not equal)
• >, <, >=, <= (numeric ordering)
• : (matches any substring)
• =~ (regex search for a pattern)
• !~ (regex search not for a pattern)

Using Conditional Filtering

LQL’s regex capabilities enable filtering based on multiple conditions. For example:

resource.type = 'gce_instance' 
jsonPayload.message =~ "Error" AND jsonPayload.message =~ "failure"
severity = ERROR

Complex Searches

Chain conditions within parentheses for more detailed searches. This allows for nuanced query construction, similar to advanced SQL queries.

RegEx Searches

LQL’s regex syntax is versatile, allowing for case-insensitive searches, quotation searches, and complex boolean expressions.

searchlabels.subnetwork_name =~ "(?i)Subnet" # Case insensitive
marksjsonPayload.message =~ "label=\"project.*\"" # Quotation search
labels.pod_name =~ "(error|failure)" # BOOL expression
logName =~ "/my%2Proj$" # Query using anchors

Understanding Severity Levels

LQL lets you filter logs based on severity levels like DEFAULT, INFO, WARNING, ERROR, etc. Knowing the severity level helps in prioritizing issues.

Time Filtering in LQL

Time-based filtering is essential for temporal data analysis. LQL uses ISO format timestamps, allowing for precise time range queries.

Examples of Time Filters

-- ISO format timestamp using >= operator.
timestamp >= "2022-05-10T09:00:00Z"

-- Specific date range query.
timestamp >= "2022-05-10" AND timestamp <= "2022-05-13"

Key Takeaway: Know Your Metadata

The effectiveness of LQL depends on your understanding of the metadata. Knowing the specifics of what you’re querying is crucial to avoid overwhelming results.

LQL vs SQL

While simpler than SQL, LQL’s power lies in its application. Understanding your data’s intricacies is key to effective querying.

Tips for Python Users

If using the Python client, be mindful of specifying severity levels and other parameters to avoid excessive read requests errors.

Conclusion

LQL offers a simplified yet powerful approach to querying logs in GCP. By understanding your data and utilizing LQL’s features effectively, you can gain meaningful insights from your logs.