The problem's critical requirements are: centralized collection of all audit logs (including Data Access logs) from a production folder to a central BigQuery dataset, with long-term retention, and most importantly, the ability to intercept and override any project-level log sinks to prevent duplicate storage or misrouting.
Aggregated Log Sink at Folder Level: To centralize logs from all current and future projects within the "production folder," an aggregated sink configured at the folder level is the correct approach. Logs generated in child projects will flow up to the folder level and be matched by this sink.
Extract Reference: "Aggregated exports allow you to export logs from multiple Google Cloud projects, folders, or your entire organization. An aggregated export can include all logs from all included resources, or you can use queries to include only specific logs." (Google Cloud Documentation: "Route logs to supported destinations | Cloud Logging" -
Intercepting Sink (--intercept-logs / overrideDestinations): This is the crucial feature to meet the "intercept and override" requirement. When an aggregated sink is configured as an "intercepting" sink, any log entries that match its filter are immediately routed to its destination and are not processed by any lower-level sinks (e.g., project-level sinks). This ensures that logs are not inadvertently routed elsewhere and prevents duplicate storage.
Extract Reference: "An intercepting sink is an aggregated sink that, if it includes the overrideDestinations field set to true, stops matched log entries from propagating to lower-level sinks in the Cloud Logging resource hierarchy." (Google Cloud Documentation: "Route logs to supported destinations | Cloud Logging" -
BigQuery Destination and IAM Permissions: BigQuery is specified as the long-term retention destination. The sink's writer_identity (a service account automatically created for the sink) needs appropriate IAM permissions (e.g., BigQuery Data Editor or BigQuery User) on the target BigQuery dataset to write logs.
Inclusion Filter for Audit Logs: An inclusion filter is necessary to ensure only the required audit logs, including Data Access logs (logName:"cloudaudit.googleapis.com" OR logName:"data_access"), are routed.
Let's evaluate the other options:
A. Standard aggregated log sink... Logs Bucket Writer: A standard aggregated sink does not intercept or override lower-level sinks. Also, Logs Bucket Writer role is for Cloud Logging buckets, not BigQuery. The correct role would be for BigQuery.
B. Log sink in each production project: This is not a centralized solution and would require manual configuration for every project, which is inefficient and error-prone for "all current and future projects." It also doesn't provide any override mechanism.
C. Aggregated log sink at the organization level... configure a log view: While an organization-level sink offers broad centralization, if the requirement is specifically for a production folder and not the entire organization, a folder-level intercepting sink is more targeted. A log view is for displaying logs, not for routing or overriding. The --include-children flag is implied for aggregated sinks but doesn't provide the intercepting behavior.
Therefore, creating an intercepting aggregated log sink at the production folder level, configured to send audit logs to BigQuery, is the precise solution to meet all the stated requirements, especially the crucial "intercept and override" condition.
