diff --git a/docs/audit-log-filter-compression-encryption.md b/docs/audit-log-filter-compression-encryption.md index d1601454366..6fe2e363b10 100644 --- a/docs/audit-log-filter-compression-encryption.md +++ b/docs/audit-log-filter-compression-encryption.md @@ -31,6 +31,8 @@ The following audit log filter functions are used with encryption: The `audit_log_filter.password_history_keep_days` variable is used with encryption. If the variable is not zero (0), invoking `audit_log_encryption_password_set()` causes the expiration of archived audit log passwords. +When an archived password expires, the component removes it from the keyring. If a password is removed from the keyring, you cannot decrypt audit log files that were encrypted with that password. However, passwords that are still in use for rotated audit log files do not expire, even if they are past the expiration date specified by `password_history_keep_days`. + When the component starts with encryption enabled, the component checks if the keyring has an audit log filter encryption password. If no password is found, the component generates a random password and stores this password in the keyring. Use `audit_log_encryption_password_get()` to review this password. If compression and encryption are enabled, the component applies compression before encryption. If you must manually recover a file with both settings, first decrypt the file and then uncompress the file. @@ -52,8 +54,8 @@ This function gets the encryption password, and the iterations count and returns Get the keyring password: -```mysql -mysql> SELECT audit_log_encryption_password_get('audit-log-20190414T223342-2'); +```sql +SELECT audit_log_encryption_password_get('audit-log-20190414T223342-2'); ``` The return value of this function may look like the following: diff --git a/docs/audit-log-filter-formats.md b/docs/audit-log-filter-formats.md index 8a67ca57ed1..ffdb2e427c7 100644 --- a/docs/audit-log-filter-formats.md +++ b/docs/audit-log-filter-formats.md @@ -16,8 +16,8 @@ Set with the `audit_log_filter.format` system variable at startup. The available By default, the file contents in the new-style XML format are not compressed or encrypted. -Changing the `audit_log_filter.format`, you should also change -the `audit_log_filter.file` name. For example, changing the `audit_log_filter.format` +When changing the `audit_log_filter.format`, it is recommended that you also change +the `audit_log_filter.file` name. For example, when changing the `audit_log_filter.format` to JSON, change the `audit_log_filter.file` to `audit.json`. If you don't change the `audit_log_filter.file` name, then all audit log filter files have the same base name and you won't be able to easily find when the format changed. diff --git a/docs/audit-log-filter-migration.md b/docs/audit-log-filter-migration.md new file mode 100644 index 00000000000..a1259675ff8 --- /dev/null +++ b/docs/audit-log-filter-migration.md @@ -0,0 +1,637 @@ +# Migrating from 8.0 Plugin to 8.4 Component + +This guide helps you migrate from the Percona Server 8.0 audit log filter plugin to the 8.4 audit log filter component. + +## Overview of changes + +### Architecture change + +8.0: Audit log filter was implemented as a plugin + +* Installed via `INSTALL PLUGIN` + +* Plugin-specific variables and functions + +* Coexisted with older `audit_log` plugin + +8.4: Audit log filter is implemented as a component + +* Installed via `INSTALL COMPONENT` + +* Component-based variables (`audit_log_filter.*`) + +* Plugin was replaced by the component in 8.4 + +### Key differences + +| Aspect | 8.0 Plugin | 8.4 Component | +|--------|-----------|---------------| +| Installation | `INSTALL PLUGIN` | `INSTALL COMPONENT` or installation script | +| Variables | `audit_log_filter_*` (underscore) | `audit_log_filter.*` (dot notation) | +| Format | OLD, NEW, JSON, CSV | OLD, NEW, JSON (CSV removed) | +| Wildcards | Limited support | Full support in 8.4.4+ | +| Functions | Plugin functions | Component functions (similar names) | + +### Removed features + +* CSV format: No longer available in 8.4 + +* Plugin installation method: `INSTALL PLUGIN` no longer works + +### Note about the deprecated audit_log plugin + +In Percona Server for MySQL 8.4.7-7 and later, the older `audit_log` plugin (not to be confused with `audit_log_filter`) has been reintroduced but is marked as deprecated and will be removed in a future release. The `audit_log_filter` component is the recommended replacement for both: + +* The `audit_log_filter` plugin (replaced in 8.4) +* The deprecated `audit_log` plugin (deprecated in 8.4.7-7) + +If you are currently using the deprecated `audit_log` plugin, you should migrate to the `audit_log_filter` component. The component provides equivalent functionality with enhanced flexibility, performance, and filtering capabilities. + +**Important:** Do not install both the `audit_log` plugin and the `audit_log_filter` component simultaneously. They use different configuration variables and options, and running both can cause conflicts. + +## Pre-migration checklist + +Before starting migration, complete these steps: + + +- [ ] Backup current configuration + * Document all `audit_log_*` variables + + * Export filter definitions from `mysql.audit_log_filter` table + + * Export user assignments from `mysql.audit_log_user` table + + * Backup configuration files (my.cnf) + +- [ ] Review current setup + + * List all active filters + + * Document user filter assignments + + * Note any custom configurations + + * Identify CSV format usage (if any) + +- [ ] Plan migration + + * Schedule maintenance window + + * Plan for server restart (required) + + * Prepare rollback procedure + + * Test in non-production environment first + +- [ ] Verify prerequisites + + * Percona Server 8.4 is installed + + * Installation script is available + + * Required privileges are available + + * Sufficient disk space for logs + +## Step-by-step migration procedure + +### Step 1: Backup current configuration + +Export filter definitions: + +```sql +-- Connect to 8.0 server +SELECT * FROM mysql.audit_log_filter INTO OUTFILE '/tmp/audit_filters_backup.txt'; +SELECT * FROM mysql.audit_log_user INTO OUTFILE '/tmp/audit_users_backup.txt'; +``` + +Document variables: + +```sql +SHOW VARIABLES LIKE 'audit_log%'; +-- Save output to file +``` + +Backup configuration files: + +```bash +cp /etc/mysql/my.cnf /etc/mysql/my.cnf.backup +``` + +### Step 2: Install 8.4 component + +Using installation script (recommended): + +```bash +mysql -u root -p -D mysql < /path/to/mysql/share/audit_log_filter_linux_install.sql +``` + +Verify installation: + +```sql +-- Check component is installed +SELECT * FROM mysql.component WHERE component_urn LIKE '%audit_log_filter%'; + +-- Verify tables exist +SHOW TABLES IN mysql LIKE 'audit%'; +``` + +### Step 3: Convert variable names + +Update all variable names from plugin format to component format. + +Variable name mapping: + +| 8.0 Plugin Variable | 8.4 Component Variable | Notes | +|---------------------|------------------------|-------| +| `audit_log_filter_database` | `audit_log_filter.database` | Database for tables | +| `audit_log_filter_file` | `audit_log_filter.file` | Log file name | +| `audit_log_filter_format` | `audit_log_filter.format` | Log format (CSV removed) | +| `audit_log_filter_rotate_on_size` | `audit_log_filter.rotate_on_size` | Rotation size | +| `audit_log_filter_buffer_size` | `audit_log_filter.buffer_size` | Buffer size | + +Update configuration file: +```ini +# Old 8.0 format +audit_log_filter_database=mysql +audit_log_filter_file=audit_filter.log +audit_log_filter_format=NEW + +# New 8.4 format +audit_log_filter.database=mysql +audit_log_filter.file=audit_filter.log +audit_log_filter.format=NEW +``` + +### Step 4: Convert include/exclude variables to JSON filters + +If you used `audit_log_include_*` or `audit_log_exclude_*` variables, convert them to JSON filters. + +Example conversion: + +8.0 Configuration: +```ini +audit_log_include_accounts = admin@localhost, dba@% +audit_log_exclude_databases = test, temp +audit_log_include_commands = SELECT, INSERT, UPDATE, DELETE +``` + +8.4 JSON Filter: +```json +{ + "filter": { + "class": [ + { + "name": "general", + "user": ["admin", "dba"], + "database": ["test", "temp"], + "negate": true, + "event": [{"name": "command"}] + } + ] + } +} +``` + +Create filter: +```sql +SELECT audit_log_filter_set_filter('migrated_filter', '{ + "filter": { + "class": [ + { + "name": "general", + "user": ["admin", "dba"], + "database": ["test", "temp"], + "negate": true, + "event": [{"name": "command"}] + } + ] + } +}'); +``` + +See [Filter Migration Guide](filter-audit-log-filter-files.md#translating-80-includeexclude-variables-to-json-filters) for detailed conversion examples. + +### Step 5: Migrate existing filters + +If you have existing filters in `mysql.audit_log_filter` table: + +Option 1: Filters are compatible (recommended) + +* If filters are already in JSON format, they should work as-is + +* Verify filters after migration + +* Test functionality + +Option 2: Recreate filters + +* Export filter definitions + +* Recreate using `audit_log_filter_set_filter()` + +* Verify each filter works correctly + +Migrate user assignments: +```sql +-- Reassign filters to users +-- Use data from backup or mysql.audit_log_user table + +SELECT audit_log_filter_set_user('admin@localhost', 'filter_name'); +SELECT audit_log_filter_set_user('user@%', 'filter_name'); +``` + +### Step 6: Handle CSV format (if used) + +If you were using CSV format in 8.0: + +Options: + + +1. Switch to JSON format (recommended for programmatic access) + +2. Switch to NEW XML format (human-readable) + +3. Switch to OLD XML format (compatibility) + +Migration: +```sql +-- Change format to JSON +SET GLOBAL audit_log_filter.format = 'JSON'; +SET GLOBAL audit_log_filter.file = 'audit_filter.json'; +-- Restart server +``` + +Note: Old CSV format files cannot be converted. Only new logs use the new format. + +### Step 7: Update wildcard usage (8.4.4+) + +If you're on 8.4.4 or later, you can use wildcards in `audit_log_filter_set_user()`: + +```sql +-- Old method (specific hosts) +SELECT audit_log_filter_set_user('user@192.168.1.1', 'filter'); +SELECT audit_log_filter_set_user('user@192.168.1.2', 'filter'); + +-- New method (wildcard - 8.4.4+) +SELECT audit_log_filter_set_user('user@192.168.1.%', 'filter'); +``` + +### Step 8: Test functionality + +Verify filters work: +```sql +-- Check filters +SELECT * FROM mysql.audit_log_filter; + +-- Check user assignments +SELECT * FROM mysql.audit_log_user; + +-- Test filter +SELECT audit_log_filter_set_filter('test_filter', '{"filter": {"log": true}}'); +SELECT audit_log_filter_set_user('%', 'test_filter'); + +-- Execute test query +SELECT 1; + +-- Verify logging (if JSON format) +SELECT audit_log_read(); +``` + +Verify log rotation: +```sql +-- Check rotation settings +SHOW VARIABLES LIKE 'audit_log_filter.rotate_on_size'; + +-- Test manual rotation +SELECT audit_log_rotate(); +``` + +### Step 9: Remove old plugin (optional) + +If the old plugin is still installed: + +```sql +-- Check if plugin exists +SELECT * FROM INFORMATION_SCHEMA.PLUGINS +WHERE PLUGIN_NAME LIKE 'audit_log_filter'; + +-- Uninstall plugin (if present) +UNINSTALL PLUGIN audit_log_filter; +``` + +Note: The plugin may have been automatically removed during 8.4 upgrade. + +### Step 10: Restart server + +Required for: + +* Variable changes (format, file, database, etc.) + +* Component activation + +* Configuration changes + +```bash +# Restart MySQL server +systemctl restart mysql +# Or +service mysql restart +``` + +Verify after restart: +```sql +-- Check component is loaded +SELECT * FROM mysql.component WHERE component_urn LIKE '%audit_log_filter%'; + +-- Check variables +SHOW VARIABLES LIKE 'audit_log_filter.%'; + +-- Test functionality +SELECT audit_log_filter_set_filter('test', '{"filter": {"log": true}}'); +``` + +## Configuration mapping reference + +### Variable name changes + +| 8.0 Plugin | 8.4 Component | Change Required | +|-----------|---------------|-----------------| +| `audit_log_filter_database` | `audit_log_filter.database` | Yes - change underscore to dot | +| `audit_log_filter_file` | `audit_log_filter.file` | Yes - change underscore to dot | +| `audit_log_filter_format` | `audit_log_filter.format` | Yes - change underscore to dot | +| `audit_log_filter_rotate_on_size` | `audit_log_filter.rotate_on_size` | Yes - change underscore to dot | +| `audit_log_filter_buffer_size` | `audit_log_filter.buffer_size` | Yes - change underscore to dot | + +### Removed variables + +These plugin variables are no longer available in 8.4: + +* `audit_log_include_accounts` → Use JSON filters with `user` field + +* `audit_log_exclude_accounts` → Use JSON filters with `user` field and `negate: true` + +* `audit_log_include_commands` → Use JSON filters with `event: {"name": "command"}` + +* `audit_log_exclude_commands` → Use JSON filters with `event` and `negate: true` + +* `audit_log_include_databases` → Use JSON filters with `database` field + +* `audit_log_exclude_databases` → Use JSON filters with `database` field and `negate: true` + +### Function name changes + +Function names remain the same, but behavior may differ: + +* `audit_log_filter_set_filter()` - Same name, same behavior + +* `audit_log_filter_set_user()` - Same name, enhanced wildcard support in 8.4.4+ + +* `audit_log_filter_flush()` - Same name, same behavior + +* `audit_log_filter_remove_filter()` - Same name, same behavior + +* `audit_log_filter_remove_user()` - Same name, same behavior + +## Common migration issues + +### Issue: Filters not working after migration + +Symptoms: + +* Filters are created but events are not logged + +* Functions return errors + +Causes: + +* Component not properly installed + +* Tables not created + +* Missing privileges + +Solutions: +1. Verify component installation: + ```sql + SELECT * FROM mysql.component WHERE component_urn LIKE '%audit_log_filter%'; + ``` + +2. Verify tables exist: + ```sql + SHOW TABLES IN mysql LIKE 'audit%'; + ``` + +3. Re-run installation script if needed: + ```bash + mysql -u root -p -D mysql < /path/to/mysql/share/audit_log_filter_linux_install.sql + ``` + +4. Check privileges: + ```sql + GRANT AUDIT_ADMIN ON *.* TO 'user'@'host'; + ``` + +### Issue: Variables not recognized + +Symptoms: + +* Server doesn't recognize variable names + +* Configuration errors + +Causes: + +* Using old plugin variable names (underscore format) + +* Variable not updated in configuration file + +Solutions: +1. Update variable names to component format (dot notation) +2. Update my.cnf or other configuration files +3. Restart server after changes + +### Issue: CSV format not available + +Symptoms: + +* Error when setting format to CSV + +* CSV format option missing + +Causes: + +* CSV format was removed in 8.4 + +Solutions: +1. Choose alternative format: + + * JSON (recommended for programmatic access) + + * NEW XML (human-readable) + + * OLD XML (compatibility) + +2. Update configuration: + ```sql + SET GLOBAL audit_log_filter.format = 'JSON'; + SET GLOBAL audit_log_filter.file = 'audit_filter.json'; + ``` + +3. Restart server + +### Issue: Permission errors + +Symptoms: + +* `ERROR 1227 (42000): Access denied` + +* Cannot create or modify filters + +Causes: + +* Missing `AUDIT_ADMIN` privilege + +* User doesn't have required permissions + +Solutions: +```sql +-- Grant required privilege +GRANT AUDIT_ADMIN ON *.* TO 'user'@'host'; +FLUSH PRIVILEGES; +``` + +### Issue: Filters not replicating correctly + +Symptoms: + +* Filters work on source but not replica + +* Replicated filters not active + +Causes: + +* Tables replicate but filters don't activate automatically + +* Missing `audit_log_filter_flush()` on replica + +Solutions: +1. On replica, call flush after replication: + ```sql + SELECT audit_log_filter_flush(); + ``` + +2. Or configure replication to ignore audit tables: + ```ini + replicate-ignore-table=mysql.audit_log_filter + replicate-ignore-table=mysql.audit_log_user + ``` + +## Rollback procedure + +The audit log filter plugin is not available in 8.4, so rollback to the plugin is not possible. If migration fails or causes issues, use one of these approaches: + +### Option 1: Fix component configuration (recommended) + +Instead of rolling back, fix the component configuration: + +1. Review error messages and logs to identify the issue + +2. Correct the configuration: + + * Fix variable names (ensure dot notation is used) + + * Recreate filters if needed + + * Verify user assignments + +3. Restart the server if configuration changes require it + +4. Verify functionality: + + * Test filters work + + * Verify logging occurs + + * Check variables are recognized + +### Option 2: Downgrade to 8.0 (if necessary) + +If you must use the plugin version, downgrade to Percona Server 8.0: + +**Prerequisites:** + +* Backup of current 8.4 configuration + +* Access to 8.0 installation + +* Plan for downtime during downgrade + +**Downgrade steps:** + +1. Uninstall the component: + + ```sql + UNINSTALL COMPONENT 'file://audit_log_filter'; + ``` + +2. Restore old configuration: + + * Restore my.cnf from backup + + * Restore filter definitions if needed + +3. Downgrade to Percona Server 8.0 following standard downgrade procedures + +4. Reinstall the audit log filter plugin: + + ```sql + INSTALL PLUGIN audit_log_filter SONAME 'audit_log_filter.so'; + ``` + +5. Verify functionality: + + * Test filters work + + * Verify logging occurs + + * Check variables are recognized + +**Note:** Downgrading is complex and may cause data compatibility issues. Fixing the component configuration is strongly recommended instead. + +## Post-migration verification + +After migration, verify everything works correctly: + + +- [ ] Component is installed and loaded + +- [ ] Tables exist and are accessible + +- [ ] Variables are recognized (dot notation) + +- [ ] Filters can be created + +- [ ] Filters can be assigned to users + +- [ ] Events are being logged + +- [ ] Log rotation works + +- [ ] Log reading works (if JSON format) + +- [ ] Performance is acceptable + +- [ ] No errors in error log + +## Additional resources + +* [Audit Log Filter Overview](audit-log-filter-overview.md) + +* [Writing Filter Definitions](write-filter-definitions.md) + +* [Filter Functions](filter-audit-log-filter-files.md) + +* [Component Upgrade Guide](upgrade-components.md) + +--8<--- "get-help-snip.md" diff --git a/docs/audit-log-filter-overview.md b/docs/audit-log-filter-overview.md index 5deb0008529..d189141ea72 100644 --- a/docs/audit-log-filter-overview.md +++ b/docs/audit-log-filter-overview.md @@ -1,24 +1,135 @@ # Audit Log Filter overview -The Audit Log Filter component allows you to monitor, log, and block a connection or query actively executed on the selected server. +The Audit Log Filter component allows you to monitor and log database activity on the selected server. The component can also block queries or connections that match specific security rules, providing an additional layer of protection. -Enabling the component produces a log file that contains a record of server activity. The log file has information on connections and databases accessed by that connection. +Enabling the component produces a log file that contains a record of server activity. The log file has information on connections and databases accessed by that connection. The component uses the `mysql` system database to store filter and user account data. Set the [`audit_log_filter.database`](audit-log-filter-variables.md#audit_log_filterdatabase) variable at server startup to select a different database. -The `AUDIT_ADMIN` privilege is required to enable users to manage the Audit Log Filter component. +For installation instructions, see [Install the audit log filter](install-audit-log-filter.md). For detailed information on writing filter definitions, see [Write audit_log_filter definitions](write-filter-definitions.md). -## Privileges +## What is a filter? -Define the privilege at runtime at the startup of the server. The associated Audit Log Filter privilege can be unavailable if the component is not enabled. +A filter is a set of rules that determines which database events are logged or blocked by the Audit Log Filter component. Filters are defined using JSON and specify criteria such as: -### `AUDIT_ADMIN` +* Which users to monitor -This privilege is defined by the server and enables the user to configure the component. +* Which databases or tables to track -### `AUDIT_ABORT_EXEMPT` +* What types of operations to log (connections, queries, table access, etc.) + +* Whether to log or block matching events + +### How filters work + +When a database event occurs (such as a user connecting, executing a query, or accessing a table), the Audit Log Filter component checks if any active filters match that event. If a filter matches: + +* Logging: The event is written to the audit log file + +* Blocking: If the filter includes an `abort` item, the operation can be prevented from executing + +Filters are stored in the `mysql.audit_log_filter` table and can be assigned to specific user accounts or set as a default filter for all users. + +### Filter components + +A filter definition includes: + +* Event classes: Categories of events (connection, table_access, general, etc.) + +* Event subclasses: Specific types within a class (connect, disconnect, read, write, etc.) + +* Filter criteria: Conditions such as user, database, table, status, etc. + +* Action: Whether to log the event, block it, or both + +For more details on writing filter definitions, see [Write audit_log_filter definitions](write-filter-definitions.md). + +## Filter assignment and precedence + +The Audit Log Filter component uses a specific precedence order when determining which filter to apply to a user connection. + +### Filter assignment logic + +When a new connection is established, the component searches for a filter assignment in the following order: + +1. User-specific filter: If the user account has a filter directly assigned via `audit_log_filter_set_user()`, that filter is used. + +2. Default filter: If no user-specific filter exists, but a default filter is assigned to the `%` account, that filter is used. + +3. No filtering: If neither a user-specific nor default filter exists, no events are processed. + +### Default account filter + +The default account is represented by `%` as the account name. Assigning a filter to `%` makes it the default filter for all users who don't have a specific filter assigned. + +Example: + +```sql +-- Assign a default filter that logs all connection events +SELECT audit_log_filter_set_user('%', 'log_all_connections'); +``` + +### Filter precedence scenarios + +| Scenario | User Filter | Default Filter | Result | +|----------|------------|---------------|--------| +| User has specific filter | Assigned | Assigned | User's specific filter is used | +| User has no filter, default exists | None | Assigned | Default filter is used | +| Neither exists | None | None | No filtering (events not processed) | +| User filter removed | Removed | Assigned | Default filter is used for new connections | + +### Current sessions vs new connections + +Important behavior: + +* Filter changes take effect for new connections immediately + +* Current sessions continue using the filter that was active when they connected + +* To apply a new filter to current sessions, users must: + + * Execute `CHANGE_USER` command, or + + * Disconnect and reconnect + +Example: + +1. User `admin@localhost` connects → Uses filter `filter_A` + +2. Administrator changes user's filter to `filter_B` + +3. User's current session → Still uses `filter_A` -This privilege allows queries from a user account to always be executed. An `abort` item does not block them. This ability lets the user account regain access to a system if an audit is misconfigured. The query is logged due to the privilege. User accounts with the `SYSTEM_USER` privilege have the `AUDIT_ABORT_EXEMPT` privilege. +4. User disconnects and reconnects → Now uses `filter_B` + +### Removing filters + +When you remove a filter using `audit_log_filter_remove_filter()`: + +* The filter is unassigned from all users (including the default `%` account) + +* Current sessions are detached from the filter + +* New connections for affected users fall back to: + + * Default filter (if one exists), or + + * No filtering (if no default exists) + +### Best practices + +1. Always set a default filter for the `%` account to ensure all connections are logged +2. Test filter changes before deploying to production +3. Document filter assignments to track which users have specific filters +4. Use wildcards in `audit_log_filter_set_user()` (available starting from version 8.4.4, see [release notes](release-notes/8.4.4-4.md)) for flexible user matching + +## Blocking queries and connections + +The Audit Log Filter component can not only log events but also block queries or connections that match specific filter rules. This blocking capability provides an additional layer of security by preventing unauthorized or dangerous operations. + +When a filter includes an `abort` item set to `true`, matching queries or connections are blocked before execution. The blocked operation is still logged to the audit log, but the query does not execute. Users with the `AUDIT_ABORT_EXEMPT` privilege (or `SYSTEM_USER` privilege) are exempt from blocking, ensuring administrators can always regain access if filters are misconfigured. + +For detailed information on blocking syntax, examples, and best practices, see [Write audit_log_filter definitions](write-filter-definitions.md#blocking-queries-and-connections). ## Audit Log Filter tables @@ -38,3 +149,54 @@ The `audit_log_user` table stores account data and has the following column defi | USER | The account name of the user | | HOST | The account name of the host | | FILTERNAME | The account filter name | + +You can modify these tables directly using SQL, but you must call `audit_log_filter_flush()` after making changes for them to take effect. For more information on managing filters, see [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md). + +## Privileges + +The `AUDIT_ADMIN` privilege is required to enable users to manage the Audit Log Filter component. Define the privilege at runtime at the startup of the server. The associated Audit Log Filter privilege can be unavailable if the component is not enabled. + +### `AUDIT_ADMIN` + +The `AUDIT_ADMIN` privilege is defined by the server and enables the user to configure the component. Users with this privilege can: + +* Create, modify, and remove filters + +* Assign filters to user accounts + +* Manage audit log configuration + +* Rotate and manage log files + +### `AUDIT_ABORT_EXEMPT` + +The `AUDIT_ABORT_EXEMPT` privilege allows queries from a user account to always be executed. An `abort` item does not block them. This exemption ability lets the user account regain access to a system if an audit is misconfigured. The query is logged due to the privilege. User accounts with the `SYSTEM_USER` privilege have the `AUDIT_ABORT_EXEMPT` privilege. + +## Privilege interactions + +The following table shows how privileges interact with audit log filter functionality: + +| Privilege | Can Configure Filters | Can Block Queries | Exempt from Blocks | Notes | +|-----------|---------------------|-------------------|-------------------|-------| +| `AUDIT_ADMIN` | Yes | Yes | No | Required for filter management functions | +| `AUDIT_ABORT_EXEMPT` | No | No | Yes | Queries always execute even if blocked by filter | +| `SYSTEM_USER` | No | No | Yes | Automatically has `AUDIT_ABORT_EXEMPT` privilege | +| `SUPER` | Yes (legacy) | Yes | No | Deprecated in MySQL 8.0+, use `AUDIT_ADMIN` instead | + +!!! note "Privilege precedence" + + Users with `AUDIT_ABORT_EXEMPT` (or `SYSTEM_USER`) can always execute queries, even when filters are configured to block them. This privilege precedence ensures administrators can regain access if filters are misconfigured. + +## Next steps + +Now that you understand the basics of the Audit Log Filter component: + +* [Install the audit log filter](install-audit-log-filter.md) - Set up the component + +* [Write audit_log_filter definitions](write-filter-definitions.md) - Learn how to create filters + +* [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md) - Manage filters and assignments + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) - Reference for all functions and variables + +--8<--- "get-help-snip.md" diff --git a/docs/audit-log-filter-restrictions.md b/docs/audit-log-filter-restrictions.md index 014d44f5822..ad2e248878d 100644 --- a/docs/audit-log-filter-restrictions.md +++ b/docs/audit-log-filter-restrictions.md @@ -26,3 +26,228 @@ The Audit Log Filter has the following general restrictions: Please notice that just changing the content of these tables (via replication channel) is not enough to automatically make changes to in-memory data structures in the `audit_log_filter` component that store information about active audit log filtering rules. However, this may happen after component reloading / server restart or manually calling `audit_log_filter_flush()`. * Filter only on string values. The audit log filter does not filter on integer values. All filter criteria must be specified as strings, even when the underlying value is numeric. For example, `connection_id` values must be specified as strings (for example, `"123"` rather than `123`), and status values must be specified as `"0"` or `"1"` rather than `0` or `1`. If you use integer values in your filter definition, you will see the error: `ERROR: Incorrect rule definition.` + +## Replication and cluster behavior + +### Replication scenarios + +The audit log filter tables (`mysql.audit_log_filter` and `mysql.audit_log_user`) are replicated by default, but filter rules stored in these tables do not automatically become active on replica servers. + +**Scenario 1: Source server creates filter** + +1. On source server: Administrator creates a filter and assigns it to users +2. Filter is stored in `mysql.audit_log_filter` and `mysql.audit_log_user` tables +3. Filter is immediately active on source server (in-memory) +4. Table changes replicate to replica server +5. On replica server: Filter exists in tables but is NOT active in memory +6. To activate on replica: Call `audit_log_filter_flush()` or restart server + +**Example:** +```sql +-- On source server +SELECT audit_log_filter_set_filter('log_all', '{"filter": {"log": true}}'); +SELECT audit_log_filter_set_user('%', 'log_all'); +-- Filter is immediately active + +-- On replica server (after replication) +-- Filter exists in tables but is not active +-- Must run: +SELECT audit_log_filter_flush(); +-- Now filter is active on replica +``` + +**Scenario 2: Direct table modification on source** + +1. On source server: Administrator modifies tables directly (INSERT/UPDATE/DELETE) +2. Changes replicate to replica +3. On source: Must call `audit_log_filter_flush()` to activate changes +4. On replica: Must also call `audit_log_filter_flush()` to activate changes + +**Recommended replication configuration:** + +To avoid conflicts and ensure independent filter configuration on each server, configure replication to ignore audit log filter tables: + +```ini +# In my.cnf on replica server +[mysqld] +replicate-ignore-table=mysql.audit_log_filter +replicate-ignore-table=mysql.audit_log_user +``` + +**Benefits:** +* Each server can have independent filter configuration +* No conflicts between source and replica filters +* Better control over audit logging on each server +* Avoids accidental filter activation from replication + +### Cluster scenarios + +When using audit log filter in a cluster environment: + +**Each server needs independent configuration:** +* Install the component on each server in the cluster +* Configure filters separately on each server +* Filters are not shared between cluster nodes + +**Aggregating logs:** +* Each server generates its own audit log files +* You must aggregate logs from all servers manually +* Consider using external log aggregation tools +* Ensure consistent time synchronization across servers + +**Example cluster setup:** +``` +Server 1 (Primary): audit_filter.log +Server 2 (Replica): audit_filter.log +Server 3 (Replica): audit_filter.log + +Aggregate all three files for complete audit trail +``` + +## What gets logged vs not logged + +### Top-level statements only + +The audit log filter logs only top-level SQL statements. Statements executed within stored procedures, functions, or triggers are not logged. + +**What gets logged:** +* `CALL stored_procedure_name();` → Logged (top-level statement) +* `SELECT * FROM table;` → Logged (top-level statement) +* `INSERT INTO table VALUES (...);` → Logged (top-level statement) + +**What does NOT get logged:** +* Statements inside stored procedures: + ```sql + CREATE PROCEDURE my_proc() + BEGIN + SELECT * FROM table1; -- NOT logged + INSERT INTO table2 ...; -- NOT logged + END; + ``` +* Statements inside triggers: + ```sql + CREATE TRIGGER my_trigger + BEFORE INSERT ON table1 + FOR EACH ROW + BEGIN + INSERT INTO table2 ...; -- NOT logged + END; + ``` +* Statements inside functions: + ```sql + CREATE FUNCTION my_func() RETURNS INT + BEGIN + SELECT COUNT(*) INTO @count FROM table1; -- NOT logged + RETURN @count; + END; + ``` + +**Why this limitation exists:** +* Performance: Logging every statement in procedures/triggers would generate excessive log volume +* Complexity: Nested statement execution makes it difficult to track context +* Focus: Top-level statements represent user-initiated actions + +**Workaround if full statement capture is needed:** +* Enable general query log for complete statement capture +* Use application-level logging +* Modify stored procedures to include logging statements + +### LOAD_DATA file content + +When using `LOAD DATA` statements: + +**What gets logged:** +* The `LOAD DATA` statement itself is logged +* File path may be logged (depending on configuration) +* Number of rows affected is logged + +**What does NOT get logged:** +* File contents are NOT logged (for security and privacy) +* Data values being loaded are NOT logged + +**Example:** +```sql +LOAD DATA INFILE '/path/to/data.csv' INTO TABLE customers; +``` + +**Logged information:** +* Statement: `LOAD DATA INFILE '/path/to/data.csv' INTO TABLE customers` +* User, host, database, table +* Execution status (success/failure) +* Rows affected + +**Not logged:** +* Contents of `/path/to/data.csv` +* Individual data values being inserted + +This behavior protects sensitive data that might be in the file and prevents log files from becoming excessively large. + +## Performance considerations + +### Impact of logging all events + +Logging all events can have significant performance impact: + +**Typical performance impact:** +* Query performance: 30-70% reduction in queries per second (QPS) when logging everything +* Disk I/O: Substantial increase in write operations +* CPU usage: Increased CPU usage for log processing +* Memory: Buffer usage for asynchronous logging + +**Recommendations:** +* Start with selective logging (specific users, databases, or event types) +* Monitor performance metrics using status variables +* Gradually increase logging scope if needed +* Use asynchronous logging strategy (default) + +### Impact of complex filters + +Complex filters with multiple conditions can have additional overhead: + +**Factors affecting performance:** +* Number of filter conditions +* Number of event classes being filtered +* Use of wildcards in filters +* Filter evaluation for each event + +**Best practices:** +* Keep filters as simple as possible +* Use specific conditions rather than broad wildcards when possible +* Test filter performance in non-production environment +* Monitor `audit_log_filter_events_filtered` to see filter effectiveness + +### Impact of blocking queries + +Blocking queries (using `abort: true`) has minimal additional overhead compared to logging: + +**Performance impact:** +* Similar to logging, but query execution is prevented +* Slightly faster than logging (no disk write needed for blocked queries) +* Network overhead for error messages to client + +**Considerations:** +* Blocking can affect application behavior +* Ensure applications handle blocked query errors gracefully +* Test blocking filters thoroughly before production deployment + +### Monitoring performance impact + +Use status variables to monitor performance: + +**Key metrics:** +* `audit_log_filter_events` - Total events processed +* `audit_log_filter_events_written` - Events written to log +* `audit_log_filter_events_filtered` - Events filtered out +* `audit_log_filter_events_lost` - Events lost (buffer overflow) +* `audit_log_filter_write_waits` - Buffer wait count (async mode) + +**Thresholds:** +* `audit_log_filter_events_lost > 0` → Increase buffer size or reduce logging +* `audit_log_filter_write_waits` increasing → Performance degradation, consider reducing logging scope +* High ratio of `events_filtered` to `events` → Filters are working effectively + +**Performance tuning:** +* If `events_lost > 0`: Increase `audit_log_filter.buffer_size` +* If high `write_waits`: Reduce logging scope or use `PERFORMANCE` strategy +* If high CPU usage: Use more selective filters +* If high disk I/O: Enable compression or reduce logging scope diff --git a/docs/audit-log-filter-security.md b/docs/audit-log-filter-security.md index 24800aee287..6ad1b7b67d7 100644 --- a/docs/audit-log-filter-security.md +++ b/docs/audit-log-filter-security.md @@ -10,3 +10,210 @@ that contains these files should be accessible only to the following: The files are not encrypted by default and may contain sensitive information. The default name for the file in the data directory is `audit_filter.log`. If needed, use the `audit_log_filter.file` system variable at server startup to change the location. Due to the log rotation, multiple audit log files may exist. + +## Encryption best practices + +### When to use encryption + +Enable encryption for audit log files when: +* Logs contain sensitive data (PII, financial information, etc.) +* Compliance requirements mandate encrypted audit logs +* Logs are stored in shared or untrusted storage +* Regulatory requirements (HIPAA, PCI DSS, etc.) require encryption + +### Enabling encryption + +**Step 1: Install keyring component** +```sql +INSTALL COMPONENT 'file://component_keyring_file'; +-- Or use another keyring component (vault, KMIP, AWS KMS) +``` + +**Step 2: Set encryption password** +```sql +SELECT audit_log_encryption_password_set('your_secure_password'); +``` + +**Step 3: Enable encryption (requires restart)** +```sql +SET GLOBAL audit_log_filter.encryption = 'AES'; +-- Restart server for change to take effect +``` + +### Key management + +**Using keyring components:** +* Store encryption passwords in keyring (recommended) +* Keyring provides secure password storage +* Supports password rotation +* Multiple keyring backends available (file, vault, KMIP, AWS KMS) + +**Password rotation:** +```sql +-- Set new password (old password is archived) +SELECT audit_log_encryption_password_set('new_secure_password'); + +-- Old password remains available for reading old files +-- New password is used for new log files +``` + +**Password history:** +```sql +-- Configure password retention (in days) +SET GLOBAL audit_log_filter.password_history_keep_days = 90; +-- Passwords older than 90 days may be removed +-- But remain available if needed for old files +``` + +### Encryption performance + +**Impact:** +* Minimal performance overhead (typically < 5%) +* Encryption happens during log write operations +* CPU usage increases slightly +* Disk I/O remains similar + +**Best practices:** +* Use hardware acceleration if available +* Monitor performance after enabling encryption +* Test in non-production environment first + +## Access control recommendations + +### File permissions + +**Recommended permissions:** +```bash +# Audit log directory +chmod 750 /var/lib/mysql/audit +chown mysql:mysql /var/lib/mysql/audit + +# Audit log files +chmod 640 /var/lib/mysql/audit/audit_filter.log* +chown mysql:mysql /var/lib/mysql/audit/audit_filter.log* +``` + +**Permissions breakdown:** +* Directory: `750` - Owner (mysql) can read/write/execute, group can read/execute, others have no access +* Files: `640` - Owner (mysql) can read/write, group can read, others have no access + +### Directory access + +**Who should have access:** +* MySQL server process (must have write access) +* Database administrators (read access for log analysis) +* Security team (read access for audits) +* Backup systems (read access for backups) + +**Who should NOT have access:** +* Application users +* General system users +* Unauthorized personnel + +### Protecting encrypted files + +**Additional security measures:** +1. **Separate storage:** Store encrypted logs on separate, secure storage +2. **Backup encryption:** Ensure backups are also encrypted +3. **Access logging:** Log who accesses audit log files +4. **Network security:** If logs are on network storage, use encrypted connections +5. **Physical security:** Secure physical access to log storage + +### Log file integrity + +**Verification:** +* Use file system integrity checks +* Monitor for unauthorized modifications +* Implement file checksums/hashes +* Regular integrity audits + +**Tamper detection:** +* Encrypted files are harder to tamper with +* Monitor file modification times +* Compare file sizes and checksums +* Alert on unexpected changes + +## Compliance considerations + +### PCI DSS compliance + +**Requirements:** +* Audit all access to cardholder data +* Log all administrative access +* Protect audit logs from modification +* Retain logs for specified period + +**How audit log filter helps:** +* Logs all database access +* Encryption protects log integrity +* Access controls prevent unauthorized modification +* Supports retention requirements + +### HIPAA compliance + +**Requirements:** +* Audit access to protected health information (PHI) +* Log all database access +* Protect audit logs +* Retain logs as required + +**How audit log filter helps:** +* Logs access to PHI-containing tables +* Encryption ensures log security +* Access controls meet HIPAA requirements +* Supports audit trail requirements + +### SOX compliance + +**Requirements:** +* Audit financial data access +* Log all changes to financial data +* Protect audit logs +* Demonstrate compliance + +**How audit log filter helps:** +* Logs all financial database operations +* Tracks who accessed what data +* Encryption protects audit trail +* Provides compliance evidence + +### General compliance best practices + +1. **Define audit requirements:** Identify what must be logged +2. **Configure filters:** Set up filters to meet requirements +3. **Enable encryption:** Protect audit logs +4. **Control access:** Limit who can view/modify logs +5. **Retain logs:** Configure retention policies +6. **Monitor compliance:** Regularly verify logging is working +7. **Document procedures:** Document audit log management procedures + +### Retention requirements + +**Configure retention:** +```sql +-- Time-based retention (30 days) +SET GLOBAL audit_log_filter.prune_seconds = 2592000; + +-- Size-based retention (10GB total) +SET GLOBAL audit_log_filter.max_size = 10737418240; +``` + +**Compliance considerations:** +* Ensure retention meets regulatory requirements +* Some regulations require longer retention (7 years for SOX) +* Balance retention with storage costs +* Archive old logs before pruning if needed + +### Immutability considerations + +**Challenges:** +* MySQL audit logs are files that can be modified +* File system permissions help but don't guarantee immutability + +**Mitigation strategies:** +1. **Encryption:** Makes tampering more difficult +2. **Access controls:** Limit who can modify files +3. **Backup immediately:** Copy logs to immutable storage +4. **External logging:** Send logs to external SIEM systems +5. **Write-once storage:** Use WORM (Write Once Read Many) storage +6. **Integrity checks:** Regular checksums/hash verification diff --git a/docs/audit-log-filter-variables.md b/docs/audit-log-filter-variables.md index 199d70412fc..1cd9dcfb44f 100644 --- a/docs/audit-log-filter-variables.md +++ b/docs/audit-log-filter-variables.md @@ -695,49 +695,285 @@ This read-only variable defines the `priority` value for the syslog. This variab The audit log filter component exposes status variables. These variables provide information on the operations. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NameDescription
audit_log_filter_current_sizeThe current size of the audit log filter file. If the log is rotated, the size is reset to 0.
audit_log_filter_direct_writesIdentifies when the log_strategy_type = ASYNCHRONOUS and messages bypass the write buffer and are written directly to the log file.
audit_log_filter_max_drop_sizeIn the performance logging mode, the size of the largest dropped event.
audit_log_filter_eventsThe number of audit log filter events.
audit_log_filter_events_filteredThe number of filtered audit log filter component events.
audit_log_filter_events_lostIf the event is larger than the available audit log filter buffer space, the event is lost.
audit_log_filter_events_writtenThe number of audit log filter events written.
audit_log_filter_total_sizeThe total size of the events written to all audit log filter files. The number increases even when a log is rotated.
audit_log_filter_write_waitsIn the asynchronous logging mode, the number of times an event waited for space in the audit log filter buffer.
+| Name | Description | +|------|-------------| +| `audit_log_filter_current_size` | The current size of the audit log filter file. If the log is rotated, the size is reset to 0. | +| `audit_log_filter_direct_writes` | Identifies when the `log_strategy_type` = ASYNCHRONOUS and messages bypass the write buffer and are written directly to the log file. | +| `audit_log_filter_max_drop_size` | In the performance logging mode, the size of the largest dropped event. | +| `audit_log_filter_events` | The number of audit log filter events. | +| `audit_log_filter_events_filtered` | The number of filtered audit log filter component events. | +| `audit_log_filter_events_lost` | If the event is larger than the available audit log filter buffer space, the event is lost. | +| `audit_log_filter_events_written` | The number of audit log filter events written. | +| `audit_log_filter_total_size` | The total size of the events written to all audit log filter files. The number increases even when a log is rotated. | +| `audit_log_filter_write_waits` | In the asynchronous logging mode, the number of times an event waited for space in the audit log filter buffer. | + +## Monitoring and interpreting status variables + +Status variables provide valuable insights into audit log filter performance and health. Use them to monitor, troubleshoot, and optimize your audit logging configuration. + +### Key metrics and their meanings + +#### `audit_log_filter_events` + +**What it measures:** Total number of events processed by the audit log filter component. + +**Normal values:** Varies based on database activity. Should increase steadily during normal operation. + +**When to be concerned:** +* Value is 0 when you expect events to be logged → Check filter configuration +* Sudden drop to 0 → Component may be disabled or filters misconfigured + +**How to use:** +```sql +-- Monitor event count over time +SHOW STATUS LIKE 'audit_log_filter_events'; +-- Compare with previous values to track activity +``` + +#### `audit_log_filter_events_written` + +**What it measures:** Number of events actually written to log files. + +**Normal values:** Should be less than or equal to `audit_log_filter_events` (some events may be filtered out). + +**When to be concerned:** +* Much lower than `events` → Many events are being filtered (may be expected) +* Zero when `events` is non-zero → Writing may be disabled or file issues + +**How to use:** +```sql +-- Check write ratio +SELECT + VARIABLE_VALUE as events_written +FROM performance_schema.global_status +WHERE VARIABLE_NAME = 'audit_log_filter_events_written'; +``` + +#### `audit_log_filter_events_filtered` + +**What it measures:** Number of events filtered out (not written to log). + +**Normal values:** Depends on filter configuration. Higher values indicate filters are working. + +**When to be concerned:** +* Very high ratio of filtered to total events → Filters may be too restrictive +* Zero when you expect filtering → Filters may not be working correctly + +**How to use:** +```sql +-- Calculate filter effectiveness +SELECT + (SELECT VARIABLE_VALUE FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_events_filtered') / + (SELECT VARIABLE_VALUE FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_events') * 100 +AS filter_percentage; +``` + +#### `audit_log_filter_events_lost` + +**What it measures:** Number of events lost due to buffer overflow. + +**Normal values:** Should be 0. Any non-zero value indicates a problem. + +**When to be concerned:** +* **Any value > 0** → Events are being lost, immediate action required +* Increasing value → Buffer is consistently too small + +**How to fix:** +1. Increase buffer size: + ```sql + -- Check current buffer size + SHOW VARIABLES LIKE 'audit_log_filter.buffer_size'; + -- Increase in my.cnf (requires restart) + audit_log_filter.buffer_size = 2097152 -- 2MB + ``` + +2. Reduce logging scope (more selective filters) + +3. Use PERFORMANCE strategy (drops events instead of blocking): + ```sql + -- Set strategy to PERFORMANCE (requires restart) + audit_log_filter.strategy = PERFORMANCE + ``` + +**Example monitoring:** +```sql +-- Alert if events are being lost +SELECT + CASE + WHEN VARIABLE_VALUE > 0 THEN 'ALERT: Events being lost!' + ELSE 'OK: No events lost' + END as status +FROM performance_schema.global_status +WHERE VARIABLE_NAME = 'audit_log_filter_events_lost'; +``` + +#### `audit_log_filter_write_waits` + +**What it measures:** Number of times events waited for buffer space in asynchronous mode. + +**Normal values:** Should be low or 0. Occasional waits are acceptable under high load. + +**When to be concerned:** +* Consistently increasing → Performance degradation, buffer too small +* High values during normal operation → Need to optimize configuration + +**How to use:** +```sql +-- Monitor write waits +SHOW STATUS LIKE 'audit_log_filter_write_waits'; + +-- If consistently high: +-- 1. Increase buffer size +-- 2. Reduce logging scope +-- 3. Consider PERFORMANCE strategy +``` + +#### `audit_log_filter_current_size` + +**What it measures:** Current size of the active log file in bytes. + +**Normal values:** Varies based on activity and rotation settings. + +**When to be concerned:** +* Approaching `rotate_on_size` → Rotation will occur soon +* Very large without rotation → Rotation may not be configured + +**How to use:** +```sql +-- Check current file size +SHOW STATUS LIKE 'audit_log_filter_current_size'; + +-- Compare with rotation size +SHOW VARIABLES LIKE 'audit_log_filter.rotate_on_size'; + +-- Calculate percentage +SELECT + (SELECT VARIABLE_VALUE FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_current_size') / + (SELECT VARIABLE_VALUE FROM performance_schema.global_variables + WHERE VARIABLE_NAME = 'audit_log_filter_rotate_on_size') * 100 +AS rotation_percentage; +``` + +#### `audit_log_filter_total_size` + +**What it measures:** Total size of all events written across all log files (including rotated). + +**Normal values:** Continuously increases as events are logged. + +**When to be concerned:** +* Approaching `max_size` (if configured) → Pruning will occur +* Very large → May need to adjust pruning settings + +**How to use:** +```sql +-- Monitor total size +SHOW STATUS LIKE 'audit_log_filter_total_size'; + +-- Compare with max_size limit +SHOW VARIABLES LIKE 'audit_log_filter.max_size'; +``` + +#### `audit_log_filter_max_drop_size` + +**What it measures:** Size of the largest event that was dropped (PERFORMANCE strategy only). + +**Normal values:** 0 if PERFORMANCE strategy not used, or size of largest dropped event. + +**When to be concerned:** +* Large values → Very large events are being dropped +* May indicate queries with large result sets or long statements + +**How to use:** +```sql +-- Check if large events are being dropped +SHOW STATUS LIKE 'audit_log_filter_max_drop_size'; + +-- If using PERFORMANCE strategy and seeing drops: +-- Consider increasing buffer size or using ASYNCHRONOUS strategy +``` + +### Monitoring best practices + +1. **Regular monitoring:** + ```sql + -- Create monitoring query + SELECT + 'Events Processed' as metric, + VARIABLE_VALUE as value + FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_events' + UNION ALL + SELECT + 'Events Written', + VARIABLE_VALUE + FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_events_written' + UNION ALL + SELECT + 'Events Lost', + VARIABLE_VALUE + FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_events_lost'; + ``` + +2. **Set up alerts:** + * Alert if `events_lost > 0` + * Alert if `write_waits` increases rapidly + * Alert if `current_size` approaches rotation limit + +3. **Track trends:** + * Monitor values over time + * Identify patterns (peak hours, etc.) + * Adjust configuration based on trends + +4. **Performance tuning:** + * If `events_lost > 0`: Increase buffer size + * If high `write_waits`: Reduce logging scope or increase buffer + * If high CPU: Use more selective filters + * If high disk I/O: Enable compression + +### Troubleshooting with status variables + +**Problem: No events being logged** + +```sql +-- Check if events are being processed +SHOW STATUS LIKE 'audit_log_filter_events'; +-- If 0: Check if component is enabled, filters are configured + +-- Check if events are being written +SHOW STATUS LIKE 'audit_log_filter_events_written'; +-- If 0 but events > 0: Check file permissions, disk space +``` + +**Problem: Events being lost** + +```sql +-- Check lost events +SHOW STATUS LIKE 'audit_log_filter_events_lost'; +-- If > 0: Increase buffer size immediately + +-- Check buffer size +SHOW VARIABLES LIKE 'audit_log_filter.buffer_size'; +-- Increase if too small +``` + +**Problem: Performance degradation** + +```sql +-- Check write waits +SHOW STATUS LIKE 'audit_log_filter_write_waits'; +-- If high: Reduce logging scope or increase buffer + +-- Check filter effectiveness +SELECT + (SELECT VARIABLE_VALUE FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_events_filtered') / + (SELECT VARIABLE_VALUE FROM performance_schema.global_status + WHERE VARIABLE_NAME = 'audit_log_filter_events') * 100 +AS filter_percentage; +-- High percentage means filters are working well +``` diff --git a/docs/filter-audit-log-filter-files.md b/docs/filter-audit-log-filter-files.md index 6d321e895e8..5bbcc0fff10 100644 --- a/docs/filter-audit-log-filter-files.md +++ b/docs/filter-audit-log-filter-files.md @@ -31,6 +31,240 @@ Filter definitions are `JSON` values. The function, `audit_log_filter_flush()`, forces reloading all filters and should only be invoked when modifying the audit tables. This function affects all users. Users in current sessions must either execute change-user or disconnect and reconnect. +## Function behavior and edge cases + +### audit_log_filter_set_filter() + +**Behavior when filter name already exists:** +* If a filter with the same name already exists, the function returns an error +* To update an existing filter, you must first remove it, then create a new one with the same name +* Alternatively, modify the filter directly in the `mysql.audit_log_filter` table and call `audit_log_filter_flush()` + +**Example:** +```sql +-- First attempt creates the filter +SELECT audit_log_filter_set_filter('my_filter', '{"filter": {"log": true}}'); +-- Returns: OK + +-- Second attempt with same name fails +SELECT audit_log_filter_set_filter('my_filter', '{"filter": {"log": false}}'); +-- Returns: ERROR: Filter 'my_filter' already exists +``` + +### audit_log_filter_set_user() + +**Wildcard support (8.4.4+):** +Starting from Percona Server for MySQL 8.4.4, `audit_log_filter_set_user()` accepts wildcard characters (`%` and `_`) in the host part of user accounts. + +**Examples:** +```sql +-- Match user from any host +SELECT audit_log_filter_set_user('admin@%', 'log_all'); + +-- Match user from specific IP subnet +SELECT audit_log_filter_set_user('user@192.168.0.%', 'log_all'); + +-- Match user from specific domain +SELECT audit_log_filter_set_user('user@%.example.com', 'log_all'); + +-- Match user with underscore in hostname +SELECT audit_log_filter_set_user('user@db_server_1', 'log_all'); +``` + +**Behavior when assigning filter to non-existent user:** +* The function succeeds even if the user account doesn't exist +* The filter assignment is stored in `mysql.audit_log_user` table +* When the user connects, the filter is applied +* If the user never connects, the assignment remains but has no effect + +**Behavior when user is in current session:** +* If a user is currently connected, changing their filter assignment does not affect their current session +* The new filter takes effect when: + * The user disconnects and reconnects, or + * The user executes `CHANGE_USER` command + +### audit_log_filter_remove_filter() + +**Behavior when filter is assigned to users:** +* When you remove a filter, it is automatically unassigned from all users (including the default `%` account) +* Current sessions are detached from the filter immediately +* New connections for affected users fall back to: + * Default filter (if one exists), or + * No filtering (if no default exists) + +**Example:** +```sql +-- Remove filter that's assigned to multiple users +SELECT audit_log_filter_remove_filter('log_all'); +-- All users assigned to 'log_all' are now unassigned +-- Current sessions continue without filter +-- New connections use default filter or no filtering +``` + +### audit_log_filter_remove_user() + +**Behavior:** +* Removing a user's filter assignment does not affect their current session +* New sessions for the user will use: + * Default filter (if one exists), or + * No filtering (if no default exists) + +**Removing default filter:** +* To remove the default filter, use: `SELECT audit_log_filter_remove_user('%');` +* This affects all users without specific filter assignments + +## Modifying filters via tables vs functions + +You can modify audit log filters in two ways: using functions or by directly modifying the audit tables. + +### Using functions (recommended) + +**Advantages:** +* Validation of filter definitions +* Immediate effect (no flush required) +* Error messages for invalid configurations +* Atomic operations + +**Example:** +```sql +SELECT audit_log_filter_set_filter('my_filter', '{"filter": {"log": true}}'); +-- Filter is immediately active +``` + +### Direct table modification + +**When to use:** +* Bulk updates of multiple filters +* Script-based configuration management +* Replication scenarios (with caution) + +**Important:** +* Changes to `mysql.audit_log_filter` or `mysql.audit_log_user` tables do not take effect automatically +* You must call `audit_log_filter_flush()` after making table changes +* No validation is performed on filter definitions +* Invalid JSON in filter definitions can cause errors + +**Example:** +```sql +-- Direct table modification +INSERT INTO mysql.audit_log_filter (NAME, FILTER) +VALUES ('my_filter', '{"filter": {"log": true}}'); + +-- Must flush to activate +SELECT audit_log_filter_flush(); +``` + +### Replication implications + +**Default behavior:** +* By default, `mysql.audit_log_filter` and `mysql.audit_log_user` tables are replicated +* Changes on the source server replicate to replica servers +* However, replicated table changes do not automatically activate filters on the replica + +**What happens:** +1. Source server: Filter is created and activated +2. Replication: Table changes are replicated to replica +3. Replica server: Filter exists in table but is NOT active in memory +4. Replica server: Must call `audit_log_filter_flush()` or restart server to activate + +**Recommendation:** +Configure replication to ignore audit log filter tables: +```ini +# In my.cnf +replicate-ignore-table=mysql.audit_log_filter +replicate-ignore-table=mysql.audit_log_user +``` + +This ensures: +* Each server has independent filter configuration +* No conflicts between source and replica filters +* Better control over audit logging on each server + +## Translating 8.0 include/exclude variables to JSON filters + +If you're migrating from Percona Server 8.0 plugin to 8.4 component, you need to convert old include/exclude variables to JSON filter definitions. + +### Variable mapping table + +| 8.0 Plugin Variable | 8.4 Component JSON Filter | Example | +|---------------------|---------------------------|---------| +| `audit_log_include_accounts` | `"user": ["user1", "user2"]` | Include specific users | +| `audit_log_exclude_accounts` | `"user": ["user1"], "negate": true` | Exclude specific users | +| `audit_log_include_commands` | `"class": {"name": "general"}, "event": {"name": "command"}` | Include specific commands | +| `audit_log_exclude_commands` | `"class": {"name": "general"}, "event": {"name": "command"}, "negate": true` | Exclude specific commands | +| `audit_log_include_databases` | `"database": ["db1", "db2"]` | Include specific databases | +| `audit_log_exclude_databases` | `"database": ["db1"], "negate": true` | Exclude specific databases | + +### Migration examples + +**8.0 Configuration:** +```ini +audit_log_include_accounts = admin@localhost, dba@% +audit_log_exclude_databases = test, temp +``` + +**8.4 JSON Filter:** +```json +{ + "filter": { + "class": [ + { + "name": "general", + "user": ["admin", "dba"], + "database": ["test", "temp"], + "negate": true + } + ] + } +} +``` + +**8.0 Configuration:** +```ini +audit_log_include_commands = SELECT, INSERT, UPDATE, DELETE +audit_log_exclude_accounts = readonly_user@% +``` + +**8.4 JSON Filter:** +```json +{ + "filter": { + "class": [ + { + "name": "general", + "event": [{"name": "command"}], + "user": ["readonly_user"], + "negate": true + } + ] + } +} +``` + +### Complete migration procedure + +1. **Document current 8.0 configuration:** + * List all `audit_log_include_*` and `audit_log_exclude_*` variables + * Note their values + +2. **Convert to JSON filters:** + * Use the mapping table above + * Create filter definitions for each include/exclude rule + * Combine related rules into single filters where possible + +3. **Test filters:** + * Create filters in a test environment + * Verify they match the behavior of old plugin configuration + * Adjust as needed + +4. **Deploy to production:** + * Install 8.4 component + * Create JSON filters + * Assign filters to users + * Verify functionality + +For complete migration instructions, see [Migration Guide](audit-log-filter-migration.md). + ## Constraints The `component_audit_log_filter` component must be enabled and the audit tables must exist to use the audit log filter functions. The user account must have the required privileges. diff --git a/docs/install-audit-log-filter.md b/docs/install-audit-log-filter.md index e7b9819d722..9ee2fd33dc5 100644 --- a/docs/install-audit-log-filter.md +++ b/docs/install-audit-log-filter.md @@ -4,6 +4,22 @@ The recommended way to install the component is to use the `audit_log_filter_linux_install.sql` script, located in the `share` directory, which creates the required tables before installing the component. +### What the script does + +When you run the `audit_log_filter_linux_install.sql` script, it performs the following operations: + +1. Creates the `mysql.audit_log_filter` table: This table stores filter definitions. Each filter has a name and a JSON definition that specifies which events to log or block. + +2. Creates the `mysql.audit_log_user` table: This table stores user account assignments. It links user accounts (user@host combinations) to specific filter names. + +3. Installs the component: The script executes `INSTALL COMPONENT 'file://audit_log_filter'` to load the audit log filter component into the MySQL server. + +4. Verifies the installation: The script ensures that both tables are created successfully and the component is properly installed. + +The script can be run safely multiple times: if the component is already installed, it will not reinstall it. If the tables already exist, it will not recreate them. + +For detailed information about the table structures and column definitions, see [Audit Log Filter tables](audit-log-filter-overview.md#audit-log-filter-tables) in the overview. + ### Prerequisites The `plugin_dir` system variable defines the component library location. If needed, set the `plugin_dir` variable at server startup. @@ -22,7 +38,7 @@ You can also designate a different database with the `audit_log_filter.database` ### Install the component -To install the component using the script, you must specify the `mysql` database. You can do this in two ways: +To install the component using the script, you must specify the `mysql` database. You can specify the `mysql` database in two ways: * Option 1: Run the script from the command line with the `-D mysql` option: @@ -59,11 +75,13 @@ mysql> show tables in mysql like 'aud%'; 2 rows in set (0.00 sec) ``` +For detailed information about these tables, including column definitions and how to modify them, see [Audit Log Filter tables](audit-log-filter-overview.md#audit-log-filter-tables) in the overview. + ## Alternative: INSTALL COMPONENT method -You can also install the component using the `INSTALL COMPONENT` command, but this method does not create the required tables and will cause filter operations to fail. +You can also install the component using the `INSTALL COMPONENT` command, but the `INSTALL COMPONENT` method does not create the required tables and will cause filter operations to fail. -```mysql +```sql INSTALL COMPONENT 'file://audit_log_filter' ``` @@ -71,13 +89,13 @@ INSTALL COMPONENT 'file://audit_log_filter' Check that the component is properly installed: -```{.bash data-prompt="mysql>"} -mysql> select * from mysql.component; +```sql +select * from mysql.component; ``` ??? example "Expected output" - ```{.text .no-copy} + ```text +--------------+--------------------+------------------------------------+ | component_id | component_group_id | component_urn | +--------------+--------------------+------------------------------------+ @@ -91,13 +109,13 @@ mysql> select * from mysql.component; Test that the audit log filter is working correctly: -```{.bash data-prompt="mysql>"} -mysql> SELECT audit_log_filter_set_filter('log_all', '{"filter": {"log": true}}'); +```sql +SELECT audit_log_filter_set_filter('log_all', '{"filter": {"log": true}}'); ``` ??? example "Expected output" - ```{.text .no-copy} + ```text +---------------------------------------------------------------------+ | audit_log_filter_set_filter('log_all', '{"filter": {"log": true}}') | +---------------------------------------------------------------------+ @@ -108,27 +126,287 @@ mysql> SELECT audit_log_filter_set_filter('log_all', '{"filter": {"log": true}}' !!! note - This error occurs when the component is installed without the required tables. Using the SQL script prevents this issue. + This error occurs when the component is installed without the required tables. Using the SQL script prevents this error. ### Fix missing tables If you have already installed the audit log component but are missing the required tables, you can run the `audit_log_filter_linux_install.sql` script to create the audit tables in the `mysql` database: -```{.bash data-prompt="$"} +```shell mysql -u root -p -D mysql < /path/to/mysql/share/audit_log_filter_linux_install.sql ``` Or interactively: -```{.bash data-prompt="mysql>"} -mysql> use mysql; -mysql> source /path/to/mysql/share/audit_log_filter_linux_install.sql; +```sql +USE mysql; +source /path/to/mysql/share/audit_log_filter_linux_install.sql; +``` + +Running the script creates the missing tables without reinstalling the component. + +## Choosing a database for audit tables + +By default, audit log filter tables are created in the `mysql` system database. You can use a different database if needed. + +### When to use a non-default database + +Consider using a custom database when: + +* You want to separate audit tables from system tables + +* You have specific backup/restore requirements + +* You need different replication configuration + +* Compliance requirements specify separate database + +### Setting a custom database + +Before installation: + +```sql +-- Set custom database at server startup +-- Add to my.cnf: +[mysqld] +audit_log_filter.database=audit_db +``` + +After installation: +Changing the database requires: + +1. Creating new database + +2. Migrating tables + +3. Updating configuration + +4. Restarting server + +### Database limitations + +Constraints: + +* Database name cannot be NULL + +* Database name cannot exceed 64 characters + +* Database must exist before component installation + +* Invalid database name prevents component from working + +### Migration between databases + +Procedure: + +1. Create new database + +2. Export tables from old database + +3. Import tables to new database + +4. Update `audit_log_filter.database` variable + +5. Restart server + +6. Verify component works with new database + +## Post-installation configuration + +After installing the audit log filter component, configure it for your environment. + +### Step 1: Create your first filter + +Simple filter to log all events: + +```sql +SELECT audit_log_filter_set_filter('log_all', '{"filter": {"log": true}}'); +``` + +Filter to log only connections: + +```sql +SELECT audit_log_filter_set_filter('log_connections', '{ + "filter": { + "class": { + "name": "connection" + } + } +}'); +``` + +### Step 2: Assign filter to users + +Assign to default account (all users): + +```sql +SELECT audit_log_filter_set_user('%', 'log_all'); +``` + +Assign to specific user: + +```sql +SELECT audit_log_filter_set_user('admin@localhost', 'log_all'); +``` + +Assign with wildcard (8.4.4+): + +```sql +SELECT audit_log_filter_set_user('user@192.168.%', 'log_all'); +``` + +### Step 3: Test filter functionality + +Verify filter is working: + +```sql +-- Check filters +SELECT * FROM mysql.audit_log_filter; + +-- Check user assignments +SELECT * FROM mysql.audit_log_user; + +-- Test by executing a query +SELECT 1; + +-- Check audit log file (if format is JSON) +SELECT audit_log_read(); +``` + +### Step 4: Configure log management + +Set rotation size: + +```sql +SET GLOBAL audit_log_filter.rotate_on_size = 1073741824; -- 1GB +``` + +Set pruning: + +```sql +-- Size-based pruning (10GB total) +SET GLOBAL audit_log_filter.max_size = 10737418240; + +-- Time-based pruning (30 days) +SET GLOBAL audit_log_filter.prune_seconds = 2592000; ``` -This operation creates the missing tables without reinstalling the component. +### Step 5: Optional security configuration + +Enable encryption: + +```sql +-- Install keyring component first +INSTALL COMPONENT 'file://component_keyring_file'; + +-- Set encryption password +SELECT audit_log_encryption_password_set('secure_password'); + +-- Enable encryption (requires restart) +SET GLOBAL audit_log_filter.encryption = 'AES'; +``` + +Enable compression: + +```sql +SET GLOBAL audit_log_filter.compression = 'GZIP'; +-- Requires restart +``` + +## Upgrading from 8.0 plugin + +If you're upgrading from Percona Server 8.0 with the audit log filter plugin, follow these steps. + +### Prerequisites + +1. Backup current configuration: + + * Document all `audit_log_*` variables + + * Export filter definitions if using plugin filters + + * Backup `mysql.audit_log_filter` and `mysql.audit_log_user` tables + +2. Review plugin configuration: + + ```sql + -- Check if plugin is installed + SELECT * FROM INFORMATION_SCHEMA.PLUGINS + WHERE PLUGIN_NAME LIKE 'audit%'; + + -- Document current variables + SHOW VARIABLES LIKE 'audit_log%'; + ``` + +### Upgrade procedure + +1. Install component: + + ```bash + mysql -u root -p -D mysql < /path/to/mysql/share/audit_log_filter_linux_install.sql + ``` + +2. Verify installation: + + ```sql + SELECT * FROM mysql.component WHERE component_urn LIKE '%audit_log_filter%'; + SHOW TABLES IN mysql LIKE 'audit%'; + ``` + +3. Migrate filter definitions: + + * Convert old include/exclude variables to JSON filters + + * See [Migration Guide](audit-log-filter-migration.md) for details + + * Create new filters using `audit_log_filter_set_filter()` + +4. Update variable names: + + * Change `audit_log_filter_*` to `audit_log_filter.*` + + * Update configuration files (my.cnf) + + * Remove deprecated plugin variables + +5. Test functionality: + + * Verify filters work correctly + + * Check that events are being logged + + * Verify log rotation and pruning + +6. Remove old plugin (optional): + + ```sql + UNINSTALL PLUGIN audit_log_filter; + ``` + +### Common upgrade issues + +Issue: Filters not working + +* Cause: Tables not created or component not loaded + +* Solution: Run installation script, verify component is installed + +Issue: Variables not recognized + +* Cause: Using old plugin variable names + +* Solution: Update to component variable names (`audit_log_filter.*`) + +Issue: Permission errors + +* Cause: Missing `AUDIT_ADMIN` privilege + +* Solution: Grant `AUDIT_ADMIN` privilege to users + +For complete migration instructions, see [Migration Guide](audit-log-filter-migration.md). ## Additional information -To upgrade from `audit_log_filter` plugin in Percona Server 8.4 to `component_audit_log_filter` component in Percona Server {{vers}}, do the [manual upgrade](upgrade-components.md). +To upgrade from `audit_log_filter` plugin in Percona Server 8.0 to `component_audit_log_filter` component in Percona Server 8.4, do the [manual upgrade](upgrade-components.md). --8<--- "get-help-snip.md" diff --git a/docs/manage-audit-log-filter.md b/docs/manage-audit-log-filter.md index 533111a464d..78195eb1ed1 100644 --- a/docs/manage-audit-log-filter.md +++ b/docs/manage-audit-log-filter.md @@ -18,3 +18,284 @@ The `SELECT audit_log_rotate()` command renames the file and creates a new audit The files are pruned if either `audit_log_filter.max_size` or `audit_log_filter.prune_seconds` have a value greater than 0 (zero) and `audit_log_filter.rotate_on_size` > 0. After the files have been renamed, you must manually remove any archived audit log filter files. The renamed audit log filter files can be read by `audit_log_read()`. The `audit_log_read()` does not find the logs if the name pattern differs from the current pattern. + +## Changing log format + +When you change the `audit_log_filter.format` system variable, the component handles the transition automatically. + +### Format change behavior + +**What happens:** +1. Current log file is rotated (renamed with timestamp) +2. New log file is created with the new format +3. Old format files remain accessible but are not appended to + +**Example:** +```sql +-- Current format: NEW (XML) +-- Current file: audit_filter.log + +-- Change to JSON format +SET GLOBAL audit_log_filter.format = 'JSON'; +-- Requires server restart to take effect + +-- After restart: +-- Old file: audit_filter.log.20240101120000 (XML format) +-- New file: audit_filter.log (JSON format) +``` + +### Backward compatibility + +**Reading old format files:** +* `audit_log_read()` function only works with JSON format +* XML format files (OLD or NEW) must be read manually or with external tools +* Old format files remain readable after format change + +**Recommendation:** +Change the log filename when changing format to avoid confusion: + +```sql +-- Before changing format, update filename +SET GLOBAL audit_log_filter.file = 'audit_filter.json'; +-- Then change format +SET GLOBAL audit_log_filter.format = 'JSON'; +-- Restart server +``` + +This ensures: +* Clear distinction between format types +* Easier log file management +* Better organization of historical logs + +### Format change procedure + +1. **Plan the change:** + * Decide on new format (OLD, NEW, or JSON) + * Choose new filename if changing format type + * Schedule during maintenance window (requires restart) + +2. **Update configuration:** + ```sql + -- Optionally change filename first + SET GLOBAL audit_log_filter.file = 'audit_filter.json'; + -- Change format + SET GLOBAL audit_log_filter.format = 'JSON'; + ``` + +3. **Restart server:** + * Format change requires server restart + * Old file is automatically rotated + * New file is created with new format + +4. **Verify:** + * Check that new log file exists + * Verify format is correct + * Test reading logs (if using JSON format) + +## Rotation scenarios and examples + +### Manual rotation + +**When to use:** +* `audit_log_filter.rotate_on_size` is set to 0 +* You need to rotate logs on demand +* Before maintenance operations +* When changing log format + +**Example:** +```sql +-- Rotate log file manually +SELECT audit_log_rotate(); +``` + +**Result:** +* Current file is renamed (e.g., `audit_filter.log.20240101120000`) +* New file is created with original name (`audit_filter.log`) +* Old file can be read by `audit_log_read()` if format is JSON + +### Automatic rotation + +**Configuration:** +```sql +-- Enable automatic rotation at 1GB +SET GLOBAL audit_log_filter.rotate_on_size = 1073741824; +``` + +**Behavior:** +* When log file reaches specified size, it's automatically rotated +* New file is created immediately +* Rotation happens transparently during normal operation + +**Example scenario:** +``` +audit_filter.log (1GB) → audit_filter.log.20240101120000 +New audit_filter.log created (0 bytes) +``` + +### Rotation with compression + +**Configuration:** +```sql +-- Enable compression +SET GLOBAL audit_log_filter.compression = 'GZIP'; +-- Requires server restart +``` + +**Behavior:** +* Rotated files are compressed automatically +* File extension may change (e.g., `.log.gz`) +* Compression happens after rotation +* Reading compressed files requires decompression + +### Rotation with encryption + +**Configuration:** +```sql +-- Enable encryption +SET GLOBAL audit_log_filter.encryption = 'AES'; +-- Requires server restart and password setup +``` + +**Behavior:** +* Rotated files remain encrypted +* Need encryption password to read old files +* Each rotated file uses same encryption key +* Password rotation affects new files, not old ones + +### Reading rotated files + +**JSON format:** +```sql +-- audit_log_read() can read rotated files +-- if they match the current file pattern +SELECT audit_log_read(); +-- Reads from current and rotated files matching pattern +``` + +**XML format:** +* Must read manually or with external tools +* Cannot use `audit_log_read()` function + +**File naming pattern:** +* Rotated files: `audit_filter.log.YYYYMMDDHHMMSS` +* Current file: `audit_filter.log` +* Pattern matching is based on base filename + +## Pruning behavior and configuration + +Pruning removes old audit log files based on size or age limits. + +### Pruning requirements + +To enable pruning, you must configure at least one of the following: + +1. **Enable rotation:** + ```sql + SET GLOBAL audit_log_filter.rotate_on_size = 1073741824; + ``` + Pruning only works when rotation is enabled. + +2. **Configure size-based pruning:** + ```sql + SET GLOBAL audit_log_filter.max_size = 5368709120; -- 5GB + ``` + +3. **Configure time-based pruning:** + ```sql + SET GLOBAL audit_log_filter.prune_seconds = 2592000; -- 30 days + ``` + +### Size-based pruning + +**Configuration:** +```sql +SET GLOBAL audit_log_filter.max_size = 10737418240; -- 10GB +``` + +**Behavior:** +* When combined size of all audit log files exceeds `max_size`, oldest files are pruned +* Pruning continues until total size is below limit +* Current active file is never pruned +* Files are pruned in order of age (oldest first) + +**Example:** +``` +Total size: 12GB (max_size: 10GB) +Files: + audit_filter.log.20240101 (2GB) → Pruned (oldest) + audit_filter.log.20240102 (3GB) → Pruned + audit_filter.log.20240103 (4GB) → Kept + audit_filter.log (3GB) → Kept (current file) +``` + +### Time-based pruning + +**Configuration:** +```sql +SET GLOBAL audit_log_filter.prune_seconds = 604800; -- 7 days +``` + +**Behavior:** +* Files older than `prune_seconds` are automatically pruned +* Age is calculated from file modification time +* Current active file is never pruned +* Pruning happens during rotation operations + +**Example:** +``` +prune_seconds: 7 days (604800 seconds) +Current date: 2024-01-08 + +Files: + audit_filter.log.20240101 (7 days old) → Pruned + audit_filter.log.20240102 (6 days old) → Kept + audit_filter.log.20240103 (5 days old) → Kept + audit_filter.log (current) → Kept +``` + +### Combined size and time pruning + +You can configure both size and time-based pruning: + +```sql +SET GLOBAL audit_log_filter.max_size = 10737418240; -- 10GB +SET GLOBAL audit_log_filter.prune_seconds = 2592000; -- 30 days +``` + +**Behavior:** +* Files are pruned if they exceed EITHER limit +* Size limit: Total size of all files +* Time limit: Age of individual files +* Whichever condition is met first triggers pruning + +**Recommendation:** +When both are configured, set `max_size` to at least 7 times `rotate_on_size` to allow multiple rotated files before pruning. + +### Pruning with encryption + +**Behavior:** +* Encrypted files can be pruned normally +* Pruning does not require decryption +* Pruned files are permanently deleted +* Ensure you have backups before enabling aggressive pruning + +### Pruning with compression + +**Behavior:** +* Compressed files are pruned based on compressed size +* Pruning considers total compressed size of all files +* Compression reduces storage, allowing more files before pruning + +### Monitoring pruning + +**Check current file sizes:** +```sql +SHOW STATUS LIKE 'audit_log_filter_current_size'; +SHOW STATUS LIKE 'audit_log_filter_total_size'; +``` + +**Verify pruning is working:** +* Monitor disk space usage +* Check that old files are being removed +* Verify files older than `prune_seconds` are pruned +* Ensure total size stays below `max_size` diff --git a/docs/reading-audit-log-filter-files.md b/docs/reading-audit-log-filter-files.md index 09f886b2029..4671a96ae7e 100644 --- a/docs/reading-audit-log-filter-files.md +++ b/docs/reading-audit-log-filter-files.md @@ -26,3 +26,236 @@ mysql> SELECT audit_log_read(); Reading a file is closed when the session ends or calling `audit_log_read()` with another argument. +## Reading different formats + +### JSON format reading + +The `audit_log_read()` and `audit_log_read_bookmark()` functions work only with JSON format files. + +**Requirements:** +* `audit_log_filter.format` must be set to `JSON` +* Files must be in JSON format +* Function returns error if format is not JSON + +**Example:** +```sql +-- Ensure format is JSON +SHOW VARIABLES LIKE 'audit_log_filter.format'; +-- Should show: JSON + +-- Read audit log +SELECT audit_log_read(); +``` + +### XML format reading + +XML format files (OLD or NEW style) cannot be read using `audit_log_read()` function. + +**Options for reading XML files:** +1. **Manual parsing:** Use text editors or XML parsers +2. **External tools:** Use log analysis tools that support XML +3. **Convert to JSON:** Change format to JSON for future logs (requires restart) + +**XML file structure:** +* OLD style: Traditional XML format with specific schema +* NEW style: Updated XML format with enhanced structure +* Both formats are human-readable but require XML parsing tools + +### Converting between formats + +**To read logs programmatically:** +1. Change `audit_log_filter.format` to `JSON` +2. Restart server +3. Future logs will be in JSON format +4. Use `audit_log_read()` function + +**Note:** Old XML format files remain in XML and cannot be converted. Only new logs use the new format. + +## Reading historical log files + +### Reading rotated files + +The `audit_log_read()` function can read rotated files if they match the current file naming pattern. + +**Requirements:** +* Files must have the same base name as `audit_log_filter.file` +* Files must be in JSON format +* Files must be in the same directory as the current log file + +**Example:** +``` +Current file: audit_filter.log +Rotated files: + audit_filter.log.20240101120000 + audit_filter.log.20240102120000 + audit_filter.log.20240103120000 + +audit_log_read() reads from all matching files +``` + +**File pattern matching:** +* Base name must match: `audit_filter.log` +* Timestamp suffix is ignored for pattern matching +* Files with different base names are ignored + +### Reading specific time ranges + +Use bookmarks to read from specific positions: + +**Read from bookmark:** +```sql +-- Get bookmark for most recent event +SELECT audit_log_read_bookmark(); +-- Returns: {"timestamp": "2024-01-01 12:00:00", "id": 12345} + +-- Read from that bookmark +SELECT audit_log_read('{"timestamp": "2024-01-01 12:00:00", "id": 12345}'); +``` + +**Read with max array length:** +```sql +-- Limit number of events returned +SELECT audit_log_read(NULL, 100); +-- Returns maximum 100 events +``` + +### Limitations of reading old files + +**File naming:** +* If rotated files are renamed with different patterns, they won't be found +* Files moved to different directories won't be accessible +* Files with different base names are ignored + +**Format compatibility:** +* Only JSON format files can be read +* XML format files cannot be read with functions +* Mixed format files cause errors + +**File access:** +* Files must be readable by MySQL server process +* Encrypted files require password/keyring access +* Compressed files may need decompression + +## Reading errors and troubleshooting + +### File doesn't exist + +**Error:** Function returns NULL or empty result + +**Causes:** +* Log file hasn't been created yet (no events logged) +* File was deleted or moved +* Incorrect file path configuration + +**Solution:** +```sql +-- Check file location +SHOW VARIABLES LIKE 'audit_log_filter.file'; + +-- Verify file exists +-- Check data directory or specified path + +-- Generate some events to create file +-- Then try reading again +``` + +### Format is not JSON + +**Error:** Function returns error or NULL + +**Causes:** +* `audit_log_filter.format` is set to OLD or NEW (XML) +* File is in XML format + +**Solution:** +```sql +-- Check current format +SHOW VARIABLES LIKE 'audit_log_filter.format'; + +-- Change to JSON (requires restart) +SET GLOBAL audit_log_filter.format = 'JSON'; +-- Restart server + +-- Note: Old XML files still cannot be read +-- Only new logs will be in JSON format +``` + +### File is corrupted + +**Error:** Function returns error or partial data + +**Causes:** +* File system corruption +* Incomplete writes +* Disk errors + +**Solution:** +1. Check file system for errors +2. Verify disk health +3. Check MySQL error log for I/O errors +4. Restore from backup if available +5. Corrupted files may need manual repair or recreation + +### File is encrypted + +**Error:** Cannot read encrypted files without proper setup + +**Causes:** +* Encryption is enabled but keyring/password not accessible +* Encryption password not set +* Keyring component not loaded + +**Solution:** +```sql +-- Check encryption status +SHOW VARIABLES LIKE 'audit_log_filter.encryption'; + +-- If encryption is AES, ensure: +-- 1. Keyring component is loaded +-- 2. Encryption password is set +-- 3. Keyring has access to password + +-- Get encryption password (if accessible) +SELECT audit_log_encryption_password_get(); +``` + +### File is compressed + +**Behavior:** Compressed files may need special handling + +**Causes:** +* `audit_log_filter.compression` is set to GZIP +* Files are compressed with gzip + +**Solution:** +* `audit_log_read()` should handle compressed files automatically +* If issues occur, check compression configuration +* Verify gzip support is available + +### Reading performance issues + +**Symptoms:** +* Slow read operations +* Timeout errors +* High CPU usage during reads + +**Causes:** +* Very large log files +* Many rotated files to process +* Complex JSON parsing + +**Solutions:** +1. Use `max_array_length` parameter to limit results +2. Read in smaller chunks using bookmarks +3. Consider archiving old files +4. Use external tools for bulk analysis +5. Increase `audit_log_filter.read_buffer_size` if available + +### Best practices for reading logs + +1. **Use JSON format** for programmatic access +2. **Read in chunks** using bookmarks and max_array_length +3. **Archive old files** to reduce reading overhead +4. **Monitor file sizes** to prevent performance issues +5. **Use external tools** for complex analysis of large log sets +6. **Test reading** in non-production environment first diff --git a/docs/write-filter-definitions.md b/docs/write-filter-definitions.md index a85cffafda1..3299f99007b 100644 --- a/docs/write-filter-definitions.md +++ b/docs/write-filter-definitions.md @@ -237,6 +237,46 @@ The table shows the available event classes and their subclasses: This setup gives you the flexibility to monitor the exact events that are important to you while controlling logging behavior in a detailed way. +## Filterable fields by event class + +The following table shows all filterable fields available for each event class and subclass. All numeric fields must be specified as strings in filter definitions. + +| Event Class | Event Subclass | Field Name | Data Type | Always Present | Filterable | Example Values | Notes | +|-------------|---------------|------------|-----------|----------------|------------|----------------|-------| +| `connection` | `connect` | `user` | string | Yes | Yes | `"admin"`, `"user1"` | Username | +| `connection` | `connect` | `host` | string | Yes | Yes | `"192.168.1.1"`, `"localhost"` | Host/IP address | +| `connection` | `connect` | `connection_id` | integer (as string) | Yes | Yes | `"12345"` | Connection identifier | +| `connection` | `connect` | `status` | integer (as string) | Yes | Yes | `"0"` (success), `"1"` (failure) | Connection status | +| `connection` | `change_user` | `user` | string | Yes | Yes | `"admin"`, `"user1"` | New username | +| `connection` | `change_user` | `host` | string | Yes | Yes | `"192.168.1.1"` | Host/IP address | +| `connection` | `change_user` | `connection_id` | integer (as string) | Yes | Yes | `"12345"` | Connection identifier | +| `connection` | `disconnect` | `user` | string | Yes | Yes | `"admin"` | Username | +| `connection` | `disconnect` | `host` | string | Yes | Yes | `"192.168.1.1"` | Host/IP address | +| `connection` | `disconnect` | `connection_id` | integer (as string) | Yes | Yes | `"12345"` | Connection identifier | +| `table_access` | `read` | `user` | string | Yes | Yes | `"admin"`, `"readonly_user"` | Username | +| `table_access` | `read` | `database` | string | Yes | Yes | `"mydb"`, `"financial_db"` | Database name | +| `table_access` | `read` | `table` | string | Yes | Yes | `"customers"`, `"orders"` | Table name | +| `table_access` | `read` | `status` | integer (as string) | Yes | Yes | `"0"` (success), `"1"` (failure) | Query status | +| `table_access` | `insert` | `user` | string | Yes | Yes | `"admin"` | Username | +| `table_access` | `insert` | `database` | string | Yes | Yes | `"mydb"` | Database name | +| `table_access` | `insert` | `table` | string | Yes | Yes | `"customers"` | Table name | +| `table_access` | `insert` | `status` | integer (as string) | Yes | Yes | `"0"`, `"1"` | Query status | +| `table_access` | `update` | `user` | string | Yes | Yes | `"admin"` | Username | +| `table_access` | `update` | `database` | string | Yes | Yes | `"mydb"` | Database name | +| `table_access` | `update` | `table` | string | Yes | Yes | `"customers"` | Table name | +| `table_access` | `update` | `status` | integer (as string) | Yes | Yes | `"0"`, `"1"` | Query status | +| `table_access` | `delete` | `user` | string | Yes | Yes | `"admin"` | Username | +| `table_access` | `delete` | `database` | string | Yes | Yes | `"mydb"` | Database name | +| `table_access` | `delete` | `table` | string | Yes | Yes | `"customers"` | Table name | +| `table_access` | `delete` | `status` | integer (as string) | Yes | Yes | `"0"`, `"1"` | Query status | +| `general` | `status` | `user` | string | Yes | Yes | `"admin"` | Username | +| `general` | `status` | `status` | integer (as string) | Yes | Yes | `"0"` (success), `"1"` (failure) | Query execution status | +| `general` | `command` | `user` | string | Yes | Yes | `"admin"` | Username | +| `general` | `command` | `command` | string | Yes | Yes | `"SELECT"`, `"INSERT"`, `"UPDATE"` | SQL command type | + +!!! note "Field filtering" + All numeric fields (like `connection_id`, `thread_id`, `status`) must be specified as strings in filter definitions. Use `"123"` not `123`, and `"0"` not `0`. + ### Inclusive filters Inclusive filters capture specific database events you want to log. They allow you to precisely target and record only the actions you care about. @@ -340,6 +380,426 @@ This example defines a filter that `excludes` (negate: true) all table access ev This filter captures failed update/delete modifications by admin and developer users in the financial database and successful connections for the `external_service` user +## Advanced filter patterns + +The following examples demonstrate complex filter patterns for real-world scenarios. + +### Combining multiple conditions + +**Log all failed queries from specific users:** +```json +{ + "filter": { + "class": [ + { + "name": "general", + "user": ["app_user", "web_user"], + "event": [{"name": "status"}], + "status": ["1"] + } + ] + } +} +``` + +**Block all DDL operations on production database:** +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "database": ["production_db"], + "event": [{"name": "ddl"}], + "abort": true + } + ] + } +} +``` + +**Log all connections except from specific IP range:** +```json +{ + "filter": { + "log": false, + "class": [ + { + "name": "connection", + "event": [{"name": "connect"}], + "host": ["192.168.100.%"], + "negate": true, + "log": true + } + ] + } +} +``` + +**Log all table access in financial database, but exclude read operations:** +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "database": ["financial_db"], + "event": [ + {"name": "insert"}, + {"name": "update"}, + {"name": "delete"} + ] + } + ] + } +} +``` + +### Filtering by connection ID or thread ID + +**Log all operations from a specific connection:** +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "connection_id": ["12345"] + } + ] + } +} +``` + +**Log all operations from specific threads:** +```json +{ + "filter": { + "class": [ + { + "name": "general", + "thread_id": ["67890", "67891"] + } + ] + } +} +``` + +!!! note "Connection and thread IDs" + Connection IDs and thread IDs must be specified as strings in filter definitions, even though they are numeric values. + +### Nested filters with multiple classes + +**Comprehensive security audit filter:** +```json +{ + "filter": { + "class": [ + { + "name": "connection", + "event": [ + {"name": "connect"}, + {"name": "disconnect"} + ], + "user": ["admin", "dba", "security_team"] + }, + { + "name": "table_access", + "database": ["financial_db", "customer_db"], + "event": [ + {"name": "update"}, + {"name": "delete"} + ], + "status": ["0", "1"] + }, + { + "name": "general", + "event": [{"name": "status"}], + "status": ["1"] + } + ] + } +} +``` + +This filter logs: +* All connections and disconnections from admin, dba, and security_team users +* All update and delete operations (successful and failed) in financial_db and customer_db +* All failed queries (status = 1) from any user + +### Using wildcards in filters + +**Log all operations on tables matching a pattern:** +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "table": ["customer_%", "order_%", "payment_%"] + } + ] + } +} +``` + +**Exclude operations on temporary tables:** +```json +{ + "filter": { + "log": false, + "class": [ + { + "name": "table_access", + "table": ["tmp_%", "temp_%"], + "negate": true, + "log": true + } + ] + } +} +``` + +## Blocking queries and connections + +The Audit Log Filter component can not only log events but also block queries or connections that match specific filter rules. This blocking capability provides an additional layer of security by preventing unauthorized or dangerous operations. + +### How blocking works + +When a filter includes an `abort` item set to `true`, matching queries or connections are blocked before execution. The blocked operation is still logged to the audit log, but the query does not execute. + +### Blocking syntax + +Add an `abort` item to your filter definition to enable blocking: + +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "event": [{"name": "ddl"}], + "abort": true + } + ] + } +} +``` + +This filter example blocks all Data Definition Language (DDL) operations such as `DROP TABLE`, `ALTER TABLE`, etc. + +### Which events can be blocked + +Most event classes support blocking, including: + +* `table_access` - Block table operations (read, write, DDL) + +* `general` - Block general commands + +* `connection` - Block connection attempts + +* `query` - Block specific queries + +!!! note "Blocking limitations" + Some events may not support blocking, or blocking may have limited effect. Test your blocking filters in a non-production environment before deploying. + +### Interaction with AUDIT_ABORT_EXEMPT + +Users with the `AUDIT_ABORT_EXEMPT` privilege (or `SYSTEM_USER` privilege) are exempt from blocking. Their queries execute even if they match a blocking filter. This exemption behavior ensures administrators can always regain access if filters are misconfigured. + +Example scenario: + +1. A filter is configured to block all `DROP TABLE` statements + +2. A regular user attempts `DROP TABLE customers;` → Query is blocked and logged + +3. An administrator with `AUDIT_ABORT_EXEMPT` attempts the same query → Query executes and is logged + +### Blocking examples + +Block all DROP TABLE statements: +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "event": [{"name": "ddl"}], + "abort": true + } + ] + } +} +``` + +Block connections from specific IP range: +```json +{ + "filter": { + "class": [ + { + "name": "connection", + "event": [{"name": "connect"}], + "host": ["192.168.100.%"], + "abort": true + } + ] + } +} +``` + +Block failed queries from specific users: +```json +{ + "filter": { + "class": [ + { + "name": "general", + "event": [{"name": "status"}], + "user": ["suspicious_user"], + "status": ["1"], + "abort": true + } + ] + } +} +``` + +!!! warning "Blocking configuration" + + Misconfigured blocking filters can lock you out of the system. Always test blocking filters in a non-production environment first, and ensure at least one user account has `AUDIT_ABORT_EXEMPT` privilege. + +## Common filter definition errors + +The following examples show common mistakes and how to fix them. + +### Using integers instead of strings + +**Incorrect:** +```json +{ + "filter": { + "class": [ + { + "name": "general", + "status": [0, 1] + } + ] + } +} +``` + +**Error:** `ERROR: Incorrect rule definition.` + +**Correct:** +```json +{ + "filter": { + "class": [ + { + "name": "general", + "status": ["0", "1"] + } + ] + } +} +``` + +### Invalid event class or subclass name + +**Incorrect:** +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "event": [{"name": "select"}] + } + ] + } +} +``` + +**Error:** Invalid event subclass. Use `read` instead of `select`. + +**Correct:** +```json +{ + "filter": { + "class": [ + { + "name": "table_access", + "event": [{"name": "read"}] + } + ] + } +} +``` + +### Malformed JSON + +**Incorrect:** +```json +{ + "filter": { + "class": [ + { + "name": "connection" + "event": [{"name": "connect"}] + } + ] + } +} +``` + +**Error:** Missing comma after `"connection"`. + +**Correct:** +```json +{ + "filter": { + "class": [ + { + "name": "connection", + "event": [{"name": "connect"}] + } + ] + } +} +``` + +### Missing required fields + +**Incorrect:** +```json +{ + "filter": { + "class": [ + { + "event": [{"name": "connect"}] + } + ] + } +} +``` + +**Error:** Missing `name` field in class definition. + +**Correct:** +```json +{ + "filter": { + "class": [ + { + "name": "connection", + "event": [{"name": "connect"}] + } + ] + } +} +``` + ## Best practices Following a systematic approach helps ensure successful deployment and maintenance when implementing audit log filters. Start by creating broad, inclusive filters that capture a wide range of events, giving you a comprehensive view of your database activity. For example, you might begin by logging all actions from administrative users or all changes on critical databases. As you analyze the captured data, you can refine these filters to focus on specific events, users, or changes that matter most to your organization.