Optimizing disk storage space for the monthly database of urchin profiles

Summary: Urchin's report data is stored in a monthly database unique to each profile (note: Urchin data is archived on a monthly basis), which is typically located in the Urchin data/reports directory. Per configuration processed database size is original day

Urchin's report data is stored in a monthly database unique to each profile (note: The urchin data is archived on a monthly basis), which is typically located in the Urchin data/reports directory. Each configuration has a processed database size of 5% to 10% of the original log size.

By default, Urchin retains these monthly profile databases, but the amount of data accumulated over a long period of time becomes larger, resulting in more space occupied by urchin processing, and reduced urchin responsiveness when viewed by the user. Therefore, you need to optimize the disk storage space for the monthly database of the Urchin profile.

There are usually five ways to optimize the disk storage space of a urchin profile monthly database:

1. Set the configuration file to automatically delete the original trace data after processing the log

2. Set up profiles to archive history data

3. Limit the number of months that retention history reports data.

4. Compress the configuration file database.

5. Reasonable set up database automatic backup.

Method 1: Delete the original trace data after processing the log

Configuration files can be configured to delete the original audience and session information after processing is complete. This can improve the performance of large Web sites and reduce the amount of data stored. Note: When this configuration is selected, a cross date session is displayed as two sessions (one session a day) instead of a session. For most websites, the differences in results are negligible.

Configure the configuration file to delete the original audience and session information after processing completes:

1. In the admin interface, click "Configure" and then click "Urchin Profile" – "Configuration file".

2. Modify the required configuration file.

3. In the Storage/Database tab, set the Keep original trace data field to off.

4. Click "Update".

Method 2: AutoArchive history data

Configuration files can be configured to compress monthly history data into an archive file. Reports can view archived data, but no additional clicks are processed for the archived months.

Configure profiles to archive history data:

1. In the admin interface, click "Configure" and then click "Urchin Profile" – "Configuration file".

2. Modify the required configuration file.

3. In the Storage/Database tab, set the Archive database field to open.

4. Specify the month number for the archive database after this field (this option specifies how many months after the data is retained to start AutoArchive).

5. Click "Update".

Method 3: Periodically remove unused profile data

The data in the Urchin profile Data/reports/profile-name directory is removable, so regular removal of data information that is not used is removed. This is the simplest straightforward method, and is recommended for automated scripting.

Method 4: Compress the profile database

The size of the archive produced by the old urchin monthly database compression is generally only about 20% to 30% of the database set before it was uncompressed. Although the Urchin reporting engine cannot read the ZIP archive directly, it can extract the required database from the ZIP archive at any time. The report engine does not delete the uncompressed database, which increases the speed at which users can access data when they view the Urchin report. However, the original ZIP archive remains in place, so periodic cleanup can remove the uncompressed database directly to regain disk space.

Method 5: Properly set up database automatic backup

The database backup and Purge feature provides settings for profile backup information:

1. Enable automatic rollback of the database, if the process stopped or interrupted, the data can be automatically repaired. Urchin automatically detects this and rolls back the data to the most recent backup (if any) before continuing.

2. Clear backup, use this option to keep the number of backups in the next option as needed to automatically purge previous backups. If this feature is enabled, urchin will automatically delete previous backups to control the amount of storage.

3. To keep the number of backups, this option specifies the number of backups that are reserved for each month for the above cleanup feature.

The backup feature is typically turned off from the point of view of optimizing server footprint, but the backup functionality can be enabled from the security perspective of the configuration file.

Urchin Database Storage Technology Overview

For each urchin configuration file, Urchin maintains a monthly set of database files stored in a directory named Yyyymm. These directories contain approximately 50 files that provide data for the reporting engine. These directories and database files are named after the month in which the data is stored. The complete list of databases is:

Yyyymm-uhed–> Database Header

Yyyymm-usti–> String Index

yyyymm-ustd–> String Data

Yyyymm-udai–> Summary Table Index

Yyyymm-udxx–> Summary datasheet (XX is replaced by the table number for the data map).

Yyyymm-uvii–> Visitor Index

Yyyymm-uvid–> Visitor Data

yyyymm-used–> Session Data

yyyymm-upad–> Path Data

Yyyymm-utrd–> Trading Data (ecommerce)

Yyyymm-uitd–> Project Data (ecommerce)

Yyyymm-ulti–> Log Tracking Index

Yyyymm-ultd–> Log Trace data

Yyyymm-utod–> Total Data

Yyyymm-uhid–> Column Chart data

Yyyymm-umad–> Visitor Matrix Data

Each group of databases is complete for the month in which the data is contained. Because there is no correlation between the monthly database sets, each database set can be archived and trimmed independently, and data from other months will not be affected.

The full monthly database file for each month is maintained under normal operation. However, the Urchin log processing engine uses only 4 files of these database files. These database files are:

Yyyymm-usti

Yyyymm-udai

Yyyymm-ulti

Yyyymm-ultd

The Urchin log processing engine uses the following database files to process cross Group and visitor Drill-down reports. Deleting these items will only affect these reporting capabilities.

Yyyymm-uvii

Yyyymm-uvid

Yyyymm-used

Yyyymm-upad

Yyyymm-utrd

Yyyymm-uitd

These databases contain information about visitors, sessions, paths, transactions, and products. These files will use a percentage of the total storage space required for the current month, about 10% to 50%. Therefore, you can win larger disk space if you set the Keep original trace data option on the storage/Database screen of profile configuration to shut down.

It is recommended that you disable the Keep original trace data option only if you have a very high volume of traffic and you keep the original trace data from causing disk or CPU resource consumption problems.