Pgsql Source Analysis · Pg_dump Analysis

PostgreSQL itself provides the logical export tool Pg_dumpall and pg_dump, where pg_dumpall export all the databases, Pg_dump export a single database, the usage and parameters of two tools are no longer detailed, this article from the code level to analyze this process.

In summary, the logical export to do is to connect to the corresponding database, read out the definitions and data of each database object, but also include comment, server configuration and permission control, and so on, these database object definitions of SQL statements will be written to the corresponding dump file. You can set export-only mode or export data only, by default, export mode and data, so that you can support step-up export and recovery. The data table data can be written to the backup file in either copy or INSERT statements.

This process mainly involves several documents, including PG_DUMPALL.C,PG_DUMP.C,PG_BACKUP_DB.C. Where PG_DUMPALL.C export all the database, PG_DUMP.C export a single database, will be pg_dumpall.c constantly called, thereby exporting all the database, where the focus of the analysis of PG_DUMP.C work.

Pg_dump Process Analysis

The main function of the pg_dump.c file is to perform the following tasks:

(1) Analysis of various parameters, including the corresponding variable assignment and the parameters are compatible with each other, if not compatible, the error exits.

(2) Call the Createarchive function, open the output file, the output stream for G_fout,g_fout is the archive type, here the more ingenious place is based on different file formats, will produce different g_fout, The correspondence also uses different. c files to separate the processing functions under different exported file formats, which makes it easy to add new export file formats, improve maintainability and extensibility, and implement the methods we will analyze below. Currently supports four export file formats and is:

Custom
Pg_backup_custom.c
Export objects are stored in a binary format file

file
Pg_backup_files.c
Export objects are stored in the specified file

Plain
Pg_backup_null.c
Export files to standard output

Tar
Pg_backup_tar.c
Export files in a compressed file format

(3) Call the Connectdatabase function, connect the destination database, and execute some SQL statements on this database, such as setting the encoding between C/s, setting the database for Date type usage format, and setting some version-related information for different versions of the server.

(4) Open a transaction on the database connection (3), guarantee the consistency of all the exported data, and set the correct floating-point output format in order to ensure the ability to export floating-point numbers:

do_sql_command(g_conn, "BEGIN");do_sql_command(g_conn, "SET TRANSACTION ISOLATION LEVEL SERIALIZABLE");do_sql_command(g_conn, "SET extra_float_digits TO 2");

(5) In order to keep the Pg_dump tool compatible with the lower version, determine the value of some variables based on the server version number.

(6) Querying and storing information about the schemas and tables that need to be exported.

(7) Call the Getschemadata function, decide which database objects to export, and invoke the following function to save the specific database object:

Proclanginfo = Getproclangs (&numproclangs); agginfo = Getaggregates (&numaggregates); oprinfo = GetOpera Tors (&numoperators); oprinfoindex = Buildindexarray (Oprinfo, numoperators, sizeof (Oprinfo)); opcinfo = GetOpclass ES (&numopclasses);p rsinfo = Gettsparsers (&numtsparsers); tmplinfo = Gettstemplates (&numtstemplates); Dictinfo = Gettsdictionaries (&numtsdicts); cfginfo = Gettsconfigurations (&numtsconfigs); fdwinfo =      Getforeigndatawrappers (&numforeigndatawrappers); srvinfo = Getforeignservers (&numforeignservers);d Aclinfo = Getdefaultacls (&numdefaultacls); opfinfo = Getopfamilies (&numopfamilies); convinfo = GetConversions ( &numconversions); tblinfo = Gettables (&numtables); tblinfoindex = Buildindexarray (Tblinfo, numTables, sizeof ( Tableinfo)); inhinfo = Getinherits (&numinherits); ruleinfo = GetRules (&numrules); castinfo = GetCasts (&numcasts); FlaginhtaBles (Tblinfo, Numtables, Inhinfo, numinherits); Gettableattrs (Tblinfo, numtables); Flaginhattrs (Tblinfo, numTables); Getindexes (Tblinfo, numtables); Getconstraints (Tblinfo, numtables); Gettriggers (Tblinfo, numtables);

It is important to note that the object's export order is not fully determined here, and the principle is that the object being relied upon is piloted. In these functions, note the sequence of calls similar to the following:

tblinfo = getTables(numTables);tblinfoindex = buildIndexArray(tblinfo, numTables, sizeof(TableInfo));

This indicates that the table is piloted, and then the index information attached to the table is exported.

flagInhTables(tblinfo, numTables, inhinfo, numInherits);

The previous sentence indicates that the parent table should be exported before the child table.

(8) Each GETXXXS function will perform the following procedure:

• According to the server version number, query the system table, read out the object's meta-data information.
• Iterates through each database object's metadata information, computes its dependent objects through Pg_depend system tables for each database object, and records the metadata information of all objects and the metadata information of its dependent objects.

(9) Call the Gettabledata function, "Export" the data in the table. Note that the export quoted here is not the real export data, but a linked list to store each of the database objects need to export the basic information, to really need to export the time to traverse the list and then make corresponding processing. This uses the idea of a placeholder, which does not occupy a lot of memory space.

(10) If you need to export large objects, call Getblobs, ibid is also a linked list, and did not really go to do the export.

(11) According to the steps (8) to get the dependency of each object, call the Getdependencies function, rearrange the dependencies between objects, Call Sortdumpableobjects to determine the order in which each database object is exported (the export priority of objects of different types depends on the newobjecttypepriority array; objects of the same type are sorted by name).

(12) Information such as storage encoding and the destination database for the connection.

(13) Iterate through all objects and "Export" the object (called the Dumpdumpableobject function, this function calls a bunch of objects such as Dumpnamespace, dumptable, etc.). If it is an "export" table, then according to the information of the "Export" object, query system tables, look up the column information corresponding to each table, generate the corresponding SQL statement of the Table object, output the SQL statement to G_fou; if it is "export" table data, call Dumptabledata, there are two ways to choose, The first is to generate an INSERT statement, and the default is to generate a copy statement of PostgreSQL itself. It is no longer specific to the introduction.

(14) When "Export" each object, usually call Archiveentry, do the real SQL statement generation work. In addition, functions such as dumpcomment, Dumpseclabel, Dumpacl, and other related information such as annotations, permissions, and so on are called "exported".

(15) Call the Restorearchive function, really export the data, note that this is based on the different export file format to choose different restorearchive functions.

(16) Close handle to release resources and so on.

Next, we briefly analyze the four types of export formats currently supported and how to implement different processing functions for different export formats. Currently PostgreSQL provides four export file formats, as follows:

Custom
Pg_backup_custom.c
Export the backup file to a binary format, including the file header and file body. The file body is a linked list that holds each backup object, each backed up object has a uniform structure identifier that supports compression (compression relies on the system compilation option and the macro definition switch in the pg_config.h file).

Plain
Pg_backup_null.c
Output SQL script content to standard output, by default.

file
Pg_backup_files.c
The export includes backing up one master file and some auxiliary files, the main file is similar to the custom file format, the auxiliary file is a data file, and each auxiliary file corresponds to a table in the Backup object.

Tar
Pg_backup_tar.c
File backups are basically similar to "file", but all of the files that were backed up in the last file are archived to a tar file. The maximum file size is 8GB (limited by tar file format).

PostgreSQL implements these four export file formats to correspond to different processing functions through function pointers. In the Pg_backup_archiver.h file, there are a number of function pointers defined, such as:

typedef void (*CLOSEPTR) (struct _archivehandle * AH); typedef void (*REOPENPTR) (struct _archivehandle * AH); typedef void (*ARCHIVEENTRYPTR)      (Struct _archivehandle * AH, struct _tocentry * te); These function pointers are used in the following file (file-and called function): Pg_backup_custom.c->initarchivefmt_custom (archivehandle *ah) pg_backup_null.c ->initarchivefmt_null (Archivehandle *ah) pg_backup_files.c->initarchivefmt_files (ArchiveHandle *AH) Pg_backu P_tar.c->initarchivefmt_tar (Archivehandle *ah) uses a large number of function pointers in the data structure archivehandle, Makes it possible to assign a handler function to a different processing function when initializing a archive structure with different export file formats.  So in pg_dump.c, as long as the user-specified file format parameters, you can call the corresponding handler function, the code is as follows:/* Open the output file */if (PG_STRCASECMP (format, "a") = = 0 | | PG_STRCASECMP (Format, "append") = = 0) {/* this are used by Pg_dumpall, and are not documented */Plaintex        t = 1;    G_fout = createarchive (filename, archnull, 0, archmodeappend); } else if (pg_strcasecmp (format, "c") = = 0 | | pg_strcasecmp (format, "custom") = = 0) G_fout =Createarchive (filename, Archcustom, compresslevel, archmodewrite); else if (pg_strcasecmp (format, "f") = = 0 | | pg_strcasecmp (format, "file") = = 0) {/* * Dump files into t He current directory;         For demonstration only, not * documented.    */g_fout = createarchive (filename, archfiles, compresslevel, archmodewrite);        } else if (pg_strcasecmp (format, "P") = = 0 | | pg_strcasecmp (format, "plain") = = 0) {plaintext = 1;    G_fout = createarchive (filename, archnull, 0, Archmodewrite); } else if (pg_strcasecmp (format, "t") = = 0 | | pg_strcasecmp (format, "tar") = = 0) g_fout = createarchive (filename    , Archtar, Compresslevel, archmodewrite);        else {write_msg (NULL, "Invalid output format \"%s\ "specified\n", format);    Exit (1); }

In summary, the contents of pg_dump export can be divided into database object definition and object data. The definition of the database object is exported by querying the system table to read the corresponding meta-information, the object's various types of information on a linked list, including its dependent object OID. and the specific data, that is, each data table data, is also abstracted for a database object (which we can call a data object), saved in this list (all the objects on the list have their own type, tocentry structure has a member "Tesection section", Is the type that identifies this node). By adjusting the export order, the definition of the database object is exported, and then the data object is exported, as long as the corresponding data object node in the linked list information, execute the appropriate SQL statement, read out the data from the table, and then write out the data. So, in memory is just the definition of the object on the list, the data is written at the edge of the read side, can be fully implemented streaming export, as follows:

Export data, import to destination library via pipeline and Psql tools

pg_dump -h host1 dbname | psql -h host2 dbname

Export data to standard output

pg_dump dbname > outfile

Exporting large amounts of data larger than the maximum file supported by the operating system

pg_dump dbname | gzip > filename.gz

Recover data from large data to the destination library

gunzip -c filename.gz | psql dbname

Or

cat filename.gz | gunzip | psql dbname

Of course, in addition to the above analysis, there are many other detailed content needs to be analyzed, such as different versions of the database operation, version compatibility issues, how to deal with object permissions, how to deal with the constraint relationship. These questions are worth the next step of concrete analysis.

Pgsql Source Analysis · Pg_dump Analysis