MongoDB source code overview-start processing

After MongoDB is started,ProgramThe corresponding parameters, lock files left over from the last time, log files, and so on will be processed accordingly. At the same time, some service threads supporting other parts of the operation will be enabled. In order to read the MongoDBCodeAnd understand its global design concept. Therefore, I will give myself a summary of these aspects through the blog to facilitate future reference.

 

The program processes all input parameters in the mian function. The program uses the boost library to implement cross-platform command line parameter compatibility. This part of the code is very huge and messy, so there is no need to record it too much. It is done at the bottom of the main function.Initandlisten (Response line. Port, response rvpath );Call. This function is our key part.

 

In void _ initandlisten (INT listenport, const char * appserverloc = NULL), the function first checks the running environment, for example, to determine whether the current system is a 32bit system: bool is32bit = sizeof (int *) = 4; then output some version information (this part of the code will not be pasted)The acquirepathlock () method is called later.

Void acquirepathlock (){

StringName = (boost: filesystem: path (dbpath )/"Mongod. Lock"). Native_file_string ();

Bool Oldfile = False ;
// The concept of oldfile true is that there is mongod. Lock and Its filesize is greater than 0 ,. A normal process will set the MongoDB. Lock size to 0 (including Ctrl + C)
If (Boost: filesystem: exists (name) & boost: filesystem: file_size (name)> 0 ){
Oldfile = True ;
}
// We check this here because we want to see if we can get the lock
// If we can't, then its probably just another running D running

Lockfile = open (name. c_str (), o_rdwr | o_creat, s_irwxu | s_irwxg | s_irwxo );
If (Lockfile <= 0 ){
Uasserted ( 10309 , STR: stream () < " Unable to create/open lock file for lockfilepath: " <Name < '   ' <Errnowithdescription ());
}
If (Flock (lockfile, lock_ex | lock_nb )! = 0 ){ // MAN 2 flock viewing Parameters
Close (lockfile ); // If another mongod program locks the file, flock lock_ex will fail (that is, return results! = 0). The following exception is thrown. This ensures that a dbpath cannot be opened by two mongod programs at the same time.
Lockfile = 0 ;
Uassert ( 10310 , " Unable to acquire lock for lockfilepath: " + Name, 0 );
}

If (Oldfile ){
// If the last exception ends
String Errmsg;
If (Cmdline. Dur ){
// If the-dur attribute is enabled
If (! Dur: havejournalfiles ()){

Vector < String > Dbnames;
Getdatabasenames (dbnames ); // Obtain the XX of all XX. ns in the same directory and save it to dbnames.

If (Dbnames. Size () = 0 ){ // Without any XX. NS, you do not need to fix it, so you do not need to handle it. You can directly run the program.
// This means that your d crashed
// Between Initial Startup and when journaling was initialized
// It is safe to continue
}
Else {
Errmsg = STR: stream ()
< " * ************* \ N "
< " Old lock file: " <Name < " . Probably means unclean shutdown, \ n "
< " But there are no journal files to recover. \ n "
< " This is likely human error or filesystem upload uption. \ n "
< " Found " <Dbnames. Size () < " DBS. \ n "
< " See: http://dochub.mongodb.org/core/repair for more information \ n "
< " ************* " ;
}

}
}
Else {
// If the-dur attribute is not enabled and the. Lock file exists, the error message is returned to errmsg.
Errmsg = STR: stream ()
< " * ************* \ N "
< " Old lock file: " <Name < " . Probably means unclean shutdown \ n "
< " Recommend removing file and running -- repair \ n "
< " See: http://dochub.mongodb.org/core/repair for more information \ n "
< " ************* " ;
}
// If the preceding errmsg has any error information, stop the program.
If (! Errmsg. Empty ()){
Cout <errmsg <Endl;
Close (lockfile );

Lockfile =0;
Uassert (12596,"Old lock file",0);
}
}

// Not related to lock file, but this is where we handle unclean Shutdown
If (! Cmdline. Dur & dur: havejournalfiles ()){ // If it is not started in-dur mode but there is another journal log file, it exits unexpectedly because it cannot process non-persistent data.
Cout < " ************** " <Endl;
Cout < " Error: Journal files are present in Journal directory, yet starting without -- dur enabled. " <Endl;
Cout < " It is recommended that you start with journaling enabled so that recovery may occur. " <Endl;
Cout < " Alternatively (not recommended), you can backup everything, then delete the journal files, and run -- repair " <Endl;
Cout < " ************** " <Endl;
Uasserted (13597 , " Can't start without -- dur enabled when journal/files are present " );
}
Uassert ( 13342 , " Unable to truncate lock file " , Ftruncate (lockfile, 0 ) = 0 );
Writepid (lockfile ); // Write the process number to the mongod. Lock file.
Fsync (lockfile );

}

For the logic of this code, I am not so arrogant here. I drew a flowchart, and I believe that the image is easier to understand than the code.

 

When oldfile is true, mongod exists. lock and Its filesize is greater than 0. For this, it may not be easy to understand the meaning of filesize in this specific context. Let's look at it in dB. the signal processing function registered in CPP will be clear.

Void Ctrlcterminate (){

Log () <"Got kill or Ctrl-C signal, will terminate after current cmd ends"<Endl;
Client: initthread ("Ctrlcterminate");
Exitcleanly (exit_kill );
}

Finally, the shutdownserver function is called in exitcleanly to take charge of all the scanning tasks. The processing of the Mongo. Lock that we are concerned with is as follows:

If (Lockfile ){

Log () < " Shutdown: removing FS lock... " <Endl;
If (Ftruncate (lockfile, 0 )) // Man ftruncate sets the file size to 0
Log () < " Couldn't remove FS lock " <Errnowithdescription () <Endl;
Flock (lockfile, lock_un );
}

That is, as long as it Exits normally (including Ctrl + C), the size of Mongo. Lock will be 0, so you can use it to determine how the last service ended.

The following call is also made in the _ initandlisten method:

Fileallocator: Get ()-> Start ();

Once you see this call entry, we can infer that the singleton mode is definitely used here.

In the start method we call, fileallocator creates a separate thread to specifically run its run Method

1 Void Fileallocator: Start (){
2 Boost: thread t (boost: BIND (& fileallocator: Run, This ));

}

 

Fileallocator: The run method is similar to a separate service thread, which provides a task to create a specified file size.

When other running parts of the system need to use its service, you only need to callFileallocator: requestallocation Function

This function will put the file name you want to createList <string>_ PENDINGStore the file name and size inMutable Map <string, long>

At the same time, this function is still used.Boost: condition to perform synchronization operations (similar to producer and consumer). In the run function, when the file creation Requirement List (_ PENDING) is empty, it will wait.

If (Fa-> _ PENDING. Size () = 0)

Fa-> _ pendingupdated. Wait (LK. Boost ()); 

This prevents the abuse of CPU time caused by the infinite loop while.

If there is an item to be created in the Requirement List, the system API is called to create a file. 

Long FD = open (name. c_str (), o_creat | o_rdwr | o_noatime, s_irusr | s_iwusr ); 

 

Through the above call, the file is created, but the requirement is not met, because the file we created is not of the specified size, it is necessary to explain why we need to create a file of the specified size first, because operations on database files are likely to be very frequent, which puts forward requirements for the files where I store data, we always hope that the database file can get a continuous space (just like the memory allocation) on the disk. In this way, when reading and writing filesFragments are relatively small and can be read quickly (the system API level is uncontrollable ).Handles allocation of contiguous files on disk

Let's take a look at how MongoDB handles this problem:

After you call open to create a fileEnsurelength (FD, size) call, inEnsurelength will carry out the following loop internally

1 Char * Buf = buf_holder. Get ();
2 Memset (BUF, 0 , Z );
3 Long Left = size;
4 While (Left> 0 ){
5 Long Towrite = left;
6 If (Towrite> Z)
7 Towrite = z;
8
9 Int Written = write (FD, Buf, towrite );
10 Uassert ( 10443 , Errnowithprefix ( " Fileallocator: file write failed " ), Written> 0 );
11 Left-= written;

}

After reading it, will you be surprised to see that creating a file of the specified size is done through the while loop? No matter whether you are surprised or not, I am surprised.

After careful consideration, this is actually a good solution. After all, I have not found any method for creating a file with a specified size and continuous space, although the above method does not ensure that the space is continuous, but the file size is occupied at a time, you can write it every time you need it,Fragments will be much less.

 

OK. To sum up, fileallocator is created as much as possible.Fragments has a small number of files with a specified size. The implementation method is: a thread is used to provide services that create files and populate the files with a specified size. Other threads callThe requestallocation function registers a requirement, and the service thread obtains the requirement and then works. It can also be understood as a producer consumer mode. Other threads produce the requirement (file name, size ), this can be understood as a producer, while a service thread can be understood as a consumer. It runs only when the demand arrives, otherwise it waits ....