MIT6.828 Lab 5:file System, Spawn and Shell

Introduction
In this experiment, you will implement the create process and call library functions to load and run executables on disk. Run the shell on the console of the operating system kernel at the same time. These features require the implementation of the filesystem, where we will implement 1 simple and readable file systems.
The new additions to this experiment are as follows:
FS/FS.C the structure of the operating file system on disk.
FS/BC.C block caching based on the user-level page error handling mechanism.
FS/IDE.C most simple PIO-based IDE disk drive.
Server-side code that fs/serv.c the file system to interact with the client process
LIB/FD.C implements the traditional UNIX file descriptor interface.
File system driver for LIB/FILE.C disk type
LIB/CONSOLE.C Console type File system driver
LIB/SPAWN.C Spawn system Call implementation

File System Preliminaries
This section mainly introduces the structure of general file system, including the concept of sector, block, super block, block bitmap, file metadata and directory. The file system behind the Jos implementation is designed to these things and needs to be read carefully.

The File System
The goal of this experiment is not to let you implement the entire file system, but to implement the key parts. In particular, how to read blocks to the block cache and write back to the disk, the mapping file is offset to the disk block, the file reads, writes, and opens the IPC interface call.

Disk Access
The Jos file system needs to be able to access the disk, but we are not yet accessing the disk in the kernel implementation. To simplify this, we are abandoning the traditional single-core operating system to implement disk-driven as a system call, and disk drive as a user process to access the disk.
This will be simple to implement disk access in user space by polling instead of interrupts. In the x86 processor, you can allow the user state process to execute IO instructions such as in and out by setting the IOPL bit in the EFlags register.
Exercise 1:
The I386_init function creates 1 file system processes, passing the ENV_TYPE_FS flag to the Env_create function, which allows the file system process to execute IO instructions.
Reply:
Modify the Eflag value of the process in the Env_create function.

        if (type = = Env_type_fs)
              e->env_tf.tf_eflags |= fl_iopl_mask;

Question 1:
How to ensure that the IO privilege settings are saved and overloaded while the process is switching.
Reply:
The ENV.POP_TF function was called during the process switchover, where registers were recovered and registers such as Eip,cs,eflags were restored in the iret instruction.

The Block Cache
In Jos, 1 simple disk block caching mechanisms are implemented. This mechanism supports a maximum disk size of 3GB, and can be used similar to the Cow page mechanism implemented in Lab 4.
The implementation mechanism is as follows:
1, the virtual address space (0x10000000 (Diskmap) to 0xd0000000 (Diskmap+diskmax)) of the file system service process corresponds to the address space (3GB) of the disk.
2, the initial file system service process does not map the page, if you want to access the address space of 1 disks, a page error occurs.
3, in the page error handler, in memory request a block of space mapped to the corresponding file system virtual address, and then go to the actual physical disk to read the region's data to the memory area, and finally restore the file system service process.
EXERCISE2:
Implements the Bc_pgfault and Flush_block functions, where bc_pgfault is a page fault handler that loads pages from disk.
Reply:
The main purpose is to implement disk block buffer page processing and writeback parts, mainly used in direct disk interaction with the IDE driver function.

int Ide_read (uint32_t secno, void *dst, size_t nsecs)
int ide_write (uint32_t secno, void *dst, size_t nsecs)

The secno corresponds to the sector number on the IDE disk, and DST is the corresponding address in the current file system service program space, nsecs the number of sectors to read and write.

static void Bc_pgfault (struct utrapframe *utf) {void *addr = (void *) utf->utf_fault_va;
        uint32_t Blockno = ((uint32_t) addr-diskmap)/blksize;

        int R; Check that the fault is within the block cache region if (Addr < (void*) diskmap | | addr >= (void*) (diskm AP + disksize)) Panic ("Page fault in Fs:eip%08x, VA%08x, Err%04x", utf->utf_e

        IP, addr, utf->utf_err);
        Sanity check the block number. if (Super && blockno >= super->s_nblocks) Panic ("Reading non-existent block%08x\n", Blockno

        );
        Allocate a page in the disk map region, read the contents//for the block from the disk to that page. Hint:first round addr to page boundary.
        FS/IDE.C have code to read//the disk.
        addr = RoundDown (addr, pgsize); if ((r = Sys_page_alloc (0, addr, Pte_u | pte_p | Pte_w)) < 0) Panic ("inBc_pgfault, Sys_page_alloc:%e ", R);

        if (r = Ide_read (Blockno * blksects, addr, blksects)) < 0) Panic ("in Bc_pgfault, Ide_read:%e", R); Clear the dirty bit for the disk block page since we just read the//block from disk if (R = Sys_page_map (0, addr, 0, addr, uvpt[pgnum (addr)] & Pte_syscall)) < 0) Panic ("in Bc_pgfault, Sys_pag

        E_map:%e ", R);
if (bitmap && block_is_free (blockno)) Panic ("Reading free Block%08x\n", blockno); }

First calculate the corresponding blockno according to the address, and then check the correctness of whether the address is within the map range, the corresponding block exists, and so on.

void
flush_block (void *addr)
{
        uint32_t blockno = ((uint32_t) addr-diskmap)/blksize;

        if (addr < (void*) diskmap | | addr >= (void*) (Diskmap + disksize))
                panic ("Flush_block of Bad va%08x", addr); 
  
   int R;

        addr = RoundDown (addr, pgsize);
        if (va_is_mapped (addr) && va_is_dirty (addr)) {
                if (R = Ide_write (Blockno * blksects, addr, blksects)) < 0)
                        panic ("in Flush_block, Ide_write:%e", R);
                if (r = Sys_page_map (0, addr, 0, addr, uvpt[pgnum (addr)] & Pte_syscall)) < 0)
                        panic ("in Flush_block, Sys_page _map:%e ", R);
        }
}
  

First calculate the corresponding blockno according to the address, then check the correctness, and finally determine whether it is a dirty block, if it is to write back to the disk and clear the dirty bit.
　　
The Block Bitmap
After the Fs_init function sets the block bitmap, we can treat the bitmap as a bit array.
Exercise 3:
Using Free_block as a reference to implement Alloc_block, the function is to find 1 free disk blocks in the bitmap, marked to occupy and return block sequence numbers. When you allocate 1 blocks, in order to maintain file system consistency, you need to use the Flush_block function quickly to write back your modifications to the bitmap.
Reply:
This part is relatively simple, refer to the implementation of the Free_block function.

int
alloc_block (void)
{
        //The bitmap consists of one or more blocks.  A single bitmap block
        //contains the in-use bits for blkbitsize blocks.  There is
        //super->s_nblocks blocks in the disk altogether.

        uint32_t Blockno;

        for (blockno = 0; blockno < super->s_nblocks; blockno++) {
                if (Block_is_free (Blockno)) {
                        bitmap[blockno/ ^= 1<< (blockno%32);
                        Flush_block (bitmap);
                        return blockno;
                }
        }
        Return-e_no_disk;
}

File Operations
Jos has provided 1 series functions to manipulate and manage file structures, browse and manage directories, and parse filenames in FS/FS.C. The functions of the individual functions are as follows:
File_block_walk (struct File *f, uint32_t filebno, uint32_t **ppdiskbno, bool alloc):
Look for a file structure F in section Fileno block number to point to the disk block numbers put into ppdiskbno. If Filebno is less than ndirect, the corresponding link in F.direct[ndirect] is returned, otherwise the block found in F_indirect is returned. If the alloc is true and the corresponding disk block does not exist, allocate 1.
Dir_lookup (struct file *dir, const char *name, struct file **file):
Look for a file named name in Directory Dir and, if found, point to the file structure.
Dir_alloc_file (struct file *dir, struct file **file):
Assign 1 file pointers to file in the dir corresponding file struct to add files to the operation.
Skip_slash (const char *p):
Used for string processing in the path, skipping slashes.
Walk_path (const char *path, struct file **pdir, struct file **pf, char *lastelem):
Path is the file name from the absolute path, and if the file is successfully found, the corresponding file structure is assigned to PF, and the file structure of the directory in which it is located is assigned to Pdir,lastlem as the last remaining file name when it fails.
File_free_block (struct File *f, uint32_t filebno):
Frees the Filebno disk blocks in 1 files. This function is called in File_truncate_blocks.
File_truncate_blocks (struct File *f, off_t newsize):
Set the file to the new size after shrinking, emptying the physical blocks that were freed.
　　
Exercise 4:
Implement the File_block_walk function and the File_get_block function.
Reply:
The File_block_walk function looks for a file structure F where the Fileno block points to the disk block number placed in Ppdiskbno.

static int
file_block_walk (struct file *f, uint32_t filebno, uint32_t **ppdiskbno, bool alloc)
{
       int r;

       if (filebno >= ndirect + nindirect)
               return-e_inval;
       if (Filebno < Ndirect) {
               if (ppdiskbno)
                       *ppdiskbno = F->f_direct + filebno;
               return 0;
       }
       if (!alloc &&!f->f_indirect)
               return-e_not_found;
       if (!f->f_indirect) {
               if (R = Alloc_block ()) < 0)
                       Return-e_no_disk;
               F->f_indirect = R;
               Memset (Diskaddr (R), 0, blksize);
               Flush_block (DISKADDR (R));
       }
       if (ppdiskbno)
               *ppdiskbno = (uint32_t*) diskaddr (f->f_indirect) + filebno-ndirect;
       return 0;
}

The File_get_block function calls the File_walk_block function to find the target block in the file and then converts it to an address in the address space assigned to the BLK.

int file_get_block (struct file *f, uint32_t filebno, char **blk) {//LAB 5:your code her
       E. int r;

       uint32_t *ppdiskbno;
       if (R = File_block_walk (f, Filebno, &ppdiskbno, 1)) < 0) return R;
                if (*ppdiskbno = = 0) {if (R = Alloc_block ()) < 0) Return-e_no_disk;
                *ppdiskbno = R;
                Memset (Diskaddr (R), 0, blksize);

       Flush_block (DISKADDR (R));
       } *blk = Diskaddr (*ppdiskbno);
return 0; }

the file system interface
Now that we have implemented the necessary functions of the filesystem, we need to make it available to other process calls. We will use the IPC method implemented in LAB4 to allow other processes to interact with the file system service process for file operations. The


under dotted line section is the mechanism by which a normal process can send a read request to the file system service process. First read operation file descriptor, distributed to the appropriate device read function Devfile_read. The Devfile_read function implements the read disk file as a client file operation function. The parameters of the request structure are then established, and the FSIPC function is called to send the IPC request and parse the returned result.
The code for the file system server is in FS/SERV.C, and the service process loops until it receives 1 IPC requests. The appropriate processing functions are then distributed and the results are sent back through the IPC. For read requests, the server is distributed to the Serve_read function
in the IPC mechanism implemented by Jos, allowing the process to send a 32-digit and 1-page. To implement sending 1 requests from the client to the server, we use 32-bit numbers to represent the request type, and the storage parameters are in the federated FSIPC on the shared page. On the client side we always share Fsipcbuf page, on the server we map the request page to the Fsreq address (0x0ffff000). The
server also sends results through the IPC. We use a 32-digit number as the return code for the function. The Fsreq_read and Fsreq_stat functions also return data, which writes data to the shared page back to the client.

Union FSIPC {struct Fsreq_open {char req_path[maxpathlen];
        int Req_omode;
        } Open;
                struct Fsreq_set_size {int req_fileid;
        off_t req_size;
        } set_size;
                struct Fsreq_read {int req_fileid;
        size_t Req_n;
        } read;
        struct Fsret_read {char ret_buf[pgsize];
        } Readret;
                struct Fsreq_write {int req_fileid;
                size_t Req_n;
        Char Req_buf[pgsize-(sizeof (int) + sizeof (size_t))];
        } write;
        struct Fsreq_stat {int req_fileid;
        } stat;
                struct Fsret_stat {char ret_name[maxnamelen];
                off_t ret_size;
        int ret_isdir;
        } Statret;
        struct Fsreq_flush {int req_fileid;
        } flush;
  struct Fsreq_remove {char req_path[maxpathlen];      } remove;
Ensure FSIPC is one page char _pad[pgsize]; };

Here you need to know the union FSIPC, the file system in the client and server communication through the IPC, the data format of the communication is the Union FSIPC, each of its members corresponds to a file system operation request. Each time a client sends a request, it puts the parameter into a union FSIPC mapped physical page to the server. At the same time the service side will also put the processed results into the FSIPC, passed to the client. The address space layout of the file server is as follows:
　　
The OpenFile structure is a mapping of the service-side process maintenance, which corresponds to a file descriptor struct FD that is opened by a real-world file and a user client. The struct FD corresponding to each open file is mapped to 1 physical pages up to Fileeva (0xd0000000), and the server and the client process that opened the file share this physical page. Use 0_fileid to specify the files to manipulate when the client process and the file system server communicate.

struct OpenFile {
        uint32_t o_fileid;      File ID
        struct file *o_file;    Mapped descriptor for open file
        int o_mode;             Open mode
        struct Fd *o_fd;        Fd page
};

The file system default maximum number of files can be opened is 1024, so there are 1024 strcut Openfile, corresponding to the server process address space 0xd0000000 up to 1024 physical pages, to map these corresponding struct Fd.
struct FD is a 1 abstraction layer, Jos and Linux, all the IO is a file, so the user sees is the FD represents the file. But FD records its corresponding objects, such as real files, sockets, pipelines, and so on. Now only the files are used, so there are only 1 fdfile in the Union.

    struct FD {
        int fd_dev_id;
        off_t Fd_offset;
        int fd_omode;
        Union {
                //File server files
                struct fdfile fd_file;};
};

Exercise 5:
Implement the Serve_read function in the fs/serv.c file.
Reply:
First, we need to clarify the internal structure of the service-side process, working mechanism.

void serve (void) {uint32_t req, whom;
        int Perm, R;

        void *pg;
                while (1) {perm = 0;
                req = IPC_RECV ((int32_t *) &whom, fsreq, &perm); if (Debug) cprintf ("FS req%d from%08x [page%08x:%s]\n", req, W

                Hom, Uvpt[pgnum (Fsreq)], fsreq); All requests must contain an argument page if (! (
                                Perm & Pte_p) {cprintf ("Invalid request from%08x:no argument page\n",
                        whom); Continue

                Just leave it hanging ...}
                pg = NULL;
                if (req = = Fsreq_open) {r = Serve_open (whom, (struct fsreq_open*) fsreq, &AMP;PG, &perm); } else if (req < nhandlers && Handlers[req]) {r = Handlers[req] (whom, FSR
                EQ);
} else {                        cprintf ("Invalid request code%d from%08x\n", req, whom);
                R =-e_inval;
                } ipc_send (whom, R, pg, PERM);
        Sys_page_unmap (0, Fsreq); }
}

The server-side main loop accepts the file operation request of the client process using a polling method. Each operation is as follows:
1. Accept 1 Request type req and data page from IPC Fsreq
2, and then according to Req to perform the corresponding service-side processing function
3. The execution result of the corresponding service-side function (PG if the data is generated) is sent back to the calling process via IPC
4, Map the physical page Fsreq unmap
　　
The service-side function is defined in the handler array, called by the request number.

typedef int (*fshandler) (envid_t envid, Union FSIPC *req);

Fshandler handlers[] = {
        //Open is handled specially because it passes pages
        /* [Fsreq_open] =       (fshandler) ser Ve_open, *
        /[Fsreq_read] =          serve_read,
        [fsreq_stat] =          serve_stat,
        [Fsreq_flush] =         ( Fshandler) Serve_flush,
        [fsreq_write] =         (Fshandler) serve_write,
        [fsreq_set_size] =      (fshandler ) serve_set_size,
        [Fsreq_sync] =          serve_sync
};
#define Nhandlers (sizeof (handlers)/sizeof (Handlers[0]))

For read file requests, call the Serve_read function to process.

int
serve_read (envid_t envid, Union FSIPC *IPC)
{
        struct fsreq_read *req = &ipc->read;
        struct Fsret_read *ret = &ipc->readRet;

        if (Debug)
                cprintf ("Serve_read%08x%08x%08x\n", Envid, Req->req_fileid, req->req_n);

        struct OpenFile *o;
        int R, Req_n;

        if (r = Openfile_lookup (Envid, Req->req_fileid, &o)) < 0)
                return R;
        Req_n = req->req_n > pgsize? pgsize:req->req_n;
        if (r = File_read (O->o_file, Ret->ret_buf, Req_n, O->o_fd->fd_offset)) < 0)
                return R;
        O->o_fd->fd_offset + = r;

        return r;
}

The structure of the read request is first fetched from the FSIPC, then the openfile structure of the Fileid is found in the OpenFile, followed by reading from OpenFile's o_file to save the returned result and moving the file offset pointer.
Then we can look at the user process to send a read request function Devfile_read, the main operation is to encapsulate the FSIPC set request type Fsreq_read, after receiving the return, will return the results copied to their own buf.

Static ssize_t
devfile_read (struct Fd *fd, void *buf, size_t N)
{
        int r;

        Fsipcbuf.read.req_fileid = fd->fd_file.id;
        Fsipcbuf.read.req_n = n;
        if (r = FSIPC (Fsreq_read, NULL)) < 0)
                return R;
        ASSERT (R <= N);
        ASSERT (R <= pgsize);
        Memmove (buf, Fsipcbuf.readRet.ret_buf, R);
        return r;
}

Exercise 6:
Emulate the read request to implement the Serve_write function and the Devfile_write function.
Reply:
The implementation is similar to the read request.

fs/serv.c int Serve_write (envid_t envid, struct fsreq_write *req) {if (Debug) cprintf ("Serve_w

        Rite%08x%08x%08x\n ", Envid, Req->req_fileid, req->req_n);
        struct OpenFile *o;

        int R, Req_n;
        if (r = Openfile_lookup (Envid, Req->req_fileid, &o)) < 0) return R; Req_n = req->req_n > pgsize?
        pgsize:req->req_n;
        if (r = File_write (O->o_file, Req->req_buf, Req_n, O->o_fd->fd_offset)) < 0) return R;

        O->o_fd->fd_offset + = r;
return R;

        }//Lib/file.c static ssize_t devfile_write (struct Fd *fd, const void *buf, size_t n) {int r;
        if (n > sizeof (FSIPCBUF.WRITE.REQ_BUF)) n = sizeof (FSIPCBUF.WRITE.REQ_BUF);
        Fsipcbuf.write.req_fileid = fd->fd_file.id;
        Fsipcbuf.write.req_n = n;
        Memmove (Fsipcbuf.write.req_buf, buf, N);
     if ((R = FSIPC (Fsreq_write, NULL)) < 0)           return R;
return R; }

spawning Processes
The Spawn function has been implemented in LIB/SPAWN.C to create 1 new processes and to load 1 programs from the file system, and then the parent process continues execution. This is similar to the Unix fork function, which executes exec immediately after creating a new process.
Exercise 7:
The Spawn function relies on the new system call Sys_env_set_trapframe to initialize the state of the newly created process. Implement the Sys_env_set_trapframe function (do not forget to distribute the corresponding call number in syscall).
Reply:
The Sys_env_set_trapframe function is simple to implement, primarily to copy the registers of the parent process.

static int
sys_env_set_trapframe (envid_t envid, struct trapframe *tf)
{
        struct env *e;
        int R;

        if (r = envid2env (Envid, &e, True)) < 0)
                return-e_bad_env;
        User_mem_assert (E, TF, sizeof (struct trapframe), Pte_u);
        E->ENV_TF = *TF;
        E->env_tf.tf_cs |= 3;
        E->env_tf.tf_eflags |= fl_if;

        return 0;
}

sharing library state across fork and Spawn
Unix file descriptors include Pipe,console I/O. In Jos, these device types have 1 struct Dev associated with it, with function pointers that implement file operations such as Read/write. The traditional UNIX file descriptor interface is implemented in LIB/FD.C. The
also includes a file descriptor layout for each client process in LIB/FD.C, starting at fstable. This space retains 1 pages of address space for each descriptor. At any time, the page is mapped in the File descriptor table only if the file descriptor is in use.
We want to share the state of the file descriptor by calling fork and spawn to create a new process. Now, the fork function uses cow to copy the state 1 copies instead of sharing. In Spawn, the state is not copied but completely discarded.
So we'll change the fork to share the state. The Pte_share bit is newly defined in Inc/lib.h to identify the page share. When the bit is set in the page table portal, the PTEs should be copied from the parent process when the PTE is mapped to the child process at fork and spawn.
Exercise 8:
Change the duppage function to achieve the above changes, if the page table entry has a pte_share bit set, then copy the map directly. Similarly, implement the Copy_shared_pages function.
Answer:

static int Duppage (envid_t envid, unsigned pn) {int r;
        void *addr;
        PTE_T Pte;

        int perm;
        addr = (void *) ((uint32_t) pn * pgsize);
        Pte = UVPT[PN]; if (Pte & Pte_share) {if (R = Sys_page_map (Sys_getenvid (), addr, Envid, addr, Pte & Pte_syscall)) &L T
                        0) {Panic ("Duppage:page mapping failed%e", R);
                return R; }} else {perm = Pte_p |
                Pte_u; if (Pte & Pte_w) | |
                        (Pte & Pte_cow))
                Perm |= Pte_cow;  if (r = Sys_page_map (thisenv->env_id, addr, envid, addr, perm)) < 0) {panic ("duppage:page
                        Remapping failed%e ", R);
                return R; } if (perm & Pte_cow) {if (R = Sys_page_map (thisenv->env_id, addr, Thisen
    v->env_id, addr, perm)) < 0) {                            Panic ("Duppage:page remapping failed%e", R);
                        return R;
}}} return 0; }

New Pte_share-bit judgment is added on the original basis.

Copy_shared_pages (envid_t child)
{
        int i, J, PN, R;

        for (i = PDX (utext), I < PDX (uxstacktop), i++) {
                if (Uvpd[i] & pte_p) {for
                        (j = 0; J < Nptentries; J + +) {
                                pn = pgnum (P