NSQ source Reading (iv) Diskqueue file read/write

This is a creation in Article, where the information may have evolved or changed.

Introduction

As we mentioned in the first article in this series, if you need a message to land on the selection of the storage subsystem, Speed 文件系统>分布式KV（持久化）>分布式文件系统>数据库 . Instead, NSQ selected the file system as the storage subsystem. This article will focus on the operation of NSQ for files.

When is the file written?

When the memory msg Chan buffer is full, MSG is written to the file with the following code:

func (c *Channel) put(m *Message) error {    select {    case c.memoryMsgChan <- m:    default:        b := bufferPoolGet()        err := writeMessageToBackend(b, m, c.backend)        bufferPoolPut(b)        c.ctx.nsqd.SetHealth(err)        if err != nil {            c.ctx.nsqd.logf(LOG_ERROR, "CHANNEL(%s): failed to write message to backend - %s",                c.name, err)            return err        }    }    return nil}

Write a message

func (d *diskQueue) writeOne(data []byte) error {

diskQueueMaintains offset for writing files and writing files

// diskQueue implements a filesystem backed FIFO queuetype diskQueue struct {    ...    writePos     int64    ...    writeFile *os.File    ...}

Use the Seek function to set the offset of the write file to writePos :

        if d.writePos > 0 {            _, err = d.writeFile.Seek(d.writePos, 0)

The size of data is then written in binary form:

dataLen := int32(len(data))d.writeBuf.Reset()err = binary.Write(&d.writeBuf, binary.BigEndian, dataLen)

The trick here is binary.Write to write a fixed-size piece of data based on the type of data being written. The Datalen here is Int32, so a 4 byte data is written to represent the size of the database. The size of the data is known by reading a byte of 4 bytes first.
Then write to data:

_, err = d.writeBuf.Write(data)_, err = d.writeFile.Write(d.writeBuf.Bytes())

Read a message

readOneThe function reads a message in the form of a byte array.

func (d *diskQueue) readOne() ([]byte, error) {

diskQueueMaintains the offset of files and files currently being read

// diskQueue implements a filesystem backed FIFO queuetype diskQueue struct {readPos      int64...readFile  *os.File

Use the Seek function to set the offset of the current file to Readpos:

        if d.readPos > 0 {            _, err = d.readFile.Seek(d.readPos, 0)            if err != nil {                d.readFile.Close()                d.readFile = nil                return nil, err            }        }

Read the size of a message first:

err = binary.Read(d.reader, binary.BigEndian, &msgSize)

msgSizeAnd when you write a file, dataLen it's all int32 type.
Yes msgSize , define a buffer of a msgSize size, read a piece of data from a file to fill up the data inside the Buffer,buffer is a message

  Readbuf: = Make ([]byte, Msgsize) _, err = Io. Readfull (D.reader, readbuf)