Nginx cluster for uploading and downloading large WCF files (6 GB transmission supported), nginxwcf

Nginx cluster for uploading and downloading large WCF files (6 GB transmission supported), nginxwcf

Directory

1 General idea... 1

2 Nginx cluster-Uploading and downloading large WCF files... 1

3 BasicHttpBinding-related configuration parsing... 2

4. Compile the WCF Service and client program... 3

5 URL reserved items... 8

6. Deploy the WCF Service Program to one PC in the LAN... 8

7. Nginx cluster configuration and setup... 9

8 running results of the WCF client program... 11

9 Conclusion... 13

 

1. General idea

L Nginx cluster-Uploading and downloading large WCF files

L BasicHttpBinding-related Configuration Analysis

TransferMode, messageEncoding, maxcompute edmessagesize, receiveTimeout, sendTimeout

L compile the WCF Service and client programs

L URL reserved items

L deploy the WCF Service Program to one PC in the LAN

L Nginx cluster configuration and Setup

L running result of the WCF client program

L Summary

2 Nginx cluster-Uploading and downloading large WCF files

Nginx matching rules can easily help us to divide the network segments of the WCF Service, so as to achieve multi-regional division of the enterprise data information system, such as small data microservices, data file transmission services, instant messaging services, or email services, it is equivalent to building an enterprise's internal information-based Data Bus (DataBus ).

The main structure diagram described in this article is as follows:

The client accesses the Nginx domain name using BasicHttpBinding.

For "Cold data", you can divide directories, create separate FTP servers and WCF servers, and perform operations. As shown in:

The architecture of the WCF upload/download server is as follows:

 

3 BasicHttpBinding-related Configuration Analysis

BasicHttpBinding configuration. For details, refer:

Https://msdn.microsoft.com/zh-cn/library/system.servicemodel.basichttpbinding.aspx

The following are the main application configurations:

TransferMode	Gets or sets a value indicating whether to send messages through buffering or stream processing. (Inherited from HttpBindingBase .)
MessageEncoding	Get or set whether to use MTOM or text to encode the SOAP message.
Maxcompute edmessagesize	Obtains or sets the maximum size, in bytes. You can use the channel configured for this binding to receive messages. (Inherited from HttpBindingBase .)
ReceiveTimeout	Gets or sets the maximum interval between connections that remain inactive before undo. No application messages are received during this interval. (Inherited from Binding .)
SendTimeout	Gets or sets the time interval that can be used to complete write operations before transmission raises an exception. (Inherited from Binding .)

 

To improve the performance of MTOM encoding and asynchronous calling, see:

Optimize the performance of stream transmission by using mtom and different requests

 

4. Compile the WCF Service and client programs

L WCF Service Program

Program. cs

Using System. serviceModel; using Service; using System; namespace FileTransferHosting {class Program {static void Main (string [] args) {using (ServiceHost host = new ServiceHost (typeof (FileTransferOperation ))) {host. opened + = delegate {Console. writeLine (host. description. endpoints [0]. address. uri + "started. Press any key to terminate the service! ") ;}; Host. Open (); Console. Read ();}}}}

FileTransferOperation. cs

Using Service. interface; using System. IO; using System. serviceModel; namespace Service {[ServiceBehavior (ConcurrencyMode = ConcurrencyMode. multiple)] public class FileTransferOperation: IFileTransferOperation {// <summary> // upload a file // </summary> /// <param name = "remoteFile"> </param> public void UploadLoad (RemoteFileInfo remoteFile) {StreamToFile (remoteFile);} // <summary> // write a file/ /// </Summary> /// <param name = "remoteFile"> </param> public void StreamToFile (RemoteFileInfo remoteFile) {string fileFullPath = Path. combine (System. environment. currentDirectory, remoteFile. fileName); Stream sourceStream = remoteFile. fileByteStream; if (sourceStream = null) {return;} if (! SourceStream. canRead) {return;} // create a file stream and read the data in the stream to generate the file using (FileStream fs = new FileStream (fileFullPath, FileMode. create, FileAccess. write, FileShare. none) {const int bufferLength = 4096; byte [] myBuffer = new byte [bufferLength]; // data buffer int count; while (count = sourceStream. read (myBuffer, 0, bufferLength)> 0) {fs. write (myBuffer, 0, count);} fs. close (); sourceStream. close () ;}/// <summar Y> // download the file /// </summary> /// <param name = "fileName"> </param> /// <returns> </returns> public stream DownLoad (string fileName) {string fileFullPath = Path. combine (System. environment. currentDirectory, fileName); if (! File. exists (fileFullPath) // checks whether the File Exists {return null;} try {Stream myStream = File. openRead (fileFullPath); return myStream;} catch {return null ;}}}}

Server configuration file:

<?xml version="1.0" encoding="utf-8" ?><configuration>  <system.serviceModel>    <behaviors>      <serviceBehaviors>        <behavior name="FileTransferBehavior">          <serviceMetadata httpGetEnabled="true" />          <serviceDebug includeExceptionDetailInFaults="true" />        </behavior>      </serviceBehaviors>    </behaviors>    <bindings>      <basicHttpBinding>        <binding name="FileTransferdBinding" receiveTimeout="00:30:00" sendTimeout="00:30:00" maxReceivedMessageSize="6442450944" transferMode="Streamed"            messageEncoding="Mtom" />      </basicHttpBinding>    </bindings>    <services>      <service behaviorConfiguration="FileTransferBehavior" name="Service.FileTransferOperation">        <endpoint binding="basicHttpBinding" bindingConfiguration="FileTransferdBinding"            contract="Service.Interface.IFileTransferOperation" />        
L client program
Program. cs
Using System; using System. collections. generic; using System. IO; using System. linq; using System. text; using FileTransferClient. fileTransferService; namespace FileTransferClient {class Program {static void Main (string [] args) {Console. writeLine ("Enter the complete path of the file:"); string fullFilePath = Console. readLine (). trim (); using (FileTransferOperationClient proxy = new FileTransferOperationClient () {DateTime datetime1 = DateTime. now; byte [] buffer = System. IO. file. readAllBytes (fullFilePath); Stream streamBuffer = new MemoryStream (buffer); // upload the file proxy. uploadLoad (buffer. length, System. IO. path. getFileName (fullFilePath), streamBuffer); OutPutDiffTime (DateTime. now, datetime1, "uploaded successfully"); Console. writeLine ("start downloading file:"); DateTime datetime2 = DateTime. now; string filename = System. IO. path. getFileName (fullFilePath); // download the object Stream sourceStream = proxy. DownLoad (filename); if (sourceStream = null) {return;} if (! SourceStream. canRead) {return;} // create a file stream and read the data in the stream to generate the file using (FileStream fs = new FileStream (Path. combine (System. environment. currentDirectory, filename), FileMode. create, FileAccess. write, FileShare. none) {const int bufferLength = 4096; // data buffer byte [] myBuffer = new byte [bufferLength]; int count; while (count = sourceStream. read (myBuffer, 0, bufferLength)> 0) {fs. write (myBuffer, 0, count);} fs. close (); sourceStream. close ();} OutPutDiffTime (DateTime. now, datetime2, "Download successful"); Console. read () ;}} public static void OutPutDiffTime (DateTime datetime2, DateTime datetime1, string mesg) {TimeSpan ts = (datetime2-datetime1); string dateDiff = ts. hours. toString () + "Hour" + ts. minutes. toString () + "Minute" + ts. seconds. toString () + "seconds"; Console. writeLine (String. format ("spent {0}, {1}", dateDiff, mesg ));}}}
Client configuration file (note that the client transferMode = "Streamed" must use streaming data for processing and change the endpoint address to "http://zhyongfeng.com/fileupload "):
<?xml version="1.0" encoding="utf-8" ?><configuration>  <system.serviceModel>    <bindings>      <basicHttpBinding>        <binding name="BasicHttpBinding_IFileTransferOperation" receiveTimeout="00:30:00"          sendTimeout="00:30:00" maxReceivedMessageSize="6442450944" transferMode="Streamed"          messageEncoding="Mtom" />      </basicHttpBinding>    </bindings>    <client>      <endpoint address="http://zhyongfeng.com/fileupload" binding="basicHttpBinding"        bindingConfiguration="BasicHttpBinding_IFileTransferOperation"        contract="FileTransferService.IFileTransferOperation" name="BasicHttpBinding_IFileTransferOperation" />    </client>  </system.serviceModel></configuration>
As shown in:

5. URL reserved items
See: http://www.cnblogs.com/yongfeng/p/7851039.html
6. Deploy the WCF Service Program to one PC in the LAN
Remote Deployment of a WCF server program to a PC

7. Configure and build an Nginx Cluster
Access the Server Load balancer cluster through the self-built domain name zhyongfeng.com: 80, access C: \ Windows \ System32 \ drivers \ etc \ hosts, and add the following "Custom Domain Name of the local IP Address ":
10.93.85.66    zhyongfeng.com
The Nginx matching principle is used to configure one PC deployed on WCF as follows:
worker_processes  1;events {    worker_connections  1024;}http {    include       mime.types;    default_type  application/octet-stream;    sendfile        on;    keepalive_timeout  65;    upstream zhyongfeng.com {        server    10.92.202.56:5600;        server    10.92.202.57:5700;         server    10.92.202.58:5800;    }    server {        listen       80;        server_name  zhyongfeng.com;        location / {            proxy_pass   http://zhyongfeng.com;            proxy_connect_timeout       10s;        }     location /fileupload {            proxy_pass   http://10.92.202.56:5600/fileupload;            proxy_connect_timeout       10s;        }     }}
Run CMD:
D:\DTLDownLoads\nginx-1.10.2>start nginxD:\DTLDownLoads\nginx-1.10.2>nginx -s reload
Access the WCF server: http://zhyongfeng.com/fileuploadand run the result:
8. Running results of the WCF client program
Start the WCF client program and directly upload the Linux ubuntu System iso image. The total size is about GB. It takes about 7 minutes to upload the Local Area Network (the server line is 10 Gpbs, the download time is about 3 minutes.
The Network Status of the client server. The running effect is as follows:




 
 
9 Summary
The Nginx matching principle can effectively allocate URLs, distribute streaming data to corresponding services for processing, and support large upload/download functions in the LAN. The BasicHttpBinding configuration allows you to control the size of streaming data uploads and supports downloading streaming data to upload and download large files in WCF.
Of course, the specific application scenario should be based on the data size. Here is just a reference for a solution.
For example, if some video files are processed at 1 TB every day and PB-level big data files are processed, it is recommended that hadoop HDFS be used for better implementation.
On the other hand, invoices and report files with a data size of several MB or several KB are mostly large but not large. hadoop MapReduce is obviously a little useless, you can use the "Business Domain-Year-month-day" method to establish your own data structure storage method.
 
Source code download:
Http://download.csdn.net/download/ruby_matlab/10131307
 
PDF download:
Upload and download the nginxcluster WCF large file (supporting 6g ).pdf