使用s3 java sdk 分區檔案上傳API 報‘SignatureDoesNotMatch’ 異常的定位及解決方案

標籤：key   ack   amazon   9.png   roo   str   hub   encode   decode   

import java.io.File;

  import  com.amazonaws.AmazonClientException;

import  com.amazonaws.auth.profile.ProfileCredentialsProvider;

import  com.amazonaws.services.s3.transfer.TransferManager;

import  com.amazonaws.services.s3.transfer.Upload;

 

public  class  UploadObjectMultipartUploadUsingHighLevelAPI {

       public  static  void  main(String[] args)  throws  Exception {

         String existingBucketName =  "*** Provide existing bucket name ***" ;

         String keyName            =  "*** Provide object key ***" ;

         String filePath           =  "*** Path to and name of the file to upload ***" ; 

         

         TransferManager tm =  new  TransferManager( new  ProfileCredentialsProvider());       

         System.out.println( "Hello" );

         // TransferManager processes all transfers asynchronously,

         // so this call will return immediately.

         Upload upload = tm.upload(

                 existingBucketName, keyName,  new  File(filePath));

         System.out.println( "Hello2" );

 

         try  {

             // Or you can block and wait for the upload to finish

             upload.waitForCompletion();

             System.out.println( "Upload complete." );

         }  catch  (AmazonClientException amazonClientException) {

             System.out.println( "Unable to upload file, upload was aborted." );

             amazonClientException.printStackTrace();

         }

     }

}

問題描述：使用s3 java sdk 的如上代碼分區檔案上傳，報‘SignatureDoesNotMatch’ 異常如下

Unable to abort multipart upload, you may need to manually remove uploaded parts: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000003ddf-005853b76e-1a56f9b-default)
com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000003ddf-005853b76e-1a56f9b-default), S3 Extended Request ID: 1a56f9b-default-default

 

 

 

定位過程：

下載s3 java sdk 源碼，並配置好 log4j, 列印aws debug日誌； 

# For JBoss: Avoid to setup Log4J outside $JBOSS_HOME/server/default/deploy/log4j.xml!# For all other servers: Comment out the Log4J listener in web.xml to activate Log4J.#log4j.rootLogger=INFO, stdout, logfilelog4j.rootLogger=INFO,stdout,logfilelog4j.logger.com.amazonaws = DEBUGlog4j.appender.stdout=org.apache.log4j.ConsoleAppenderlog4j.appender.stdout.layout=org.apache.log4j.PatternLayoutlog4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - <%m>%n

RGW設定檔/etc/ceph/ceph.conf 中的記錄層級調整為20，重啟RGW進程

debug rgw = 20

 

執行分區上傳代碼，分別截獲用戶端和服務端的請求日誌。 如下2個圖片可以發現uploadId用戶端和服務端不一致，導致服務端進行簽名校正失敗。

 

 

 

  

那麼問題來了，到底是RGW服務端，還是java s3 sdk實現有bug呢？ 通過查看java 代碼和aws文檔描述，我認為是RGW簽名實現存在bug。 RGW 應該在

擷取從用戶端url中的參數中uploadId值（已經被URLEncode）後，先decode，再根據aws文檔描述的步驟進行encode（~特殊字元不encode），得到符合規則的參數字串。 

http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

 

 

規避方案：

因為修改RGW的代碼周期比較長，暫時先修改好修改的java代碼，編譯一個新的jar包。 如下是修改方法，不按照aws的規則來，對參數值只執行URLencode。

最終還是要提交一個patch到rgw git庫。

 

 https://github.com/BodihTao/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AbstractAWSSigner.java

 

為了方便大家，我上傳修改後編譯好的class檔案，把這2個檔案用winrar 放到aws-java-sdk-1.11.69.jar包中com/amazonaws/auth目錄下 替換原來檔案即可得到新的jar包。

https://github.com/BodihTao/aws-sdk-java/raw/master/aws-java-sdk/AbstractAWSSigner%241.class

https://github.com/BodihTao/aws-sdk-java/raw/master/aws-java-sdk/AbstractAWSSigner.class

 

使用s3 java sdk 分區檔案上傳API 報‘SignatureDoesNotMatch’ 異常的定位及解決方案