Web services development experience and project lessons

Last year, Web services was used in a large project (1500 million). Now the project is coming to an end, so I will summarize my previous development experience.
I am sure you will ask? Why do you need web services in your project because the customer has the following requirements:
1. The customer requires the project to use the C/S architecture, and the server is IBM's: WebSphere appserver + DB2 + aix5.3 + RS/6000.
2. the end user reports data. For Network reasons, such as modem surfing, the data can be operated offline. After dozens of reports are filled in, the data can be submitted at one time. At the same time, you can synchronize the server data to the local access or MSSQL database during login, which improves the client response speed.
3. Some reports may need to be modified in the future, or some new reports need to be added without re-development, so that the customer's staff can customize them through the client.

If you have the above requirements, I think everyone should agree with this heterogeneous Distributed Solution: the client is developed with C #. Net and the Java component on the server is called through Web Services.

In fact, the above solution is too ideal, and we have to face the cruel reality: two of the three clients are forced to change to B/S.
In the project, I am mainly responsible for various problems encountered in the development of Web Services and server components, which is equivalent to technical support and development of some modules.

The following are the actual problems we encountered during development. Although they were all solved in the end, they encountered several bottlenecks that could not be broken through: unstable client, slow client response, and great difficulties in subsequent testing and maintenance.

1. Web Services compatibility of heterogeneous platforms
In the development process, we use axis as the Web Services engine and tomcat as the container. Because we only have a 60-day trial version of rad6.0 provided by IBM. This tool occupies the memory, and the built-in WebSphere development test is extremely slow, seriously affecting the development efficiency. After my initial trial, it was basically discarded. We recommend that 20 or 30 of developers in the project team use lomboz eclipse3.12 for development, which is basically satisfactory.

Since axis is an embedded engine, it can be packaged into the final WebSphere appserver (was). That is to say, we do not use the Web Services engine provided by was, this leads to the following question: how to deploy Web Services security?

When axis is used, axis has always had a bug or defect, and the official documentation also details it, but we didn't find it at the time and took a lot of detours: We used the Web Services released by axis to give it. net client calls must use the RPC style, but cannot use the Web Services standard cross-platform style document. The latter is the default lomboz axis plug-in method. That is to say, the web services we publish are always inexplicably unavailable. We released the axis plug-in that comes with jbuilder2007, and it went quite smoothly.

Ii. Server Components in Web Services Development
We use spring + hibernate for server-side development and encapsulate a layer on the service layer of spring, that is, the fa c Ade mode. The Fa C Ade is directly published as Web Services, this conversion is required because of the performance, and because of the complex model object of hibernate. the net Client recognizes some problems, and the list and map may also have problems. In short, these objects are too complicated and we wrap them into simple VO objects.

Another problem is that our service method, if we use a framework such as webwork on the server side, will not cause any problems. net client, the lazy loading error will occur, because. the. NET client cannot receive proxy objects. It must load all the data, but the hibernate session is closed. Many people in the project team encountered these problems. In the end, they all used the eager mode, leading to the final consequence: serious performance problems. Because I am not a leader, I found this problem and couldn't ask others. After all, a large team.
Experience: if a team is unfamiliar with hibernate, the technical risk is very high. The development efficiency brought about by Hibernate is based on the knowledge of the team members.

Of course, the performance problem is not only caused by hibernate, but also caused by the severe performance of Web Services: time-consuming XML serialization and deserialization, and network transmission caused by XML file expansion, the stateless HTTP results in network I/O performance. Experience: if the system must be distributed, rather than pursuing the so-called SOA architecture, Web services should be the next policy, because there are many protocols and methods to choose from: IIOP, RMI, Hessian, burlap, and rpc. In addition, there is a message method for system integration.

There are few other problems in Web services development, because Web services do not need to be programmed, but they are only deployed. development tools and servers will automatically do it for us, we only need to understand the principles and usage of the SOAP engine. If a problem occurs, we can monitor the soap data packets through the tcpmonitor of axis.
During development ,. on the Net client side, vsstudio is very intelligent. It will generate everything we need based on the WSDL file. Of course, changes to the WSDL file will cause vsstudio to regenerate all classes and interfaces, it is also time-consuming and prone to problems.

Iii. Web Services security issues
At that time, it took me nearly a month to solve the web services security problem, mainly to learn and handle the following four problems:
XML and web services security specifications
Secure deployment of was Web Services Engine
Axis and the xfire engine Web Services Security
. Net client wse3.0 Security and was communication

In the end, these problems are basically solved, but they are still useless, because it is very troublesome to deploy several clients and servers that we have developed, and we need to test them. In addition, customers cannot accept this either.

Of course, we also avoided a serious problem: Our Web services are released on the axis engine, and have not been transplanted to the was Web Services engine, but released on this platform, there must be development tools such as rad, and it is almost impossible to do it manually. Was engine's Web Services security configuration is exceptionally complicated: We only configured authentication and integration, and didn't do encryption, but it was enough. At that time, I used Sun's netbean development tool to release web Serivces, which is also quite useful, but not configured for security. By the way, sun's Web Services engine connector sdp2.0 is designed to be a bit similar to an EJB container, which is not easy to use and has poor portability.
At that time, we used the axis engine version 1.3, which does not support the standard oasis WS-Security. It was only available in version 2.0, and almost all of them were hand-written configuration files.

Compared with IBM redbooks, was's Web Services security configuration is not very difficult, but complicated. security-related XML Code contains hundreds of lines and several files. During configuration. net Client Communication encountered a problem, how can it not interwork,. net and. net client can interwork, Java and Java client can also interwork, finally I intercepted the soap package, find the solution: must manually change the http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3 after V3, IBM tool generation is V1, this is caused by incompatible standards, because rad was at the end of 04, while Microsoft's wse3.0 was relatively new. Later I found this article has a similar experience: http://pluralsight.com/blogs/kirillg/archive/2005/04/13/7315.aspx

During the process of processing web services security, I downloaded 10 books in this aspect through eMule and IBM. I think the following information is of the greatest help to me:

Axis documents: reference, developers-guide, architecture-guide, etc.
IBM redbooks: WebSphere version 6 Web Services handbook development and deployment.pdf and WebSphere Application Server V6 Security handbook.pdf, which specifically describes the principles and specific configurations of Web Services security, it is very easy to understand.
Securing web services with WS-Security: This book is very theoretical, but I think it is very good, although Amazon's ranking is not high.
Wse3.0 msdn documentation.

 

Iv. Communication issues during development
This should not be a technical issue, but a software development methodology issue, but it has a great impact on the entire software development process.
The reality we face:. Net client developers do not understand server-side Java, and server-side Java developers do not understand. net.
In addition to technical barriers, there are also business cohesion issues, because we are not vertically divided into modules, and the premise of horizontal development is that our server developers are very familiar with the business and know the interfaces required by the client, but in fact, the business is mainly driven by the client. Therefore, developers at both ends have encountered great communication barriers.

The technical point of view is: the interface required by the client developers. The server developers are unclear, and the server developers do not know how to grasp the granularity. For example, there is an updateuser method, but when updating user information, a lot of information may need to be updated: user information, user role, user group ..... Loaduser has the same On-Demand Loading Problem.

Of course, from a technical point of view, the development of Web Serivces has two development modes: bottom-up and top-down. We chose the former, which is also the most common method. Maybe the latter is more suitable for our project: Starting from the definition of the WSDL file, both client and server development follow it. But the problem is: how do we determine the WSDL, that is, the interface we require, because we are not very familiar with the business.

V. client and server development and testing methods
At that time, we were very stupid and most direct: After the server component was released, we notified the client developers, and then the client developers used the Web Services generation tool provided by vsstudio, generate the required. net object, and then use it like a local call.

But the problem is:
WSDL changes at any time, which means that the components generated by the client are always changing and compilation errors often occur.
The problems encountered during client development will be the client itself and the server components at the same time: the methods I want do not contain enough information.

Once the server component is tested, the container is slow and the client call is slow.
What kind of Quagmire was involved in our test: for example, to test a report, we manually enter the Report on the client, and then observe the server logs and responses. Someone may ask, isn't it OK to use automatic testing tools such as LoadRunner or function tester? I have used them. They are really useful for Web UI, and the latter is also useful for Swing clients, but they are not realistic for. Net clients.
In addition, debug is very difficult because it requires developers at both ends to work closely together.

I think the solution is not necessarily useful:
The server service method must be written to the unit test testcase. The code may be large and published as Web services only after the test.
At the same time, the server provides the mock Implementation of the same interface for client development and testing to solve the problem of parallel development.

6. Other problems
Of course, I have omitted all the above points. In short, there are many problems: technical problems, management problems, methods, and process problems.

We have developed almost two "service layer + Persistence" solutions, because the offline client also uses nhib.pdf for persistence, which leads to a huge development and testing workload, let's just put it one thing: both sides are synchronized through packaged SQL statements and transmitted through soap, but access and DB2 SQL are incompatible. To be compatible, the cost of performance and flexibility is reduced.

In addition, we were passive when writing project proposals, but there was no way because there were several companies competing. Several problems that have a great impact on us:

1. IBM was is far easier to use than WebLogic Server, resulting in extremely time-consuming deployment. There were also some bugs, such as connection pool resources, which had to be consulted with IBM engineers. In fact, we only use a very small part of the powerful was: Web container.
2. We do not have a development tool RAD for was. But to be honest, the trial version of rad is also a word: slow, and super-large during installation, about 4G. It cannot be integrated with the version control tool VSS that we are already using.
3. The project's C/S architecture is not very reasonable, that is, the customer's B/S architecture also runs well and runs on a PC server using ASP. We are technical to a certain extent. At last, it could not meet customer requirements: performance + stability.
4. The custom report is not put into use at the end, but a half-finished product. This custom report is very difficult. If it is easy, a developer of the software can easily solve the problem of persistence of the presentation layer. Generally, MIS developers should not be unemployed, MDA is not that strong. Many OA platforms have been solving this problem and have not found it particularly useful. We tried Ms infopath, Adobe designer, and Excel server during our technical research.

 

Of course, this subsystem is only part of our huge system. It's also a lesson to make a summary of the above! However, from a personal perspective, there are still a lot of things to learn.

This subsystem took us nearly 200 man-months. If we say that the waste was at least 100 man-months. Why? From this article, I can only look at one corner, because the entire system involves CMS, OA, Bi, e-commerce, GIS, Im, and MIS. I will summarize the following reasons:
1. The project proposal is empty and impractical: The company is helpless and the customer is not mature.
2. Demand Analysis after demand investigation closed doors: the customer's contract is divided into stages. We pay 20% yuan after submitting the demand statement and 20% yuan after submitting the Design book. All are developed in one waterfall, though written in accordance with the RUP document. During actual development half a year later, many demands were found unreasonable.
3. The whole process has not been communicated with the customer, so the customer can see it after the final development. At that time, the customer also failed: This is not the product I want. Change, change, stay up late.
4. The overall technical strength of the project team was weak. At that time, only a few people were involved in Java development, most of which were on demand. Think of hibernate. In addition, was and Aix problems have emerged in the product environment: system instability and downtime.
5. the entire development process is not well grasped, such as project specifications, test methods, logs, and version control. These problems have emerged in the future and are very serious. Let's talk about the log. I don't know how to check the log at last, and the log is messy.
6. Lack of experience in big projects, the entire system architecture is loose, and many people do not know where to start at the beginning of the project, and it is also very hasty.
7. The project lasted for more than a year, and several batches were changed, which caused a great deal of work handover.
.....

But to be honest, the project team, especially the employees who have been in the company for one or two years, worked very hard and no one complained about anything. I was very happy to work with him or her.