Troubleshooting Guide Smartfoxserver 2X

Troubleshooting Guide

In this guide, we provide tips and tricks to overcome the typical problems encountered when setting up or running a server.

Unable to reach server
Flash cross-domain Policy issue
server startup problem
Delete mail
ghost user

» Unable to reach server

A common run-time problem is that you cannot connect to a test or production server after the first installation.

Once your sfs2x instance is running, you should make sure that no firewall (software or hardware) is blocking the TCP port being used. Specifically, you should open TCP ports 9933 and 8080 to "world". If this is not possible, you should ask your system administrator to read this document.

If your server is not connected directly to the Internet, you should also make sure that the NAT service (port forwarding) is configured correctly.
The NAT service allows you to expose one or more services from your computer to a public Internet address. For example, a well-configured router sends all traffic from Port 9933 to a server computer running Smartfoxserver 2X.

To make sure something is blocking your server, you can try to telnet its public address from the outside. Telnet is a common command-line tool available on most operating systems and allows you to establish a TCP connection to a remote host. Perform the following steps to test the connection to the server.

Open the System console:
    use Windows Key + R under Windows, and then type cmd in the dialog box that appears.
    under MacOS X, the terminal is in/applications/utilities/terminal.
    under Linux ... Well, every Linux user should know how to start the terminal. :)

Type the following:
telnet <ip-address> 9933

Which is the IP address to test.

You should get something like this:
lapo$ telnet localhost 9933
Trying:: 1 ...
Connected to localhost.
Escape character is ' ^] '.
The server has started a new connection, so it is accessible.
If the attempt fails, you should carefully check that no firewall is blocking traffic and that port forwarding is configured correctly.

» Flash cross-Domain policy issues

If the server is up to its public address, but you are still experiencing connectivity problems, make sure that you are not affected by any Flash player sandbox restrictions. For security reasons, Flash Player does not allow connections to external domains, rather than setting explicit permissions through cross-domain policy files.

You can learn more about Flash Player security settings in this whitepaper (it is in the SFS 1 documentation, but the same concept applies to sfs2x).

Attention
SFS2X can provide policy files automatically via sockets. The file is located in Sfs2x/config/crossdomain.xml and can be edited to meet your requirements.
» Server Startup Issues

If the server does not complete the startup phase, we recommend that you check the log files.
SFS2X runs a two-phase boot sequence: Start the low-level network engine first, and then the Smartfox service starts in turn until the server is ready.

The first low-level boot stage generates detailed reports in {Sfs2x-install-dir}/sfs2x/logs/boot/folders. The Smartfoxserver operation log is located in {sfs2x-install-dir}/sfs2x/logs/folder.

Log files are rotated every day and are easily identified by the date appended to the file name.
» Discarded Messages

Many of the questions raised by our support Committee revolve around the topic of abandoning information. Sometimes you may notice that the count of dashboard modules from Admintool is very high, especially for outgoing deleted messages.

The incoming delete message is simple: sfs2x discards any messages that are malformed or that do not conform to their protocols. Additionally, it discards messages that have a value set in the Config/core.xml file with a size greater than the maxincomingrequestsize parameter.

The outgoing Delete message value retains the count of server messages that were not sent to their respective recipients. Before we discuss possible reasons for discarding information, let's look at the following chart:

The server associates an outgoing message queue to each connected session to store data that cannot be written immediately. In fact, the server attempts to pass each packet as fast as possible, but sometimes the user is connected to congestion and rarely even sends data.
In this case, the server is forced to store the remaining bits of data in the queue and wait for the network pipeline to be available again for the new transport.

Depending on the user configuration, Smartfoxserver retains the data in each queue until its capacity is exhausted: The message is discarded at this time. This mechanism allows the server to protect its own unrestricted memory allocations, which can eventually cause the Java VM to crash.

You can fine-tune the server tolerance for dropped messages from the Server Configurator module in Admintool.
» Reasons to discard outgoing messages

Bad or slow client connection: Slow user response, insufficient bandwidth to keep up with server response. This causes the socket to quickly become busy and force the server to keep the data in the queue. Too

much data is sent too often: this is a slight change in previous topics. Only this time, we don't necessarily blame client connections, but this may depend on the server logic sending too much data or too frequent updates. It is important to remember that on the Internet, each connection experiences a certain amount of latency, typically between 50 and 200 milliseconds, but can also be achieved in a few seconds.

It is important to remember these limitations and to use specific client and server algorithms to address these limitations to reduce lag. In general, it is not recommended to send more than 10-20 updates/sec over TCP, while the value of UDP packets may be higher.

If you are interested in learning more about network latency, we recommend that you read this article.

server-side bandwidth is low: when you reach the capacity of the host bandwidth, you will experience general network performance degradation. Your players may start complaining that the game or app is getting slower, and you should see an increasing number of messages being uploaded.

you can easily detect this issue by monitoring bandwidth usage daily.

»ghost Users

Ghost users can be generated when an incomplete TCP connection is disconnected. The TCP transport protocol uses a 4-way interchange between the client and the server to close a previously opened connection. If the communication suddenly breaks during the interchange, the disconnection does not complete and the client is in a partially fractured state.

The low-level TCP stack uses several flags to maintain the status of each connection that can be checked using the Netstat utility in the command line. To describe the technical details of TCP internal work, this section is outside the scope of this section, but if you want more information, you can review the following resources:
TCP/IP transport protocol
Netstat tools

Common causes of»ghost users

Ghost users are extremely rare when working in a Lan/wan environment because connections are straightforward and very congested. On the other hand, on the Internet, the situation is very different, because the client undergoes many network jumps, which represent the packets from one end of the connection (for example, the client) to the router/firewall/gateway (for example, the server) on the other end.

In this case, there is a consistent number of variables that can cause connection loss, including network congestion, timeouts, resets, non-recoverable packet loss, and so on. When this happens, the user socket can remain in a "delayed state" for a long time, as both parties wait for the last point of communication.

The Disconnect event cannot be notified until this process is complete. Therefore, the server does not know what happened, and the developer's custom extension code is also waiting for the User_disconnect event.

Depending on the operating system, each TCP implementation has slightly different rules and timeout settings to handle deferred connections: Eventually all of these sockets will be forcibly closed, causing the disconnect event to bubble to Smartfoxserver and its extension.
» How to deal with ghosts

Since version 2.4, Smartfoxserver 2X has tried to reduce the impact of a pending connection by running a scheduling task called Ghost Hunter, which checks the integrity of each connection and removes those stale connections.

The configuration of the server is also important to reduce this problem, especially with two settings that can be adjusted through Admintool:

serverconfigurator> Session Maximum idle time: Adjusts the maximum idle time for a single session. The session is used only by clients to log on to the server. Once they are logged in, the client becomes a user. Therefore, it is recommended that this value be kept within the range of 10-40 seconds. Any other clients that might open a connection to the server without logging on will be quickly removed.
zoneconfigurator> User Maximum idle time: Adjusts the user's maximum idle time. This value can vary depending on how the application works and how long the user is allowed to remain idle in the event of a continuous drive. In general, we recommend avoiding values that exceed 30 minutes.

Detecting network problems

It is important to note that ghost users should rarely appear in the production environment and should not exceed the 1-2% of the total CCU. If this occurs at a high rate, a comprehensive inspection of the production environment is required, including:

Server Configuration
Bandwidth resources (monitoring usage and peak)
network configuration (kernel settings, routers, firewalls)
security (DDoS attacks)