Tuesday, December 22, 2009

The Ephemeral Ports

Ephemeral Ports: I picked it up on wikipedia, The definiton goes as this -
"An ephemeral (short-lived) port is a transport protocol port for Internet Protocol (IP) communications allocated automatically from a predefined range by the TCP/IP stack software. It is typically used by the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or the Stream Control Transmission Protocol (SCTP) as port for the client end of a client-server communication when the application doesn't bind the socket to a specific port number, or by a server application to free up a service's well-known listening port and establish a service connection to the client host. The allocations are temporary and only valid for the duration of the connection. After completion of the communication session the ports become available for reuse, although most implementations simply increment the last used port number until the ephemeral port range is exhausted."

"Ephemeral port range gets exhausted -- > " This is one of the problems I faced during the performance and scalability testing of one of our product services. A very common mistake that caused it but took us a lot to figure out what was happening actually.

The error message said
"java.net.SocketException: No buffer space available (maximum connections reached?): listen failedjava.net.SocketException: No buffer space available (maximum connections reached?): listen failed"
We were really clueless as on a windows machine it took more then 3days of running the test case that this error would arise and on a Linux machine it would come back within an hour.
But thanks to the Linux machine settings we could nail this problem down.
So here is the list of suggested ideas that you can ponder on when you face such a problem

Some suggested ideas

1. The buffer the error message is talking about refers to the TCP/IP stack's buffers. The error you're seeing can be caused by multiple things: too much data queued up for send viaTCP/IP or you're out of ephemeral sockets. The big problem is that the number of available ephmeral sockets doesn't change just because you have a massive amount of RAM.

Download tcpview.exe from sysinternals.com: when the problem occurs, fire it up and see what process has the greatest number of sockets out there. You'll probably see a bunch of sockets associated with one or a few processes (and they'll probably be in the TIME_WAIT state...)



2. This can mean that you have run out of local TCP ports to use for outgoing connections at the applet host(s). I believe Windows client platforms limit these to ports 1024-5000. In turn this seems to mean that you aren't closing old connections (or that you really need this many outgoing connections at the same time, which seems hard to believe). See.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;196271

3. If an exception occurs while a binded socket tries to make a connection to a machine, the socket does not get unbounded and closed. Try adding a java try catch block to ensure all sockets get unbounded and closed if an exception is thrown while either
i) trying to establish a connection to the machine or
ii) performing file transfer.

4. A memory or file handle leak can also be the source of the problem. When a program opens up too many handles rapidly, it exhausts the system resources - hence throwing the "No Buffer Space" error. In our case, we are opening up too many handles for each request. To check and see how many handles the processes on your computer has opened up, look at your Task Manager and click View -> Select Columns -> Handle Count.