COMP2310/COMP6310:
Concurrent and Distributed Systems
Semester 2 2013

Laboratory 7

Sockets and UNIX Select

You are expected to use the man pages! Remember that man pages with the same title may appear in multiple sections. If you want to search all sections use man -a.

Earlier in the course you were introduced to the Ada rendezvous communication model. This provided a message passing based communication primitive involving two tasks. Recall that the Ada rendezvous was asymmetric, in that one task posts an accept, while any other task could call on the associated entry. This communication pattern typifies that required by the client/server model.

In this lab we explore the Unix equivalent of rendezvous, namely sockets. You will also use the Unix equivalent of the Ada select statement, also conveniently called select(). However, you will find that implementing sockets and/or using Unix select is considerably more involved than it is with Ada.

Sockets

Server Side

In the previous lab, you encountered pipes as a means of providing interprocess communication (IPC) within the Unix environment. To establish a pipe connection it was required, however, that the processes involved had common ancestry. Sockets remove this requirement, enabling point to point connections between processes that may reside on either the same, or physically separate, computer systems.

Creating and using a socket involves several steps. First you must create it using a call socket (do man 2 socket):

       int socket(int domain, int type, int protocol);

As you may deduce from the above sockets come in several flavors.

The first parameter dictates the addressing mode. The two most common schemes are AF_UNIX and AF_INET. The former uses Unix pathnames to identify the socket, while the latter uses Internet addresses (the four byte numbers usually written like 150.203.24.13 - which is the machine partch.anu.edu.au).
The second parameter is the type of socket. The two most common are SOCK_STREAM and SOCK_DGRAM. The first provides sequenced, reliable two-way communication based on byte streams (and is used for TCP communications). The second provides unreliable bunches of data called datagrams (and is used for UDP communications). We will use the former.
The third parameter is the protocol used for the communication. Giving a value of 0 results in the default protocol (IP) and will be used here (for a list of protocols look at /etc/protocols).

The return value is a socket descriptor that is used subsequently to refer to this socket.

Having created a socket we must associate it with an address. This is done using the bind() call and can be likened to assigning a telephone number to a telephone.

       int bind(int sockfd, struct sockaddr *my_addr, socklen_t addrlen);

In the above we pass a pointer to an address structure. With the AF_INET addressing mode there are two critical pieces of information required: i) the hostname ii) the port number (you may remember port numbers from COMP2300. You can view a list of port numbers for different services provided through those ports by looking at file /etc/services). When we bind a socket we will leave it up to the O/S to assign us a free port number.

For SOCK_STREAM sockets, a number of incoming messages can be queued, after which point subsequent messages are blocked. Normally the default number of queued messages is 5, but there is no requirement that this be true. As a consequence it is good practice to alway explicitly set the maximum number of queued messages. To do this we use the listen() function.

       int listen(int s, int backlog);

With the socket created, bound to an address and with a non-zero incoming queue limit, we are now in a position to establish point to point communications with other processes. This is done using the accept() call:

       int accept(int s, struct sockaddr *addr, socklen_t *addrlen);

This extracts the first connection request on the queue of pending connections, creates a new connected socket with mostly the same properties as the original, and allocates it a new file descriptor. Note that accept() can be either blocking or non-blocking, depending on the attributes of the socket. For our purposes, we will use only blocking sockets.

Client Side

In the above, the server has created a socket and is waiting to accept incoming communications. For the client to connect to that socket it must first create its own socket, binding it to an address and port number in an identical fashion. To contact the server, we then use the connect() function:

      int  connect(int  sockfd,  const  struct sockaddr *serv_addr, socklen_t addrlen);

In this call sockfd is the socket descriptor on the client side, while serv_addr is the address (host id and port number) on the server side. The obvious question - that we will return to, but you may want to consider - is how does the client know the port number of the server if that port number was dynamically assigned?

Communicating

Having established a socket connection, data can be communicated using a variety of system calls such as (send(), recv()) or (write()), read()). Essentially analogous calls to those used for writing data to a file. Moreover, just like file I/O we can terminate the socket connection by issuing a close() system call.

A Classic Server Program

The following code creates a socket and binds it to the default Internet interfaces with a port number that is determined by the O/S. It then waits for an incoming connection on that port number. When a connection is established, it receives an incoming character string, prints it, then appends to it and returns the string to the client.

Note that as we are using Internet based communications and as a result this could give rise to communicates between machines with different byte ordering (big and little endian). An IPv4 Internet address is represented as a 4 byte integer, so there would be obvious problems if one machine stored the IP representation of partch.anu.edu.au in memory as 150.203.24.13, while another effectively had it stored as 13.24.203.150. To remove this difficulty, TCP/IP defines what is called network byte ordering (which is in fact big endian) and assumes all IP related data uses this representation. In recognition of this Unix provides us with a variety of functions to convert from "host byte ordering" (be that big or little endian) to "network byte ordering". These are used in the following program. For more information do e.g. man ntohs.

#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <arpa/inet.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>

#define MAXBUF 128

int main(int argc, char* argv[])
{
  int sock_1, sock_2;                /* two socket descriptors */
  struct sockaddr_in client, server; /* address info;
                                        see /usr/include/linux/in.h */
  char buf[MAXBUF];                  /* data buffer */
  int nbytes;
  socklen_t namelen;

  /* ----Create TCP/IP socket---- */
  sock_1 = socket(AF_INET, SOCK_STREAM, 0);
  if (sock_1 == -1) {
    perror("socket() Socket was not created");
    exit(-1);
  }
  printf("Socket created successfully.\n");

  /* ----Address information for use with bind---- */
  server.sin_family = AF_INET;         /* it is an IP address */
  server.sin_port   = 0;               /* use O/S defined port number */
  server.sin_addr.s_addr = INADDR_ANY; /* use any interface on this host*/

  /* ----Bind socket to address and port---- */
  if (bind(sock_1, (struct sockaddr *) &server, sizeof(server))) {
    perror("Server bind error");
    exit(-1);
  }

  /* ----Find out what port number was assigned---- */
  namelen = sizeof(server);
  if (getsockname(sock_1, (struct sockaddr *) &server, &namelen)) {
    perror("Server get port number");
    exit(-1);
  }
  printf("The assigned server port number is %d\n", ntohs(server.sin_port));


  /* ----Set queue limits on socket---- */
  if (listen(sock_1, 1)) {
    perror("Server listen error");
    exit(-1);
  }

  /* ----Now we block waiting for a connection---- */
  namelen = sizeof(client);
  sock_2 = accept(sock_1, (struct sockaddr *) &client, &namelen);
  if (sock_2 < 0) {
    perror("Server accept failed");
    exit(-1);
  }
  getsockname(sock_2, (struct sockaddr *) &server, &namelen);
  printf("Server received connection from %s, now on port %d\n",
          inet_ntoa(client.sin_addr), ntohs(server.sin_port));

  /* ----Wait to receive some data---- */
  nbytes = recv(sock_2, buf, sizeof(buf)-1, 0);
  if (nbytes < 0) {
    perror("Server recv error");
    exit(-1);
  }
  buf[nbytes] = 0; // do not trust client to send a null-terminated string!
  printf("Server received message: %s", buf);

  /* ----Add string and return to client---- */
  strncat(buf, " and hello to you client\n", MAXBUF-strlen(buf));
  nbytes = send(sock_2, buf, strlen(buf), 0);
  if (nbytes != strlen(buf)) {
    perror("Server send error ");
    exit(-1);
  }

  /* ----Close sockets and terminate---- */
  close(sock_2);
  close(sock_1);

  printf("Server finished.\n");
  return 0;
}

A Classic Client Program

The following code is the client companion of the server program. Again we create a pipe, but to make a connection we need to know the IP address and the port ID of the server program. As we will be running the client and server on the same physical computer we can again use INADDRs_ANY as the IP address. The port number is, however, required as command line input. Thus this program can only be run after the server program has been started and printed out its assigned port number.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <arpa/inet.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h> //close()
#define MAXBUF 128

int main(int argc, char* argv[])
{
  int sock_1;                /* client socket */
  struct sockaddr_in server; /* server address */
  uint16_t server_port;      /* server port number */
  char buf[MAXBUF];          /* data buffer */
  int nbytes;

  if (argc != 2) {
    printf("Usage: %s port_number_of_server \n", argv[0] );
    exit(-1);
  }
  /* ----Port number on server---- */
  server_port = (unsigned short) atoi(argv[1]); 

  /* ----Create TCP/IP socket---- */
  sock_1 = socket(AF_INET, SOCK_STREAM, 0);
  if (sock_1 < 0) {
    perror("Client socket creation");
    exit(-1);
  }
  printf("Client socket created\n");

  /* ----Address and port information for server---- */
  server.sin_family      = AF_INET;
  server.sin_port        = htons(server_port);
  server.sin_addr.s_addr = INADDR_ANY; /* use any interface on this host*/

  /* ----Attempt to connect to the server---- */
  if (connect(sock_1, (struct sockaddr *) &server, sizeof(server))) {
    perror("Client connection failure");
    exit(-1);
  }

  /* ----Create a message---- */
  strncpy(buf, "Hello Server", MAXBUF);
  printf("Client sending message: %s\n", buf);

  /* ----Send the message---- */
  nbytes = send(sock_1, buf, sizeof(buf), 0);
  if (nbytes < 0) {
    perror("Client failed to send data");
    exit(-1);
  }

  /* ----Receive reply---- */
  nbytes = recv(sock_1, buf, sizeof(buf), 0);
  if(nbytes < 0) {
    perror("Client receive fail");
    exit(-1);
  }
  printf("Return message from server: %s\n", buf);

  /* ----Close socket and terminate---- */
  close(sock_1);
  printf("Client terminating\n");
  return 0;
}

ACTIONS

Compile both the server and client codes given above. Start the server program and wait until it prints out the port number. Now either background the server process, or in another shell window on the same computer run the client program providing on the command line the server port number. Make sure you are happy with how this is working.
* Modify the server program so that it uses fork() to create a new process that then issues an exec() call to execute the client program. The port number of the server is still passed to the client but as one of the arguments in the exec() call, i.e. there is no longer any need for you to type in the port number. Use wait() to ensure that the server terminates after the client (if in doubt about how to do this, go back to Lab 7). Show your solution to your tutor when done. Worth 1 Prac Point.
Now further extend the code so that the server spawns N client tasks (still via fork() and exec()), where N is given on the command line when the server is started. Have each client send a message to the server and the server respond. The server should again only terminate when all clients have terminated. Hint: the server can accept / read / write / close a new (sock_2) connection for each client.
Now run the original server program on partch and note the port number. We can communicate with this server process from your desktop machine using telnet. To do this do telnet partch server_port_number. When you do this, you should find that the window on partch where you are running the server responds to say that it has received a connection from the IP address of your desktop machine. Now type in the telnet window some short phrase - like hello server and press return. You should receive back your initial phrase augmented with and hello to you client.
Now modify the original client program so that it will connect with the server process running on partch. This will require you to change from using INADDR_ANY in the client program to explicitly giving the IP address of partch (note that you need the address in integer form; either you can convert by hand, or, better, use inet_pton() (recall dd2hex.c from COMP2300 Lab 09); gethostbyname() can also be used. However, you will still need to provide your client program with the relevant port number. You now have a truly distributed system!

The `select` Function

Now for the really tricky stuff. The function select() enables a process to wait for a change of status on any one of a variety of file descriptors. It also enables the process to set time limits similar to the use of "or delay" with the Ada select construction. The file descriptors of interest are grouped into what is called sets, with separate sets for read, write and exceptions. The format is as follows:

    int select(int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout);

The parameter n is one higher than the maximum file descriptor to watch. On return the function reports the number of file descriptors that have changed. Thus the user is then required to interrogate each file descriptor in turn to determine what has happened.

To manipulate the file descriptor sets the user is provided with a variety of macros: FD_CLR, FD_ISSET, FD_SET and FD_ZERO. For complete details, do man select

Below is an example of using select() with pipes. The program forks two children. Each child sends a request to the parent, who responds by sending them the value of a counter. After receiving the counter, they sleep for either 1 or 2 seconds. The parent terminates everything after the counter has reached a value of 20.

 
#include <sys/types.h> 
#include <sys/time.h> 
#include <unistd.h> 
#include <stdlib.h> 
#include <stdio.h>
#include <sys/wait.h>
#define MAXBUF 32
#define NUM_CHILD 2
#define COUNT_MAX 20
     
int main(int argc, char* argv[]) {
  int fd_pc[NUM_CHILD][2], fd_cp[NUM_CHILD][2];
  int maxfd;
  fd_set fd_read_set;
  char buffer[MAXBUF];
  int i, counter;
      
  /* ----Create the pipes---- */
  for (i=0; i < NUM_CHILD; i++) {
    if (pipe(fd_pc[i]) || pipe(fd_cp[i])) {
      perror("pipe create");
      exit(-1);
    }
  }
      
  for (i=0; i < NUM_CHILD; i++) {
    /* ----Create child i ---- */
    if (fork()==0) { /* ---- perform child i  ---- */
      int count;
      close(fd_pc[i][1]); /*Close parent to child write*/
      close(fd_cp[i][0]); /*Close child to parent read*/
      sprintf(buffer, "Request from child %d", i);
      do {
        write(fd_cp[i][1], buffer, MAXBUF);
        read(fd_pc[i][0], &count, sizeof(count));
        printf("Value of counter on child %d: %d\n", i, count);
        fflush(stdout);
        sleep(i+1);
      } while (count > 0);
      printf("Child %d terminates\n", i);
      fflush(stdout);
      exit(0);
    }
  }
 
  /* ----Now for the parent---- */
  for (i=0; i < NUM_CHILD; i++) {
    close(fd_pc[i][0]); /*Close parent to child read*/  
    close(fd_cp[i][1]); /*Close child to parent write*/
  }
      
  /* ----What is the maximum file descriptor for select---- */
  maxfd = fd_cp[0][0];
  for (i=1; i < NUM_CHILD; i++)
    if (fd_cp[i][0] > maxfd)
      maxfd = fd_cp[i][0];
  maxfd = maxfd + 1;
      
  counter = 0;
  /* ----Clear the read set---- */
  FD_ZERO(&fd_read_set);
  while (counter < COUNT_MAX) {
    /* ----Set file descriptor set to NUM_CHILD read descriptors---- */
    for (i=0; i < NUM_CHILD; i++) {
      FD_SET(fd_cp[i][0], &fd_read_set);
    }

    /* ----Wait in select until file descriptors change---- */
    select(maxfd, &fd_read_set, NULL, NULL, NULL);

    for (i=0; i < NUM_CHILD; i++) {
      /* ----Was it child i---- */
      if (FD_ISSET(fd_cp[i][0], &fd_read_set)) {
        read(fd_cp[i][0], buffer, MAXBUF);
        printf("%s\n", buffer);
        counter++;
        write(fd_pc[i][1], &counter, sizeof(counter));
      }
    }
  }
    
  /* ----Terminate the children with waiting---- */
  counter = -1;
  for (i=0; i < NUM_CHILD; i++)
    write(fd_pc[i][1], &counter, sizeof(counter));
  for (i=0; i < NUM_CHILD; i++)
    wait(NULL);
  exit(0);
}