Link to a very basic tutorial about TCP (evolved to: TCP-socket democode to send / receive data / characters similar to serial

Projects Programming
1

Hi everybody,

I'm somehow advanced in programming techniques like non-blocking timing, state-machines, registering and using call-back-functions, using parameters etc.
But I'm still a total noob about the subject of clients and servers.

Now I'm looking for a very very very basic tutorial about TCP-connections.

Whatever I have found so far by googling were demo-codes that immidiately jump up to the level of doing http-requests.

I explicitly want to stay

below HTTP

plain simple sending and receiving characters "ABC" , "123" by using the TCP-protocol

The information that comes closest to this provided by user @Juraj is this

But it is only a code-snippet and not a full demo-code

Of course I'm interested in a working demo-code but I'm also interested in understanding what TCP TCP/IP is itself

And that is the reason why I'm asking for a TCP-basics tutorial
ideally for ESP/arduino but not limited to them as TCP is used across a lot of programming languages. I think the underlying basic principles is alway the same. Just differencies in the code-syntax.

In this my focus is on code that shows the code that is nescessary to

  • establish a connection
  • send / receive plain simple sending / receiving characters over this TCP-connection

best regards Stefan

3 Likes
2

You need to read about the layer model to understand where you play

first google hit: ➜ https://www.ad-net.com.tw/osi-model-tcp-ip-network-models-must-concept-understand-move-deeper-networking-adventures/

As you can see in their table, the TCP/IP model offers multiple layers
2_scheme

So where you want to play is not the Application layer, but at the transport layer, in the world of TCP and UDP.

The web is full of articles explaining the difference, here are a couple ones from a google hit: ➜ TCP vs UDP: Key Difference Between Them and TCP vs. UDP: What’s the Difference? - Lifesize (many more to read)

So if you want to focus on TCP (connection-oriented versus the connectionless UDP), you need to be able to

  • establish the connection (ie identify who you are going to talk to)
  • discuss over the connection (both ways)
  • terminate the connection

So the question becomes how do you identify who you are going to talk to.

You are discussing with an application (or a service). TCP has something known as a port which is a unique number assigned to different applications. Standard services (applications) have well known ports that are documented

  • HTTP ➜ 80
  • HTTPS ➜ 443
  • FTP ➜ 21
  • FTPS / SSH ➜ 22
  • POP3 ➜ 110
  • POP3 SSL ➜ 995
  • IMAP➜ 143
  • IMAP SSL ➜ 993
  • SMTP ➜ 25 (Alternate: 26)
  • SMTP SSL ➜ 587
  • MySQL ➜ 3306
  • cPanel ➜ 2082
  • cPanel SSL ➜ 2083
  • WHM (Webhost Manager) ➜ 2086
  • WHM (Webhost Manager) SSL ➜ 2087
  • Webmail ➜ 2095
  • Webmail SSL➜ 2096
  • WebDAV/WebDisk➜ 2077
  • WebDAV/WebDisk SSL ➜ 2078
  • ...

so you need to know what PORT is used by the remote service you are trying to reach out to. You can create a service on the port you want, best is of course to stay out of well know ports if you want to offer a new service.

Then you need to know where is that service being provided from. That's where you need to read about the layer below. You will need an IP address (from the network Layer).

Once you have gathered the target IP and port number, you are all set. What you send/receive over that link is custom and depends on the application and usually there is a formally documented protocol to manage the conversation.

For example, HTTP is an extensible application layer protocol that you use over TCP* but as you have seen from the list of ports, there are many more frequently used services that have they own specification and of course you can build your own

The Arduino library offers the WiFi library to abstract that in the Client class. (there are also capabilities to support UDP)

Here are the methods you can use:

Client class

The client class creates clients that can connect to servers and send and receive data.

Assume you successfuly connected to the network then you can do

IPAddress remoteHost(xxx,xxx,xxx,xxx);  // IP address of the remote service
uint16_t portNumber = 21;               // FTP standard port number
WiFiClient client;
    if (client.connect(remoteHost, portNumber)) {
           // here you are connected, you can use the documented FTP protocol to discuss with the service
    }

The next question might be, "Ok, I need to know the IP, port and protocol of the service I want to reach out to in order to have something useful, but how do I create the service ?"

that's where the Server class is useful. It offers many methods too

Server class

The Server class creates servers which can send data to and receive data from connected clients (programs running on other computers or devices).

so you create a server on one of your ESP specifying the port you want to listen to (a uint16_t number) and use begin() to start the service.

uint16_t portNumber = 21;               // FTP standard port number
WiFiServer ftpService(portNumber);
...
void setup() {
  ...
  ftpService.begin(); // starts the service that will listen on port portNumber
  ...

makes sense? hopes it helps you get started.

have fun.

*footnote: HTTP can actually be sent over a TLS-encrypted TCP connection or theoretically any reliable transport protocol

7 Likes
3

Hi JML,

wow very extensive response, answering multiple questions.

The actual thing I want to learn is how to bi-directional send/receive characters just like send/receive in a serial-connection.

As the client-"object" (is "object" the correct name for it?
has functions
write(), print(), println() for sending
and
available(), read(), etc. for receiving
the line of code to send/receive is clear to me.

I guess if I want to use two ESP32 to make them send/receive characters I have to connect them to my local WiFi-network as always with some code like this

// helper-function for non-blocking timing
boolean TimePeriodIsOver (unsigned long &expireTime, unsigned long TimePeriod) {
  unsigned long currentMillis  = millis();  
  if ( currentMillis - expireTime >= TimePeriod )
  {
    expireTime = currentMillis; // set new expireTime
    return true;                // more time than TimePeriod) has elapsed since last time if-condition was true
  } 
  else return false;            // not expired
}

void ConnectToWiFi() {
  WiFi.mode(WIFI_STA);
  Serial.println("WiFi.mode(WIFI_STA)");
  Serial.print("trying to connect to #");
  Serial.print(ssid);
  Serial.println("#");
  
  WiFi.begin(ssid, password);
  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    static unsigned long MyTestTimer;
    if ( TimePeriodIsOver(MyTestTimer,1000) ) {
      Serial.print(".");    
    }
  }
  Serial.println("");
  Serial.print("Connected to #");
  Serial.print(ssid);
  Serial.print("# IP address: ");
  Serial.println(WiFi.localIP());
  WiFi.setAutoReconnect(true); 
  WiFi.persistent(true); 
}

again comparing it with a serial connection as something I know very well

Though it is unclear to me what will be the TCP-equivalent to the
Serial-code

Serial.begin(baudrate)

and what are the details that equivalent to the functionality of a bytewise receiving characters like in the serial-input-basics tutorial provided by robin2

void recvWithStartEndMarkers() {
  static boolean recvInProgress = false;
  static byte ndx = 0;
  char startMarker = '<';
  char endMarker = '>';
  char rc;

  while (Serial.available() > 0 && newData == false) {
    rc = Serial.read();

    if (recvInProgress == true) {
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else {
        receivedChars[ndx] = '\0'; // terminate the string
        recvInProgress = false;
        ndx = 0;
        newData = true;
      }
    }

    else if (rc == startMarker) {
      recvInProgress = true;
    }
  }
}

In most examples the connection is closed after receiving
Is this a must?
What would happen if I keep the connection "opened" all the time?

As I want to send / receive data BI-directional do I have to create a "server" and a "client" on both ESP32s?

additional there are WiFi-"servers" and Async-servers.
Async servers seem to work "in the backound"
As long as I don't flood a device with always new data using something that can do the receiving "in the background" would be more conveniant.

Can I simply replace a "synchronised" server with an async-Server?

best regards Stefan

4

yes two connected WiFiClients behave exactly the same way as Serial to Serial over RX/TX pins.

the server is there as dispatcher to listen to incoming connection and to create the local client object,

but here I only repeat what you already read in my SE answer.

5

I think that most examples are showing how to use the connection to get something and once it is delivered, the link can be dropped.

There's no reason why you can't keep a connection open forever if you want to, but you should probably add code to reestablish it if it drops for some reason.

If you write the code at each end, the connection should stay up, but a connection to some other service may be closed if there is no activity.

Firewalls may be problematic too. My boss and I spent several "happy" weeks trying to debug a new system that network operations insisted on adding a firewall to, between two of our servers. We eventually learned that the firewall killed any TCP connection that had no traffic for thirty minutes.

6

'async server' is arduino esp82666/esp32 terminology for specific web (http) server implementation. nothing related to what you need.

1 Like
7

yes you would handle that the exact same way as an asynchronous communication. Same technics apply.

if you never close the connection, it stays there but of course you are dependant on the layers that are lower than TCP for this to work. If at the network layer for example you loose the connection, then your TCP connection is dead. So that's why you need to make sure you listen to what happens across all layers and be able to recover from those possible issues. (The remote end can also decide to unilaterally close the connection

8

I took the examples in here and in ESP32 simple TCP-Democode Windows 10 telnet-TCP-connection closes instantly together with arduino ide - Multiple client server over wifi - Arduino Stack Exchange , played around with them and came up with the sample code below. This has been written and tested on an ESP8266. I don't have any other WiFi capable board to try it on but I imagine with minor modifications it should work on other WiFi or Ethernet capable boards.

Server
The server code creates a server capable of supporting multiple clients (limited to 10 in the example). For demonstration purposes when the server receives a single character it replies with the current value of millis(). The server should be given a static IP address and that IP address should be put in the client code so the client knows where to connect to.

// Demonstration code to transfer data from server to client using TCP.
// This code receives a 'request' in for form of a single character from the a connected, then responds with the value of Millis.
// The server can have multiple clients connected at the same time, in this example the number is limited to 10 clients.

#include <ESP8266WiFi.h>

#define STASSID "Put your WiFi SSID here"
#define STAPSK  "Put your WiFi password here"

const char * ssid = STASSID;
const char * pass = STAPSK;

#define MAX_CLIENTS 10
#define portNumber 7005
WiFiServer server(portNumber);
WiFiClient *clients[MAX_CLIENTS] = { NULL };

void setup() {
    Serial.begin(9600);
    //Serial.begin(115200);
    setup_wifi();
    //delay(500);
    setupServer();
    Serial.println("Setup complete");
}

void loop() {
    TCP_Server_multiple_Client();
}

void TCP_Server_multiple_Client() {
    uint32_t currentMillis = millis();
    const uint32_t timeOutPeriod = 15000;
    static uint32_t clientTimeout[MAX_CLIENTS];

    char dataToSend[50];
    
    // Check if a new client has connected
    WiFiClient newClient = server.available();
  
  if (newClient) {
    Serial.print("new client ");
    Serial.println(newClient);
    
    // Find the first unused space
    for (int i=0 ; i<MAX_CLIENTS ; ++i) {
        if (NULL == clients[i]) {
            clients[i] = new WiFiClient(newClient);
            clientTimeout[i] = currentMillis;
            break;
        }
     }
  }

  // Check whether each client has some data
  for (int i=0 ; i<MAX_CLIENTS ; ++i) {
    // If the client is in use, and has some data...
    if (NULL != clients[i] && clients[i]->available() ) {
      // Read the data 
        char newChar = clients[i]->read();
        Serial.print(i);
        Serial.print(" ");
        Serial.println(newChar);

        clientTimeout[i] = currentMillis;

        // Send response to client
        sprintf(dataToSend, "Server Millis is: %1ul\r\n", currentMillis);
        clients[i]->print(dataToSend);
    }
// Deletes clients that have disconnected properly
    if (clients[i] != NULL) {
        if (!clients[i]->connected()) {
            Serial.print("Client ");
            Serial.print(i);
            Serial.print(" is no longer connected and will be deleted");
            clients[i]->stop();
            delete clients[i];
            clients[i] = NULL;
            
        }
    }
// Deletes clients that have no activity for duration of timeOutPeriod
    if (currentMillis - clientTimeout[i] > timeOutPeriod) {
        if (clients[i] != NULL) {

            Serial.print("Deleting client ");
            Serial.println(i);
            
            clients[i]->stop();
            delete clients[i];
            clients[i] = NULL;
        }
    }
    
  }
}

//Setup functions
void setup_serial_port() {
    //uint32_t baudrate = 115200;
    uint32_t baudrate = 9600;
    Serial.begin(baudrate);
    Serial.println("");
    Serial.print("Serial port connected: ");
    Serial.println(baudrate);
}

void setup_wifi() {
    uint8_t wait = 0;
    while (WiFi.status() != WL_CONNECTED) {
        if (wait) {
            --wait;
        } else {
            wait = 40;
            WiFi.mode(WIFI_STA);
            WiFi.begin(ssid, pass);
        }
    delay(100);
    }
    Serial.println("");
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
}

void setupServer() {
    server.begin();
    //server.setNoDelay(true); // Sends small packets immediately without consolidating a number of small packets into one big packet
}

Client
The client connects to the server and sends single characters to it, then prints the response from the server to the serial port. I have created 2 clients in the code, which send characters at different intervals, to demonstrate both how to have more than one client on the same hardware and to show that the server can deal with more than one client.

// Demonstration code to transfer data from server to client using TCP
// This code sends a 'request' in for form of a single character to the corresponding server, which responds with the value of Millis

#include <ESP8266WiFi.h>

WiFiClient client_A;
WiFiClient Client_B;

#define STASSID "Put your WiFi SSID here"
#define STAPSK  "Put your WiFi password here"

const char * ssid = STASSID;
const char * pass = STAPSK;
IPAddress serverIP(192, 168, 2, 21);  // Replace this IP address with the IP address of your server
#define portNumber 7005

void setup() {
    setup_serial_port();
    setup_wifi();
}

void loop() {
    TCP_Client_A();
    TCP_Client_B();
}

void TCP_Client_A() {
    uint32_t currentMillis = millis();
    static uint32_t lastMillis;
    const uint32_t interval = 5000;
  
  if (!client_A.connected()) {
    if (client_A.connect(serverIP, portNumber)) {                                         // Connects to the server
      Serial.print("Connected to Gateway IP = "); Serial.println(serverIP);
    } else {
      Serial.print("Could NOT connect to Gateway IP = "); Serial.println(serverIP);
      delay(500);
    }
  } else {
    while (client_A.available()) Serial.write(client_A.read());                             // Receives data from the server and sends to the serial port
    
    if (currentMillis - lastMillis >= interval) {                                       // Sends the letter A (could be anything) to the server once every 'interval'
        lastMillis += interval;
        client_A.print("A");
    }
  }
}

void TCP_Client_B() {
    uint32_t currentMillis = millis();
    static uint32_t lastMillis;
    const uint32_t interval = 3000;
  
  if (!Client_B.connected()) {
    if (Client_B.connect(serverIP, portNumber)) {                                         // Connects to the server
      Serial.print("Connected to Gateway IP = "); Serial.println(serverIP);
    } else {
      Serial.print("Could NOT connect to Gateway IP = "); Serial.println(serverIP);
      delay(500);
    }
  } else {
    while (Client_B.available()) Serial.write(Client_B.read());                             // Receives data from the server and sends to the serial port
    
    if (currentMillis - lastMillis >= interval) {                                       // Sends the letter B (could be anything) to the server once every 'interval'
        lastMillis += interval;
        Client_B.print("B");
    }
  }
}

//Setup functions
void setup_serial_port() {
  //uint32_t baudrate = 115200;
  uint32_t baudrate = 9600;
  Serial.begin(baudrate);
  Serial.println("");
  Serial.print("Serial port connected: ");
  Serial.println(baudrate);
}

void setup_wifi() {
  uint8_t wait = 0;
  while (WiFi.status() != WL_CONNECTED) {
    if (wait) {
      --wait;
    } else {
      wait = 40;
      WiFi.mode(WIFI_STA);
      WiFi.begin(ssid, pass);
    }
    delay(100);
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

Assigning a static IP address to the server
The code above gets an IP address using DHCP. To assign a static IP address to the server I suggest that in your router you link an IP address to the MAC address of the ESP8266 (or whatever) so that the DHCP server in the router always assigns the same IP address to the ESP8266. Put this IP address in the client where indicated.

Acknowledgements and thanks
Thank you to @StefanL38 for asking the question, which made me investigate this more.
Thank you to @J-M-L , @Juraj and @wildbill whose ideas and code I have read then modified to make the above example. I make no claims that my example is the definitive example of how to use TCP for simple data transfer and I am certainly not an expert on such things.
Thank you to Mark Smith and Bhushan Mahajan for their answers on Stack Exchange, which is the origin of the code for a multiple server that I took and modified into something I can understand.

4 Likes
9

https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/src/ArduinoWiFiServer.h

10

@Juraj

The example code has a short description

The example is a simple server that echoes any incoming
  messages to all connected clients. Connect two or more
  telnet sessions to see how server.available() and
  server.print() work.

what is needed in your example to send/receive data to a second ESP8266 / ESP32?

How do I start "telnet-session" on a windows PC?
Are there any additional adjustements to do? Portnumber?
Does this line specifiy the portnumber?

ArduinoWiFiServer server(2323);

best regards Stefan

11

Just write a client program similar to @PerryBebbington's example and load it on one or more ESPs.

12

TCP is very simple. You don't have to know how it is implemented to use it.
Arduino uses the word 'Client' but the standard terminology is 'socket', which is less confusing if you use it on server's side.

Two connected TCP sockets are a bidirectional communication line. What you write on one end you read on the other end.

But first you must establish the connection if you want to talk to some other socket like you have to dial a number on a phone. For sockets, first number is the number of the device: IP address, second number is the 'port' number for a specific service.

To make a service available on a device, you say to TCP stack, that you want to have a 'listening socket' on a specific port. Then remote sockets can request a connection to that port. Arduino uses the name 'Server' for 'listening socket', which again is confusing.

If the TCP/IP stack receives a request for a socket connection for a specific port, it checks is a listening socket is defined for that port, and if yes, then a new socket is created, paired with the remote socket and the connection is established.

Port numbers <1024 are reserved to be used with standard services. For example 23 is for Telnet. 80 is for HTTP. 443 is for HTTPS. Port numbers > 1023 are free to use.

TCP doesn't care what kind of data are transported. It is the receiving code which requires a protocol. Like a HTTP server's code only understands if you send valid HTTP.

4 Likes
13

Hi @Juraj ,

thank you very very much for this explanation. This makes a lot of things much clearer now.

best regards Stefan

14

Unless you are a glutton for punishment, understanding the details of TCP/IP implementation is a waste of time. It is VERY complex. What you should instead focus on is how to USE the existing Arduino implementations, which is really quite simple. Understanding all the underlying machinations of TCP is completely unnecessary. Perhaps start with doing something simple with UDP, which is about the simplest way of sending a message, and getting a response. Or, alternatively, just work at the socket level, which is also very simple. Each requires only a very few lines of code to make a connection, and send/receive data, and there are TONS of examples and tutorials out there. I've done LOTS of rather complex networking apps over the last 20+ years without ever once having to understand exactly how the low-level networking stacks actually work.

15

Hi Ray,

I understand. It is not nescessary to dive into TCP itself. Understanding the "socket"-level is enough.

All the tutorials I have looked into so far (maybe 10) all immidiately jump up to use html.
And this confused me. The Demo-Code that @PerryBebbington posted is easy to understand because it stays below using http.

At least the tutorials I was looking into all suffer massively from what I call
"the experts blindness for beginner difficulties"

If it is unclear to a beginner that TCP-send/receive is really equal to a serial send/receive connection,
if you don't understand this, then how it works remains unclear.

The expert wonders: why doesn't she/he understand this short code???
Well, it has become so self-evident to the expert that he no longer notices that a beginner could have difficulties with it. = experts blindness for beginner difficulties

Again I want to thank @Juraj for his very basic explanation and explaining the basics on the level of the "socket".

I have my own head. And if in my opinion an explanation or a common term is confusing.
I change it! I'm planning to create a basic example that has exhaustive explanations. With the ambition to explain it in a very easy to understand way. If I can do that, then this example will prevail. If any of the "old experts" would complain about using other names than the common. My answer is "You are free to write down an even more easier to understand tutorial and better commented demo-codes" I'm very sure those "experts" won't do that. Because they are just coding-experts but not explaining-experts.

best regards Stefan

16

Not just you, it has confused me many times!

Thank you! Is there anything about it that's not clear? Any comment or explanation I missed? Anything I need to explain better?

17 
void setupServer(uint16_t port) {
    server.begin(port);
    //server.setNoDelay(true); // Sends small packets immediately without consolidating a number of small packets into one big packet
}

Probably don't need to provide the port number to begin() since it was provided when the WiFiServer object was created:

WiFiServer server(portNumber);
18

Just google "Arduino sockets tutorial", and the first two hits are a tutorial on the client, and server.

19

Arduino EthernetClient and WiFiClient add two features to a basic socket:
First is printing and reading different types of strings and numbers, which is same as on Serial thanks to Arduino Stream and Print base classes.
Second feature is connect with hostname as parameter. The function first resolves the name to IP address and then connects with the IP address.

The esp8266 and esp32 WiFiServer class is just a listening socket with no additional functionality, but the Arduino EthernetServer and WiFiServer in proper Arduino libraries manage the sockets for you. Calling server.available() here only returns a Client if it has data available. And then you can throw away the returned Client. At next server.available() you again get only a Client with data available. Maybe the same Client as before. And the Arduino Server has print-to-all-client function. The Server implements the Print class. Everything printed to Server is sent to all sockets managed by the Server.

The server.available() as I described above is not always good. Some protocols require that the server side sends data as first. This can't work with server.available() since the sketch doesn't get the Client if the remote side didn't send data. For that EthernetServer has server.acccept(). If the sketch uses server.accept(), then the Server is only a 'socket listener'. The WiFiServer in WiFiNINA and WiFi101 libraries, doesn't have accept() yet. But my 'pull requests' with accept() implementation are waiting in both libraries on GitHub.

So the esp8266 and esp32 WiFiServer class has server.available() implemented as EthernetServer's accept(). The 'rename' of server.available() to server.accept() is already done in esp8266 Arduino for the next release. (Of course available() stays, but with deprecation warning.) Here If you want to hold the connection you have to store the WiFiClient in sketch, otherwise the socket connection is closed as the WiFiClient object goes out of scope or is deleted.

For ESP8266WiFi library I wrote ArduinoWiFiServer. This manages the sockets as the Arduino Server classes should do. It is demonstrated by the PagerServer example.

2 Likes
20

Thanks, I've tried that and it works perfectly well, I'll modify the code I posted before you can type 'use code tags </>'.

The problem I had was that until I read @Juraj 's explanation in reply #12 I had no idea that what I needed was a tutorial about 'sockets'. Even my friend who knows a lot more about web stuff than I ever will, but knows nothing about Arduino, never suggested that the word I needed was 'socket'.

Does that mean I need to modify the code I used to use server.accept() instead of server.available()?