RemoteOperations::exec_command explicitly transfers LANG, LANGUAGE and LC_* envvars to the server side (#187)

* RemoteOperations::exec_command updated

- Exact enumeration of supported 'cmd' types
- Refactoring

* RemoteOperations::exec_command explicitly transfers LANG, LANGUAGE and LC_* envvars to the server side

It should help resolve a problem with replacing a LANG variable by ssh-server.

History.

On our internal tests we got a problem on the Debian 11 and PostgresPro STD-13.

One test returned the error from initdb:

initdb: error: collations with different collate and ctype values ("en_US.UTF-8" and "C.UTF-8" accordingly) are not supported by ICU

- TestRunner set variable LANG="C"
- Python set variable LC_CTYPE="C.UTF-8"
- Test call inidb through command "ssh test@localhost inidb -D ...."
- SSH-server replaces LANG with value "en_US.UTF-8" (from etc/default/locale)
- initdb calculate collate through this value of LANG variable and get en_US.UTF-8

So we have that:
- ctype is C.UTF-8
- collate is en_US.UTF-8

ICU on the Debuan-11 (uconv v2.1  ICU 67.1) does not suppot this combination and inidb rturns the error.

This patch generates a new command line for ssh:

ssh test@localhost "LANG=\"...\";LC_xxx=\"...\";<command>"

It resolves this problem with initdb and should help resolve other problems with execution of command through SSH.

Amen.

* New tests in TestgresRemoteTests are added

New tests:
 - test_init__LANG_С
 - test_init__unk_LANG_and_LC_CTYPE

* TestgresRemoteTests.test_init__unk_LANG_and_LC_CTYPE is updated

Let's test bad data with '\' and '"' symbols.

* Static methods are marked with @staticmethod [thanks to Victoria Shepard]

The following methods of RemoteOperations were corrected:
 - _make_exec_env_list
 - _does_put_envvar_into_exec_cmd
 - _quote_envvar

* TestRemoteOperations::_quote_envvar is updated (typification)

Author: dmitry-lipetsk

testgres/operations/remote_ops.py

@@ -87,7 +87,12 @@ def exec_command(self, cmd, wait_exit=False, verbose=False, expect_error=False,
 
         assert type(cmd_s) == str  # noqa: E721
 
-        ssh_cmd = ['ssh', self.ssh_dest] + self.ssh_args + [cmd_s]
+        cmd_items = __class__._make_exec_env_list()
+        cmd_items.append(cmd_s)
+
+        env_cmd_s = ';'.join(cmd_items)
+
+        ssh_cmd = ['ssh', self.ssh_dest] + self.ssh_args + [env_cmd_s]
 
         process = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         assert not (process is None)
@@ -510,6 +515,45 @@ def db_connect(self, dbname, user, password=None, host="localhost", port=5432):
         )
         return conn
 
+    @staticmethod
+    def _make_exec_env_list() -> list[str]:
+        result = list[str]()
+        for envvar in os.environ.items():
+            if not __class__._does_put_envvar_into_exec_cmd(envvar[0]):
+                continue
+            qvalue = __class__._quote_envvar(envvar[1])
+            assert type(qvalue) == str   # noqa: E721
+            result.append(envvar[0] + "=" + qvalue)
+            continue
+
+        return result
+
+    sm_envs_for_exec_cmd = ["LANG", "LANGUAGE"]
+
+    @staticmethod
+    def _does_put_envvar_into_exec_cmd(name: str) -> bool:
+        assert type(name) == str  # noqa: E721
+        name = name.upper()
+        if name.startswith("LC_"):
+            return True
+        if name in __class__.sm_envs_for_exec_cmd:
+            return True
+        return False
+
+    @staticmethod
+    def _quote_envvar(value: str) -> str:
+        assert type(value) == str  # noqa: E721
+        result = "\""
+        for ch in value:
+            if ch == "\"":
+                result += "\\\""
+            elif ch == "\\":
+                result += "\\\\"
+            else:
+                result += ch
+        result += "\""
+        return result
+
 
 def normalize_error(error):
     if isinstance(error, bytes):

tests/test_simple_remote.py

@@ -119,6 +119,79 @@ def test_custom_init(self):
             # there should be no trust entries at all
             self.assertFalse(any('trust' in s for s in lines))
 
+    def test_init__LANG_С(self):
+        # PBCKP-1744
+        prev_LANG = os.environ.get("LANG")
+
+        try:
+            os.environ["LANG"] = "C"
+
+            with get_remote_node(conn_params=conn_params) as node:
+                node.init().start()
+        finally:
+            __class__.helper__restore_envvar("LANG", prev_LANG)
+
+    def test_init__unk_LANG_and_LC_CTYPE(self):
+        # PBCKP-1744
+        prev_LANG = os.environ.get("LANG")
+        prev_LANGUAGE = os.environ.get("LANGUAGE")
+        prev_LC_CTYPE = os.environ.get("LC_CTYPE")
+        prev_LC_COLLATE = os.environ.get("LC_COLLATE")
+
+        try:
+            # TODO: Pass unkData through test parameter.
+            unkDatas = [
+                ("UNKNOWN_LANG", "UNKNOWN_CTYPE"),
+                ("\"UNKNOWN_LANG\"", "\"UNKNOWN_CTYPE\""),
+                ("\\UNKNOWN_LANG\\", "\\UNKNOWN_CTYPE\\"),
+                ("\"UNKNOWN_LANG", "UNKNOWN_CTYPE\""),
+                ("\\UNKNOWN_LANG", "UNKNOWN_CTYPE\\"),
+                ("\\", "\\"),
+                ("\"", "\""),
+            ]
+
+            for unkData in unkDatas:
+                logging.info("----------------------")
+                logging.info("Unk LANG is [{0}]".format(unkData[0]))
+                logging.info("Unk LC_CTYPE is [{0}]".format(unkData[1]))
+
+                os.environ["LANG"] = unkData[0]
+                os.environ.pop("LANGUAGE", None)
+                os.environ["LC_CTYPE"] = unkData[1]
+                os.environ.pop("LC_COLLATE", None)
+
+                assert os.environ.get("LANG") == unkData[0]
+                assert not ("LANGUAGE" in os.environ.keys())
+                assert os.environ.get("LC_CTYPE") == unkData[1]
+                assert not ("LC_COLLATE" in os.environ.keys())
+
+                while True:
+                    try:
+                        with get_remote_node(conn_params=conn_params):
+                            pass
+                    except testgres.exceptions.ExecUtilException as e:
+                        #
+                        # Example of an error message:
+                        #
+                        # warning: setlocale: LC_CTYPE: cannot change locale (UNKNOWN_CTYPE): No such file or directory
+                        # postgres (PostgreSQL) 14.12
+                        #
+                        errMsg = str(e)
+
+                        logging.info("Error message is: {0}".format(errMsg))
+
+                        assert "LC_CTYPE" in errMsg
+                        assert unkData[1] in errMsg
+                        assert "warning: setlocale: LC_CTYPE: cannot change locale (" + unkData[1] + "): No such file or directory" in errMsg
+                        assert "postgres" in errMsg
+                        break
+                    raise Exception("We expected an error!")
+        finally:
+            __class__.helper__restore_envvar("LANG", prev_LANG)
+            __class__.helper__restore_envvar("LANGUAGE", prev_LANGUAGE)
+            __class__.helper__restore_envvar("LC_CTYPE", prev_LC_CTYPE)
+            __class__.helper__restore_envvar("LC_COLLATE", prev_LC_COLLATE)
+
     def test_double_init(self):
         with get_remote_node(conn_params=conn_params).init() as node:
             # can't initialize node more than once
@@ -994,6 +1067,12 @@ def test_child_process_dies(self):
             # try to handle children list -- missing processes will have ptype "ProcessType.Unknown"
             [ProcessProxy(p) for p in children]
 
+    def helper__restore_envvar(name, prev_value):
+        if prev_value is None:
+            os.environ.pop(name, None)
+        else:
+            os.environ[name] = prev_value
+
 
 if __name__ == '__main__':
     if os_ops.environ('ALT_CONFIG'):