Remotely running commands or scripts with python

The task of communicating and running complex operations between servers can be a difficult and tedious process. Running commands or scripts remotely on a server from your local machine can usually be done quite easily using a scripting language such as bash, but doing this from within a Python application can be quite difficult. Luckily there are Python modules we can use that make the job significantly more easy, namely Paramiko and SCP.

Running commands remotely on another host from your local machine

Using the Paramiko module in Python, you can create an SSH connection to another host from within your application, with this connection you can send your commands to the host and retrieve the output.

Paramiko doesn't come installed by default in Python, but can be easily installed with pip like so:

pip install paramiko

The first thing you need to do when running a command remotely is create an SSH connection, this can be done with Paramiko's SSHClient object like so:

import paramiko   
   
# declare credentials   
host = 'hostip'   
username = 'username'   
password = 'password'   
   
# connect to server   
con = paramiko.SSHClient()   
con.load_system_host_keys()   
con.connect(host, username=username, password=password)   

Here we are importing Paramiko, instantiating an SSHClient object, loading the system's host information(known hosts, keys, etc.) then connecting to it using your credentials.

If you have an SSH key set up on the remote server for the root user, you will not need to specify a username and password. If you have your server set up differently with other options such as a passphrase, the connect method can cater for that as it has several other arguments to suit your needs, these can be found on the official documentation page.

Once you've created a connection, you can now use it to run commands on a remote server using the exec_command method. This method takes a command as its main argument and returns the stdin, stdout and stderr of the completed command. The stdout and stderr values are file-like objects and can be interacted with just like you would with files.

As an example, let's run a simple command to list the contents in the /tmp folder on our remote server.

import paramiko   
# declare credentials   
host = 'hostip'   
username = 'username'   
password = 'password'   
# connect to server   
con = paramiko.SSHClient()   
con.load_system_host_keys()   
con.connect(host, username=username, password=password)   
# run the command   
# use the -1 argument so we can split the files by line   
stdin, stdout, stderr = con.exec_command('ls -1 /tmp')   
# process the output   
if stderr.read() == b'':   
    for line in stdout.readlines():   
        print(line.strip()) # strip the trailing line breaks   
else:   
    print(stderr.read())   

In the above example we use the ls command with the -1 argument to list the files in the /tmp directory on separate lines. Once the command is complete we can see whether or not is was successful by checking the stderr value. if it is empty we know the command was successful and we can print out the files contained in stdout. If we supply an invalid directory such as /test then stderr will have a value (telling us the directory does not exist) and we can print out the error.

note: if you are accepting user input and using it in your remote commands, make sure you are properly filtering and validating that input as this method of running remote commands is susceptible to command injections.

Running scripts remotely on another host from your local machine

If you want to run an entire script (such as a bash or even a python application) on another server from your local machine, you can make use of the SCP module to upload your script, then simply execute it using the same technique we used above with Paramiko.

The SCP module requires an SSH connection to copy a file to a server, for this we can simply use the connection created with Paramiko's SSH client which we can also use to run the command to execute the script.

As with Paramiko, SCP does not come installed with Python by default, it can be installed using the following command:

pip install scp

To copy your file across, you'll need to import the SCPClient object from SCP, this takes the SSH connection from Paramiko as an argument, you can the use the 'put' method to upload your file.

import paramiko   
from scp import SCPClient   
# declare credentials   
host = 'hostip'   
username = 'username'   
password = 'password'   
# connect to server   
con = paramiko.SSHClient()   
con.load_system_host_keys()   
con.connect(host, username=username, password=password)   
# copy the file accross  
with SCPClient(con.get_transport()) as scp:   
    scp.put('source_file', 'destination_directory')   

Once you've uploaded your file, you can execute it on the server using exec_command just like we did in the first example.

As an example, let's upload and execute a simple bash script on our server that will create a file and write a "hello world" to it once executed.

test.sh

#!/bin/bash   
   
echo "hello world" > /tmp/test.txt

main.py

import paramiko
from scp import SCPClient
# declare credentials
host = 'hostip'
username = 'username'
password = 'password'
# connect to server
con = paramiko.SSHClient()
con.load_system_host_keys()
con.connect(host, username=username, password=password)
# copy the file across
with SCPClient(con.get_transport()) as scp:
    scp.put('test.sh', '/tmp')
# execute the script
stdin, stdout, stderr = con.exec_command('bash /tmp/test.sh')
if stderr.read() == b'':
    print('Success')
else:
    print('An error occurred')

If we run this script- assuming it connects properly and the the permissions of the source file and destination directory are set correctly- you'll find a new file has been created in the /tmp directory called test.txt that contains the string "hello world".

Running a lengthy command remotely and polling for its completion

If you need to run a command that may take a while to finish processing, you can run it as a background task on the remote server using '&', fetch its process ID using '$!' and then poll for its completion using the 'ps' command.

import paramiko
from time import sleep
# declare credentials
host = 'hostip'
username = 'username'
password = 'password'
# connect to server
con = paramiko.SSHClient()
con.load_system_host_keys()
con.connect(host, username=username, password=password)
# run the command
# Replace 'sleep 5' with your actual command which will write its output to /tmp/foo.log
# where we can read from later, make sure to also pipe stderr to stdout with 2>&1 otherwise this will hang
stdin, stdout, stderr = con.exec_command('sleep 5 > /tmp/foo.log 2>&1 & echo $!;')
procid = stdout.read().decode().strip()
print('Waiting for process', procid)
while True:
        # Use ps to find whether the process exists, returns 2 lines if it does
        stdin, stdout, stderr = con.exec_command('ps -p ' + procid)
        if len(stdout.readlines()) < 2:
                break
        sleep(1)  # Poll every second
# Read your output (the above sleep command won't have one)
stdin, stdout, stderr = con.exec_command('cat /tmp/foo.log')
print(stdout.read().decode())

Providing input for any arguments your remote script may ask for

If you are running a remote script that requires you to pass it arguments, you can simply pipe a list of arguments in the order they are asked for separated by line separators '\n'.

Say we have the following script on the remote server which sequentially asks for 2 arguments:

#!/bin/bash

read t
read y
echo "test1: $t"
echo "test2: $y"

We can provide the arguments it needs using exec_command like this:

stdin, stdout, stderr = con.exec_command('echo "test\ntest2" | bash test.sh')
print(stdout.read().decode())

This will return the following output:

test1: test
test2: test2

If you have any questions or want to know more about the topic, please leave a comment below!