Automate SFTP Transfers Using ‘expect’
A programmer asked recently how to automate their password-based SFTP transfer. How does one automate SFTP or any task that typically requires user interaction?
This blog post shows how to achieve that automation while providing two (2) key takeaways:
- The expect tool, which has been around the open source world for over 30 years, is found in IBM i open source (
yum install expect) and can automate tasks that usually require user interaction. - Using open source on the IBM i is a lot easier if you have the reference platform for most open source—that is, Linux—installed and at your fingertips.
‘expect’ Makes a Script Flexible
Don Libes of Nist wrote the expect tool (1990) based on John Ousterhout’s Tcl language (1988). It reads output and writes input in response to your script.
The power of expect lies in its ability to send input to a server, then take action based on the server’s response. This capability makes it easy to see how expect can be useful in automating a password-based SFTP transfer.
To make expect easier to use, there’s a tool called autoexpect that will watch you type and write an expect script based on your live interaction.
Although expect can be installed on IBM i PASE, autoexpect has not yet been ported to IBM i PASE. Therefore, I used my Linux system to execute an SFTP session under the watchful eye of autoexpect to generate an expect script which I can use later in IBM i PASE.
Using ‘autoexpect’
I issued the command autoexpect sftp fred@foobar.example.com, which ran an apparently normal session with SFTP. When finished, however, the script below had been generated. I then logged in to the IBM i and was able to use the generated script in IBM i PASE.
In the script as shown, I have changed the user id, password, and target system, as well as simplifying the script to only log in, list the current directory, then quit:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
#!/usr/bin/expect -f # # This Expect script was generated by autoexpect on Fri Dec 23 11:20:12 2022 # Expect and autoexpect were both written by Don Libes, NIST. # # (Autoexpect generates heavily commented scripts.) # (Many comments have been deleted in this blog post for brevity.) set force_conservative 0; # set to 1 to force conservative mode even if script wasn't run conservatively originally if {$force_conservative} { set send_slow {1 .1} proc send {ignore arg} { sleep .1 exp_send -s -- $arg } } set timeout -1 spawn sftp fred@foobar.example.com match_max 100000 expect -exact "\rfred@foobar.example.com's password: " send -- "maryhadalittlelamb\r" expect -exact "\r Connected to foobar.example.com.\r sftp> " send -- "ls\r" expect -exact "ls\r # autoexpect generated these 3 lines from my live session. 221220 \r # I did the ls and there was one file, called 221220 sftp> " # and then the prompt came back. No need to “expect” here. send -- "quit\r" # Instead I could have done a wait and then quit. expect eof |
What the Script Does
The script above is:
spawn-ing a program (sftp fred@foobar.example.com)expect-ing certain strings (surrounded by double-quotes across possibly many lines)- If the
expect-ed string doesn’t come,expectwill stall at that point (timeouts can be set) send-ing certain strings (again, surrounded by double-quotes, possibly many lines)
… and so forth. Read the expect manual page and various web sources to learn more, but the easiest approach is to run autoexpect on your Linux workstation and then gently modify the scripts it produces.
You can do anything in an expect script that you can do typing at the command line, so it’s easy to try things out. Echo strings to files . . . launch a REPL session in PHP or Python and then quit it . . . a little personal experimentation with expect and autoexpect goes a long way!
Making a Decision
Here’s a simple script that sends you the link to a video of a seagull bemoaning the day being Wednesday . . . but only if it’s Wednesday!
|
1 2 3 4 5 6 7 |
#!/usr/bin/expect -f set timeout -1 spawn date expect { "Wed" {send_user "It's Wednesday, my dudes!\n" send_user "https://www.youtube.com/watch?v=OVQ3qmYSLMw\n"} } |
Much more sophisticated decision branching is available in expect.
Automation Is Easier with ‘expect’
- You can create shell scripts to replace interactive sessions using Don Libes’s
expect, which can be installed on IBM i PASE via yum install expect. - When trying new open source tools and crafting IBM i solutions using such tools, you will find it handy to also have access to a server running the universal open source reference platform, Linux.
IMHO one very good alternative when integrating nowadays in modern IBMi some SFTPing in real business processes (with logging on DBMS, a little transactional behaviour, data transfer) is to use a “real” language and just ditch the shell scripting or tool patchwork altogether that tends to be somewhat flimsy and with little control.
Better IMHO to use the SFTP libraries offered, for example, by the great Seiden PHP distro.
Just run the PHP in PASE like a normal script (or Python if you wish).
A PHP script handling SFTP can be even more compact than a “shell script”, can integrate with the local DB2 and RPG, and you have an integrated “real” language at your disposal.
remember, we are running IBMi, that, compared by the “poor” unix fellows ;)), have by assumption always a full fledge relational system below, with robust RPG in case for business logic (ie after receiving a order file, process it invoking an RPG in normal ERP), there is no need to mess with strings/text parsing/shell bricolage ;)
All good suggestions, Ema Tissani.
Your solution PHP+SFTP libraries is a full-fledged solution for a complex and rich business process.
My “just run it on Linux under
autoexpectand copy the script up to the IBM i” is more ad-hoc, however, the user request I was addressing was equally ad-hoc, so I found it to be a useful solution!Thanks for commenting!
This looks cool but you’ve lost me. On the IBM i, we’ve always had FTP script files to automate the FTP process. When is this needed? What tasks require user interaction?
Hi Glenn —
It’s a little more complicated with some of the more modern stuff. SFTP for instance expects Unix-style terminal interaction.
expectknows how to act like a terminal. Andexpectcan take decision branches based on what is returned by the program it is interacting with. All our old FTP scripting on IBM i is based on “shouting into the darkness” without regard for what is returned.Also, advanced vendor tools like GoAnywhere can do some of this behavior, but they tend to be focused on certain facilities like ftp, sftp, scp, etc.
expectis a generic solution that will write input and read output with decision branching for any terminal interaction in PASE.