SFTP Upload Hangs

[KCC]

Hello,

I am encountering an issue with the SFTP portion of phpseclib where the put() function will sometimes hang indefinitely.

The put call uses the local file method, and is wrapped around logging to show the beginning and end, like this:

print "SFTP START PUT $file\n";
$put = $sftp->put($destination,$file,NET_SFTP_LOCAL_FILE);
print "SFTP PUT COMPLETE\n";

When the SFTP connection is made, the socket timeout is set like this:
stream_set_timeout($sftp->fsock, 30);

It is relatively rare, but perhaps 1 in 500 times the put command will never complete and the script continues on for hours or days until stopped manually. I suspect it is caused by the remote server either dropping the connection or failing in some way but I have not seen a way to program for that scenario.

Thank you.

[TerraFrost]

I'm gonna try to add some real-time logging to SSH2.php. SFTP.php has it but not SSH2.php. The problem with logging as is is that you can only get it after the script has finished running. You could use register_shutdown_function() as well, with the Net_SSH2 object being declared global within it, but then if you do Ctrl + C, it won't be called, so it seems like real-time logging is the only option here.

[TerraFrost]

I've added real-time logging to the latest SVN of Net_SSH2. Quoting from the comments in the code:

Code: Select all

(NET_SSH2_LOG_REALTIME)
                // dump the output out realtime; packets may be interspersed with non packets,
                // passwords won't be filtered out and select other packets may not be correctly
                // identified

(NET_SSH2_LOG_REALTIME_FILE)
                // basically the same thing as NET_SSH2_LOG_REALTIME with the caveat that NET_SSH2_LOG_REALTIME_FILE
                // needs to be defined and that the resultant log file will be capped out at NET_SSH2_LOG_MAX_SIZE.
                // the earliest part of the log file is denoted by the first <<< START >>> and is not going to necessarily
                // at the beginning of the file

An example of how to use it:

Code: Select all

<?php
include('Net/SSH2.php');

define('NET_SSH2_LOG_REALTIME_FILE', '/path/to/test2.txt');
define('NET_SSH2_LOGGING', NET_SSH2_LOG_REALTIME_FILE);

$ssh = new Net_SSH2('www.domain.tld');
$ssh->login('username', 'password');

$ssh->exec('ls -la');

I haven't done an equivalent mode for SFTP because SFTP runs on top of SSH2 and because, in so far as we know, the problem could be with the SSH2 layer and not at the SFTP layer. If the problem were at the SFTP layer the SSH2 logs would reveal that but not vice versa.

[KCC]

Thank you, I will update to the latest SVN code and let you know what happens.

The issue actually occurred yesterday but I noticed something I haven't seen before, the put() command never completed and these lines appeared in the STDERR stream:
[put() started here]
PHP Notice: Invalid HMAC in /usr/share/pear/Net/SSH2.php on line 1890
PHP Notice: Connection closed by server in /usr/share/pear/Net/SSH2.php on line 2017
PHP Notice: Expected SSH_FXP_STATUS in /usr/share/pear/Net/SFTP.php on line 1136


The put() command never returned and the code was hung indefinitely after those 3 error lines.

Thanks

[TerraFrost]

That actually helps a lot I think. I'll try to make an update tomorrow. Thanks!!

[TerraFrost]

Does this do the trick?:

Code: Select all

#
#-----[ FIND ]------------------------------------------
#
        $this->_read_put_responses($i);

        if ($mode == NET_SFTP_LOCAL_FILE) {
            fclose($fp);
        }
#
#-----[ REPLACE WITH ]----------------------------------
#
        if (!$this->_read_put_responses($i)) {
            return false;
        }

        if ($mode == NET_SFTP_LOCAL_FILE) {
            fclose($fp);
        }

[KCC]

I applied the changes you quoted above, the circumstances to reproduce are somewhat random so I will watch it for the next few days and post back the outcome.

Thank you

[KCC]

Hello,

The issue popped up again today, this was the error log:
PHP Notice: Invalid HMAC in /usr/share/pear/Net/SSH2.php on line 1890
PHP Notice: Connection closed by server in /usr/share/pear/Net/SSH2.php on line 2017
PHP Notice: Expected SSH_FXP_STATUS in /usr/share/pear/Net/SFTP.php on line 1139
PHP Fatal error: Out of memory (allocated 1569193984) (tried to allocate 1565705251 bytes) in /usr/share/pear/Net/SSH2.php on line 1875

I have not noticed the memory line until today, the max memory configuration is 4GB.

Thanks

[TerraFrost]

Sorry for the delay - my main laptop has been undergoing a warranty repair and meh.

Anyway, your line 1139 doesn't match my line 1139 (latest SVN). Can you post your SFTP.php?

My line 1139 is in the _chmod_recursive() function. It's "return false;". Here's the surrounding code:

Code: Select all

                if ($i >= 50) {
                    if (!$this->_read_put_responses($i)) {
                        return false;
                    }
                    $i = 0;
                }

[KCC]

Sorry for the delay, here is the attachment.

Thanks again

[TerraFrost]

Looks like you forgot to include the attachment in your post.

[KCC]

Sorry, looks like it doesn't take .php extensions, the file should be attached now, thanks.

[TerraFrost]

Maybe try this?:

Code: Select all

#
#-----[ FIND ]------------------------------------------
#
Net/SFTP.php
#
#-----[ FIND ]------------------------------------------
#
            extract(unpack('Nstatus', $this->_string_shift($response, 4)));
            if ($status != NET_SFTP_STATUS_OK) {
                extract(unpack('Nlength', $this->_string_shift($response, 4)));
                $this->sftp_errors[] = $this->status_codes[$status] . ': ' . $this->_string_shift($response, $length);
                break;
            }
#
#-----[ REPLACE WITH ]----------------------------------
#
            extract(unpack('Nstatus', $this->_string_shift($response, 4)));
            if ($status != NET_SFTP_STATUS_OK) {
                extract(unpack('Nlength', $this->_string_shift($response, 4)));
                $this->sftp_errors[] = $this->status_codes[$status] . ': ' . $this->_string_shift($response, $length);
                return false;
            }

If that doesn't work... I'm curious what the output of this would be:

Code: Select all

#
#-----[ FIND ]------------------------------------------
#
Net/SFTP.php
#
#-----[ FIND ]------------------------------------------
#
        while ($sent < $size) {
#
#-----[ AFTER, ADD ]------------------------------------
#
            echo "$sent < $size\r\n";

Also, would you happen to know the size of the file that you're trying to upload that's causing the script to hang?

[KCC]

I have applied the first patch and will monitor it for changes. I've seen the hang occur on file sizes from 2-20MB's.
Thanks again.

[KCC]

Hello,

The issue has continued to occur, these 3 lines always output when it happens now:
PHP Notice: Invalid HMAC in /usr/share/pear/Net/SSH2.php on line 1890
PHP Notice: Connection closed by server in /usr/share/pear/Net/SSH2.php on line 2017
PHP Notice: Expected SSH_FXP_STATUS in /usr/share/pear/Net/SFTP.php on line 1139

I will add in the second part that logs the size and see what happens with that.
Thanks