SFTP Upload Hangs

[KCC]

With the logging put in the lockup occured but the logging stops when it happens. This just happened tonight, a file is started and the line given to log is:

while ($sent < $size) {
error_log("SFTP $sent < $size");


20120711 11:34:46 PM: SFTP START File=(...)
SFTP 0 < 8131920
SFTP 4096 < 8131920
SFTP 8192 < 8131920
SFTP 12288 < 8131920
SFTP 16384 < 8131920
SFTP 20480 < 8131920
.....(all the other increments)
SFTP 2453504 < 8131920


It stayed stuck on the last line given for over 2 hours until quit manually.

Thanks

[TerraFrost]

What about this?:

Code: Select all

If you could do this...


#
#-----[ FIND ]------------------------------------------
#
Net/SFTP.php
#
#-----[ FIND ]------------------------------------------
#
        if (!$this->_read_put_responses($i)) {
            return false;
        }
#
#-----[ BEFORE, ADD ]-----------------------------------
#
echo "ABOUT TO READ PUT RESPONSES\r\n";
#
#-----[ FIND ]------------------------------------------
#
    function _read_put_responses($i)
    {
        while ($i--) {
#
#-----[ AFTER, ADD ]------------------------------------
#
echo "READ PUT RESPONSE ($i)\r\n";

[KCC]

I will add that and test it out.

One other thing, sometimes when it gets stuck it ends up exiting due to memory:
...
SFTP 782336 < 2389467
SFTP 786432 < 2389467
SFTP 790528 < 2389467
SFTP 794624 < 2389467
SFTP 798720 < 2389467
SFTP 802816 < 2389467
SFTP 806912 < 2389467
SFTP 811008 < 2389467
SFTP 815104 < 2389467
PHP Fatal error: Out of memory (allocated 1930166272) (tried to allocate 1926713082 bytes) in /usr/share/pear/Net/SSH2.php on line 1875


Me line 1875 is:
$temp = fread($this->fsock, $remaining_length);

[KCC]

Here is a log with the above included:
(SFTP $sent < $size)
SFTP 6332416 < 7058939
SFTP 6336512 < 7058939
SFTP 6340608 < 7058939
SFTP 6344704 < 7058939
READ PUT RESPONSE (49)
READ PUT RESPONSE (48)
READ PUT RESPONSE (47)
READ PUT RESPONSE (46)
READ PUT RESPONSE (45)
READ PUT RESPONSE (44)
READ PUT RESPONSE (43)
READ PUT RESPONSE (42)
READ PUT RESPONSE (41)
READ PUT RESPONSE (40)
READ PUT RESPONSE (39)
READ PUT RESPONSE (38)
READ PUT RESPONSE (37)
READ PUT RESPONSE (36)
READ PUT RESPONSE (35)
READ PUT RESPONSE (34)
READ PUT RESPONSE (33)
READ PUT RESPONSE (32)
READ PUT RESPONSE (31)
READ PUT RESPONSE (30)
READ PUT RESPONSE (29)
READ PUT RESPONSE (28)
READ PUT RESPONSE (27)
READ PUT RESPONSE (26)
READ PUT RESPONSE (25)
READ PUT RESPONSE (24)
READ PUT RESPONSE (23)
READ PUT RESPONSE (22)
READ PUT RESPONSE (21)
READ PUT RESPONSE (20)
READ PUT RESPONSE (19)
PHP Notice: Invalid HMAC in /usr/share/pear/Net/SSH2.php on line 1890
PHP Notice: Connection closed by server in /usr/share/pear/Net/SSH2.php on line 2017
PHP Notice: Expected SSH_FXP_STATUS in /usr/share/pear/Net/SFTP.php on line 1142
ABOUT TO READ PUT RESPONSES
PHP Fatal error: Out of memory (allocated 1427111936) (tried to allocate 1423453887 bytes) in /usr/share/pear/Net/SSH2.php on line 1875

[KCC]

Here is the same problem, but without the memory exhaustion occuring:
READ PUT RESPONSE (13)
READ PUT RESPONSE (12)
READ PUT RESPONSE (11)
READ PUT RESPONSE (10)
READ PUT RESPONSE (9)
READ PUT RESPONSE (8)
READ PUT RESPONSE (7)
READ PUT RESPONSE (6)
READ PUT RESPONSE (5)
READ PUT RESPONSE (4)
READ PUT RESPONSE (3)
READ PUT RESPONSE (2)
READ PUT RESPONSE (1)
READ PUT RESPONSE (0)
SFTP 204800 < 801528
SFTP 208896 < 801528
SFTP 212992 < 801528
SFTP 217088 < 801528
SFTP 221184 < 801528
SFTP 225280 < 801528
SFTP 229376 < 801528
SFTP 233472 < 801528
SFTP 237568 < 801528
SFTP 241664 < 801528
SFTP 245760 < 801528
SFTP 249856 < 801528
SFTP 253952 < 801528
SFTP 258048 < 801528
SFTP 262144 < 801528
SFTP 266240 < 801528
SFTP 270336 < 801528
SFTP 274432 < 801528
SFTP 278528 < 801528
SFTP 282624 < 801528
SFTP 286720 < 801528
SFTP 290816 < 801528
SFTP 294912 < 801528
SFTP 299008 < 801528
SFTP 303104 < 801528
SFTP 307200 < 801528
SFTP 311296 < 801528
SFTP 315392 < 801528
SFTP 319488 < 801528
SFTP 323584 < 801528
SFTP 327680 < 801528
SFTP 331776 < 801528
SFTP 335872 < 801528
SFTP 339968 < 801528
SFTP 344064 < 801528
SFTP 348160 < 801528
SFTP 352256 < 801528
SFTP 356352 < 801528
SFTP 360448 < 801528
SFTP 364544 < 801528
SFTP 368640 < 801528
SFTP 372736 < 801528
SFTP 376832 < 801528
SFTP 380928 < 801528
SFTP 385024 < 801528
SFTP 389120 < 801528
SFTP 393216 < 801528
SFTP 397312 < 801528
SFTP 401408 < 801528
SFTP 405504 < 801528
READ PUT RESPONSE (49)
READ PUT RESPONSE (48)
READ PUT RESPONSE (47)
READ PUT RESPONSE (46)
READ PUT RESPONSE (45)
READ PUT RESPONSE (44)
READ PUT RESPONSE (43)
READ PUT RESPONSE (42)
READ PUT RESPONSE (41)
READ PUT RESPONSE (40)
READ PUT RESPONSE (39)
READ PUT RESPONSE (38)
READ PUT RESPONSE (37)
READ PUT RESPONSE (36)
READ PUT RESPONSE (35)
READ PUT RESPONSE (34)
READ PUT RESPONSE (33)
READ PUT RESPONSE (32)
READ PUT RESPONSE (31)
READ PUT RESPONSE (30)
READ PUT RESPONSE (29)
READ PUT RESPONSE (28)
READ PUT RESPONSE (27)
READ PUT RESPONSE (26)
READ PUT RESPONSE (25)
READ PUT RESPONSE (24)
READ PUT RESPONSE (23)
READ PUT RESPONSE (22)
READ PUT RESPONSE (21)
READ PUT RESPONSE (20)
READ PUT RESPONSE (19)
READ PUT RESPONSE (18)
READ PUT RESPONSE (17)
READ PUT RESPONSE (16)
READ PUT RESPONSE (15)
READ PUT RESPONSE (14)
READ PUT RESPONSE (13)
READ PUT RESPONSE (12)
READ PUT RESPONSE (11)
READ PUT RESPONSE (10)
READ PUT RESPONSE (9)
READ PUT RESPONSE (8)
READ PUT RESPONSE (7)
READ PUT RESPONSE (6)
READ PUT RESPONSE (5)
READ PUT RESPONSE (4)
READ PUT RESPONSE (3)
READ PUT RESPONSE (2)
READ PUT RESPONSE (1)
READ PUT RESPONSE (0)
SFTP 409600 < 801528
SFTP 413696 < 801528
SFTP 417792 < 801528
SFTP 421888 < 801528
SFTP 425984 < 801528
SFTP 430080 < 801528
SFTP 434176 < 801528
SFTP 438272 < 801528
SFTP 442368 < 801528
SFTP 446464 < 801528
SFTP 450560 < 801528
SFTP 454656 < 801528
SFTP 458752 < 801528
SFTP 462848 < 801528
SFTP 466944 < 801528
SFTP 471040 < 801528
SFTP 475136 < 801528
SFTP 479232 < 801528
SFTP 483328 < 801528
SFTP 487424 < 801528
SFTP 491520 < 801528
SFTP 495616 < 801528
SFTP 499712 < 801528
SFTP 503808 < 801528
SFTP 507904 < 801528
SFTP 512000 < 801528
SFTP 516096 < 801528
SFTP 520192 < 801528
SFTP 524288 < 801528
SFTP 528384 < 801528
SFTP 532480 < 801528
SFTP 536576 < 801528
SFTP 540672 < 801528
SFTP 544768 < 801528
SFTP 548864 < 801528
SFTP 552960 < 801528
SFTP 557056 < 801528
SFTP 561152 < 801528
SFTP 565248 < 801528
SFTP 569344 < 801528
SFTP 573440 < 801528
SFTP 577536 < 801528
SFTP 581632 < 801528
SFTP 585728 < 801528
SFTP 589824 < 801528
SFTP 593920 < 801528
SFTP 598016 < 801528
SFTP 602112 < 801528
SFTP 606208 < 801528
SFTP 610304 < 801528
READ PUT RESPONSE (49)
READ PUT RESPONSE (48)
READ PUT RESPONSE (47)
READ PUT RESPONSE (46)
READ PUT RESPONSE (45)
READ PUT RESPONSE (44)
READ PUT RESPONSE (43)
READ PUT RESPONSE (42)
READ PUT RESPONSE (41)
READ PUT RESPONSE (40)
READ PUT RESPONSE (39)
READ PUT RESPONSE (38)
READ PUT RESPONSE (37)
READ PUT RESPONSE (36)
READ PUT RESPONSE (35)
READ PUT RESPONSE (34)
READ PUT RESPONSE (33)
READ PUT RESPONSE (32)
READ PUT RESPONSE (31)
READ PUT RESPONSE (30)
READ PUT RESPONSE (29)
READ PUT RESPONSE (28)
READ PUT RESPONSE (27)
READ PUT RESPONSE (26)
READ PUT RESPONSE (25)
READ PUT RESPONSE (24)
READ PUT RESPONSE (23)
READ PUT RESPONSE (22)
READ PUT RESPONSE (21)
READ PUT RESPONSE (20)
READ PUT RESPONSE (19)
READ PUT RESPONSE (18)
READ PUT RESPONSE (17)
READ PUT RESPONSE (16)
READ PUT RESPONSE (15)

It never produced any output past response(15), it was killed manually 12 hours later.

Thanks

[TerraFrost]

How long did it run for / how much data was transferred?

Also, what's the output of $ssh->getServerIdentification()?

[KCC]

It had been running for roughly half an hour and successfully transferred 80-100 files before locking up. The output is "SSH-2.0-SFTP Server".

Thanks

[TerraFrost]

I was more curious how many MB / GB you uploaded lol.

SSH-2.0-SFTP Server is a pretty generic name. My guess is that maybe your server handles key reexchanges badly.

To test that try this:

Code: Select all

#
#-----[ FIND ]------------------------------------------
#
Net/SSH2.php
#
#-----[ FIND ]------------------------------------------
#
    function _key_exchange($kexinit_payload_server)
    {
#
#-----[ AFTER, ADD ]------------------------------------
#
echo "STARTING KEY EXCHANGE\r\n";

[KCC]

The sizes range from several hundred MB's to over a GB per day. After putting in the key exchange log I've only seen it output once per session when the connection is opened, but the freeze issue has continues to sporadically occur.

Thanks

[TerraFrost]

Can you do the following?:

Code: Select all

#
#-----[ FIND ]------------------------------------------
#
Net/SSH2.php
#
#-----[ FIND ]------------------------------------------
#
        while ($remaining_length > 0) {
            $temp = fread($this->fsock, $remaining_length);
            $buffer.= $temp;
            $remaining_length-= strlen($temp);
        }
#
#-----[ REPLACE WITH ]----------------------------------
#
        while ($remaining_length > 0) {
            $temp = fread($this->fsock, $remaining_length);
            $buffer.= $temp;
if (!strlen($temp)) {
   echo "TEMP = 0; REMAINING LENGTH = $remaining_length\r\n";
}
            $remaining_length-= strlen($temp);
        }

My expectation is that that'd flood the console with the string that it's echo'ing out.

Also, looking at the previous posts it doesn't look like you ever gave me a log file? eg.

Code: Select all

<?php
include('Net/SSH2.php');

define('NET_SSH2_LOG_REALTIME_FILE', '/path/to/test2.txt');
define('NET_SSH2_LOGGING', NET_SSH2_LOG_REALTIME_FILE);

$ssh = new Net_SSH2('www.domain.tld');
$ssh->login('username', 'password');

$ssh->exec('ls -la');