These forums are archived

See this post for further info

get_iplayer forums

Forum archived. Posting disabled.

get_iplayer forums > Linux & Unix Forums > Linux & Unix General Help > Resuming broken?

Resuming broken?

user-270

04-12-2013, 10:29 PM
I've installed get_iplayer on my headless server, instead of being on Windows.

It's been a day or so now, and what I've discovered is that if there is any interruption of the stream for any reason, when the download is resumed, it is always corrupt:

Code:
Resuming download at: 498988.494 kB / 1481.040 sec (54.6%)
INFO: Metadata:
INFO:   duration              2711.08
INFO:   moovPosition          36.00
INFO:   width                 832.00
INFO:   height                468.00
INFO:   videocodecid          avc1
INFO:   audiocodecid          mp4a
INFO:   avcprofile            77.00
INFO:   avclevel              31.00
INFO:   aacaot                2.00
INFO:   videoframerate        25.00
INFO:   audiosamplerate       48000.00
INFO:   audiochannels         2.00
INFO: trackinfo:
INFO:   length                67777000.00
INFO:   timescale             25000.00
INFO:   language              und
INFO: sampledescription:
INFO:   sampletype            avc1
INFO:   length                130131968.00
INFO:   timescale             48000.00
INFO:   language              und
INFO: sampledescription:
INFO:   sampletype            mp4a
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested frame, ignoring data...
WARNING: Stream does not start with requested FLV frame, ignoring data...
WARNING: Stream does not start with requested FLV frame, ignoring data...
WARNING: Stream does not start with requested FLV frame, ignoring data...
WARNING: Stream does not start with requested FLV frame, ignoring data...
WARNING: Stream does not start with requested FLV frame, ignoring data...



It will constantly keep trying to resume, and constantly fail with that stream of warnings.

Under Windows, there were very few occasions where there was a proper error that needed the partial file deleted, but the only way to solve this seems to be kill the process, delete the file, then restart the download.

Anyone got any pointers?

user-2

04-12-2013, 11:27 PM
You can try to limit the number of retries by running with, e.g.,
Code:
--attempts=3
. The default is 50 per stream - read this:

http://www.mail-archive.com/[email protected]

If that doesn't help, there is nothing you can do except kill rtmpdump and delete the file. Even if rtmpdump quits of its own accord, the file is useless if rtmpdump couldn't find a spot to cleanly resume. The resume functionality can easily be thwarted if garbage comes over the wire or the connection is dropped - it's hit-and-miss. You don't say what OS or rtmpdump version you are using, so make sure it's of recent vintage.

user-2

05-12-2013, 12:59 AM
The OP of the mailing list thread I referenced implied he could make a partial file resumable with this:

https://raw.github.com/K-S-V/Scripts/mas...vFixer.php

I've never tried it.

user-493

30-06-2014, 11:03 PM
Quote:<div class="d4p-bbp-quote-title">Quote:</div>The OP of the mailing list thread I referenced implied he could make a partial file resumable with this:

https://raw.github.com/K-S-V/Scripts/mas...vFixer.php

I’ve never tried it.

How would you use this script? Thx.

user-2

01-07-2014, 11:08 AM
Code:
php FlvFixer.php --help

user-699

04-11-2014, 10:24 PM
rtmpdump does not resume properly.

It attempts to find the last video keyframe by reading backwards from the end of the file without checking that its looking at a valid header.

I have a small patch to find the last tag so that the original logic will work properly.

rtmpdump now resumes after you kill it in mid download.

congrats on finxing the get_iplayer awesome work!



Code:
    fseek(file, 5, SEEK_SET);
    if (fread(buffer, 1, 4, file) != 4) {
        RTMP_Log(RTMP_LOGERROR, "Couldn't read headerSize from file!");
        return RD_FAILED;
    }
    headerSize = AMF_DecodeInt32(buffer);
    offset = headerSize;

    while (offset < size) {
        fseeko(file, offset + 5, SEEK_SET);

        if (fread(buffer, 1, 4, file) != 4) {
//            RTMP_Log(RTMP_LOGERROR,
//                    "Couldn't read tag header offset %d last good tag %d!",
//                    (int) offset, (int) lastGoodTag);
            break;
        }
        payloadSize = AMF_DecodeInt24(buffer);
//        int lt, pt, ps, ts, sid;
//        lt = AMF_DecodeInt32(buffer);
//        pt = buffer[4];
//        ps = AMF_DecodeInt24(buffer + 5);
//        ts = AMF_DecodeInt24(buffer + 8);
//        ts |= (buffer[11] << 24);
//        sid = AMF_DecodeInt24(buffer + 12);
//        RTMP_Log(RTMP_LOGERROR, "Packet header @%d lt:%d pt:%d ps:%d ts:%d sid:%d!", offset,
//                lt, pt, ps, ts, sid);
//
//
//        if (((bAudioOnly && buffer[4] == 0x08) // audio frame
//        || (!bAudioOnly && (buffer[4] == 0x09 && (buffer[15] & 0xf0) == 0x10)) // video keyframe in forst byte of data!
//        ) && offset + payloadSize + 15 < size) {
//            lastGoodTag = offset;
//
//        }
        lastGoodTag = offset;
        offset += payloadSize + 15; // the back pointer the tag header and the payload
    }
    tsize = size - lastGoodTag - 4; // set tsize to point to offset from end of file of the header of last good tag

user-699

05-11-2014, 01:08 AM
same patch as a git diff sent to the rtmpdump mailing list

Code:
diff --git a/rtmpdump.c b/rtmpdump.c
index 13741a7..5a69395 100644
--- a/rtmpdump.c
+++ b/rtmpdump.c
@@ -303,8 +303,29 @@ GetLastKeyframe(FILE * file,       // output file [in]
   // find the last seekable frame
   off_t tsize = 0;
   uint32_t prevTagSize = 0;
+  uint32_t headerSize = 0;
+  uint32_t payloadSize = 0;
+  off_t offset = 0;
+  off_t lastGoodTag = 0;

   // go through the file and find the last video keyframe
+  fseek(file, 5, SEEK_SET);
+  if (fread(buffer, 1, 4, file) != 4) {
+       RTMP_Log(RTMP_LOGERROR, "Couldn't read headerSize from file!");
+       return RD_FAILED;
+  }
+  headerSize = AMF_DecodeInt32(buffer);
+  offset = headerSize;
+  while (offset < size) {
+       fseeko(file, offset + 5, SEEK_SET);
+       if (fread(buffer, 1, 4, file) != 4) {
+               break;
+       }
+       payloadSize = AMF_DecodeInt24(buffer);
+       lastGoodTag = offset;
+       offset += payloadSize + 15; // the back pointer the tag header and the payload
+  }
+  tsize = size - lastGoodTag - 4; // set tsize to point to offset from end of file of the header of last good tag
   do
     {
       int xread;

user-712

07-11-2014, 06:53 PM
Just wanted to say thanks for the patch. I was quite frustrated with having a lot of failed downloads with failed resumes. And your patch does seem to have corrected this.

user-745

28-11-2014, 09:39 AM
Improved resume patch

Is seems that the flv file can have bad headers when downloads abort to the iplayer site. This validates each header prevTagSize against the payload size of the preceding tag and starts a couple of tags back just in case.

Could also validate the flags I suppose anyone have any better ideas?

Code:
diff --git a/rtmpdump.c b/rtmpdump.c
index 13741a7..b9523bd 100644
--- a/rtmpdump.c
+++ b/rtmpdump.c
@@ -303,8 +303,48 @@ GetLastKeyframe(FILE * file,    // output file [in]
   // find the last seekable frame
   off_t tsize = 0;
   uint32_t prevTagSize = 0;
+  uint32_t headerSize = 0;
+  uint32_t payloadSize = 0;
+  off_t offset = 0;
+  off_t oldoffset = 0;
+  off_t lastGoodTag = 0;

   // go through the file and find the last video keyframe
+  fseek(file, 5, SEEK_SET);
+  if (fread(buffer, 1, 4, file) != 4) {
+      RTMP_Log(RTMP_LOGERROR, "Couldn't read headerSize from file!");
+      return RD_FAILED;
+  }
+  headerSize = AMF_DecodeInt32(buffer);
+  offset = headerSize;
+  while (offset < size) {
+      fseeko(file, offset , SEEK_SET);
+      if (fread(buffer, 1, 8, file) != 8) {
+          break;
+      }
+        prevTagSize = AMF_DecodeInt32(buffer);
+    // if prevTagSize dosent match previous payloadSize this packet is fubar
+    if ( ( prevTagSize != 0 ) && ( prevTagSize != ( payloadSize + 11 ) ) )
+        break;
+/*
+avoid this
+
+DEBUG: Packet: prevTagSize 5535d : lastGoodTag 997137924 oldoffset 997143463 offset 1003257860 payloadSize 6114382
+DEBUG: Packet: prevTagSize 3067505264d : lastGoodTag 997143463 oldoffset 1003257860 offset 1006408882 payloadSize 3151007
+DEBUG: Packet: prevTagSize 1520529135d : lastGoodTag 1003257860 oldoffset 1006408882 offset 1021507501 payloadSize 15098604
+DEBUG: Packet: prevTagSize 1933000270d : lastGoodTag 1006408882 oldoffset 1021507501 offset 1037784596 payloadSize 16277080
+DEBUG: Packet: prevTagSize 4182630141d : lastGoodTag 1021507501 oldoffset 1037784596 offset 1040989210 payloadSize 3204599
+DEBUG: Packet: prevTagSize 2634447075d : lastGoodTag 1037784596 oldoffset 1040989210 offset 1052125896 payloadSize 11136671
+DEBUG: Readback Packet: size 1045180416 tzize 7395816 offset 1037784596 prevTagSize -112337155
+
+*/
+      payloadSize = AMF_DecodeInt24(buffer+5);
+      lastGoodTag = oldoffset;
+      oldoffset = offset;
+      offset += payloadSize + 15; // the back pointer (4)  the tag header (11) and the payload
+      // RTMP_Log(RTMP_LOGDEBUG, "Packet: prevTagSize %u : lastGoodTag %d oldoffset %d offset %d payloadSize %d", (uint32_t)prevTagSize, (int)lastGoodTag, (int)oldoffset, (int)offset, (int)payloadSize);
+  }
+  tsize = size - lastGoodTag - 4; // set tsize to point to offset from end of file of the header of last good tag
   do
     {
       int xread;
@@ -324,6 +364,7 @@ GetLastKeyframe(FILE * file,    // output file [in]
    }

       prevTagSize = AMF_DecodeInt32(buffer);
+      // RTMP_Log(RTMP_LOGDEBUG, "Readback Packet: size %d tzize %d offset %d prevTagSize %d", (int)size, (int)tsize, (int)(size - tsize - 4), (int)prevTagSize);
       //RTMP_Log(RTMP_LOGDEBUG, "Last packet: prevTagSize: %d", prevTagSize);

       if (prevTagSize == 0)

user-745

28-11-2014, 09:40 AM
Trying to upload the patch

user-745

28-11-2014, 09:42 AM
Oh well patches are a security risk. Here is the patched c file.

user-779

16-12-2014, 01:50 AM
Hello! wow thanks a lot for posting a patch to this issue. I have been experiencing similar issues regarding those same warning messages. I am just using rtmpdump without iplayer and I am getting those messages. I would like to patch my rtmpdump.exe with this patch, but I do not know how to do this. If someone could post the patched rtmpdump.exe, it would be very much apprecaited :D
get_iplayer forums > Linux & Unix Forums > Linux & Unix General Help > Resuming broken?

Site design Copyright © SquarePenguin; some rights reserved.

Attribution of Licensed Works

These forums are archived

See this post for further info