Problem with programmes that are over 7 days old
#1
Have started having problems with programmes that are over 7 days old again today.

It seems that some programmes that have gone past 7 days are refusing to download the default version even though it shows as available in
Code:
--info
and shows modes as available for the default version.

An example is http://www.bbc.co.uk/iplayer/episode/b00...-episode-1 which is fully playable on iPlayer but cannot seem to be downloaded.

Output with
Code:
--info

<pre>
Code:
C:\Program Files (x86)\get_iplayer>get_iplayer --modes=best --output=i:/videos --p
id b0079ckz --info
get_iplayer v2.90, Copyright (C) 2008-2010 Phil Lewis
  This program comes with ABSOLUTELY NO WARRANTY; for details use --warranty.
  This is free software, and you are welcome to redistribute it under certain
  conditions; use --conditions for details.

INFO Trying to stream pid using type tv
INFO: pid found in cache
Matches:
1992:   The Slammer: Series 1 - Episode 1, CBBC, Children's,Entertainment & Come
dy, default
INFO: File name prefix = The_Slammer_Series_1_-_Episode_1_b0079ckz_default

available:      Unknown
brand:          The Slammer
categories:     Children's,Entertainment & Comedy
category:       Children's
channel:        CBBC
desc:           Welcome to The Slammer, a prison for entertainers. Each week fou
r acts are chosen to perform for their freedom in front of a live studio audienc
e.
descmedium:     Welcome to The Slammer, a prison for entertainers. Each week fou
r acts are chosen to perform for their freedom in front of a live studio audienc
e.
descshort:      Welcome to the prison for entertainers. Each week, four acts per
form for their freedom.
dir:            I:\videos
dldate:         2014-12-01
dltime:         16:10:36
duration:       1800
durations:      default: 1800
durations:      signed: 1680
episode:        Episode 1
episodenum:     1
episodeshort:   Episode 1
expiry:         2014-12-24T11:30:00Z
expiryrel:      in 22 days 19 hours
ext:            EXT
filename:       I:\videos\The_Slammer_Series_1_-_Episode_1_b0079ckz_default.EXT
filepart:       I:\videos\The_Slammer_Series_1_-_Episode_1_b0079ckz_default.partia
l.EXT
fileprefix:     The_Slammer_Series_1_-_Episode_1_b0079ckz_default
firstbcast:     default: 2006-09-22T16:55:00+01:00
firstbcast:     signed: 2008-03-25T10:00:00Z
firstbcastdate: default: 2006-09-22
firstbcastdate: signed: 2008-03-25
firstbcastrel:  default: 8 years 70 days 0 hours ago
firstbcastrel:  signed: 6 years 251 days 6 hours ago
index:          1992
lastbcast:      default: 2014-11-24T11:00:00Z
lastbcast:      signed: 2008-04-10T10:00:00+01:00
lastbcastdate:  default: 2014-11-24
lastbcastdate:  signed: 2008-04-10
lastbcastrel:   default: 7 days 5 hours ago
lastbcastrel:   signed: 6 years 235 days 7 hours ago
longname:       The Slammer: Series 1
modes:          default: flashhigh1,flashhigh2,flashstd1,flashstd2,flashvhigh1,f
lashvhigh2,rtsphigh1,rtsphigh2,rtspstd1,rtspstd2,rtspvhigh1,rtspvhigh2,subtitles
1
modes:          signed:
modesizes:      default: flashhigh1=175MB,flashhigh2=175MB,flashstd1=105MB,flash
std2=105MB,flashvhigh1=330MB,flashvhigh2=330MB,rtsphigh1=175MB,rtsphigh2=175MB,r
tspstd1=105MB,rtspstd2=105MB,rtspvhigh1=330MB,rtspvhigh2=330MB
modesizes:      signed:
name:           The Slammer: Series 1
nameshort:      The Slammer
pid:            b0079ckz
player:         http://www.bbc.co.uk/iplayer/episode/b0079ckz
senum:          s01e01
series:         Series 1
seriesnum:      1
thumbfile:      I:\videos\The_Slammer_Series_1_-_Episode_1_b0079ckz_default.jpg
thumbnail:      http://ichef.bbci.co.uk/images/ic/150x84/p01lcfwg.jpg
timeadded:      6 days 17 hours ago (1416869634)
title:          The Slammer: Series 1: Episode 1
type:           tv
verpids:        default: b006qyfg
verpids:        signed: b009nsp7
version:        default
versions:       default,signed
web:            http://www.bbc.co.uk/programmes/b006mkqb

INFO: 1 Matching Programmes

C:\Program Files (x86)\get_iplayer>
</pre>



Shows that default does exist and has modes available.

Output from download attempt with
Code:
--verbose

<pre>
Code:
C:\Program Files (x86)\get_iplayer>get_iplayer --modes=best --output=i:/videos --p
id b0079ckz --get --verbose
get_iplayer v2.90, Copyright (C) 2008-2010 Phil Lewis
  This program comes with ABSOLUTELY NO WARRANTY; for details use --warranty.
  This is free software, and you are welcome to redistribute it under certain
  conditions; use --conditions for details.

INFO: encodinglocale = cp1252
INFO: encodinglocalefs = cp1252
INFO: encodingconsoleout = cp850
INFO: encodingconsolein = cp850
INFO: ${^UNICODE} = 0
INFO: User prefs dir: C:\Users\User1/.get_iplayer
INFO: System options dir: C:\ProgramData/get_iplayer/options
Current options:
        encodingconsolein = cp850
        encodingconsoleout = cp850
        encodinglocale = cp1252
        encodinglocalefs = cp1252
        expiry = 604800
        get = 1
        mmsnothread = 1
        modes = best
        nopurge = 1
        output = i:/videos
        pid = b0079ckz
        rtmpdump = .\RTMPDump\rtmpdump-20140302-git-79459a2-win32\bin\rtmpdump.e
xe
        verbose = 1
        vlc = .\VLC\vlc-2.1.5\vlc.exe

INFO: Search args: ''
INFO: Will try prog types: tv
INFO: Loaded history for first check.
INFO: Loading recordings history
INFO: Programme not in history
INFO: Got 2290 file cache entries for tv
INFO Trying to stream pid using type tv
INFO: pid found in cache
Matches:
1992:   The Slammer: Series 1 - Episode 1, CBBC, Children's,Entertainment & Come
dy, default

INFO: 1 Matching Programmes
INFO: Programme not in history
INFO: iPlayer metadata URL = http://www.bbc.co.uk/iplayer/playlist/b0079ckz
INFO: Getting page http://www.bbc.co.uk/iplayer/playlist/b0079ckz
INFO: iPlayer metadata URL (JSON) = http://www.bbc.co.uk/programmes/b0079ckz/pla
ylist.json
INFO: Getting page http://www.bbc.co.uk/programmes/b0079ckz/playlist.json

WARNING: No programmes are available for this pid with version(s): default
ERROR: Could not get version pid metadata

C:\Program Files (x86)\get_iplayer>
</pre>



Also http://www.bbc.co.uk/iplayer/episode/b03...d-galloway is refusing to download saying that only the
Code:
shortened,audiodescribed
versions are available. The
Code:
--info
shows that there is a default version there but there are no modes for it.
#2
The Slammer downloads fine here. There are no default version streams for Escape to the Country. It happens sometimes, particular with old programmes - blame the Beeb. If you see a reference to a shortened version it suggests you are using a recent version of get_iplayer from GitHub, but the output from your Slammer download suggests that you are using 2.90 as released. Save us both some time and go get the very latest version of get_iplayer from GitHub. See Git HEAD section in wiki installation page. If those programmes still don't download, run with
Code:
--verbose
and attach the log as a .txt file. I can't tell anything from the above.
#3
The version I was using was the git HEAD version from 28/11/2014. Have now updated to the most recent version and still have the same problem.

When I update from the git HEAD I save the file and then replace it manually in the get_iplayer folder rather than running the copy and rename script in the wiki. Could that be why the update doesn't show in the output?

The
Code:
--verbose
output for the Slammer is the same as above.

Code:
ERROR: Could not get version pid metadata
is the line in there that I'm unsure of though. Seems to be after it's checked the JSON playlist.

Interestingly, if I change the script for a slightly older git HEAD version which I downloaded on the 12th, the Slammer downloads fine.
#4
That's not the output you should get from Git HEAD, so something is going wrong when you update. HEAD doesn't pull JSON playlists unless you don't have the XML::Simple Perl module installed. If that is the case, then you somehow have an incomplete or damaged installation. A version from 12 Nov won't have those changes, so perhaps you do have an incomplete installation. I can't duplicate the problem anyway - any version works for me. Grab Git HEAD from today (2 Dec) or later, then run
Code:
--info
for that programme. If the
Code:
--info
output has a "desclong" field, then you're running the correct script. And in your get_iplayer console window, run
Code:
perl -MXML::Simple -e 1
. If you don't see an error message, then the module is installed.
#5
"desclong" is showing in the --info and
Code:
perl -MXML::Simple -e 1
shows no errors.

Have also installed 2.90 on another machine and updated it with the most recent script with the same error.
#6
Try Git HEAD a few more times with
Code:
--verbose
and these additional options:

<pre>
Code:
Attempt 1: --force --overwrite
Attempt 2: --no-playlist-metadata --force --overwrite
Attempt 3: --scrape-versions --force --overwrite
Attempt 4: --playlist-metadata --scrape-versions --force --overwrite
</pre>


Post the output as .txt files. Don't worry about the file names. I can tell from the contents which is which.
#7
Attempts 3 & 4 worked and the programme downloaded. All outputs attached.


Attached Files
.txt   attempts12.txt (Size: 4.15 KB / Downloads: 164)
.txt   attempt3.txt (Size: 27.78 KB / Downloads: 213)
.txt   attempt4.txt (Size: 27.78 KB / Downloads: 212)
#8
Thanks. Those results at least confirm what is happening, but not why it is happening. It's still impossible for me to duplicate. Instead of this:

<pre>
Code:
INFO: 1 Matching Programmes
INFO: iPlayer metadata URL = http://www.bbc.co.uk/iplayer/playlist/b0079ckz
INFO: Getting page http://www.bbc.co.uk/iplayer/playlist/b0079ckz
INFO: iPlayer metadata URL (JSON) = http://www.bbc.co.uk/programmes/b0079ckz/playlist.json
</pre>




- which means you are getting the old, semi-busted metadata extraction - you should see something like:

<pre>
Code:
INFO: 1 Matching Programmes
INFO: Getting page http://www.bbc.co.uk/programmes/b0079ckz.xml
</pre>




Check a few BBC One or Two programmes to see if you get the same results, or whether the new metadata extraction is used. The implication of your test is that a check of the PID value is failing for some reason in your version of get_iplayer. Just to check if sanity still reigns, run the following in your get_iplayer console window:

<pre>
Code:
perl -e "print 'b0079ckz' =~ /^[bp]0[a-z0-9]{6}$/"
</pre>




It should print "1".

I'm going to remove the
Code:
--scrape...
options and re-enable the screen scraping by default when using the old metadata extraction method. It turns out that many CBBC programmes now appear to be missing any playlist data, so screen scraping is the only way to go in that case. Scraping is not needed with the new method.

Also: you don't need
Code:
--get
with
Code:
--pid
.
#9
Sanity check comes back with "1"

I still can't see any reference to the .xml playlist though when it tries to download a programme. Is seems to go straight for the JSON.

It does use mediaselector5 though to get the actual download data.

Have tried with last weeks Watchdog and Not Going Out. Output attached.


Attached Files
.txt   watchdog-not-going-out-output.txt (Size: 54.63 KB / Downloads: 179)
#10
The mediaselector reference was irrelevant - copy/paste error. "iPlayer metadata URL" refers to the XML playlist. The URL doesn't have a .xml extension.

Still unable to reproduce this behaviour, so whatever it is appears to be peculiar to your system. If you know some Perl, I can point you towards the place in the code to investigate. Otherwise I'm afraid I must throw in the towel. Can't fix a problem I can't see.

I've updated GitHub and removed the
Code:
--scrape...
options. TV episode pages will be scraped by default (turn off with
Code:
--no-scrape-versions
), so hopefully no more problems with CBBC programmes.
#11
Thanks for your help.

I don't know any Perl.

The odd thing is that it is affecting 2 machines in the same way. One is Win 7 and the other is XP.

Maybe when version 2.91 is released, whatever it is that's causing the problem might be solved by a proper install?

I will update to the latest HEAD version and see how it goes.

Thanks
#12
What do you mean by "proper install"? If you didn't use the get_iplayer installer and then update over it with Git HEAD, or if you're not using the version of Perl delivered with the installer, then all bets are off. And you shouldn't count on the next release to fix anything. The particular bit of code in question isn't going to change before then. It already works fine here on XP/7/8. Even so, the old metadata code will keep working for you - it will be just be a bit slower and less accurate.
#13
Quote:Dinky Pumpkin wrote: What do you mean by “proper install”?
I mean get_iplayer_latest.exe. It could be something I've done when updating to the HEAD version that's causing me these problems. The next installer will do it properly rather than me replacing the script.

I upgraded to 2.90 and then updated to the HEAD version over the top of that. All I did was save the git HEAD file and replace the original one with that.

Is there a command I can use to force the new metadata method to be used?
#14
No. It's the default. It only requires the programme have recognisable a PID. So in your case, either something is wrong with all your PIDs, or as you suggest, you've mad an error in installation. You can try replacing this line:

<pre>
Code:
} elsif ( $prog->{pid} =~ /^[bp]0[a-z0-9]{6}$/ && ! $opt->{playlistmetadata} ) {
</pre>



with this:

<pre>
Code:
} elsif ( ! $opt->{playlistmetadata} ) {
</pre>



to avoid the PID check, but I've no idea if that will work.
#15
I think I might of sorted it.

It seems that the new metadata method relies on AtomicParsley being installed. Without AtomicParsley, it drops back to the old method.

Now that AtomicParsley is installed I am getting

<pre>
Code:
Matches:
1849:   The One Show - 01/12/2014, BBC One, Arts Culture & the Media,Factual, de
fault

INFO: 1 Matching Programmes
INFO: Programme metadata URL = http://www.bbc.co.uk/programmes/b04snnkp.xml
INFO: Getting page http://www.bbc.co.uk/programmes/b04snnkp.xml
INFO: Getting page http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0
/mediaset/pc/vpid/b04snnjm?cb=51123
</pre>



Had a problem yesterday though with an episode of Escape to the Country from last weekend. Unfortunately it is now no longer available but when I tried to download, there was no default version but there was a "lengthened" version available but this only had flashhigh and flashstd available. I had to revert to an older script to get the flashvhigh stream to download. (I think it was the original script that came with 2.90 that I reverted back to)
#16
Quote:tvfan wrote:
It seems that the new metadata method relies on AtomicParsley being installed. Without AtomicParsley, it drops back to the old method.

Well-spotted. Fixed in Git.

Quote:Had a problem yesterday though with an episode of Escape to the Country from last weekend. Unfortunately it is now no longer available but when I tried to download, there was no default version but there was a “lengthened” version available but this only had flashhigh and flashstd available. I had to revert to an older script to get the flashvhigh stream to download. (I think it was the original script that came with 2.90 that I reverted back to)

If you don't give me an exact episode, I can't check it.
#17
Quote:Dinky Pumpkin wrote:
If you don’t give me an exact episode, I can’t check it.

I thought that would probably be the case but I thought I'd report it anyway in case it came up again.

It was http://www.bbc.co.uk/programmes/b00psvz4 but was only available for 7 days.
#18
From the metadata that remains, it looks like one version of the programme was marked as both lengthened and audio described, so the streams for the audio described version were used instead (flashhigh best available). get_iplayer should now be able to pick out the audio described version separately, at least in some cases.
#19
I think you are right. Default had no modes, lengthened only showed flashhigh and flashstd and I think the audiodescribed version showed the same.

I did try
Code:
--versions=lengthened
but it told me that flashvhigh was not available and I should try flashhigh or flashstd.

The
Code:
--info
data reflected this.

Possibly if I had run
Code:
--playlist-metadata
on it it may well have found the flashvhigh version because that would also have scraped for versions wouldn't it?

Reverting to the original 2.90 .pl file probably did this for me so the flashvhigh download worked. It downloaded it as the default version using the original .pl
#20
Code:
--playlist-metadata
would probably work in this case. It doesn't work in every case, so I'm trying to work out the correct juju to avoid any extra scraping. Latest code will discard any versions that don't have streams, and tries to to use any alternate version (e.g., "shortened") as default if listed default version doesn't have any streams, but there may be some as-yet-unseen permutations that will confuse it.


Possibly Related Threads…
Thread Author Replies Views Last Post
  Radio: 30 days and everything 320kbps? radiodaze 6 10,156 21-05-2016, 12:05 PM
Last Post: jaybeee
  problem getting episodes after 7 days piddigger 2 2,759 01-11-2014, 06:04 PM
Last Post: tvfan
  get_iplayer Cannot Search Programmes Older Than 7 Days dinky 1 3,155 29-10-2014, 06:14 PM
Last Post: SquarePenguin
  Problem downloading certain programmes tvfan 5 5,043 03-10-2014, 07:22 PM
Last Post: dinky