Hi

I'm running Windows 7 Enterprise SP1 64-bit OS.

I've been recording Radio Live Streams with the following example command:


I've run this manually from the get_iplayer comand-line and also I've run it via the Windows task scheduler program. Most of the time all is ok. But sometimes it fails with it's 1st or even 2nd attempts. Here is an example of a failed one last night:

First attempt error message:


.ts file created and was fixed with: 


Second attempt error message:


.ts file created and was fixed with same ffmpeg commands as attempt 1 above.

Third attempt error message:


The above was in red and thus an error, but it did complete successfully in that a valid m4a file was created.

--

Now this would seem to be an error that ffmpeg is giving me. But does that mean the route of the problem is ffmpeg's handling of live streaming data and converting/muxing on the fly or is it caused by get_iplayer?

If it's ffmpeg, then would simply adding the --raw option to the command to leave ffmpeg out of the process be sensible?

I have just tried this with recording a minute duration and it leaves me with a .ts file (as expected) that requires the following ffmpeg command run on it to make it a valid m4a file:


I suppose the ultimate question is that since I want to record live radio (multiple recordings may need to be run targeting different radio stations at the same time) what is the best way?
Manually doing it and figuring out the --start --stop values is doable but not ideal.
Running via Windows Task Scheduler works, but still incurs problem as described here and is also not elegant.
 
I know there is the get_iplayer PVR and there are cron jobs, but I'm on Windows here so unsure if that is a easy solution for me.

I see there's a also the beta web PVR; maybe that is the solution I need??


Thank you for your help.

And thank you for making get_iplayer still available for us. 

btw I put this in the General Topics forum as I thought the errors are more of a holistic issue rather than a specific OS one... but appreciate that my ultimate desire for a Windows OS live recording solution is likely to be OS specific. Feel free to move if more appropriate.

Are there Radio 1 programmes that you can't get via get_iplayer after broadcast?  If not, there is absolutely no point in faffing around with live recording.

Don't use --start.  It doesn't really make sense for scheduled live recording. It works better with ffmpeg that it used to with rtmpdump, but it might still lead to the beginning of the file being garbaged and cause problems (though errors could still occur without --start) . As you saw, it's unpredictable.  If you really need to remove the first minute for some reason, do it on the finished file.

You can use --ffmpeg-liveradio-opts="-bsf:a aac_adtstoasc" (or add it to preferences) to have that filter always applied. It usually isn't necessary, but should be harmless. Using --raw is also an option if you don't need M4A files, but ffmpeg is still required to produce the .ts file.

Since you're on Windows, Task Scheduler iis your first port of call for automation.  There are third-party cron-like applications that might be easier to use. The web pvr is not an option.  It isn't really intended for live recording, and it can only run one download at a time (though you could run separate instances on different ports).  The CLI pvr isn't intended for live recording either. Although you could run multiple instances of the CLI pvr, there is no advantage over just running normal get_iplayer commands.

Thank you for the quick response user-2.

The reason for recording live radio streams is that they are transmitted at 320kbps rather than the 128kbps on the listen again player; so I'm going for the better quality version.

I think that when the BBC finally fully rollout their Audio Factory project the listen again iplayer will also be 320kbps.

For info, only Radio 3 & 6 Music broadcast in 320kbps AND a full frequency spectrum (>=20kHz), whereas they self-impose a 15-16kHz frequency cut-off for other stations including Radio 1 (as has been the case for FM broadcasts).

I do 100% agree that getting content from the listen again iplayer is far easier and I've had zero failures, but as I say it's a desire to get the best quality available right now.

The --start option is mainly used on the Windows Task Scheduler in an attempt to be "nice" to get_iplayer and let it do it's searches first and be ready to run the other commands... but I'm sure that is likely to be BS lol.

That being said, I have used --start in the manual running of get_iplayer and leave the window open until the time comes to start recording (say I manually set it at 10pm for a live recording to start at 4am). 

I do want to end up with m4a files after I've losslessly edited them in their aac format (using mp3directcut). Can I leave them as aac files then?

So, the errors are just down to the live recording not being 100% stable?

many thanks

320k - I get it.  However, I'm not betting on any more 320k AOD being available to get_iplayer in the future.  When Audio Factory arrived, they removed all the previous 320k AOD resources except for Radio 3 Flash streams. 

That isn't what --start does.  It just tells ffmpeg to discard X amount of the stream before writing to output.  It doesn't have anything to do with scheduled execution and it doesn't impact get_iplayer itself.

Yes, keep AAC for editing if you can.  Converting to MP3 (or anything else) would defeat the purpose. I don't know if mp3directcut will rewrite the MP4 container and thus stomp the metadata tags. 

AFAICT, the errors are probably down to using --start and perhaps a hiccup at the CDN server, though you can't really call that an error if the output file was OK.

aaaahhh I didn't realise that is what --start did. Ok, I get it now, thanks. I'll remove it's use from now on.

Can I still use the --stop option?
It means that when the recording completes, get_iplayer ends correctly and a valid m4a file is produced, rather than ending "abruptly" leaving me with a .ts file that requires some bitstream fixing (although I see I can add that to the command options as you kindly suggested).

No, I don't convert to mp3 as that as you say defeats the purpose. mp3directcut introduced aac lossless editing three years ago (it used to only work for mp3s). But it has to have the aac file and cannot work on the m4a/mp4 container formats. So currently I will extract the raw aac file from the m4a using mp4box, do the editing, and put it back into the m4a container with mp4box. The tags are removed when extracting the aac file from the m4a file.

what options(s) would I need to add to the command to keep the output file as an .aac file?
or should I leave as is with it being output as an m4a?
or even reduce extra work by the program and leave as a raw .ts file?

Fingers crossed they provide 320kbps on listen again, but you might well be right and they keep as 128kbps.

As you say, you need --stop for get_iplayer to finish cleanly, so use it.

I see - you need raw AAC.  You can use --command to automatically dump the AAC to a separate file after download with ffmpeg or mp4box.  If you use ffmpeg you might as well use --raw since the AAC stream can just as easily be extracted from the the TS file.

thanks Dinky, I shall get onto refining the live recording command options :)

Hi user-2

A few of us are still using the live stream recording functionality. Over the past couple of weeks we have noticed more errors and thus unsuccessful recordings.

Now, I'm fairly certain this is all at the BBC end, since one of the errors we've been getting recently is a "HTTP error 500 Internal Server Error".

Am I right is assuming that as get_iplayer is a command-line program it's error handling (esp with these live streams) is minimal to non-existent?

If so, is there any way for get_iplayer to ignore errors and simply push on recording for the set time?

My guess is that improving stability of get_iplayer for live stream recording is a low priority for you, but I just wonder if a "quick fix" could be to relax/remove it's aborting when it encounters an error. Perhaps it's an option that can be set to say "ignore errors".

Many thanks

How do you imagine get_iplayer could "push on" through server errors over which it has no control?  If the servers won't play ball, there is nothing get_iplayer can do about it. In this context, get_iplayer can only do what ffmpeg can do, and ffmpeg can only do what the CDN servers and your connection will support at a given time. It's not a question of get_iplayer's stability.

get_iplayer doesn't simply abort when an error occurs during live recording. Use --attempts if you want more than one try for each recording mode for live streams. Only you can know how poor your connection may be or how recalcitrant the servers are being at any point in time, so adjust to suit.

Ok, understood.

No offence was meant.

Everything you have said I agree with and already knew... I had a taken a moment to think about it rather than blurt out my question. I simply made the incorrect assumption that why can't get_iplayer behave like how we capture DVB-S live streams. These DVB-S very rarely fail to record and indeed when a iplayer live stream fails the same show's DVB-S stream is ok.

So whilst these come from the same original source, I'm guessing the architecture and system and servers are different for the DVB-S streams??

I have used the --attempt option and with these 500 server errors it doesn't restart. With other errors like the "ADTS format and missing data" it does restart.

Thanks, I appreciate your help and patience :-)

(13-10-2015, 02:19 PM)No offence was meant.

None taken, but your expectations are unrealistic.

(13-10-2015, 02:19 PM)So whilst these come from the same original source, I'm guessing the architecture and system and servers are different for the DVB-S streams??

And now I know why. People have paid a lot of money for a lot of infrastructure to ensure satellite broadcasts are reliable. The BBC has paid a lot of money for a lot of infrastructure for iPlayer and IPTV, but there is no guaranteed service level. There is no way they can anticipate all the vagaries of every user, computer, network link, etc., and servers are going to have problems. Only one stream needs to get to a satellite.

(13-10-2015, 02:19 PM)I have used the --attempt option and with these 500 server errors it doesn't restart. With other errors like the "ADTS format and missing data" it does restart.

If ffmpeg is getting 500 errors when attempting to access a playlist, there will be nothing to record and thus nothing to restart. But I have no idea if that is what is happening.  You can't just cryptically allude to errors and expect me to have a clue what you're talking about. Put your output in a .txt file and attach it to a post.

Two error logs attached.

1. get_iplayer_http-error-500.txt
2. get_iplayer_aac-bitstream-error.txt

The output above comes from the command window that get_iplayer runs in.

The redirected output from get_iplayer simply gives
INFO: Command exit code 1 (raw code = 256)

Thanks

Thanks. A few tips for the future:

• Redirect both stdout and stderr to a file to capture all output
  
• Use --verbose, but only for radio with HLS. HLS TV downloads will fill the log with ffmpeg warnings. 
  
• Don't do testing from Windows Scheduler (if that's what the text at the top of one of your files means).  Run from the command prompt.  Your scheduler job arguments look wrong to me as well, as if you pointed it to both the start menu shortcut and get_iplayer script somehow.
  

In the first case, you're getting a 500 error from the server when ffmpeg attempts to download a stream segment.  There is nothing get_iplayer can do about that. ffmpeg may not signal an error condition when that happens, so get_iplayer is none the wiser and runs to finish. In the other case, ffmpeg received some junk data and gave up.  Again, there is nothing get_iplayer can do about it if ffmpeg doesn't signal an error condition. This problem could be a result of poor connectivity. The 500 errors point to server problems, though there is a small possibility that they could result from corrupted requests (poor connectivity again). Not surprisingly, I can't replicate either problem.  You might want to grab a newer version of ffmpeg and try it with --ffmpeg. The Apple HLS support may have improved in some way that might help you, though I can only speculate. If these gremlins plague you consistently, you probably need to create a wrapper script that will keep launching get_iplayer until the desired amount of time has expired.

Great, thanks for the info.

I cannot recreate these all the time. It only happens maybe 25% of the time.

The number 2 log (bitstream error) was run at cmd line. The other one failed whilst being run from the scheduler. Most have to run by the scheduler due to being in the early hours of the morning. Most of the time there are no errors with them. Suffice to say I've had the same errors occur when they have been run via the scheduler and direct by me adding the commands into the cmd line.

Thanks for the feedback. My next stop was to try a new version of ffmpeg, so thanks for the nudge.

Cheers

Full sdtout & stderr with the verbose mode on with a recording that failed with the http 500 error a couple of hours ago: get_iplayer_http-error-500-2.txt

Thanks. As I suspected, ffmpeg doesn't signal an error. No idea why you would encounter this even 25% of the time. One possibility is that there is a caching proxy between you and the CDN that is overloaded or failing, but that is just a guess. It looks like ffmpeg may collapse all 50x HTTP codes to 500, so a little information is getting lost as well.

I have also been seeing a few error 500 internal server errors when visiting radio pages on iPlayer recently and this is on the site itself. Sometimes when I try and load a programme, it takes a while and then eventually returns the BBCs Internal Server Error (500) page or just a standard error 500 page.

Haven't seen this with TV so far so it only seems to be affecting the radio side of things but it does seem like something is going on there.

(15-10-2015, 07:32 PM)I have also been seeing a few error 500 internal server errors when visiting radio pages on iPlayer recently and this is on the site itself. Sometimes when I try and load a programme, it takes a while and then eventually returns the BBCs Internal Server Error (500) page or just a standard error 500 page.

Haven't seen this with TV so far so it only seems to be affecting the radio side of things but it does seem like something is going on there.

now you mention it I have also had a few server errors whilst looking at BBC pages over the past couple of weeks. I hadn't made the connection to the live streaming recordings until now.

(15-10-2015, 07:22 PM)Thanks.  As  I suspected, ffmpeg doesn't signal an error. No idea why you would encounter this even 25% of the time. One possibility is that there is proxy between you and the CDN that is overloaded or failing, but that is just a guess.  It looks like ffmpeg may collapse all 50x HTTP codes to 500, so a little information is getting lost as well.

Does that mean ffmpeg should signal an error? 
And if it did, I'm guessing get_iplayer would still cease recording??
Which would imply that until the connection at the BBC is more reliable, this will continue right?

btw this is also happening with two of my friends, so that would lower the chances of this being an individual connection issue. In fact, a couple of weeks we were all recording the same radio show and it failed for us all at the same time with the 500 HTTP code.

(15-10-2015, 07:32 PM)I have also been seeing a few error 500 internal server errors when visiting radio pages on iPlayer recently and this is

500 errors for programme pages on the iPlayer Radio site are a different story, and different servers. A programme page pulls in all sorts of stuff from other systems - metadata from Nitro, etc. - and the servers appear to return 500 if all that stuff can't be assembled within some time limit.  These errors come when accessing an actual piece of the media stream from a CDN server.  AFAIK, 320k HLS live streams aren't used for any public BBC services, so maybe they get fewer resources or lower priority - more wild guessing. I don't use them enough to know.

(15-10-2015, 07:59 PM)Does that mean ffmpeg should signal an error? 

Seems like it, but hard to say. The upstream error could be anything, so if ffmpeg has recorded everything up to that point, has it really failed? One for the philosophers.

(15-10-2015, 07:59 PM)And if it did, I'm guessing get_iplayer would still cease recording??

No, it would retry as normal

(15-10-2015, 07:59 PM)Which would imply that until the connection at the BBC is more reliable, this will continue right?

One would think.  Perhaps take a look at livestreamer and see if it is more resilient than get_iplayer+ffmpeg.