Start moving to ytdl-org

2019-03-09 19:15:16 +07:00
35 changed files with 554 additions and 791 deletions
@@ -6,8 +6,8 @@
 ---
-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2019.03.18*. If it's not, read [this FAQ entry](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2019.03.09*. If it's not, read [this FAQ entry](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2019.03.18**
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2019.03.09**
 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through the [README](https://github.com/ytdl-org/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/ytdl-org/youtube-dl#faq) and [BUGS](https://github.com/ytdl-org/youtube-dl#bugs) sections
@@ -36,7 +36,7 @@ Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2019.03.18
+[debug] youtube-dl version 2019.03.09
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
@@ -1,37 +1,3 @@
 version 2019.03.18
 Core
 * [extractor/common] Improve HTML5 entries extraction
 + [utils] Introduce parse_bitrate
 * [update] Hide update URLs behind redirect
 * [extractor/common] Fix url meta field for unfragmented DASH formats (#20346)
 Extractors
 + [yandexvideo] Add extractor
 * [openload] Improve embed detection
 + [corus] Add support for bigbrothercanada.ca (#20357)
 + [orf:radio] Extract series (#20012)
 + [cbc:watch] Add support for gem.cbc.ca (#20251, #20359)
 - [anysex] Remove extractor (#19279)
 + [ciscolive] Add support for new URL schema (#20320, #20351)
 + [youtube] Add support for invidiou.sh (#20309)
 - [anitube] Remove extractor (#20334)
 - [ruleporn] Remove extractor (#15344, #20324)
 * [npr] Fix extraction (#10793, #13440)
 * [biqle] Fix extraction (#11471, #15313)
 * [viddler] Modernize
 * [moevideo] Fix extraction
 * [primesharetv] Remove extractor
 * [hypem] Modernize and extract more metadata (#15320)
 * [veoh] Fix extraction
 * [escapist] Modernize
 - [videomega] Remove extractor (#10108)
 + [beeg] Add support for beeg.porn (#20306)
 * [vimeo:review] Improve config url extraction and extract original format
  (#20305)
 * [fox] Detect geo restriction and authentication errors (#20208)
 version 2019.03.09
 Core
@@ -44,7 +44,9 @@
 - **AmericasTestKitchen**
 - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
 - **AnimeOnDemand**
 - **anitube.se**
 - **Anvato**
 - **AnySex**
 - **APA**
 - **Aparat**
 - **AppleConnect**
@@ -696,6 +698,7 @@
 - **PornoXO**
 - **PornTube**
 - **PressTV**
 - **PrimeShareTV**
 - **PromptFile**
 - **prosiebensat1**: ProSiebenSat.1 Digital
 - **puhutv**
@@ -715,7 +718,7 @@
 - **radio.de**
 - **radiobremen**
 - **radiocanada**
- - **radiocanada:audiovideo**
+ - **RadioCanadaAudioVideo**
 - **radiofrance**
 - **RadioJavan**
 - **Rai**
@@ -762,6 +765,7 @@
 - **RTVS**
 - **Rudo**
 - **RUHD**
 - **RulePorn**
 - **rutube**: Rutube videos
 - **rutube:channel**: Rutube channels
 - **rutube:embed**: Rutube embedded videos
@@ -1006,6 +1010,7 @@
 - **video.mit.edu**
 - **VideoDetective**
 - **videofy.me**
 - **VideoMega**
 - **videomore**
 - **videomore:season**
 - **videomore:video**
@@ -1122,7 +1127,6 @@
 - **yandexmusic:album**: Яндекс.Музыка - Альбом
 - **yandexmusic:playlist**: Яндекс.Музыка - Плейлист
 - **yandexmusic:track**: Яндекс.Музыка - Трек
 - **YandexVideo**
 - **YapFiles**
 - **YesJapan**
 - **yinyuetai:video**: 音悦Tai
@@ -107,184 +107,6 @@ class TestInfoExtractor(unittest.TestCase):
        self.assertRaises(ExtractorError, self.ie._download_json, uri, None)
        self.assertEqual(self.ie._download_json(uri, None, fatal=False), None)
    def test_parse_html5_media_entries(self):
        # from https://www.r18.com/
        # with kpbs in label
        expect_dict(
            self,
            self.ie._parse_html5_media_entries(
                'https://www.r18.com/',
                r'''
                <video id="samplevideo_amateur" class="js-samplevideo video-js vjs-default-skin vjs-big-play-centered" controls preload="auto" width="400" height="225" poster="//pics.r18.com/digital/amateur/mgmr105/mgmr105jp.jpg">
                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_sm_w.mp4" type="video/mp4"  res="240" label="300kbps">
                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dm_w.mp4" type="video/mp4"  res="480" label="1000kbps">
                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dmb_w.mp4" type="video/mp4"  res="740" label="1500kbps">
                    <p>Your browser does not support the video tag.</p>
                </video>
                ''', None)[0],
            {
                'formats': [{
                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_sm_w.mp4',
                    'ext': 'mp4',
                    'format_id': '300kbps',
                    'height': 240,
                    'tbr': 300,
                }, {
                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dm_w.mp4',
                    'ext': 'mp4',
                    'format_id': '1000kbps',
                    'height': 480,
                    'tbr': 1000,
                }, {
                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dmb_w.mp4',
                    'ext': 'mp4',
                    'format_id': '1500kbps',
                    'height': 740,
                    'tbr': 1500,
                }],
                'thumbnail': '//pics.r18.com/digital/amateur/mgmr105/mgmr105jp.jpg'
            })
        # from https://www.csfd.cz/
        # with width and height
        expect_dict(
            self,
            self.ie._parse_html5_media_entries(
                'https://www.csfd.cz/',
                r'''
                <video width="770" height="328" preload="none" controls  poster="https://img.csfd.cz/files/images/film/video/preview/163/344/163344118_748d20.png?h360" >
                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327358_eac647.mp4" type="video/mp4" width="640" height="360">
                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327360_3d2646.mp4" type="video/mp4" width="1280" height="720">
                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327356_91f258.mp4" type="video/mp4" width="1920" height="1080">
                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327359_962b4a.webm" type="video/webm" width="640" height="360">
                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327361_6feee0.webm" type="video/webm" width="1280" height="720">
                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327357_8ab472.webm" type="video/webm" width="1920" height="1080">
                    <track src="https://video.csfd.cz/files/subtitles/163/344/163344115_4c388b.srt" type="text/x-srt" kind="subtitles" srclang="cs" label="cs">
                </video>
                ''', None)[0],
            {
                'formats': [{
                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327358_eac647.mp4',
                    'ext': 'mp4',
                    'width': 640,
                    'height': 360,
                }, {
                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327360_3d2646.mp4',
                    'ext': 'mp4',
                    'width': 1280,
                    'height': 720,
                }, {
                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327356_91f258.mp4',
                    'ext': 'mp4',
                    'width': 1920,
                    'height': 1080,
                }, {
                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327359_962b4a.webm',
                    'ext': 'webm',
                    'width': 640,
                    'height': 360,
                }, {
                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327361_6feee0.webm',
                    'ext': 'webm',
                    'width': 1280,
                    'height': 720,
                }, {
                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327357_8ab472.webm',
                    'ext': 'webm',
                    'width': 1920,
                    'height': 1080,
                }],
                'subtitles': {
                    'cs': [{'url': 'https://video.csfd.cz/files/subtitles/163/344/163344115_4c388b.srt'}]
                },
                'thumbnail': 'https://img.csfd.cz/files/images/film/video/preview/163/344/163344118_748d20.png?h360'
            })
        # from https://tamasha.com/v/Kkdjw
        # with height in label
        expect_dict(
            self,
            self.ie._parse_html5_media_entries(
                'https://tamasha.com/v/Kkdjw',
                r'''
                <video crossorigin="anonymous">
                        <source src="https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4" type="video/mp4" label="AUTO" res="0"/>
                                <source src="https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4" type="video/mp4"
                                        label="240p" res="240"/>
                                <source src="https://s-v2.tamasha.com/statics/videos_file/20/00/Kkdjw_200041c66f657fc967db464d156eafbc1ed9fe6f_n_144.mp4" type="video/mp4"
                                        label="144p" res="144"/>
                </video>
                ''', None)[0],
            {
                'formats': [{
                    'url': 'https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4',
                }, {
                    'url': 'https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4',
                    'ext': 'mp4',
                    'format_id': '240p',
                    'height': 240,
                }, {
                    'url': 'https://s-v2.tamasha.com/statics/videos_file/20/00/Kkdjw_200041c66f657fc967db464d156eafbc1ed9fe6f_n_144.mp4',
                    'ext': 'mp4',
                    'format_id': '144p',
                    'height': 144,
                }]
            })
        # from https://www.directvnow.com
        # with data-src
        expect_dict(
            self,
            self.ie._parse_html5_media_entries(
                'https://www.directvnow.com',
                r'''
                <video id="vid1" class="header--video-masked active" muted playsinline>
                    <source data-src="https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4" type="video/mp4" />
                </video>
                ''', None)[0],
            {
                'formats': [{
                    'ext': 'mp4',
                    'url': 'https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4',
                }]
            })
        # from https://www.directvnow.com
        # with data-src
        expect_dict(
            self,
            self.ie._parse_html5_media_entries(
                'https://www.directvnow.com',
                r'''
                <video id="vid1" class="header--video-masked active" muted playsinline>
                    <source data-src="https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4" type="video/mp4" />
                </video>
                ''', None)[0],
            {
                'formats': [{
                    'url': 'https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4',
                    'ext': 'mp4',
                }]
            })
        # from https://www.klarna.com/uk/
        # with data-video-src
        expect_dict(
            self,
            self.ie._parse_html5_media_entries(
                'https://www.directvnow.com',
                r'''
                <video loop autoplay muted class="responsive-video block-kl__video video-on-medium">
                    <source src="" data-video-desktop data-video-src="https://www.klarna.com/uk/wp-content/uploads/sites/11/2019/01/KL062_Smooth3_0_DogWalking_5s_920x080_.mp4" type="video/mp4" />
                </video>
                ''', None)[0],
            {
                'formats': [{
                    'url': 'https://www.klarna.com/uk/wp-content/uploads/sites/11/2019/01/KL062_Smooth3_0_DogWalking_5s_920x080_.mp4',
                    'ext': 'mp4',
                }],
            })
    def test_extract_jwplayer_data_realworld(self):
        # from http://www.suffolk.edu/sjc/
        expect_dict(
@@ -752,8 +574,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                # Also tests duplicate representation ids, see
                # https://github.com/ytdl-org/youtube-dl/issues/15111
                'float_duration',
-                'http://unknown/manifest.mpd',  # mpd_url
+                'http://unknown/manifest.mpd',
                None,  # mpd_base_url
                [{
                    'manifest_url': 'http://unknown/manifest.mpd',
                    'ext': 'm4a',
@@ -833,8 +654,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
            ), (
                # https://github.com/ytdl-org/youtube-dl/pull/14844
                'urls_only',
-                'http://unknown/manifest.mpd',  # mpd_url
+                'http://unknown/manifest.mpd',
                None,  # mpd_base_url
                [{
                    'manifest_url': 'http://unknown/manifest.mpd',
                    'ext': 'mp4',
@@ -913,61 +733,15 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                    'width': 1920,
                    'height': 1080,
                }]
            ), (
                # https://github.com/ytdl-org/youtube-dl/issues/20346
                # Media considered unfragmented even though it contains
                # Initialization tag
                'unfragmented',
                'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',  # mpd_url
                'https://v.redd.it/hw1x7rcg7zl21',  # mpd_base_url
                [{
                    'url': 'https://v.redd.it/hw1x7rcg7zl21/audio',
                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
                    'ext': 'm4a',
                    'format_id': 'AUDIO-1',
                    'format_note': 'DASH audio',
                    'container': 'm4a_dash',
                    'acodec': 'mp4a.40.2',
                    'vcodec': 'none',
                    'tbr': 129.87,
                    'asr': 48000,
                }, {
                    'url': 'https://v.redd.it/hw1x7rcg7zl21/DASH_240',
                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
                    'ext': 'mp4',
                    'format_id': 'VIDEO-2',
                    'format_note': 'DASH video',
                    'container': 'mp4_dash',
                    'acodec': 'none',
                    'vcodec': 'avc1.4d401e',
                    'tbr': 608.0,
                    'width': 240,
                    'height': 240,
                    'fps': 30,
                }, {
                    'url': 'https://v.redd.it/hw1x7rcg7zl21/DASH_360',
                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
                    'ext': 'mp4',
                    'format_id': 'VIDEO-1',
                    'format_note': 'DASH video',
                    'container': 'mp4_dash',
                    'acodec': 'none',
                    'vcodec': 'avc1.4d401e',
                    'tbr': 804.261,
                    'width': 360,
                    'height': 360,
                    'fps': 30,
                }]
            )
        ]
-        for mpd_file, mpd_url, mpd_base_url, expected_formats in _TEST_CASES:
+        for mpd_file, mpd_url, expected_formats in _TEST_CASES:
            with io.open('./test/testdata/mpd/%s.mpd' % mpd_file,
                         mode='r', encoding='utf-8') as f:
                formats = self.ie._parse_mpd_formats(
                    compat_etree_fromstring(f.read().encode('utf-8')),
-                    mpd_base_url=mpd_base_url, mpd_url=mpd_url)
+                    mpd_url=mpd_url)
                self.ie._sort_formats(formats)
                expect_value(self, formats, expected_formats, None)
@@ -55,7 +55,6 @@ from youtube_dl.utils import (
    parse_count,
    parse_iso8601,
    parse_resolution,
    parse_bitrate,
    pkcs1pad,
    read_batch_urls,
    sanitize_filename,
@@ -1031,13 +1030,6 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(parse_resolution('4k'), {'height': 2160})
        self.assertEqual(parse_resolution('8K'), {'height': 4320})
    def test_parse_bitrate(self):
        self.assertEqual(parse_bitrate(None), None)
        self.assertEqual(parse_bitrate(''), None)
        self.assertEqual(parse_bitrate('300kbps'), 300)
        self.assertEqual(parse_bitrate('1500kbps'), 1500)
        self.assertEqual(parse_bitrate('300 kbps'), 300)
    def test_version_tuple(self):
        self.assertEqual(version_tuple('1'), (1,))
        self.assertEqual(version_tuple('10.23.344'), (10, 23, 344))
@@ -1,28 +0,0 @@
 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
 <MPD mediaPresentationDuration="PT54.915S" minBufferTime="PT1.500S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" type="static" xmlns="urn:mpeg:dash:schema:mpd:2011">
    <Period duration="PT54.915S">
        <AdaptationSet segmentAlignment="true" subsegmentAlignment="true" subsegmentStartsWithSAP="1">
            <Representation bandwidth="804261" codecs="avc1.4d401e" frameRate="30" height="360" id="VIDEO-1" mimeType="video/mp4" startWithSAP="1" width="360">
                <BaseURL>DASH_360</BaseURL>
                <SegmentBase indexRange="915-1114" indexRangeExact="true">
                    <Initialization range="0-914"/>
                </SegmentBase>
            </Representation>
            <Representation bandwidth="608000" codecs="avc1.4d401e" frameRate="30" height="240" id="VIDEO-2" mimeType="video/mp4" startWithSAP="1" width="240">
                <BaseURL>DASH_240</BaseURL>
                <SegmentBase indexRange="913-1112" indexRangeExact="true">
                    <Initialization range="0-912"/>
                </SegmentBase>
            </Representation>
        </AdaptationSet>
        <AdaptationSet>
            <Representation audioSamplingRate="48000" bandwidth="129870" codecs="mp4a.40.2" id="AUDIO-1" mimeType="audio/mp4" startWithSAP="1">
                <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
                <BaseURL>audio</BaseURL>
                <SegmentBase indexRange="832-1007" indexRangeExact="true">
                    <Initialization range="0-831"/>
                </SegmentBase>
            </Representation>
        </AdaptationSet>
    </Period>
 </MPD>
@@ -0,0 +1,30 @@
 from __future__ import unicode_literals
 from .nuevo import NuevoBaseIE
 class AnitubeIE(NuevoBaseIE):
    IE_NAME = 'anitube.se'
    _VALID_URL = r'https?://(?:www\.)?anitube\.se/video/(?P<id>\d+)'
    _TEST = {
        'url': 'http://www.anitube.se/video/36621',
        'md5': '59d0eeae28ea0bc8c05e7af429998d43',
        'info_dict': {
            'id': '36621',
            'ext': 'mp4',
            'title': 'Recorder to Randoseru 01',
            'duration': 180.19,
        },
        'skip': 'Blocked in the US',
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        key = self._search_regex(
            r'src=["\']https?://[^/]+/embed/([A-Za-z0-9_-]+)', webpage, 'key')
        return self._extract_nuevo(
            'http://www.anitube.se/nuevo/econfig.php?key=%s' % key, video_id)
@@ -0,0 +1,61 @@
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    parse_duration,
    int_or_none,
 )
 class AnySexIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?anysex\.com/(?P<id>\d+)'
    _TEST = {
        'url': 'http://anysex.com/156592/',
        'md5': '023e9fbb7f7987f5529a394c34ad3d3d',
        'info_dict': {
            'id': '156592',
            'ext': 'mp4',
            'title': 'Busty and sexy blondie in her bikini strips for you',
            'description': 'md5:de9e418178e2931c10b62966474e1383',
            'categories': ['Erotic'],
            'duration': 270,
            'age_limit': 18,
        }
    }
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        video_id = mobj.group('id')
        webpage = self._download_webpage(url, video_id)
        video_url = self._html_search_regex(r"video_url\s*:\s*'([^']+)'", webpage, 'video URL')
        title = self._html_search_regex(r'<title>(.*?)</title>', webpage, 'title')
        description = self._html_search_regex(
            r'<div class="description"[^>]*>([^<]+)</div>', webpage, 'description', fatal=False)
        thumbnail = self._html_search_regex(
            r'preview_url\s*:\s*\'(.*?)\'', webpage, 'thumbnail', fatal=False)
        categories = re.findall(
            r'<a href="http://anysex\.com/categories/[^"]+" title="[^"]*">([^<]+)</a>', webpage)
        duration = parse_duration(self._search_regex(
            r'<b>Duration:</b> (?:<q itemprop="duration">)?(\d+:\d+)', webpage, 'duration', fatal=False))
        view_count = int_or_none(self._html_search_regex(
            r'<b>Views:</b> (\d+)', webpage, 'view count', fatal=False))
        return {
            'id': video_id,
            'url': video_url,
            'ext': 'mp4',
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'categories': categories,
            'duration': duration,
            'view_count': view_count,
            'age_limit': 18,
        }
@@ -9,8 +9,8 @@ from ..utils import (
 class BeegIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?beeg\.(?:com|porn(?:/video)?)/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www\.)?beeg\.com/(?P<id>\d+)'
-    _TESTS = [{
+    _TEST = {
        'url': 'http://beeg.com/5416503',
        'md5': 'a1a1b1a8bc70a89e49ccfd113aed0820',
        'info_dict': {
@@ -24,13 +24,7 @@ class BeegIE(InfoExtractor):
            'tags': list,
            'age_limit': 18,
        }
-    }, {
+    }
        'url': 'https://beeg.porn/video/5416503',
        'only_matching': True,
    }, {
        'url': 'https://beeg.porn/5416503',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
@@ -2,96 +2,39 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from .vk import VKIE
 from ..utils import (
    HEADRequest,
    int_or_none,
 )
 class BIQLEIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?biqle\.(?:com|org|ru)/watch/(?P<id>-?\d+_\d+)'
    _TESTS = [{
-        # Youtube embed
+        'url': 'http://www.biqle.ru/watch/847655_160197695',
-        'url': 'https://biqle.ru/watch/-115995369_456239081',
+        'md5': 'ad5f746a874ccded7b8f211aeea96637',
        'md5': '97af5a06ee4c29bbf9c001bdb1cf5c06',
        'info_dict': {
-            'id': '8v4f-avW-VI',
+            'id': '160197695',
            'ext': 'mp4',
-            'title': "PASSE-PARTOUT - L'ete c'est fait pour jouer",
+            'title': 'Foo Fighters - The Pretender (Live at Wembley Stadium)',
-            'description': 'Passe-Partout',
+            'uploader': 'Andrey Rogozin',
-            'uploader_id': 'mrsimpsonstef3',
+            'upload_date': '20110605',
-            'uploader': 'Phanolito',
+        }
            'upload_date': '20120822',
        },
    }, {
-        'url': 'http://biqle.org/watch/-44781847_168547604',
+        'url': 'https://biqle.org/watch/-44781847_168547604',
        'md5': '7f24e72af1db0edf7c1aaba513174f97',
        'info_dict': {
            'id': '168547604',
            'ext': 'mp4',
            'title': 'Ребенок в шоке от автоматической мойки',
            'timestamp': 1396633454,
            'uploader': 'Dmitry Kotov',
            'upload_date': '20140404',
            'uploader_id': '47850140',
        },
        'skip': ' This video was marked as adult.  Embedding adult videos on external sites is prohibited.',
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        embed_url = self._proto_relative_url(self._search_regex(
-            r'<iframe.+?src="((?:https?:)?//daxab\.com/[^"]+)".*?></iframe>',
+            r'<iframe.+?src="((?:http:)?//daxab\.com/[^"]+)".*?></iframe>', webpage, 'embed url'))
            webpage, 'embed url'))
        if VKIE.suitable(embed_url):
            return self.url_result(embed_url, VKIE.ie_key(), video_id)
        self._request_webpage(
            HEADRequest(embed_url), video_id, headers={'Referer': url})
        video_id, sig, _, access_token = self._get_cookies(embed_url)['video_ext'].value.split('%3A')
        item = self._download_json(
            'https://api.vk.com/method/video.get', video_id,
            headers={'User-Agent': 'okhttp/3.4.1'}, query={
                'access_token': access_token,
                'sig': sig,
                'v': 5.44,
                'videos': video_id,
            })['response']['items'][0]
        title = item['title']
        formats = []
        for f_id, f_url in item.get('files', {}).items():
            if f_id == 'external':
                return self.url_result(f_url)
            ext, height = f_id.split('_')
            formats.append({
                'format_id': height + 'p',
                'url': f_url,
                'height': int_or_none(height),
                'ext': ext,
            })
        self._sort_formats(formats)
        thumbnails = []
        for k, v in item.items():
            if k.startswith('photo_') and v:
                width = k.replace('photo_', '')
                thumbnails.append({
                    'id': width,
                    'url': v,
                    'width': int_or_none(width),
                })
        return {
-            'id': video_id,
+            '_type': 'url_transparent',
-            'title': title,
+            'url': embed_url,
            'formats': formats,
            'comment_count': int_or_none(item.get('comments')),
            'description': item.get('description'),
            'duration': int_or_none(item.get('duration')),
            'thumbnails': thumbnails,
            'timestamp': int_or_none(item.get('date')),
            'uploader': item.get('owner_id'),
            'view_count': int_or_none(item.get('views')),
        }
@@ -360,7 +360,7 @@ class CBCWatchVideoIE(CBCWatchBaseIE):
 class CBCWatchIE(CBCWatchBaseIE):
    IE_NAME = 'cbc.ca:watch'
-    _VALID_URL = r'https?://(?:gem|watch)\.cbc\.ca/(?:[^/]+/)+(?P<id>[0-9a-f-]+)'
+    _VALID_URL = r'https?://watch\.cbc\.ca/(?:[^/]+/)+(?P<id>[0-9a-f-]+)'
    _TESTS = [{
        # geo-restricted to Canada, bypassable
        'url': 'http://watch.cbc.ca/doc-zone/season-6/customer-disservice/38e815a-009e3ab12e4',
@@ -386,9 +386,6 @@ class CBCWatchIE(CBCWatchBaseIE):
            'description': 'Arthur, the sweetest 8-year-old aardvark, and his pals solve all kinds of problems with humour, kindness and teamwork.',
        },
        'playlist_mincount': 30,
    }, {
        'url': 'https://gem.cbc.ca/media/this-hour-has-22-minutes/season-26/episode-20/38e815a-0108c6c6a42',
        'only_matching': True,
    }]
    def _real_extract(self, url):
@@ -65,8 +65,8 @@ class CiscoLiveBaseIE(InfoExtractor):
 class CiscoLiveSessionIE(CiscoLiveBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?ciscolive(?:\.cisco)?\.com/[^#]*#/session/(?P<id>[^/?&]+)'
+    _VALID_URL = r'https?://ciscolive\.cisco\.com/on-demand-library/\??[^#]*#/session/(?P<id>[^/?&]+)'
-    _TESTS = [{
+    _TEST = {
        'url': 'https://ciscolive.cisco.com/on-demand-library/?#/session/1423353499155001FoSs',
        'md5': 'c98acf395ed9c9f766941c70f5352e22',
        'info_dict': {
@@ -79,13 +79,7 @@ class CiscoLiveSessionIE(CiscoLiveBaseIE):
            'uploader_id': '5647924234001',
            'location': '16B Mezz.',
        },
-    }, {
+    }
        'url': 'https://www.ciscolive.com/global/on-demand-library.html?search.event=ciscoliveemea2019#/session/15361595531500013WOU',
        'only_matching': True,
    }, {
        'url': 'https://www.ciscolive.com/global/on-demand-library.html?#/session/1490051371645001kNaS',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        rf_id = self._match_id(url)
@@ -94,7 +88,7 @@ class CiscoLiveSessionIE(CiscoLiveBaseIE):
 class CiscoLiveSearchIE(CiscoLiveBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?ciscolive(?:\.cisco)?\.com/(?:global/)?on-demand-library(?:\.html|/)'
+    _VALID_URL = r'https?://ciscolive\.cisco\.com/on-demand-library/'
    _TESTS = [{
        'url': 'https://ciscolive.cisco.com/on-demand-library/?search.event=ciscoliveus2018&search.technicallevel=scpsSkillLevel_aintroductory&search.focus=scpsSessionFocus_designAndDeployment#/',
        'info_dict': {
@@ -104,9 +98,6 @@ class CiscoLiveSearchIE(CiscoLiveBaseIE):
    }, {
        'url': 'https://ciscolive.cisco.com/on-demand-library/?search.technology=scpsTechnology_applicationDevelopment&search.technology=scpsTechnology_ipv6&search.focus=scpsSessionFocus_troubleshootingTroubleshooting#/',
        'only_matching': True,
    }, {
        'url': 'https://www.ciscolive.com/global/on-demand-library.html?search.technicallevel=scpsSkillLevel_aintroductory&search.event=ciscoliveemea2019&search.technology=scpsTechnology_dataCenter&search.focus=scpsSessionFocus_bestPractices#/',
        'only_matching': True,
    }]
    @classmethod
@@ -44,7 +44,6 @@ from ..utils import (
    compiled_regex_type,
    determine_ext,
    determine_protocol,
    dict_get,
    error_to_compat_str,
    ExtractorError,
    extract_attributes,
@@ -57,16 +56,13 @@ from ..utils import (
    JSON_LD_RE,
    mimetype2ext,
    orderedSet,
    parse_bitrate,
    parse_codecs,
    parse_duration,
    parse_iso8601,
    parse_m3u8_attributes,
    parse_resolution,
    RegexNotFoundError,
    sanitized_Request,
    sanitize_filename,
    str_or_none,
    unescapeHTML,
    unified_strdate,
    unified_timestamp,
@@ -112,13 +108,10 @@ class InfoExtractor(object):
                                   for RTMP - RTMP URL,
                                   for HLS - URL of the M3U8 media playlist,
                                   for HDS - URL of the F4M manifest,
-                                   for DASH
+                                   for DASH - URL of the MPD manifest or
-                                     - HTTP URL to plain file media (in case of
+                                              base URL representing the media
-                                       unfragmented media)
+                                              if MPD manifest is parsed from
-                                     - URL of the MPD manifest or base URL
+                                              a string,
                                       representing the media if MPD manifest
                                       is parsed froma string (in case of
                                       fragmented media)
                                   for MSS - URL of the ISM manifest.
                    * manifest_url
                                 The URL of the manifest file in case of
@@ -2144,6 +2137,8 @@ class InfoExtractor(object):
                        bandwidth = int_or_none(representation_attrib.get('bandwidth'))
                        f = {
                            'format_id': '%s-%s' % (mpd_id, representation_id) if mpd_id else representation_id,
                            # NB: mpd_url may be empty when MPD manifest is parsed from a string
                            'url': mpd_url or base_url,
                            'manifest_url': mpd_url,
                            'ext': mimetype2ext(mime_type),
                            'width': int_or_none(representation_attrib.get('width')),
@@ -2282,14 +2277,10 @@ class InfoExtractor(object):
                                    fragment['duration'] = segment_duration
                                fragments.append(fragment)
                            representation_ms_info['fragments'] = fragments
-                        # If there is a fragments key available then we correctly recognized fragmented media.
+                        # NB: MPD manifest may contain direct URLs to unfragmented media.
-                        # Otherwise we will assume unfragmented media with direct access. Technically, such
+                        # No fragments key is present in this case.
                        # assumption is not necessarily correct since we may simply have no support for
                        # some forms of fragmented media renditions yet, but for now we'll use this fallback.
                        if 'fragments' in representation_ms_info:
                            f.update({
                                # NB: mpd_url may be empty when MPD manifest is parsed from a string
                                'url': mpd_url or base_url,
                                'fragment_base_url': base_url,
                                'fragments': [],
                                'protocol': 'http_dash_segments',
@@ -2300,10 +2291,6 @@ class InfoExtractor(object):
                                    f['url'] = initialization_url
                                f['fragments'].append({location_key(initialization_url): initialization_url})
                            f['fragments'].extend(representation_ms_info['fragments'])
                        else:
                            # Assuming direct URL to unfragmented media.
                            f['url'] = base_url
                        # According to [1, 5.3.5.2, Table 7, page 35] @id of Representation
                        # is not necessarily unique within a Period thus formats with
                        # the same `format_id` are quite possible. There are numerous examples
@@ -2485,43 +2472,18 @@ class InfoExtractor(object):
            media_info['thumbnail'] = absolute_url(media_attributes.get('poster'))
            if media_content:
                for source_tag in re.findall(r'<source[^>]+>', media_content):
-                    s_attr = extract_attributes(source_tag)
+                    source_attributes = extract_attributes(source_tag)
-                    # data-video-src and data-src are non standard but seen
+                    src = source_attributes.get('src')
                    # several times in the wild
                    src = dict_get(s_attr, ('src', 'data-video-src', 'data-src'))
                    if not src:
                        continue
-                    f = parse_content_type(s_attr.get('type'))
+                    f = parse_content_type(source_attributes.get('type'))
                    is_plain_url, formats = _media_formats(src, media_type, f)
                    if is_plain_url:
-                        # width, height, res, label and title attributes are
+                        # res attribute is not standard but seen several times
-                        # all not standard but seen several times in the wild
+                        # in the wild
                        labels = [
                            s_attr.get(lbl)
                            for lbl in ('label', 'title')
                            if str_or_none(s_attr.get(lbl))
                        ]
                        width = int_or_none(s_attr.get('width'))
                        height = (int_or_none(s_attr.get('height')) or
                                  int_or_none(s_attr.get('res')))
                        if not width or not height:
                            for lbl in labels:
                                resolution = parse_resolution(lbl)
                                if not resolution:
                                    continue
                                width = width or resolution.get('width')
                                height = height or resolution.get('height')
                        for lbl in labels:
                            tbr = parse_bitrate(lbl)
                            if tbr:
                                break
                        else:
                            tbr = None
                        f.update({
-                            'width': width,
+                            'height': int_or_none(source_attributes.get('res')),
-                            'height': height,
+                            'format_id': source_attributes.get('label'),
                            'tbr': tbr,
                            'format_id': s_attr.get('label') or s_attr.get('title'),
                        })
                        f.update(formats[0])
                        media_info['formats'].append(f)
@@ -13,9 +13,9 @@ class CorusIE(ThePlatformFeedIE):
                        (?:www\.)?
                        (?P<domain>
                            (?:globaltv|etcanada)\.com|
-                            (?:hgtv|foodnetwork|slice|history|showcase|bigbrothercanada)\.ca
+                            (?:hgtv|foodnetwork|slice|history|showcase)\.ca
                        )
-                        /(?:video/(?:[^/]+/)?|(?:[^/]+/)+(?:videos/[a-z0-9-]+-|video\.html\?.*?\bv=))
+                        /(?:video/|(?:[^/]+/)+(?:videos/[a-z0-9-]+-|video\.html\?.*?\bv=))
                        (?P<id>\d+)
                    '''
    _TESTS = [{
@@ -42,12 +42,6 @@ class CorusIE(ThePlatformFeedIE):
    }, {
        'url': 'http://www.showcase.ca/eyewitness/video/eyewitness++106/video.html?v=955070531919&p=1&s=da#video',
        'only_matching': True,
    }, {
        'url': 'http://www.bigbrothercanada.ca/video/1457812035894/',
        'only_matching': True
    }, {
        'url': 'https://www.bigbrothercanada.ca/video/big-brother-canada-704/1457812035894/',
        'only_matching': True
    }]
    _TP_FEEDS = {
@@ -79,10 +73,6 @@ class CorusIE(ThePlatformFeedIE):
            'feed_id': '9H6qyshBZU3E',
            'account_id': 2414426607,
        },
        'bigbrothercanada': {
            'feed_id': 'ChQqrem0lNUp',
            'account_id': 2269680845,
        },
    }
    def _real_extract(self, url):
@@ -1,11 +1,14 @@
 from __future__ import unicode_literals
 import json
 from .common import InfoExtractor
 from ..utils import (
    determine_ext,
    clean_html,
    int_or_none,
    float_or_none,
    sanitized_Request,
 )
@@ -33,7 +36,7 @@ def _decrypt_config(key, string):
 class EscapistIE(InfoExtractor):
-    _VALID_URL = r'https?://?(?:(?:www|v1)\.)?escapistmagazine\.com/videos/view/[^/]+/(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://?(?:www\.)?escapistmagazine\.com/videos/view/[^/?#]+/(?P<id>[0-9]+)-[^/?#]*(?:$|[?#])'
    _TESTS = [{
        'url': 'http://www.escapistmagazine.com/videos/view/the-escapist-presents/6618-Breaking-Down-Baldurs-Gate',
        'md5': 'ab3a706c681efca53f0a35f1415cf0d1',
@@ -58,12 +61,6 @@ class EscapistIE(InfoExtractor):
            'duration': 304,
            'uploader': 'The Escapist',
        }
    }, {
        'url': 'http://escapistmagazine.com/videos/view/the-escapist-presents/6618',
        'only_matching': True,
    }, {
        'url': 'https://v1.escapistmagazine.com/videos/view/the-escapist-presents/6618-Breaking-Down-Baldurs-Gate',
        'only_matching': True,
    }]
    def _real_extract(self, url):
@@ -77,20 +74,19 @@ class EscapistIE(InfoExtractor):
        video_id = ims_video['videoID']
        key = ims_video['hash']
-        config = self._download_webpage(
+        config_req = sanitized_Request(
-            'http://www.escapistmagazine.com/videos/vidconfig.php',
+            'http://www.escapistmagazine.com/videos/'
-            video_id, 'Downloading video config', headers={
+            'vidconfig.php?videoID=%s&hash=%s' % (video_id, key))
-                'Referer': url,
+        config_req.add_header('Referer', url)
-            }, query={
+        config = self._download_webpage(config_req, video_id, 'Downloading video config')
                'videoID': video_id,
                'hash': key,
            })
-        data = self._parse_json(_decrypt_config(key, config), video_id)
+        data = json.loads(_decrypt_config(key, config))
        video_data = data['videoData']
        title = clean_html(video_data['title'])
        duration = float_or_none(video_data.get('duration'), 1000)
        uploader = video_data.get('publisher')
        formats = [{
            'url': video['src'],
@@ -103,9 +99,8 @@ class EscapistIE(InfoExtractor):
            'id': video_id,
            'formats': formats,
            'title': title,
-            'thumbnail': self._og_search_thumbnail(webpage) or data.get('poster'),
+            'thumbnail': self._og_search_thumbnail(webpage),
            'description': self._og_search_description(webpage),
-            'duration': float_or_none(video_data.get('duration'), 1000),
+            'duration': duration,
-            'uploader': video_data.get('publisher'),
+            'uploader': uploader,
            'series': video_data.get('show'),
        }
@@ -38,7 +38,9 @@ from .alphaporno import AlphaPornoIE
 from .amcnetworks import AMCNetworksIE
 from .americastestkitchen import AmericasTestKitchenIE
 from .animeondemand import AnimeOnDemandIE
 from .anitube import AnitubeIE
 from .anvato import AnvatoIE
 from .anysex import AnySexIE
 from .aol import AolIE
 from .allocine import AllocineIE
 from .aliexpress import AliExpressLiveIE
@@ -899,6 +901,7 @@ from .puhutv import (
    PuhuTVSerieIE,
 )
 from .presstv import PressTVIE
 from .primesharetv import PrimeShareTVIE
 from .promptfile import PromptFileIE
 from .prosiebensat1 import ProSiebenSat1IE
 from .puls4 import Puls4IE
@@ -975,6 +978,7 @@ from .rtvnh import RTVNHIE
 from .rtvs import RTVSIE
 from .rudo import RudoIE
 from .ruhd import RUHDIE
 from .ruleporn import RulePornIE
 from .rutube import (
    RutubeIE,
    RutubeChannelIE,
@@ -1295,6 +1299,7 @@ from .viddler import ViddlerIE
 from .videa import VideaIE
 from .videodetective import VideoDetectiveIE
 from .videofyme import VideofyMeIE
 from .videomega import VideoMegaIE
 from .videomore import (
    VideomoreIE,
    VideomoreVideoIE,
@@ -1444,13 +1449,12 @@ from .yahoo import (
    YahooIE,
    YahooSearchIE,
 )
 from .yandexdisk import YandexDiskIE
 from .yandexmusic import (
    YandexMusicTrackIE,
    YandexMusicAlbumIE,
    YandexMusicPlaylistIE,
 )
-from .yandexvideo import YandexVideoIE
+from .yandexdisk import YandexDiskIE
 from .yapfiles import YapFilesIE
 from .yesjapan import YesJapanIE
 from .yinyuetai import YinYueTaiIE
@@ -6,12 +6,10 @@ import uuid
 from .adobepass import AdobePassIE
 from ..compat import (
    compat_HTTPError,
    compat_str,
    compat_urllib_parse_unquote,
 )
 from ..utils import (
    ExtractorError,
    int_or_none,
    parse_age_limit,
    parse_duration,
@@ -50,7 +48,6 @@ class FOXIE(AdobePassIE):
        'url': 'https://www.fox.com/watch/30056b295fb57f7452aeeb4920bc3024/',
        'only_matching': True,
    }]
    _GEO_BYPASS = False
    _HOME_PAGE_URL = 'https://www.fox.com/'
    _API_KEY = 'abdcbed02c124d393b39e818a4312055'
    _access_token = None
@@ -61,22 +58,9 @@ class FOXIE(AdobePassIE):
        }
        if self._access_token:
            headers['Authorization'] = 'Bearer ' + self._access_token
-        try:
+        return self._download_json(
-            return self._download_json(
+            'https://api2.fox.com/v2.0/' + path,
-                'https://api2.fox.com/v2.0/' + path,
+            video_id, data=data, headers=headers)
                video_id, data=data, headers=headers)
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.status == 403:
                entitlement_issues = self._parse_json(
                    e.cause.read().decode(), video_id)['entitlementIssues']
                for e in entitlement_issues:
                    if e.get('errorCode') == 1005:
                        raise ExtractorError(
                            'This video is only available via cable service provider '
                            'subscription. You may want to use --cookies.', expected=True)
                messages = ', '.join([e['message'] for e in entitlement_issues])
                raise ExtractorError(messages, expected=True)
            raise
    def _real_initialize(self):
        if not self._access_token:
@@ -97,15 +81,7 @@ class FOXIE(AdobePassIE):
        title = video['name']
        release_url = video['url']
-        try:
+        m3u8_url = self._download_json(release_url, video_id)['playURL']
            m3u8_url = self._download_json(release_url, video_id)['playURL']
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.status == 403:
                error = self._parse_json(e.cause.read().decode(), video_id)
                if error.get('exception') == 'GeoLocationBlocked':
                    self.raise_geo_restricted(countries=['US'])
                raise ExtractorError(error['description'], expected=True)
            raise
        formats = self._extract_m3u8_formats(
            m3u8_url, video_id, 'mp4',
            entry_protocol='m3u8_native', m3u8_id='hls')
@@ -1,11 +1,18 @@
 from __future__ import unicode_literals
 import json
 import time
 from .common import InfoExtractor
-from ..utils import int_or_none
+from ..compat import compat_urllib_parse_urlencode
 from ..utils import (
    ExtractorError,
    sanitized_Request,
 )
 class HypemIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?hypem\.com/track/(?P<id>[0-9a-z]{5})'
+    _VALID_URL = r'https?://(?:www\.)?hypem\.com/track/(?P<id>[^/]+)/'
    _TEST = {
        'url': 'http://hypem.com/track/1v6ga/BODYWORK+-+TAME',
        'md5': 'b9cc91b5af8995e9f0c1cee04c575828',
@@ -14,36 +21,41 @@ class HypemIE(InfoExtractor):
            'ext': 'mp3',
            'title': 'Tame',
            'uploader': 'BODYWORK',
            'timestamp': 1371810457,
            'upload_date': '20130621',
        }
    }
    def _real_extract(self, url):
        track_id = self._match_id(url)
-        response = self._download_webpage(url, track_id)
+        data = {'ax': 1, 'ts': time.time()}
        request = sanitized_Request(url + '?' + compat_urllib_parse_urlencode(data))
        response, urlh = self._download_webpage_handle(
            request, track_id, 'Downloading webpage with the url')
-        track = self._parse_json(self._html_search_regex(
+        html_tracks = self._html_search_regex(
-            r'(?s)<script\s+type="application/json"\s+id="displayList-data">(.+?)</script>',
+            r'(?ms)<script type="application/json" id="displayList-data">(.+?)</script>',
-            response, 'tracks'), track_id)['tracks'][0]
+            response, 'tracks')
        try:
            track_list = json.loads(html_tracks)
            track = track_list['tracks'][0]
        except ValueError:
            raise ExtractorError('Hypemachine contained invalid JSON.')
        key = track['key']
        track_id = track['id']
        title = track['song']
-        final_url = self._download_json(
+        request = sanitized_Request(
-            'http://hypem.com/serve/source/%s/%s' % (track_id, track['key']),
+            'http://hypem.com/serve/source/%s/%s' % (track_id, key),
-            track_id, 'Downloading metadata', headers={
+            '', {'Content-Type': 'application/json'})
-                'Content-Type': 'application/json'
+        song_data = self._download_json(request, track_id, 'Downloading metadata')
-            })['url']
+        final_url = song_data['url']
        artist = track.get('artist')
        return {
            'id': track_id,
            'url': final_url,
            'ext': 'mp3',
            'title': title,
-            'uploader': track.get('artist'),
+            'uploader': artist,
            'duration': int_or_none(track.get('time')),
            'timestamp': int_or_none(track.get('ts')),
            'track': title,
        }
@@ -1,12 +1,15 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import json
 import re
 from .common import InfoExtractor
 from ..utils import (
-    clean_html,
+    ExtractorError,
    int_or_none,
    sanitized_Request,
    urlencode_postdata,
 )
@@ -14,8 +17,8 @@ class MoeVideoIE(InfoExtractor):
    IE_DESC = 'LetitBit video services: moevideo.net, playreplay.net and videochart.net'
    _VALID_URL = r'''(?x)
        https?://(?P<host>(?:www\.)?
-        (?:(?:moevideo|playreplay|videochart)\.net|thesame\.tv))/
+        (?:(?:moevideo|playreplay|videochart)\.net))/
-        (?:video|framevideo|embed)/(?P<id>[0-9a-z]+\.[0-9A-Za-z]+)'''
+        (?:video|framevideo)/(?P<id>[0-9]+\.[0-9A-Za-z]+)'''
    _API_URL = 'http://api.letitbit.net/'
    _API_KEY = 'tVL0gjqo5'
    _TESTS = [
@@ -54,26 +57,58 @@ class MoeVideoIE(InfoExtractor):
    ]
    def _real_extract(self, url):
-        host, video_id = re.match(self._VALID_URL, url).groups()
+        mobj = re.match(self._VALID_URL, url)
        video_id = mobj.group('id')
        webpage = self._download_webpage(
-            'http://%s/video/%s' % (host, video_id),
+            'http://%s/video/%s' % (mobj.group('host'), video_id),
            video_id, 'Downloading webpage')
        title = self._og_search_title(webpage)
        thumbnail = self._og_search_thumbnail(webpage)
        description = self._og_search_description(webpage)
-        embed_webpage = self._download_webpage(
+        r = [
-            'http://%s/embed/%s' % (host, video_id),
+            self._API_KEY,
-            video_id, 'Downloading embed webpage')
+            [
-        video = self._parse_json(self._search_regex(
+                'preview/flv_link',
-            r'mvplayer\("#player"\s*,\s*({.+})',
+                {
-            embed_webpage, 'mvplayer'), video_id)['video']
+                    'uid': video_id,
                },
            ],
        ]
        r_json = json.dumps(r)
        post = urlencode_postdata({'r': r_json})
        req = sanitized_Request(self._API_URL, post)
        req.add_header('Content-type', 'application/x-www-form-urlencoded')
        response = self._download_json(req, video_id)
        if response['status'] != 'OK':
            raise ExtractorError(
                '%s returned error: %s' % (self.IE_NAME, response['data']),
                expected=True
            )
        item = response['data'][0]
        video_url = item['link']
        duration = int_or_none(item['length'])
        width = int_or_none(item['width'])
        height = int_or_none(item['height'])
        filesize = int_or_none(item['convert_size'])
        formats = [{
            'format_id': 'sd',
            'http_headers': {'Range': 'bytes=0-'},  # Required to download
            'url': video_url,
            'width': width,
            'height': height,
            'filesize': filesize,
        }]
        return {
            'id': video_id,
            'title': title,
-            'thumbnail': video.get('poster') or self._og_search_thumbnail(webpage),
+            'thumbnail': thumbnail,
-            'description': clean_html(self._og_search_description(webpage)),
+            'description': description,
-            'duration': int_or_none(self._og_search_property('video:duration', webpage)),
+            'duration': duration,
-            'url': video['ourUrl'],
+            'formats': formats,
        }
@@ -1,6 +1,7 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..compat import compat_urllib_parse_urlencode
 from ..utils import (
    int_or_none,
    qualities,
@@ -8,16 +9,16 @@ from ..utils import (
 class NprIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?npr\.org/(?:sections/[^/]+/)?\d{4}/\d{2}/\d{2}/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www\.)?npr\.org/player/v2/mediaPlayer\.html\?.*\bid=(?P<id>\d+)'
    _TESTS = [{
-        'url': 'https://www.npr.org/sections/allsongs/2015/10/21/449974205/new-music-from-beach-house-chairlift-cmj-discoveries-and-more',
+        'url': 'http://www.npr.org/player/v2/mediaPlayer.html?id=449974205',
        'info_dict': {
            'id': '449974205',
            'title': 'New Music From Beach House, Chairlift, CMJ Discoveries And More'
        },
        'playlist_count': 7,
    }, {
-        'url': 'https://www.npr.org/sections/deceptivecadence/2015/10/09/446928052/music-from-the-shadows-ancient-armenian-hymns-and-piano-jazz',
+        'url': 'http://www.npr.org/player/v2/mediaPlayer.html?action=1&t=1&islist=false&id=446928052&m=446929930&live=1',
        'info_dict': {
            'id': '446928052',
            'title': "Songs We Love: Tigran Hamasyan, 'Your Mercy is Boundless'"
@@ -31,46 +32,30 @@ class NprIE(InfoExtractor):
                'duration': 402,
            },
        }],
    }, {
        # mutlimedia, not media title
        'url': 'https://www.npr.org/2017/06/19/533198237/tigers-jaw-tiny-desk-concert',
        'info_dict': {
            'id': '533198237',
            'title': 'Tigers Jaw: Tiny Desk Concert',
        },
        'playlist': [{
            'md5': '12fa60cb2d3ed932f53609d4aeceabf1',
            'info_dict': {
                'id': '533201718',
                'ext': 'mp4',
                'title': 'Tigers Jaw: Tiny Desk Concert',
                'duration': 402,
            },
        }],
        'expected_warnings': ['Failed to download m3u8 information'],
    }]
    def _real_extract(self, url):
        playlist_id = self._match_id(url)
-        story = self._download_json(
+        config = self._download_json(
-            'http://api.npr.org/query', playlist_id, query={
+            'http://api.npr.org/query?%s' % compat_urllib_parse_urlencode({
                'id': playlist_id,
-                'fields': 'audio,multimedia,title',
+                'fields': 'titles,audio,show',
                'format': 'json',
                'apiKey': 'MDAzMzQ2MjAyMDEyMzk4MTU1MDg3ZmM3MQ010',
-            })['list']['story'][0]
+            }), playlist_id)
        playlist_title = story.get('title', {}).get('$text')
-        KNOWN_FORMATS = ('threegp', 'm3u8', 'smil', 'mp4', 'mp3')
+        story = config['list']['story'][0]
        KNOWN_FORMATS = ('threegp', 'mp4', 'mp3')
        quality = qualities(KNOWN_FORMATS)
        entries = []
-        for media in story.get('audio', []) + story.get('multimedia', []):
+        for audio in story.get('audio', []):
-            media_id = media['id']
+            title = audio.get('title', {}).get('$text')
-
+            duration = int_or_none(audio.get('duration', {}).get('$text'))
            formats = []
-            for format_id, formats_entry in media.get('format', {}).items():
+            for format_id, formats_entry in audio.get('format', {}).items():
                if not formats_entry:
                    continue
                if isinstance(formats_entry, list):
@@ -79,30 +64,19 @@ class NprIE(InfoExtractor):
                if not format_url:
                    continue
                if format_id in KNOWN_FORMATS:
-                    if format_id == 'm3u8':
+                    formats.append({
-                        formats.extend(self._extract_m3u8_formats(
+                        'url': format_url,
-                            format_url, media_id, 'mp4', 'm3u8_native',
+                        'format_id': format_id,
-                            m3u8_id='hls', fatal=False))
+                        'ext': formats_entry.get('type'),
-                    elif format_id == 'smil':
+                        'quality': quality(format_id),
-                        smil_formats = self._extract_smil_formats(
+                    })
                            format_url, media_id, transform_source=lambda s: s.replace(
                                'rtmp://flash.npr.org/ondemand/', 'https://ondemand.npr.org/'))
                        self._check_formats(smil_formats, media_id)
                        formats.extend(smil_formats)
                    else:
                        formats.append({
                            'url': format_url,
                            'format_id': format_id,
                            'quality': quality(format_id),
                        })
            self._sort_formats(formats)
            entries.append({
-                'id': media_id,
+                'id': audio['id'],
-                'title': media.get('title', {}).get('$text') or playlist_title,
+                'title': title,
-                'thumbnail': media.get('altImageUrl', {}).get('$text'),
+                'duration': duration,
                'duration': int_or_none(media.get('duration', {}).get('$text')),
                'formats': formats,
            })
        playlist_title = story.get('title', {}).get('$text')
        return self.playlist_result(entries, playlist_id, playlist_title)
@@ -243,16 +243,18 @@ class PhantomJSwrapper(object):
 class OpenloadIE(InfoExtractor):
    _DOMAINS = r'(?:openload\.(?:co|io|link|pw)|oload\.(?:tv|stream|site|xyz|win|download|cloud|cc|icu|fun|club|info|pw|live|space))'
    _VALID_URL = r'''(?x)
                    https?://
                        (?P<host>
                            (?:www\.)?
-                            %s
+                            (?:
                                openload\.(?:co|io|link|pw)|
                                oload\.(?:tv|stream|site|xyz|win|download|cloud|cc|icu|fun|club|info|pw|live|space)
                            )
                        )/
                        (?:f|embed)/
                        (?P<id>[a-zA-Z0-9-_]+)
-                    ''' % _DOMAINS
+                    '''
    _TESTS = [{
        'url': 'https://openload.co/f/kUEfGclsU9o',
@@ -357,8 +359,8 @@ class OpenloadIE(InfoExtractor):
    @staticmethod
    def _extract_urls(webpage):
        return re.findall(
-            r'<iframe[^>]+src=["\']((?:https?://)?%s/embed/[a-zA-Z0-9-_]+)'
+            r'<iframe[^>]+src=["\']((?:https?://)?(?:openload\.(?:co|io)|oload\.tv)/embed/[a-zA-Z0-9-_]+)',
-            % OpenloadIE._DOMAINS, webpage)
+            webpage)
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
@@ -176,8 +176,7 @@ class ORFRadioIE(InfoExtractor):
                'description': subtitle,
                'duration': (info['end'] - info['start']) / 1000,
                'timestamp': info['start'] / 1000,
-                'ext': 'mp3',
+                'ext': 'mp3'
                'series': data.get('programTitle')
            }
        entries = [extract_entry_dict(t, data['title'], data['subtitle']) for t in data['streams']]
@@ -36,7 +36,7 @@ class PandaTVIE(InfoExtractor):
            'https://www.panda.tv/api_room_v2?roomid=%s' % video_id, video_id)
        error_code = config.get('errno', 0)
-        if error_code != 0:
+        if error_code is not 0:
            raise ExtractorError(
                '%s returned error %s: %s'
                % (self.IE_NAME, error_code, config['errmsg']),
@@ -0,0 +1,62 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    sanitized_Request,
    urlencode_postdata,
 )
 class PrimeShareTVIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?primeshare\.tv/download/(?P<id>[\da-zA-Z]+)'
    _TEST = {
        'url': 'http://primeshare.tv/download/238790B611',
        'md5': 'b92d9bf5461137c36228009f31533fbc',
        'info_dict': {
            'id': '238790B611',
            'ext': 'mp4',
            'title': 'Public Domain - 1960s Commercial - Crest Toothpaste-YKsuFona',
        },
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        if '>File not exist<' in webpage:
            raise ExtractorError('Video %s does not exist' % video_id, expected=True)
        fields = self._hidden_inputs(webpage)
        headers = {
            'Referer': url,
            'Content-Type': 'application/x-www-form-urlencoded',
        }
        wait_time = int(self._search_regex(
            r'var\s+cWaitTime\s*=\s*(\d+)',
            webpage, 'wait time', default=7)) + 1
        self._sleep(wait_time, video_id)
        req = sanitized_Request(
            url, urlencode_postdata(fields), headers)
        video_page = self._download_webpage(
            req, video_id, 'Downloading video page')
        video_url = self._search_regex(
            r"url\s*:\s*'([^']+\.primeshare\.tv(?::443)?/file/[^']+)'",
            video_page, 'video url')
        title = self._html_search_regex(
            r'<h1>Watch\s*(?:&nbsp;)?\s*\((.+?)(?:\s*\[\.\.\.\])?\)\s*(?:&nbsp;)?\s*<strong>',
            video_page, 'title')
        return {
            'id': video_id,
            'url': video_url,
            'title': title,
            'ext': 'mp4',
        }
@@ -147,7 +147,7 @@ class RadioCanadaIE(InfoExtractor):
 class RadioCanadaAudioVideoIE(InfoExtractor):
-    IE_NAME = 'radiocanada:audiovideo'
+    'radiocanada:audiovideo'
    _VALID_URL = r'https?://ici\.radio-canada\.ca/([^/]+/)*media-(?P<id>[0-9]+)'
    _TESTS = [{
        'url': 'http://ici.radio-canada.ca/audio-video/media-7527184/barack-obama-au-vietnam',
@@ -0,0 +1,44 @@
 from __future__ import unicode_literals
 from .nuevo import NuevoBaseIE
 class RulePornIE(NuevoBaseIE):
    _VALID_URL = r'https?://(?:www\.)?ruleporn\.com/(?:[^/?#&]+/)*(?P<id>[^/?#&]+)'
    _TEST = {
        'url': 'http://ruleporn.com/brunette-nympho-chick-takes-her-boyfriend-in-every-angle/',
        'md5': '86861ebc624a1097c7c10eaf06d7d505',
        'info_dict': {
            'id': '48212',
            'display_id': 'brunette-nympho-chick-takes-her-boyfriend-in-every-angle',
            'ext': 'mp4',
            'title': 'Brunette Nympho Chick Takes Her Boyfriend In Every Angle',
            'description': 'md5:6d28be231b981fff1981deaaa03a04d5',
            'age_limit': 18,
            'duration': 635.1,
        }
    }
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        video_id = self._search_regex(
            r'lovehomeporn\.com/embed/(\d+)', webpage, 'video id')
        title = self._search_regex(
            r'<h2[^>]+title=(["\'])(?P<url>.+?)\1',
            webpage, 'title', group='url')
        description = self._html_search_meta('description', webpage)
        info = self._extract_nuevo(
            'http://lovehomeporn.com/media/nuevo/econfig.php?key=%s&rp=true' % video_id,
            video_id)
        info.update({
            'display_id': display_id,
            'title': title,
            'description': description,
            'age_limit': 18
        })
        return info
@@ -1,10 +1,13 @@
 from __future__ import unicode_literals
 import re
 import json
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
-    parse_duration,
+    ExtractorError,
-    qualities,
+    sanitized_Request,
 )
@@ -13,9 +16,9 @@ class VeohIE(InfoExtractor):
    _TESTS = [{
        'url': 'http://www.veoh.com/watch/v56314296nk7Zdmz3',
-        'md5': '9e7ecc0fd8bbee7a69fe38953aeebd30',
+        'md5': '620e68e6a3cff80086df3348426c9ca3',
        'info_dict': {
-            'id': 'v56314296nk7Zdmz3',
+            'id': '56314296',
            'ext': 'mp4',
            'title': 'Straight Backs Are Stronger',
            'uploader': 'LUMOback',
@@ -53,6 +56,29 @@ class VeohIE(InfoExtractor):
        'only_matching': True,
    }]
    def _extract_formats(self, source):
        formats = []
        link = source.get('aowPermalink')
        if link:
            formats.append({
                'url': link,
                'ext': 'mp4',
                'format_id': 'aow',
            })
        link = source.get('fullPreviewHashLowPath')
        if link:
            formats.append({
                'url': link,
                'format_id': 'low',
            })
        link = source.get('fullPreviewHashHighPath')
        if link:
            formats.append({
                'url': link,
                'format_id': 'high',
            })
        return formats
    def _extract_video(self, source):
        return {
            'id': source.get('videoId'),
@@ -67,37 +93,38 @@ class VeohIE(InfoExtractor):
        }
    def _real_extract(self, url):
-        video_id = self._match_id(url)
+        mobj = re.match(self._VALID_URL, url)
-        video = self._download_json(
+        video_id = mobj.group('id')
            'https://www.veoh.com/watch/getVideo/' + video_id,
            video_id)['video']
        title = video['title']
-        thumbnail_url = None
+        if video_id.startswith('v'):
-        q = qualities(['HQ', 'Regular'])
+            rsp = self._download_xml(
-        formats = []
+                r'http://www.veoh.com/api/findByPermalink?permalink=%s' % video_id, video_id, 'Downloading video XML')
-        for f_id, f_url in video.get('src', {}).items():
+            stat = rsp.get('stat')
-            if not f_url:
+            if stat == 'ok':
-                continue
+                return self._extract_video(rsp.find('./videoList/video'))
-            if f_id == 'poster':
+            elif stat == 'fail':
-                thumbnail_url = f_url
+                raise ExtractorError(
-            else:
+                    '%s said: %s' % (self.IE_NAME, rsp.find('./errorList/error').get('errorMessage')), expected=True)
                formats.append({
                    'format_id': f_id,
                    'quality': q(f_id),
                    'url': f_url,
                })
        self._sort_formats(formats)
-        return {
+        webpage = self._download_webpage(url, video_id)
-            'id': video_id,
+        age_limit = 0
-            'title': title,
+        if 'class="adultwarning-container"' in webpage:
-            'description': video.get('description'),
+            self.report_age_confirmation()
-            'thumbnail': thumbnail_url,
+            age_limit = 18
-            'uploader': video.get('author', {}).get('nickname'),
+            request = sanitized_Request(url)
-            'duration': int_or_none(video.get('lengthBySec')) or parse_duration(video.get('length')),
+            request.add_header('Cookie', 'confirmedAdult=true')
-            'view_count': int_or_none(video.get('views')),
+            webpage = self._download_webpage(request, video_id)
-            'formats': formats,
+
-            'average_rating': int_or_none(video.get('rating')),
+        m_youtube = re.search(r'http://www\.youtube\.com/v/(.*?)(\&|"|\?)', webpage)
-            'comment_count': int_or_none(video.get('numOfComments')),
+        if m_youtube is not None:
-        }
+            youtube_id = m_youtube.group(1)
            self.to_screen('%s: detected Youtube video.' % video_id)
            return self.url_result(youtube_id, 'Youtube')
        info = json.loads(
            self._search_regex(r'videoDetailsJSON = \'({.*?})\';', webpage, 'info').replace('\\\'', '\''))
        video = self._extract_video(info)
        video['age_limit'] = age_limit
        return video
@@ -1,16 +1,19 @@
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import (
    compat_urllib_parse_urlencode,
    compat_urlparse,
 )
 from ..utils import (
    float_or_none,
    int_or_none,
    sanitized_Request,
 )
 class ViddlerIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?viddler\.com/(?:v|embed|player)/(?P<id>[a-z0-9]+)(?:.+?\bsecret=(\d+))?'
+    _VALID_URL = r'https?://(?:www\.)?viddler\.com/(?:v|embed|player)/(?P<id>[a-z0-9]+)'
    _TESTS = [{
        'url': 'http://www.viddler.com/v/43903784',
        'md5': '9eee21161d2c7f5b39690c3e325fab2f',
@@ -75,18 +78,23 @@ class ViddlerIE(InfoExtractor):
    }]
    def _real_extract(self, url):
-        video_id, secret = re.match(self._VALID_URL, url).groups()
+        video_id = self._match_id(url)
        query = {
            'video_id': video_id,
            'key': 'v0vhrt7bg2xq1vyxhkct',
        }
        qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
        secret = qs.get('secret', [None])[0]
        if secret:
            query['secret'] = secret
-        data = self._download_json(
+        headers = {'Referer': 'http://static.cdn-ec.viddler.com/js/arpeggio/v2/embed.html'}
-            'http://api.viddler.com/api/v2/viddler.videos.getPlaybackDetails.json',
+        request = sanitized_Request(
-            video_id, headers={'Referer': url}, query=query)['video']
+            'http://api.viddler.com/api/v2/viddler.videos.getPlaybackDetails.json?%s'
            % compat_urllib_parse_urlencode(query), None, headers)
        data = self._download_json(request, video_id)['video']
        formats = []
        for filed in data['files']:
@@ -0,0 +1,60 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    decode_packed_codes,
    sanitized_Request,
 )
 class VideoMegaIE(InfoExtractor):
    _VALID_URL = r'(?:videomega:|https?://(?:www\.)?videomega\.tv/(?:(?:view|iframe|cdn)\.php)?\?ref=)(?P<id>[A-Za-z0-9]+)'
    _TESTS = [{
        'url': 'http://videomega.tv/cdn.php?ref=AOSQBJYKIDDIKYJBQSOA',
        'md5': 'cc1920a58add3f05c6a93285b84fb3aa',
        'info_dict': {
            'id': 'AOSQBJYKIDDIKYJBQSOA',
            'ext': 'mp4',
            'title': '1254207',
            'thumbnail': r're:^https?://.*\.jpg$',
        }
    }, {
        'url': 'http://videomega.tv/cdn.php?ref=AOSQBJYKIDDIKYJBQSOA&width=1070&height=600',
        'only_matching': True,
    }, {
        'url': 'http://videomega.tv/view.php?ref=090051111052065112106089103052052103089106112065052111051090',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        iframe_url = 'http://videomega.tv/cdn.php?ref=%s' % video_id
        req = sanitized_Request(iframe_url)
        req.add_header('Referer', url)
        req.add_header('Cookie', 'noadvtday=0')
        webpage = self._download_webpage(req, video_id)
        title = self._html_search_regex(
            r'<title>(.+?)</title>', webpage, 'title')
        title = re.sub(
            r'(?:^[Vv]ideo[Mm]ega\.tv\s-\s*|\s*-\svideomega\.tv$)', '', title)
        thumbnail = self._search_regex(
            r'<video[^>]+?poster="([^"]+)"', webpage, 'thumbnail', fatal=False)
        real_codes = decode_packed_codes(webpage)
        video_url = self._search_regex(
            r'"src"\s*,\s*"([^"]+)"', real_codes, 'video URL')
        return {
            'id': video_id,
            'title': title,
            'url': video_url,
            'thumbnail': thumbnail,
            'http_headers': {
                'Referer': iframe_url,
            },
        }
@@ -195,32 +195,6 @@ class VimeoBaseInfoExtractor(InfoExtractor):
            'subtitles': subtitles,
        }
    def _extract_original_format(self, url, video_id):
        download_data = self._download_json(
            url, video_id, fatal=False,
            query={'action': 'load_download_config'},
            headers={'X-Requested-With': 'XMLHttpRequest'})
        if download_data:
            source_file = download_data.get('source_file')
            if isinstance(source_file, dict):
                download_url = source_file.get('download_url')
                if download_url and not source_file.get('is_cold') and not source_file.get('is_defrosting'):
                    source_name = source_file.get('public_name', 'Original')
                    if self._is_valid_url(download_url, video_id, '%s video' % source_name):
                        ext = (try_get(
                            source_file, lambda x: x['extension'],
                            compat_str) or determine_ext(
                            download_url, None) or 'mp4').lower()
                        return {
                            'url': download_url,
                            'ext': ext,
                            'width': int_or_none(source_file.get('width')),
                            'height': int_or_none(source_file.get('height')),
                            'filesize': parse_filesize(source_file.get('size')),
                            'format_id': source_name,
                            'preference': 1,
                        }
 class VimeoIE(VimeoBaseInfoExtractor):
    """Information extractor for vimeo.com."""
@@ -685,11 +659,29 @@ class VimeoIE(VimeoBaseInfoExtractor):
            comment_count = None
        formats = []
-
+        download_request = sanitized_Request('https://vimeo.com/%s?action=load_download_config' % video_id, headers={
-        source_format = self._extract_original_format(
+            'X-Requested-With': 'XMLHttpRequest'})
-            'https://vimeo.com/' + video_id, video_id)
+        download_data = self._download_json(download_request, video_id, fatal=False)
-        if source_format:
+        if download_data:
-            formats.append(source_format)
+            source_file = download_data.get('source_file')
            if isinstance(source_file, dict):
                download_url = source_file.get('download_url')
                if download_url and not source_file.get('is_cold') and not source_file.get('is_defrosting'):
                    source_name = source_file.get('public_name', 'Original')
                    if self._is_valid_url(download_url, video_id, '%s video' % source_name):
                        ext = (try_get(
                            source_file, lambda x: x['extension'],
                            compat_str) or determine_ext(
                            download_url, None) or 'mp4').lower()
                        formats.append({
                            'url': download_url,
                            'ext': ext,
                            'width': int_or_none(source_file.get('width')),
                            'height': int_or_none(source_file.get('height')),
                            'filesize': parse_filesize(source_file.get('size')),
                            'format_id': source_name,
                            'preference': 1,
                        })
        info_dict_config = self._parse_config(config, video_id)
        formats.extend(info_dict_config['formats'])
@@ -948,7 +940,7 @@ class VimeoGroupsIE(VimeoAlbumIE):
 class VimeoReviewIE(VimeoBaseInfoExtractor):
    IE_NAME = 'vimeo:review'
    IE_DESC = 'Review pages on vimeo'
-    _VALID_URL = r'(?P<url>https://vimeo\.com/[^/]+/review/(?P<id>[^/]+)/[0-9a-f]{10})'
+    _VALID_URL = r'https://vimeo\.com/[^/]+/review/(?P<id>[^/]+)'
    _TESTS = [{
        'url': 'https://vimeo.com/user21297594/review/75524534/3c257a1b5d',
        'md5': 'c507a72f780cacc12b2248bb4006d253',
@@ -1000,8 +992,7 @@ class VimeoReviewIE(VimeoBaseInfoExtractor):
            data = self._parse_json(self._search_regex(
                r'window\s*=\s*_extend\(window,\s*({.+?})\);', webpage, 'data',
                default=NO_DEFAULT if video_password_verified else '{}'), video_id)
-            config = data.get('vimeo_esi', {}).get('config', {})
+            config_url = data.get('vimeo_esi', {}).get('config', {}).get('configUrl')
            config_url = config.get('configUrl') or try_get(config, lambda x: x['clipData']['configUrl'])
        if config_url is None:
            self._verify_video_password(webpage_url, video_id, webpage)
            config_url = self._get_config_url(
@@ -1009,13 +1000,10 @@ class VimeoReviewIE(VimeoBaseInfoExtractor):
        return config_url
    def _real_extract(self, url):
-        page_url, video_id = re.match(self._VALID_URL, url).groups()
+        video_id = self._match_id(url)
        config_url = self._get_config_url(url, video_id)
        config = self._download_json(config_url, video_id)
        info_dict = self._parse_config(config, video_id)
        source_format = self._extract_original_format(page_url, video_id)
        if source_format:
            info_dict['formats'].append(source_format)
        self._vimeo_sort_formats(info_dict['formats'])
        info_dict['id'] = video_id
        return info_dict
@@ -1,90 +0,0 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    url_or_none,
 )
 class YandexVideoIE(InfoExtractor):
    _VALID_URL = r'''(?x)
                    https?://
                        (?:
                            yandex\.ru(?:/portal/(?:video|efir))?/?\?.*?stream_id=|
                            frontend\.vh\.yandex\.ru/player/
                        )
                        (?P<id>[\da-f]+)
                    '''
    _TESTS = [{
        'url': 'https://yandex.ru/portal/video?stream_id=4dbb262b4fe5cf15a215de4f34eee34d',
        'md5': '33955d7ae052f15853dc41f35f17581c',
        'info_dict': {
            'id': '4dbb262b4fe5cf15a215de4f34eee34d',
            'ext': 'mp4',
            'title': 'В Нью-Йорке баржи и теплоход оторвались от причала и расплылись по Гудзону',
            'description': '',
            'thumbnail': r're:^https?://.*\.jpg$',
            'timestamp': 0,
            'duration': 30,
            'age_limit': 18,
        },
    }, {
        'url': 'https://yandex.ru/portal/efir?stream_id=4dbb36ec4e0526d58f9f2dc8f0ecf374&from=morda',
        'only_matching': True,
    }, {
        'url': 'https://yandex.ru/?stream_id=4dbb262b4fe5cf15a215de4f34eee34d',
        'only_matching': True,
    }, {
        'url': 'https://frontend.vh.yandex.ru/player/4dbb262b4fe5cf15a215de4f34eee34d?from=morda',
        'only_matching': True,
    }, {
        # vod-episode, series episode
        'url': 'https://yandex.ru/portal/video?stream_id=45b11db6e4b68797919c93751a938cee',
        'only_matching': True,
    }, {
        # episode, sports
        'url': 'https://yandex.ru/?stream_channel=1538487871&stream_id=4132a07f71fb0396be93d74b3477131d',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        content = self._download_json(
            'https://frontend.vh.yandex.ru/v22/player/%s.json' % video_id,
            video_id, query={
                'stream_options': 'hires',
                'disable_trackings': 1,
            })['content']
        m3u8_url = url_or_none(content.get('content_url')) or url_or_none(
            content['streams'][0]['url'])
        title = content.get('title') or content.get('computed_title')
        formats = self._extract_m3u8_formats(
            m3u8_url, video_id, 'mp4', entry_protocol='m3u8_native',
            m3u8_id='hls')
        self._sort_formats(formats)
        description = content.get('description')
        thumbnail = content.get('thumbnail')
        timestamp = (int_or_none(content.get('release_date')) or
                     int_or_none(content.get('release_date_ut')) or
                     int_or_none(content.get('start_time')))
        duration = int_or_none(content.get('duration'))
        series = content.get('program_title')
        age_limit = int_or_none(content.get('restriction_age'))
        return {
            'id': video_id,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'timestamp': timestamp,
            'duration': duration,
            'series': series,
            'age_limit': age_limit,
            'formats': formats,
        }
@@ -351,8 +351,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                            (?:www\.)?hooktube\.com/|
                            (?:www\.)?yourepeat\.com/|
                            tube\.majestyc\.net/|
-                            (?:(?:www|dev)\.)?invidio\.us/|
+                            (?:www\.)?invidio\.us/|
                            (?:www\.)?invidiou\.sh/|
                            (?:www\.)?invidious\.snopyta\.org/|
                            (?:www\.)?invidious\.kabi\.tk/|
                            (?:www\.)?vid\.wxzm\.sx/|
@@ -31,7 +31,7 @@ def rsa_verify(message, signature, key):
 def update_self(to_screen, verbose, opener):
    """Update the program file with the latest version from the repository"""
-    UPDATE_URL = 'https://yt-dl.org/update/'
+    UPDATE_URL = 'https://ytdl-org.github.io/youtube-dl/update/'
    VERSION_URL = UPDATE_URL + 'LATEST_VERSION'
    JSON_URL = UPDATE_URL + 'versions.json'
    UPDATES_RSA_KEY = (0x9d60ee4d8f805312fdb15a62f87b95bd66177b91df176765d13514a0f1754bcd2057295c5b6f1d35daa6742c3ffc9a82d3e118861c207995a8031e151d863c9927e304576bc80692bc8e094896fcf11b66f3e29e04e3a71e9a11558558acea1840aec37fc396fb6b65dc81a1c4144e03bd1c011de62e3f1357b327d08426fe93, 65537)
@@ -1798,14 +1798,6 @@ def parse_resolution(s):
    return {}
 def parse_bitrate(s):
    if not isinstance(s, compat_str):
        return
    mobj = re.search(r'\b(\d+)\s*kbps', s)
    if mobj:
        return int(mobj.group(1))
 def month_by_name(name, lang='en'):
    """ Return the number of a month by (locale-independently) English name """
@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
-__version__ = '2019.03.18'
+__version__ = '2019.03.09'
`@@ -1,3 +1,3 @@`
	`from __future__ import unicode_literals`	`from __future__ import unicode_literals`

	`__version__ = '2019.03.18'`	`__version__ = '2019.03.09'`