{"id":328,"date":"2012-10-02T09:11:00","date_gmt":"2012-10-02T09:11:00","guid":{"rendered":""},"modified":"2017-02-07T14:07:22","modified_gmt":"2017-02-07T14:07:22","slug":"using-python-and-wget-to-download-certain-file-types-from-a-web-page","status":"publish","type":"post","link":"https:\/\/perfectionatic.org\/?p=328","title":{"rendered":"Using Python and wget to download certain file types from a web-page"},"content":{"rendered":"
Recently I wanted to download all the movies in Mackay’s information theory <\/a>course. That was a great deal of files and I though it would too tedious to try to do it using the browser or even manual wget<\/code><\/a>. So digging around, I found this decent tutorial<\/a> on python HTML processing. I also dug around some information on parsing URLs<\/a>. I combined that with my knowledge of python’s subprocess<\/code><\/code><\/a> and put together this nice piece of code. <\/p>\n

 <\/p>\n

<\/h2>\n
\n
#!\/usr\/bin\/python<\/span>

import<\/span> sgmllib<\/span>

class<\/span> MyParser<\/span>(<\/span>sgmllib<\/span>.<\/span>SGMLParser<\/span>):<\/span>
    \"A simple parser class.\"<\/span>

    def<\/span> parse<\/span>(<\/span>self<\/span>,<\/span> s<\/span>):<\/span>
        \"Parse the given string 's'.\"<\/span>
        self<\/span>.<\/span>feed<\/span>(<\/span>s<\/span>)<\/span>
        self<\/span>.<\/span>close<\/span>()<\/span>

    def<\/span> __init__<\/span>(<\/span>self<\/span>,<\/span> verbose<\/span>=<\/span>0<\/span>):<\/span>
        \"Initialise an object, passing 'verbose' to the superclass.\"<\/span>

        sgmllib<\/span>.<\/span>SGMLParser<\/span>.<\/span>__init__<\/span>(<\/span>self<\/span>,<\/span> verbose<\/span>)<\/span>
        self<\/span>.<\/span>hyperlinks<\/span> =<\/span> []<\/span>

    def<\/span> start_a<\/span>(<\/span>self<\/span>,<\/span> attributes<\/span>):<\/span>
        \"Process a hyperlink and its 'attributes'.\"<\/span>

        for<\/span> name<\/span>,<\/span> value<\/span> in<\/span> attributes<\/span>:<\/span>
            if<\/span> name<\/span> ==<\/span> \"href\"<\/span>:<\/span>
                self<\/span>.<\/span>hyperlinks<\/span>.<\/span>append<\/span>(<\/span>value<\/span>)<\/span>

    def<\/span> get_hyperlinks<\/span>(<\/span>self<\/span>):<\/span>
        \"Return the list of hyperlinks.\"<\/span>

        return<\/span> self<\/span>.<\/span>hyperlinks<\/span>

import<\/span> urllib<\/span>,<\/span> sgmllib<\/span>

# Get something to work with.<\/span>
webPage<\/span>=<\/span>\"http:\/\/www.inference.phy.cam.ac.uk\/itprnn_lectures\/\"<\/span>
f<\/span> =<\/span> urllib<\/span>.<\/span>urlopen<\/span>(<\/span>webPage<\/span>)<\/span>
s<\/span> =<\/span> f<\/span>.<\/span>read<\/span>()<\/span>

# Try and process the page.<\/span>
# The class should have been defined first, remember.<\/span>
myparser<\/span> =<\/span> MyParser<\/span>()<\/span>
myparser<\/span>.<\/span>parse<\/span>(<\/span>s<\/span>)<\/span>

# Get the hyperlinks.<\/span>
links<\/span>=<\/span>myparser<\/span>.<\/span>get_hyperlinks<\/span>()<\/span>
print<\/span> links<\/span>

movies<\/span>=<\/span>[<\/span>x<\/span> for<\/span> x<\/span> in<\/span> links<\/span> if<\/span> x<\/span>.<\/span>endswith<\/span>(<\/span>'mp4'<\/span>)]<\/span>
print<\/span> movies<\/span>

import<\/span> urlparse<\/span>
movieURLs<\/span>=<\/span>[<\/span>urlparse<\/span>.<\/span>urljoin<\/span>(<\/span>webPage<\/span>,<\/span>x<\/span>)<\/span> for<\/span> x<\/span> in<\/span> movies<\/span>]<\/span>
print<\/span> movieURLs<\/span>

from<\/span> subprocess<\/span> import<\/span> call<\/span>

for<\/span> movieURL<\/span> in<\/span> movieURLs<\/span>:<\/span>
    call<\/span>([<\/span>\"wget -c \"<\/span>+<\/span>movieURL<\/span>],<\/span>shell<\/span>=<\/span>True<\/span>)<\/span>
<\/pre>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Recently I wanted to download all the movies in Mackay’s information theory course. That was a great deal of files and I though it would too tedious to try to do it using the browser or even manual wget. So digging around, I found this decent tutorial on python HTML processing. I also dug around […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[84,85,1,87,86],"tags":[],"class_list":["post-328","post","type-post","status-publish","format-standard","hentry","category-linux","category-python","category-uncategorized","category-url-parsing","category-wget"],"_links":{"self":[{"href":"https:\/\/perfectionatic.org\/index.php?rest_route=\/wp\/v2\/posts\/328","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perfectionatic.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perfectionatic.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perfectionatic.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/perfectionatic.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=328"}],"version-history":[{"count":1,"href":"https:\/\/perfectionatic.org\/index.php?rest_route=\/wp\/v2\/posts\/328\/revisions"}],"predecessor-version":[{"id":331,"href":"https:\/\/perfectionatic.org\/index.php?rest_route=\/wp\/v2\/posts\/328\/revisions\/331"}],"wp:attachment":[{"href":"https:\/\/perfectionatic.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=328"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perfectionatic.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=328"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perfectionatic.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=328"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}