python - beautiful soup parser can't find links -

-

i trying parse html document find links using beautiful soup , found weird behavior. page http://people.csail.mit.edu/gjtucker/ . here's code:

from bs4 import beautifulsoup import requests  user_agent = {'user-agent': 'mozilla/5.0 (x11; linux i686) applewebkit/537.17 (khtml, gecko) chrome/24.0.1312.52 safari/537.17'}  t=requests.get(url, headers = user_agent).text  soup=beautifulsoup(t, 'html.parser') link in soup.findall('a'):     print link['href']

this prints 2 links: http://www.amazon.jobs/team/speech-amazon , https://scholar.google.com/citations?user=-gjkphiaaaaj&hl=en, whereas there many more links in page.

can reproduce this? there specific reason happening url? few outher urls worked fine.

the html of page not well-formed, should use more lenient parser, html5lib:

soup = beautifulsoup(t, 'html5lib') link in soup.find_all('a'):     print(link['href'])

prints:

http://www.amazon.jobs/team/speech-amazon https://scholar.google.com/citations?user=-gjkphiaaaaj&hl=en http://www.linkedin.com/pub/george-tucker/6/608/3ba ... http://www.hsph.harvard.edu/alkes-price/ ... http://www.nature.com/ng/journal/v47/n3/full/ng.3190.html http://www.biomedcentral.com/1471-2105/14/299 pdfs/journal.pone.0029095.pdf pdfs/es201187u.pdf pdfs/sigtrans.pdf

Comments

Post a Comment

Popular posts from this blog

html - Styling progress bar with inline style -

-
i trying set progress bar color red using inline style. need in way because have access <body> part of document. style need apply: progress { color: red } progress::-moz-progress-bar { background: red;} progress::-webkit-progress-value { background: red; } progress[aria-valuenow]:before { background: red; } i tried <progress style="background: red;"> incorrect effect, progress bar color turns blue red background. guide right direction desired style? thank in advance. here live example: https://jsfiddle.net/z36d3f6l/
Read more

java - Oracle Sql developer error: could not install some modules -

-
i have downloaded fresh copy of sqldeveloper ( sqldeveloper-4.1.3.20.78-no-jre ) oracle website. when unzip , lauch gui, keep getting following error message: warning - not install modules: dynamic module config - no module providing capability org.netbeans.netbinox found. my system: windows 7 32 bit hp probook. jdk 1.6 , 1.7, 1.8 available in java/jdk subdirectory. netbeans 8.0.2 installed finally got way out after dirty hours. extracted sqldeveloper zip in nested directory. i.e: e://a/b/c/sqldeveloper. starting sqldeveloper gui above directory threw erros. solution: go in c:\users**username**\appdata\roaming : delete sql developer , sqldeveloper folders extract sql developer zip in new non nested directory. i.e: e:// start , should work now reference: https://community.oracle.com/thread/3870680?start=0&tstart=0
Read more

How to use autoclose brackets in Jupyter notebook? -

-
this may sound silly question, how make use of autoclose brackets in jupyter notebook? example, when type print( jupyter notebook auto-closes brakets print() and places cursor inside. type argument, say print(1 + 1) now cursor between second 1 , right bracket ) . key navigate right of ) ? of course can press end or → or ) achieve this, not seem save time, purpose of brackets autoclose suppose? thanks in advance.
Read more