scrapy - content exists, but xpath could not find it, why? -

-

i using "scrapy shell" test xpath. looked like:

scrapy shell https://item.taobao.com/item.htm?spm=a219e.1191392.1111.1.fglwuh&id=40978681727&scm=1029.newlist-0.1.50002766&ppath=&sku=&ug=#detail

the xpath looked like:

response.xpath("//a[@class='shop-name-link']")

the result none, page content contains 

<a class="shop-name-link" href="//shop103857282.taobao.com" target="_blank"      data-goldlog-id="/tbwmdd.1.044">长岛小两口创业</a>

why?

if have problems finding results xpaths use firepath or chrome browser dev tools investigate page source. remember scrapy spider sees page source unrendered. not rendered javascript. view source spider sees use firepath in browser javascript disabled.

i cannot see link class shop-name-link in page linked in question. either you're not giving proper link or element displayed after user action, or page shown in different ways different users in different countries. possible page relies on presence of cookies have dont have.

there nice shortcut:

 scrapy.utils.response import open_in_browser  open_in_browser(response)

this open response spider in browser. use when need check spider sees. in many (if not cases) differs see in browser.

if want share reproducible example of how see page chrome dev tools have useful feature "copy curl" copy request headers , cookies clipboard. if paste in question people able see page see (provided of course there no geolocation restrictions on ips).


Comments

Post a Comment

Popular posts from this blog

html - Styling progress bar with inline style -

-
i trying set progress bar color red using inline style. need in way because have access <body> part of document. style need apply: progress { color: red } progress::-moz-progress-bar { background: red;} progress::-webkit-progress-value { background: red; } progress[aria-valuenow]:before { background: red; } i tried <progress style="background: red;"> incorrect effect, progress bar color turns blue red background. guide right direction desired style? thank in advance. here live example: https://jsfiddle.net/z36d3f6l/
Read more

java - Oracle Sql developer error: could not install some modules -

-
i have downloaded fresh copy of sqldeveloper ( sqldeveloper-4.1.3.20.78-no-jre ) oracle website. when unzip , lauch gui, keep getting following error message: warning - not install modules: dynamic module config - no module providing capability org.netbeans.netbinox found. my system: windows 7 32 bit hp probook. jdk 1.6 , 1.7, 1.8 available in java/jdk subdirectory. netbeans 8.0.2 installed finally got way out after dirty hours. extracted sqldeveloper zip in nested directory. i.e: e://a/b/c/sqldeveloper. starting sqldeveloper gui above directory threw erros. solution: go in c:\users**username**\appdata\roaming : delete sql developer , sqldeveloper folders extract sql developer zip in new non nested directory. i.e: e:// start , should work now reference: https://community.oracle.com/thread/3870680?start=0&tstart=0
Read more

How to use autoclose brackets in Jupyter notebook? -

-
this may sound silly question, how make use of autoclose brackets in jupyter notebook? example, when type print( jupyter notebook auto-closes brakets print() and places cursor inside. type argument, say print(1 + 1) now cursor between second 1 , right bracket ) . key navigate right of ) ? of course can press end or → or ) achieve this, not seem save time, purpose of brackets autoclose suppose? thanks in advance.
Read more