scrapy - content exists, but xpath could not find it, why? -

-

i using "scrapy shell" test xpath. looked like:

scrapy shell https://item.taobao.com/item.htm?spm=a219e.1191392.1111.1.fglwuh&id=40978681727&scm=1029.newlist-0.1.50002766&ppath=&sku=&ug=#detail

the xpath looked like:

response.xpath("//a[@class='shop-name-link']")

the result none, page content contains 

<a class="shop-name-link" href="//shop103857282.taobao.com" target="_blank"      data-goldlog-id="/tbwmdd.1.044">长岛小两口创业</a>

why?

if have problems finding results xpaths use firepath or chrome browser dev tools investigate page source. remember scrapy spider sees page source unrendered. not rendered javascript. view source spider sees use firepath in browser javascript disabled.

i cannot see link class shop-name-link in page linked in question. either you're not giving proper link or element displayed after user action, or page shown in different ways different users in different countries. possible page relies on presence of cookies have dont have.

there nice shortcut:

 scrapy.utils.response import open_in_browser  open_in_browser(response)

this open response spider in browser. use when need check spider sees. in many (if not cases) differs see in browser.

if want share reproducible example of how see page chrome dev tools have useful feature "copy curl" copy request headers , cookies clipboard. if paste in question people able see page see (provided of course there no geolocation restrictions on ips).


Comments

Post a Comment

Popular posts from this blog

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more