javascript - Scraping dynamic content through Selenium? -

-

i'm trying scrap dynamic content blog through selenium returns un rendered javascript.

to test behavior tried wait till iframe loads , printed it's content prints fine again when move parent frame displays un rendered javascript.

i'm looking in i'm able print rendered html content

from selenium import webdriver selenium.webdriver.support.ui import webdriverwait selenium.webdriver.common.by import selenium.webdriver.support import expected_conditions  driver = webdriver.chrome("path chrome driver")    driver.get('http://justgivemechocolateandnobodygetshurt.blogspot.com/')  webdriverwait(driver, 40).until(expected_conditions.frame_to_be_available_and_switch_to_it((by.id, "navbar-iframe")))  # rendered iframe html printed. content = driver.page_source print content.encode("utf-8")  # when switch parent frame again prints non rendered javascript. driver.switch_to.parent_frame() content = driver.page_source print content.encode("utf-8")

the problem - the .page_source works in current context. there "current top-level browsing context" notation..meaning, if call on default content - not inner html of child iframeelements - have switch context of frame , call .page_source.

in other words, complete html of page including page source of iframes, have switch iframe contexts 1 one , sources separately.

see also:

old answer:

i wait @ least 1 blog entry loaded before getting page_source:

from selenium.webdriver.common.by import selenium.webdriver.support.ui import webdriverwait selenium.webdriver.support import expected_conditions ec  wait = webdriverwait(driver, 40) wait.until(ec.visibility_of_element_located((by.css_selector, ".entry-content")))  print(driver.page_source)

Comments

Post a Comment

Popular posts from this blog

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more