google chrome - Python Requests Not Returning Same Header as Browser Request/cURL -

-

i'm looking write script can automatically download .zip files bureau of transportation statistics carrier website, i'm having trouble getting same response headers can see in chrome when download zip file. i'm looking response header looks this:

http/1.1 302 object moved cache-control: private content-length: 183 content-type: text/html location: http://tsdata.bts.gov/103627300_t_t100_segment_all_carrier.zip server: microsoft-iis/8.5 x-powered-by: asp.net date: thu, 21 apr 2016 15:56:31 gmt

however, when calling requests.post(url, data=params, headers=headers) same information can see in chrome network inspector getting following response:

>>> res.headers {'cache-control': 'private', 'content-length': '262', 'content-type': 'text/html', 'x-powered-by': 'asp.net', 'date': 'thu, 21 apr 2016 20:16:26 gmt', 'server': 'microsoft-iis/8.5'}

it's got pretty except it's missing location key need in order download .zip file of data want. also content-length value different, i'm not sure if that's issue.

i think issue has fact when click "download" on page sends 2 requests can see in chrome network console. first request post request yields http response of 302 , has location in response header. second request get request url specified in location value of response header.

should sending 2 requests here? why not getting same response headers using requests in browser? fwiw used curl -x post -d /*my data*/ , got in terminal:

<head><title>object moved</title></head> <body><h1>object moved</h1>this object may found <a href="http://tsdata.bts.gov/103714760_t_t100_segment_all_carrier.zip">here</a>.</body>

really appreciate help!

i able download zip file looking using almost of headers see in google chrome web console. headers looked this:

{'connection': 'keep-alive', 'cache-control': 'max-age=0', 'referer': 'http://www.transtats.bts.gov/dl_selectfields.asp?table_id=293', 'origin': 'http://www.transtats.bts.gov', 'upgrade-insecure-requests': 1, 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'user-agent': 'mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, gecko) chrome/49.0.2623.112 safari/537.36', 'cookie': 'aspsessionidqadbbrta=cmkglhmddjiecmnglmdpokhc', 'accept-language': 'en-us,en;q=0.8', 'accept-encoding': 'gzip, deflate', 'content-type': 'application/x-www-form-urlencoded'}

and wrote:

res = requests.post(url, data=form_data, headers=headers)

where form_data copied "form data" section of chrome console. once got request, used zipfile , io modules parse content of response stored in res. this:

import zipfile, io zipfile.zipfile(io.bytesio(res.content))

and file in directory ran python code.

thanks users answered on this thread.


Comments

Post a Comment

Popular posts from this blog

ios - Memory not freeing up after popping viewcontroller using ARC -

-
hi project arc based, using uinavigationcontroller make transition between viewcontroller. using profiler analyse happening behind scene memory. noticed when push viewcontroller allocate memory components , when pop not freeing allocated memory. since using arc unable implement dealloc or release component. have analysed in detail , there no memory leak in project. i not using strong property push viewcontroller. here how pushing viewcontroller. viewcontroller *obj = [[viewcontroller alloc] init]; [self.navigationcontroller pushviewcontroller:obj animated:no]; any clue whats going on? should free memory have consumed. please advise your description indicates have retain cycle, leads objects not being deallocated. typical source of retain cycles properties assigned loading nib files (usually declared iboutlet ). two strategies break them , release objects: declare property weak : @property (nonatomic, weak) iboutlet uilabel *statuslabel; set property nil in...
Read more

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more