google chrome - Python Requests Not Returning Same Header as Browser Request/cURL -

-

i'm looking write script can automatically download .zip files bureau of transportation statistics carrier website, i'm having trouble getting same response headers can see in chrome when download zip file. i'm looking response header looks this:

http/1.1 302 object moved cache-control: private content-length: 183 content-type: text/html location: http://tsdata.bts.gov/103627300_t_t100_segment_all_carrier.zip server: microsoft-iis/8.5 x-powered-by: asp.net date: thu, 21 apr 2016 15:56:31 gmt

however, when calling requests.post(url, data=params, headers=headers) same information can see in chrome network inspector getting following response:

>>> res.headers {'cache-control': 'private', 'content-length': '262', 'content-type': 'text/html', 'x-powered-by': 'asp.net', 'date': 'thu, 21 apr 2016 20:16:26 gmt', 'server': 'microsoft-iis/8.5'}

it's got pretty except it's missing location key need in order download .zip file of data want. also content-length value different, i'm not sure if that's issue.

i think issue has fact when click "download" on page sends 2 requests can see in chrome network console. first request post request yields http response of 302 , has location in response header. second request get request url specified in location value of response header.

should sending 2 requests here? why not getting same response headers using requests in browser? fwiw used curl -x post -d /*my data*/ , got in terminal:

<head><title>object moved</title></head> <body><h1>object moved</h1>this object may found <a href="http://tsdata.bts.gov/103714760_t_t100_segment_all_carrier.zip">here</a>.</body>

really appreciate help!

i able download zip file looking using almost of headers see in google chrome web console. headers looked this:

{'connection': 'keep-alive', 'cache-control': 'max-age=0', 'referer': 'http://www.transtats.bts.gov/dl_selectfields.asp?table_id=293', 'origin': 'http://www.transtats.bts.gov', 'upgrade-insecure-requests': 1, 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'user-agent': 'mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, gecko) chrome/49.0.2623.112 safari/537.36', 'cookie': 'aspsessionidqadbbrta=cmkglhmddjiecmnglmdpokhc', 'accept-language': 'en-us,en;q=0.8', 'accept-encoding': 'gzip, deflate', 'content-type': 'application/x-www-form-urlencoded'}

and wrote:

res = requests.post(url, data=form_data, headers=headers)

where form_data copied "form data" section of chrome console. once got request, used zipfile , io modules parse content of response stored in res. this:

import zipfile, io zipfile.zipfile(io.bytesio(res.content))

and file in directory ran python code.

thanks users answered on this thread.


Comments

Post a Comment

Popular posts from this blog

html - Styling progress bar with inline style -

-
i trying set progress bar color red using inline style. need in way because have access <body> part of document. style need apply: progress { color: red } progress::-moz-progress-bar { background: red;} progress::-webkit-progress-value { background: red; } progress[aria-valuenow]:before { background: red; } i tried <progress style="background: red;"> incorrect effect, progress bar color turns blue red background. guide right direction desired style? thank in advance. here live example: https://jsfiddle.net/z36d3f6l/
Read more

java - Oracle Sql developer error: could not install some modules -

-
i have downloaded fresh copy of sqldeveloper ( sqldeveloper-4.1.3.20.78-no-jre ) oracle website. when unzip , lauch gui, keep getting following error message: warning - not install modules: dynamic module config - no module providing capability org.netbeans.netbinox found. my system: windows 7 32 bit hp probook. jdk 1.6 , 1.7, 1.8 available in java/jdk subdirectory. netbeans 8.0.2 installed finally got way out after dirty hours. extracted sqldeveloper zip in nested directory. i.e: e://a/b/c/sqldeveloper. starting sqldeveloper gui above directory threw erros. solution: go in c:\users**username**\appdata\roaming : delete sql developer , sqldeveloper folders extract sql developer zip in new non nested directory. i.e: e:// start , should work now reference: https://community.oracle.com/thread/3870680?start=0&tstart=0
Read more

How to use autoclose brackets in Jupyter notebook? -

-
this may sound silly question, how make use of autoclose brackets in jupyter notebook? example, when type print( jupyter notebook auto-closes brakets print() and places cursor inside. type argument, say print(1 + 1) now cursor between second 1 , right bracket ) . key navigate right of ) ? of course can press end or → or ) achieve this, not seem save time, purpose of brackets autoclose suppose? thanks in advance.
Read more