python - Upload PySpark RDD into BigQuery -

-

i download table bq pyspark rdd below. how upload again?

dgsconfig = {     'project_id': "project_id",     'bucket': "bucket_id" } dbqconfig = {     'gs_config': dgsconfig,     'project_id': "project_id",     'dataset_id': "dataset_id",     'table_id': "table_id" }  osc = instantiate_pyspark() rdddata, lsheadings = get_table_cloud(osc, dbqconfig)  #rdddata has list-of-lists type format     def instantiate_pyspark():     """instantiate pyspark rdd stuff"""     import pyspark      osc = pyspark.sparkcontext()     ohadoopconf = osc._jsc.hadoopconfiguration()     ohadoopconf.get("fs.gs.system.bucket")      return osc   def get_table_cloud(osc, dbqconfig):     """get table bigquery via google cloud storage     config format:         dgsconfig = {'project_id': '', 'bucket':  ''}         dbqconfig = {'project_id: '', 'dataset_id': '', 'table_id': ''}     """     dgsconfig = dbqconfig['gs_config']      dconf = {         "mapred.bq.project.id": dgsconfig['project_id'],         "mapred.bq.gcs.bucket": dgsconfig['bucket'],         "mapred.bq.input.project.id": dbqconfig['project_id'],         "mapred.bq.input.dataset.id":dbqconfig['dataset_id'],         "mapred.bq.input.table.id": dbqconfig['table_id']     }      rdddatasetraw = osc.newapihadooprdd(         "com.google.cloud.hadoop.io.bigquery.jsontextbigqueryinputformat",         "org.apache.hadoop.io.longwritable",         "com.google.gson.jsonobject",         conf=dconf     )      import json     lsheadings = json.loads(rdddatasetraw.take(1)[0][1]).keys()      rdddataset = (         rdddatasetraw         .map(lambda t, json=json: json.loads(t[1]).values() )     )      return rdddataset, lsheadings

you export intermediate files , load files bigquery.

this might help: how export table dataframe in pyspark csv?


Comments

Post a Comment

Popular posts from this blog

html - Styling progress bar with inline style -

-
i trying set progress bar color red using inline style. need in way because have access <body> part of document. style need apply: progress { color: red } progress::-moz-progress-bar { background: red;} progress::-webkit-progress-value { background: red; } progress[aria-valuenow]:before { background: red; } i tried <progress style="background: red;"> incorrect effect, progress bar color turns blue red background. guide right direction desired style? thank in advance. here live example: https://jsfiddle.net/z36d3f6l/
Read more

java - Oracle Sql developer error: could not install some modules -

-
i have downloaded fresh copy of sqldeveloper ( sqldeveloper-4.1.3.20.78-no-jre ) oracle website. when unzip , lauch gui, keep getting following error message: warning - not install modules: dynamic module config - no module providing capability org.netbeans.netbinox found. my system: windows 7 32 bit hp probook. jdk 1.6 , 1.7, 1.8 available in java/jdk subdirectory. netbeans 8.0.2 installed finally got way out after dirty hours. extracted sqldeveloper zip in nested directory. i.e: e://a/b/c/sqldeveloper. starting sqldeveloper gui above directory threw erros. solution: go in c:\users**username**\appdata\roaming : delete sql developer , sqldeveloper folders extract sql developer zip in new non nested directory. i.e: e:// start , should work now reference: https://community.oracle.com/thread/3870680?start=0&tstart=0
Read more

How to use autoclose brackets in Jupyter notebook? -

-
this may sound silly question, how make use of autoclose brackets in jupyter notebook? example, when type print( jupyter notebook auto-closes brakets print() and places cursor inside. type argument, say print(1 + 1) now cursor between second 1 , right bracket ) . key navigate right of ) ? of course can press end or → or ) achieve this, not seem save time, purpose of brackets autoclose suppose? thanks in advance.
Read more