Creating a table to query in Spark using Python -

-

i'm trying load file directly s3 , trying sparksql on it. eventually, plan on bringing in multiple files multiple tables (1:1 map between files , tables).

so i've been following this tutorial @ describing each step. i'm bit stuck on declaring proper schema , variable, if any, referred in from clause in sql statement.

here's code:

sqlcontext.sql("create table if not exists region (region_id int, name string, comment string)") region = sc.textfile("s3n://thisisnotabucketname/region.tbl")  raw_data = sc.textfile("s3n://thisisnotabucketname/region.tbl") csv_data = raw_data.map(lambda l: l.split("|")) row_data = csv_data.map(lambda p: row( region_id=int(p[0]), name=p[1], comment=p[2] ))  interactions_df = sqlcontext.createdataframe(row_data) interactions_df.registertemptable("interactions")  tcp_interactions = sqlcontext.sql(""" select region_id, name, comment region region_id > 1 """) tcp_interactions = sqlcontext.sql(""" select * region """) tcp_interactions.show()

and here's sample data. there no header

0|africa|lar deposits. blithely final packages cajole. regular waters final requests. regular accounts according | 1|america|hs use ironic, requests. s| 2|asia|ges. thinly pinto beans ca|

tcp_interactions.show() returning nothing. header of region_id|name|comment|. doing incorrectly? in sql statement, region pointing region variable declared in first line of code, or pointing else?


Comments

Post a Comment

Popular posts from this blog

html - Styling progress bar with inline style -

-
i trying set progress bar color red using inline style. need in way because have access <body> part of document. style need apply: progress { color: red } progress::-moz-progress-bar { background: red;} progress::-webkit-progress-value { background: red; } progress[aria-valuenow]:before { background: red; } i tried <progress style="background: red;"> incorrect effect, progress bar color turns blue red background. guide right direction desired style? thank in advance. here live example: https://jsfiddle.net/z36d3f6l/
Read more

java - Oracle Sql developer error: could not install some modules -

-
i have downloaded fresh copy of sqldeveloper ( sqldeveloper-4.1.3.20.78-no-jre ) oracle website. when unzip , lauch gui, keep getting following error message: warning - not install modules: dynamic module config - no module providing capability org.netbeans.netbinox found. my system: windows 7 32 bit hp probook. jdk 1.6 , 1.7, 1.8 available in java/jdk subdirectory. netbeans 8.0.2 installed finally got way out after dirty hours. extracted sqldeveloper zip in nested directory. i.e: e://a/b/c/sqldeveloper. starting sqldeveloper gui above directory threw erros. solution: go in c:\users**username**\appdata\roaming : delete sql developer , sqldeveloper folders extract sql developer zip in new non nested directory. i.e: e:// start , should work now reference: https://community.oracle.com/thread/3870680?start=0&tstart=0
Read more

How to use autoclose brackets in Jupyter notebook? -

-
this may sound silly question, how make use of autoclose brackets in jupyter notebook? example, when type print( jupyter notebook auto-closes brakets print() and places cursor inside. type argument, say print(1 + 1) now cursor between second 1 , right bracket ) . key navigate right of ) ? of course can press end or → or ) achieve this, not seem save time, purpose of brackets autoclose suppose? thanks in advance.
Read more