Creating a table to query in Spark using Python -

-

i'm trying load file directly s3 , trying sparksql on it. eventually, plan on bringing in multiple files multiple tables (1:1 map between files , tables).

so i've been following this tutorial @ describing each step. i'm bit stuck on declaring proper schema , variable, if any, referred in from clause in sql statement.

here's code:

sqlcontext.sql("create table if not exists region (region_id int, name string, comment string)") region = sc.textfile("s3n://thisisnotabucketname/region.tbl")  raw_data = sc.textfile("s3n://thisisnotabucketname/region.tbl") csv_data = raw_data.map(lambda l: l.split("|")) row_data = csv_data.map(lambda p: row( region_id=int(p[0]), name=p[1], comment=p[2] ))  interactions_df = sqlcontext.createdataframe(row_data) interactions_df.registertemptable("interactions")  tcp_interactions = sqlcontext.sql(""" select region_id, name, comment region region_id > 1 """) tcp_interactions = sqlcontext.sql(""" select * region """) tcp_interactions.show()

and here's sample data. there no header

0|africa|lar deposits. blithely final packages cajole. regular waters final requests. regular accounts according | 1|america|hs use ironic, requests. s| 2|asia|ges. thinly pinto beans ca|

tcp_interactions.show() returning nothing. header of region_id|name|comment|. doing incorrectly? in sql statement, region pointing region variable declared in first line of code, or pointing else?


Comments

Post a Comment

Popular posts from this blog

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more