git - Use Jupyter together with file share or mounted folder -

-

how can synchronize notebooks between jupyter service , other services (google cloud storage or git repository)?

some background on question:

currently on way moving google's datalab own container. motivation have more control on data region (datalab beta offered in us) , packages want use current tensorflow version.

based on ideas google (see github), build own docker image , run on kubernetes cluster in google container engine. gcp package can installed i have explained. google uses node.js server sync git datalab instance - not able running self-deployed container in eu.

second try gcsfuse driver. 1 not work non-priviliged containers of kubernetes v1.0 , google container engine. full stop.

my docker file (based on google's gce datalab image):

from debian:jessie  # setup os , core packages run apt-get clean run echo "deb-src http://ftp.be.debian.org/debian testing main" >> /etc/apt/sources.list && \ apt-get update -y && \ apt-get install --no-install-recommends -y -q \     curl wget unzip git vim build-essential ca-certificates pkg-config \     libatlas-base-dev liblapack-dev gfortran \     libpng-dev libfreetype6-dev libxft-dev \     libxml2-dev \     python2.7 python-dev python-pip python-setuptools python-zmq && \ mkdir -p /tools && \ mkdir -p /srcs && \ cd /srcs && apt-get source -d python-zmq && cd  workdir /datalab   # setup google cloud sdk run apt-get install --no-install-recommends -y -q wget unzip git -y run wget -nv https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.zip && \ unzip -qq google-cloud-sdk.zip -d tools && \ rm google-cloud-sdk.zip && \ tools/google-cloud-sdk/install.sh --usage-reporting=false \     --path-update=false --bash-completion=false \     --disable-installation-options && \ tools/google-cloud-sdk/bin/gcloud config set --scope=installation \     component_manager/fixed_sdk_version 0.9.57 && \ tools/google-cloud-sdk/bin/gcloud -q components update \     gcloud core bq gsutil compute preview alpha beta && \ rm -rf /root/.config/gcloud  # install fuse driver gce run apt-get install -y lsb-release run echo "deb http://packages.cloud.google.com/apt gcsfuse-jessie main" >     /etc/apt/sources.list.d/gcsfuse.list run curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - run apt-get update && apt-get install -y gcsfuse run mkdir /datalab/mount  # setup python packages run pip install -u \     tornado==4.2.1 pyzmq==14.4.0 jinja2==2.7.3 \     jsonschema==2.5.1 py-dateutil==2.2 pytz==2015.4 pandocfilters==1.2.4 pygments==2.0.2 \     argparse==1.2.1 mock==1.2.0 requests==2.4.3 oauth2client==1.4.12 httplib2==0.9.2 \     futures==3.0.3 && \     pip install -u numpy==1.9.2 && \     pip install -u pandas==0.16.2 && \     pip install -u scikit-learn==0.16.1 && \     pip install -u scipy==0.15.1 && \     pip install -u sympy==0.7.6 && \     pip install -u statsmodels==0.6.1 && \     pip install -u matplotlib==1.4.3 && \     pip install -u ggplot==0.6.5 && \     pip install -u seaborn==0.6.0 && \     pip install -u notebook==4.0.2 && \     pip install -u pyyaml==3.11 && \     easy_install pip && \     find /usr/local/lib/python2.7 -type d -name tests | xargs rm -rf  # path configuration env path $path:/datalab/tools/google-cloud-sdk/bin env pythonpath /env/python  # ipython configuration workdir /datalab run ipython profile create default run jupyter notebook --generate-config add ipython.py /root/.ipython/profile_default/ipython_config.py  # install tensorflow. run wget -nv https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.7.1-cp27-none-linux_x86_64.whl && \   pip install --upgrade tensorflow-0.7.1-cp27-none-linux_x86_64.whl && rm tensorflow-0.7.1-cp27-none-linux_x86_64.whl   # add build artifacts add build/lib/gcpdata-0.1.0.tar.gz /datalab/lib/ add build/lib/gcpdatalab-0.1.0.tar.gz /datalab/lib/ add setup-repo.sh /datalab add setup-env.sh /datalab add run.sh /datalab run chmod 755 /datalab/*  # install build artifacts run cd /datalab/lib/gcpdata-0.1.0 && python setup.py install run cd /datalab/lib/gcpdatalab-0.1.0 && python setup.py install  run mkdir /datalab/content workdir /datalab/content expose 6006 expose 8123 # see https://github.com/ipython/ipython/issues/7062 cmd ["/datalab/run.sh"]

ok, solved problem:

  1. use post-save hooks explained in previous post
  2. use several git commands in hook explained blog

here code (2.) archiving. goes ipython.py:

import os subprocess import check_call shlex import split  ...  def post_save(model, os_path, contents_manager):     """post-save hook doing git commit / push"""     if model['type'] != 'notebook':         return  # notebooks     workdir, filename = os.path.split(os_path)     if filename.startswith('scratch') or filename.startswith('untitled'):         return  # skip scratch , untitled notebooks     # git add / git commit / git push     check_call(split('git add {}'.format(filename)), cwd=workdir)     check_call(split('git commit -m "notebook save" {}'.format(filename)), cwd=workdir)     check_call(split('git push'), cwd=workdir)  c.filecontentsmanager.post_save_hook = post_save

my run.sh utilizes setup-env.sh , setup-repo.sh google datalab , consequently depends on gcloud commands , kubernetes deployments credentials. otherwise please ensure extend dockerfile credentials.

cd /datalab/content . /datalab/setup-env.sh . /datalab/setup-repo.sh if [ $? != "0" ];     exit 1 fi cd /datalab/content/master_branch  # multiple branches not planed here! /usr/local/bin/jupyter notebook --ip=* --no-browser --port=8123

Comments

Post a Comment

Popular posts from this blog

ios - Memory not freeing up after popping viewcontroller using ARC -

-
hi project arc based, using uinavigationcontroller make transition between viewcontroller. using profiler analyse happening behind scene memory. noticed when push viewcontroller allocate memory components , when pop not freeing allocated memory. since using arc unable implement dealloc or release component. have analysed in detail , there no memory leak in project. i not using strong property push viewcontroller. here how pushing viewcontroller. viewcontroller *obj = [[viewcontroller alloc] init]; [self.navigationcontroller pushviewcontroller:obj animated:no]; any clue whats going on? should free memory have consumed. please advise your description indicates have retain cycle, leads objects not being deallocated. typical source of retain cycles properties assigned loading nib files (usually declared iboutlet ). two strategies break them , release objects: declare property weak : @property (nonatomic, weak) iboutlet uilabel *statuslabel; set property nil in...
Read more

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more