Is it possible to process objects in a Google Cloud Storage bucket in FIFO order? -

-

in web app, need pull objects gcs 1 one , process them.
question is,
"how send request gcs next unprocessed object?"

what i’d rely on sort order provided gcs , process objects in sorted list 1 one. way, need keep track of last processed item in app. i’d rely on sort order provided timecreated timestamp on each individual object in bucket.

when query bucket via json api, notice objects returned sorted timecreated oldest newest.

for example, query ...

enter image description here

returns list ...

{  "items": [   {    "name": "cars_train/00001.jpg",    "timecreated": "2016-03-23t19:19:47.506z"   },   {    "name": "cars_train/00002.jpg",    "timecreated": "2016-03-23t19:19:49.320z"   },   {    "name": "cars_train/00003.jpg",    "timecreated": "2016-03-23t19:19:50.228z"   },   {    "name": "cars_train/00004.jpg",    "timecreated": "2016-03-23t19:19:51.377z"   },   {    "name": "cars_train/00005.jpg",    "timecreated": "2016-03-23t19:19:51.778z"   },   {    "name": "cars_train/00006.jpg",    "timecreated": "2016-03-23t19:19:52.817z"   },   {    "name": "cars_train/00007.jpg",    "timecreated": "2016-03-23t19:19:53.868z"   },   {    "name": "cars_train/00008.jpg",    "timecreated": "2016-03-23t19:19:54.925z"   },   {    "name": "cars_train/00009.jpg",    "timecreated": "2016-03-23t19:19:58.426z"   },   {    "name": "cars_train/00010.jpg",    "timecreated": "2016-03-23t19:19:59.323z"   }  ] }

this sort order timecreated need, though i’m not if can rely on being true?

so, code app process list searching first timecreated value greater last object processed.

the problem list can large , searching through huge list every single time user presses next button computationally expensive.

i able specify in query gcs filter list return single item need.

the api allow me set maxresults returned value of 1.

however, not see option allow me return objects timecreated value greater value specified.

i think trying common, i’m guessing solution may exist problem.

one work around problem physically move object has been processed bucket. way first item in list newest 1 , send request maxcount=1.

but adds complexity because forces me have have 2 separate buckets every project instead of 1.

is there way filter list of objects include ones timecreated date above specified value?

in mysql, might ...

select name bucket timecreated > x order timecreated limit 1

you can configure object change notifications on bucket, , notification each time new object arrives. allow process new objects without scanning long listing each time. avoids problem listing bucket consistent (so, uploaded objects may not show when list objects; don't know if that's problem app).

details object change notification documented @ https://cloud.google.com/storage/docs/object-change-notification.


Comments

Post a Comment

Popular posts from this blog

html - Styling progress bar with inline style -

-
i trying set progress bar color red using inline style. need in way because have access <body> part of document. style need apply: progress { color: red } progress::-moz-progress-bar { background: red;} progress::-webkit-progress-value { background: red; } progress[aria-valuenow]:before { background: red; } i tried <progress style="background: red;"> incorrect effect, progress bar color turns blue red background. guide right direction desired style? thank in advance. here live example: https://jsfiddle.net/z36d3f6l/
Read more

java - Oracle Sql developer error: could not install some modules -

-
i have downloaded fresh copy of sqldeveloper ( sqldeveloper-4.1.3.20.78-no-jre ) oracle website. when unzip , lauch gui, keep getting following error message: warning - not install modules: dynamic module config - no module providing capability org.netbeans.netbinox found. my system: windows 7 32 bit hp probook. jdk 1.6 , 1.7, 1.8 available in java/jdk subdirectory. netbeans 8.0.2 installed finally got way out after dirty hours. extracted sqldeveloper zip in nested directory. i.e: e://a/b/c/sqldeveloper. starting sqldeveloper gui above directory threw erros. solution: go in c:\users**username**\appdata\roaming : delete sql developer , sqldeveloper folders extract sql developer zip in new non nested directory. i.e: e:// start , should work now reference: https://community.oracle.com/thread/3870680?start=0&tstart=0
Read more

How to use autoclose brackets in Jupyter notebook? -

-
this may sound silly question, how make use of autoclose brackets in jupyter notebook? example, when type print( jupyter notebook auto-closes brakets print() and places cursor inside. type argument, say print(1 + 1) now cursor between second 1 , right bracket ) . key navigate right of ) ? of course can press end or → or ) achieve this, not seem save time, purpose of brackets autoclose suppose? thanks in advance.
Read more