configuring Airflow to work with CeleryExecutor -

-

i try configure airbnb airflow use celeryexecutor this:

i changed executer in airflow.cfg sequentialexecutor celeryexecutor:

# executor class airflow should use. choices include # sequentialexecutor, localexecutor, celeryexecutor executor = celeryexecutor

but following error:

airflow.configuration.airflowconfigexception: error: cannot use sqlite celeryexecutor

note sql_alchemy_conn configured this:

sql_alchemy_conn = sqlite:////root/airflow/airflow.db

i looked @ airflow's git (https://github.com/airbnb/airflow/blob/master/airflow/configuration.py)

and found following code throws exception:

def _validate(self):         if (                 self.get("core", "executor") != 'sequentialexecutor' ,                 "sqlite" in self.get('core', 'sql_alchemy_conn')):             raise airflowconfigexception("error: cannot use sqlite {}".                 format(self.get('core', 'executor')))

it seems validate method sql_alchemy_conn cannot contain sqlite.

do have idea how configure celeryexecutor without sqllite? please note downloaded rabitmq working celeryexecuter required.

it said airflow celeryexecutor requires other backend default database sqlite. have use mysql or postgresql, example.

the sql_alchemy_conn in airflow.cfg must changed follow sqlalchemy connection string structure (see sqlalchemy document)

for example, 

sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@127.0.0.1:5432/airflow

Comments

Post a Comment

Popular posts from this blog

ios - Memory not freeing up after popping viewcontroller using ARC -

-
hi project arc based, using uinavigationcontroller make transition between viewcontroller. using profiler analyse happening behind scene memory. noticed when push viewcontroller allocate memory components , when pop not freeing allocated memory. since using arc unable implement dealloc or release component. have analysed in detail , there no memory leak in project. i not using strong property push viewcontroller. here how pushing viewcontroller. viewcontroller *obj = [[viewcontroller alloc] init]; [self.navigationcontroller pushviewcontroller:obj animated:no]; any clue whats going on? should free memory have consumed. please advise your description indicates have retain cycle, leads objects not being deallocated. typical source of retain cycles properties assigned loading nib files (usually declared iboutlet ). two strategies break them , release objects: declare property weak : @property (nonatomic, weak) iboutlet uilabel *statuslabel; set property nil in...
Read more

c# - DotNetOpenAuth OAuth2 for Basecamp API -

-
i'm having difficulties getting oauth2 working basecamp api dotnetopenauth, here's have far, asp.net mvc 4 web app. public actionresult basecamp() { var server = new dotnetopenauth.oauth2.authorizationserverdescription(); server.authorizationendpoint = new uri("https://launchpad.37signals.com/authorization/new"); server.tokenendpoint = new uri("https://launchpad.37signals.com/authorization/token"); var client = new dotnetopenauth.oauth2.webserverclient( server, "my-basecamp-id", "my-basecamp-secret"); client.requestuserauthorization(returnto: new uri("http://localhost:55321/settings/basecampauth")); response.end(); return null; } [httppost] public actionresult basecampauth() { var server = new dotnetopenauth.oauth2.authorizationserverdescription(); server.authorizationendpoint = new uri("https://launchpad.37signals.com/authorization/new"); server.tokenendpoint =...
Read more