Adding a custom filter to Airflow logs

Recently I tried to add a custom logger for our Airflow deployment in kubernetes. Since we allow our logs to be shown to our customers we cannot have our secrets and other kubernetes config be shown to the users. On pod failure we get a dump of the entire pod config and that is what we want to remove today Note: Our logs are sent to S3 and remote logging is enabled for our version of airflow 2.5.0, so whatever

Continue Reading

Site Footer

Sliding Sidebar

About Me

About Me

Hey, I am Thomas Ashish Cherian. What I cannot create, I do not understand.

Social Profiles