![]() ![]() Getting started with Airflow locally and remotely Airflow has been around for a while, but it has gained a lot of traction lately. I thought airflow is not communicating with spark-master, So I tried changing the IP-address in the connection and I put nonsense like 127.127.124. Open in app last free member-only story and get an extra one. The oiginal docker-compose.yaml file was taken from the official github repo. Now the problem is when I run the DAG, it works perfectly but in the spark-master UI, there is no job running or worker executing something. About Setup Apache Airflow 2.0 locally on Windows 10 (WSL2) via Docker Compose. It will issue the airflow test mydag mytask execdate command and it will break on any of your break points. env file: AIRFLOWUID1000 AIRFLOWGID0 And the docker-compose.yaml is the default one docker-compose.yaml. It will attach to the airflow docker container. I have followed the instructions here Airflow init. You can always inspect the metadata db of airflow to check if connection is properly defined or use airflow connection CLI to inspect it. I have a keyfile generated from a suitable. The DAG from datetime import datetime, timedeltaįrom import PythonOperatorįrom .operators.spark_submit import SparkSubmitOperatorĭefault_args = ] I am trying to run a simple python script within a docker run command scheduled with Airflow. I am simply trying to create a connection to my gcp project from my airflow (running out of docker locally). ![]() I used SparkSubmitOperator to submit the job ( the job is just print a dataframe ).host = here I did docker network inspect airflow_default to get the Ip-address of the spark-master.in airflow, I added a new connection " Admin > Connections > Add new record ".Now, I would like to trigger a spark job using airflow : I think till this point the setup is correct. I connected the tow spark containers to the network airflow default using docker network connect airflow_default spark-worker1 and docker network connect airflow_default spark-master. The most common way of defining a connection is using the Airflow UI. I set up a spark-master and spark-worker1 connected through a bridge network called Spark_net.Īirflow runs using docker compose and has its own bridge network called airflow_default. I am working on airflow and Spark in two separate containers. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |