Python Flask: interacts with Docker containers

Python3 Flask: interacts with Docker containers

Last Friday, I was talking about my work with my coworker. Just about work, you know. He told me that he had to implement a server to serve his facilities. Although I didn’t understand exactly what he was going to do, it looked really interesting and I wanted to try. Since I’ve got a lot of work that creates multiple pages, I wrote a bunch of HTML things with React. It wasn’t technically interesting at the moment (I like React but the work was almost identical). I got tired of it. While I was there, the coworker told me what he was doing.

he had to

  • Create an API server that receives an image file
  • Execute a docker container and pass the image file to the container and the container will create a text file generated from the image file
  • Execute another docker container and pass text
  • The API server knows that the container is done with its work and informs the other server (the backend that servers the data to the users).

It sounds interesting, ha?, I decided to implement this system over the weekend. I thought it was going to be fun. But it’s been a really long time since I’ve coded with python. So, I had to learn a lot of things. During the weekend, I could not learn all the things, therefore, I will use minimum skills to implement the system. If you are a python developer or working with docker, this may not be on your mind at some points.

Anyway, let’s get started!


Condition

Pipenv is a tool that aims to bring all the packaging worlds (Bundler, Composer, NPM, Cargo, Yarn, etc.) to the Python world. Windows is a first class citizen in our world.
When I first used python, it was about 5 years ago, I used conda or venv to manage packages. When I was searching for package management, I found this. it seems like npmI’m not sure if this is the best solution, but in my opinion, it should be worth trying.

Docker is a platform designed to help developers build, share, and run modern applications. We handle the tedious setup, so you can focus on the code.
As I first introduced, this is the main topic of this post

Flask is a web application framework written in Python. It was developed by Armin Ronacher, who led a team of international Python enthusiasts called Poco.
To implement a simple API web server, I used Flask, There were other libraries like Fast API. To reduce my hours, I went with Flask.


This is the process I am going to implement.

  1. API server receives a file from the user

  2. Container processes A file and then saves it to B file

  3. Container B processes the B file and then saves it to the C file

  4. User can view the result through API


Python Apps and Docker Images

I created two images and uploaded them to my repository in docker-hub.

A token and word-counting app.

[Tokenizer]

import sys, json, os, requests
from textblob import TextBlob


def extract_nouns(text):
    blob = TextBlob(text)
    filtered_tags = list(filter(lambda tag: tag[1] == "NN", blob.tags))
    nouns = list(map(lambda tag: tag[0], filtered_tags))
    return nouns


def read_file(path):
    with open(path) as f:
        contents = f.read()
    return contents


def save_data(path, data):
    with open(path, "w") as f:
        json.dump(data, f)


def get_filename_from_path(path):
    return os.path.splitext(os.path.basename(path))[0]


def notify_done(url, file_name):
    requests.get(f"{url}/docker/tokenizer_done?file_name={file_name}")


if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("You must pass file path as an argument")
        print("python3 main.py [file path to read] [dir to save] [notification api]")
        print("Example) python3 main.py ./test.txt ./ http://host.docker.internal:20000")
        sys.exit()

    api_url = sys.argv[3]
    file_path = sys.argv[1]
    file_name = get_filename_from_path(file_path)
    target_path = os.path.join(sys.argv[2], file_name + ".json") 

    text = read_file(file_path)
    nouns = extract_nouns(text)

    save_data(target_path, {"nouns": nouns})
    notify_done(api_url, file_name)

    print("Done")
enter fullscreen mode

exit fullscreen mode

[word-counting]

import sys, json, os, requests


def count_word(nouns_list):
    count_dict = dict()

    for noun in nouns_list:
        if noun in count_dict:
            count_dict[noun] += 1
        else:
            count_dict[noun] = 1

    return count_dict


def load_data(path):
    with open(path) as f:
        json_data = json.load(f)
    return json_data


def save_data(path, data):
    with open(path, "w") as f:
        json.dump(data, f)


def get_filename_from_path(path):
    return os.path.splitext(os.path.basename(path))[0]


def notify_done(url, file_name):
    requests.get(f"{url}/docker/word_count_done?file_name={file_name}")


if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("You must pass file path as an argument")
        print("python3 main.py [file path to read] [dir to save] [notification api]")
        print("Example) python3 main.py ./test.txt ./ http://host.docker.internal:20000")
        sys.exit()

    api_url = sys.argv[3]
    file_path = sys.argv[1]
    file_name = get_filename_from_path(file_path)
    target_path = os.path.join(sys.argv[2], file_name + ".json") 

    json_data = load_data(file_path)
    count_dict = count_word(json_data["nouns"])

    save_data(target_path, {"result": count_dict})
    notify_done(api_url, file_name)
    print("Done")
enter fullscreen mode

exit fullscreen mode

In order to run apps from API server, I have created both python files with Dockerfiles below.

[Tokenizer]

FROM python:3.9

WORKDIR /app
COPY . .

RUN pip install pipenv
RUN pipenv install
RUN pipenv run python3 -m textblob.download_corpora

ENTRYPOINT ["pipenv", "run", "python3", "./main.py"]
enter fullscreen mode

exit fullscreen mode

[word-counting]

FROM python:3.9

WORKDIR /app
COPY . .

RUN pip install pipenv
RUN pipenv install

ENTRYPOINT ["pipenv", "run", "python3", "./main.py"]
enter fullscreen mode

exit fullscreen mode


API Server

This is the main code and it is kind of simple.

from flask import Flask
from dotenv import load_dotenv

load_dotenv()

app = Flask(__name__)

import routes
enter fullscreen mode

exit fullscreen mode

routes

[routes/docker.py]

import os
from flask import jsonify, request
from server import app
from lib import docker, json


result = []


@app.route('/docker/tokenizer_done')
def get_tokenizer_done():
    file_name = request.args.get("file_name")
    docker.run_word_count_container(file_name)
    return "run a word_count container"


@app.route('/docker/word_count_done')
def get_word_count_done():
    file_name = request.args.get("file_name")

    json_data = json.load_data(
        os.path.join(os.getenv("SHARED_VOLUME_PATH"),
        "word_count_output",
        f"{file_name}.json"
    ))
    result.append(json_data)

    return "all works done"


@app.route('/docker/result')
def get_result():
    file_name = request.args.get("file_name")
    return jsonify({
        "result": result
    })
enter fullscreen mode

exit fullscreen mode

[routes/upload.py]

import os
from flask import jsonify, request
from werkzeug.utils import secure_filename
from server import app
from lib import docker


@app.route("/upload", methods=["POST"])
def upload_file():
    f = request.files["file"]

    file_name = secure_filename(f.filename)
    f.save(os.path.join(os.getenv("SHARED_VOLUME_PATH"), "input", file_name))

    docker.run_tokenizer_container(file_name)

    return "succeed to upload"
enter fullscreen mode

exit fullscreen mode

[routes/__init__.py]

from routes import docker, upload
enter fullscreen mode

exit fullscreen mode

[lib/docker.py]

import os

API_URL = os.getenv("API_URL")
VOLUME_ROOT_PATH = os.getenv("SHARED_VOLUME_PATH")
RUN_TOKENIZER_CONTAINER = 'docker run -it --add-host=host.docker.internal:host-gateway -v "' + VOLUME_ROOT_PATH + ':/shared_volume" hskcoder/tokenizer:0.2 /shared_volume/input/{FILE_NAME_WITH_EXTENSION} /shared_volume/tokenizer_output ' + API_URL
RUN_WORD_COUNT_CONTAINER = 'docker run -it --add-host=host.docker.internal:host-gateway -v "' + VOLUME_ROOT_PATH + ':/shared_volume" hskcoder/word_count:0.2 /shared_volume/tokenizer_output/{FILE_NAME_WITHOUT_EXTENSION}.json /shared_volume/word_count_output ' + API_URL


def run_tokenizer_container(file_name):
    print(RUN_TOKENIZER_CONTAINER.format(
        FILE_NAME_WITH_EXTENSION = file_name
    ))
    os.popen(RUN_TOKENIZER_CONTAINER.format(
        FILE_NAME_WITH_EXTENSION = file_name
    ))



def run_word_count_container(file_name):
    os.popen(RUN_WORD_COUNT_CONTAINER.format(
        FILE_NAME_WITHOUT_EXTENSION = file_name
    ))
enter fullscreen mode

exit fullscreen mode

[iib/json.py]

import json


def load_data(path):
    with open(path) as f:
        json_data = json.load(f)
    return json_data
enter fullscreen mode

exit fullscreen mode

This app reads environment variables .env file, so you need to set up like this.
The variables below fit my system.

API_URL=http://host.docker.internal:20000
ROOT_PATH=C:\Users\hskco\OneDrive\바탕 화면\stuff\docker\api
SHARED_VOLUME_PATH=C:\Users\hskco\OneDrive\바탕 화면\stuff\docker\api\shared_volume
enter fullscreen mode

exit fullscreen mode


you can run the server with this script

python3 -m pipenv run flask run -h 0.0.0.0 --port 20000
enter fullscreen mode

exit fullscreen mode

Before running this command, you must set an environment variable FLASK_APP,
Since I was developing in Windows, I ran this command api dir.

$env:FLASK_APP = './server.py'
enter fullscreen mode

exit fullscreen mode

if you enter http://127.0.0.1/docker/result You must see this page.

blank result

Let’s send a file to the API server and see the result.

[1]
first step

[2]
second stage

[3]
Result


conclusion

It was really fun. I have learned many things.
Whatever your situation, I think it would be really cool to try anything that interests you.

This example is really basic, I mean.
it should have been treated as

  • authority
  • communication between containers (in this example, the API server exposes all routes to the public)
  • Management containers (containers that are done need to be removed)
  • API Server Deployment

Without these, there will be many things you need to consider. (I respect backend developers) I was just focused on implementing the system, honestly, I didn’t care much about others. If I had, I would not have been able to write this article. I’m going back to work tomorrow.

Anyway, it was fun, it’s true 🙂 I hope it’s helpful to someone.


Reference

github source code

github source code

Python

pipenv

flask

post worker

Leave a Comment