Docker - Accessing Python Application Logs

docker
python
A tutorial on getting Python application logs running on Docker inside WSL2.
Published

March 11, 2022

About

This post is about the challenges on accessing Python application logs running on a Docker container inside WSL2 linux environment.

Environment Details

  • Python = 3.8.x
  • WSL version = 2
  • WSL Ubuntu version = 20.04
  • Docker Engine version = 20.10.12
  • Docker Desktop version = 4.4.4
  • Host OS = Windows 10

Sample Application

Let us create a simple hello world application that will print “hello world” message to stdout, and also logs them in a logfile. After each message the application sleeps for 5 seconds, and keeps on doing this for 5 mins (300 sec). After this the program exists.

Project structure of this application is

app/
└── src/
    ├── commons/
    │   └── logger.py
    └── hello.py

Where * app/ is the project root folder * src/ folder contain the python application code * src/commons/logger.py is the logging module * src/hello.py is the main application

Code files are provided below

##
# app/src/commons/logger.py

import logging
import os

logformat = "%(levelname)s %(asctime)s - %(message)s"
filename = "logfile.log"

# Setting the config of the log object
logging.basicConfig(
    format=logformat,
    filename=filename,
    level=logging.INFO,
)
##
# app/src/hello.py

from datetime import datetime
import time
import commons.logger as logger


def main():
    # run for about 5 min: 300 sec
    for i in range(60):
        now = datetime.now()
        dt_string = now.strftime("%d/%m/%Y %H:%M:%S")

        # prepare message
        msg = f"hello world at {dt_string}"

        # put message to stdout and logs
        print(msg)
        logger.logging.info(msg)

        # sleep for some seconds
        time.sleep(5)


if __name__ == "__main__":
    main()

When I run the hello.py file I get the output on the termial with hello world messages like this.

helloworld_output

When we run the application a “logfile.log” will also appear in the project directory containing the logged messages.

.
├── app/
│   └── src/
│       ├── commons/
│       │   └── logger.py
│       └── hello.py
└── **logfile.log**

Contents of “logfile.log” file will look like this

INFO 2022-03-11 13:01:56,451 - hello world at 11/03/2022 13:01:56
INFO 2022-03-11 13:02:01,464 - hello world at 11/03/2022 13:02:01
INFO 2022-03-11 13:02:06,466 - hello world at 11/03/2022 13:02:06
INFO 2022-03-11 13:02:11,480 - hello world at 11/03/2022 13:02:11
INFO 2022-03-11 13:02:16,496 - hello world at 11/03/2022 13:02:16

All the code till this point can be found at GitHub repository https://github.com/hassaanbinaslam/snapshots-docker-post-11032022 * Project code files * Project zip file

Dockerize the application

Our hello-world application is ready now, and we can put it inside a docker container. For this let’s create a Dockerfile and place it in app/ folder.

##
# app/Dockerfile

FROM python:3.8-slim-buster

# set the working directory in the container
WORKDIR /app

# copy the content of the local src directory to the working directory
COPY src/ .

# command to run on container start
CMD [ "python3", "./hello.py"]

We can build our docker image by running the command from terminal at folder app/

docker build --tag python-docker .

Output of this command will look like this docker-build-cmd

We can check the created docker image using command from terminal

docker images

Output of this command will look like this docker-images-cmd

So our docker image is ready, we can now run it using command

docker run --name helloworld python-docker

After running this command you will observe that there is no output on the terminal. Even though we have run our image in an attached mode but still there is no output. We know that the container is running as control on terminal has not return back to us. We can also verify that the container is running by running command docker ps in a separate terminal. Output from this command will look like this

docker-running-ps

We can also verify that the container is running from Docker Desktop container apps menu. Running container instance will appear like this

docker-running-desktop

The reason for logs not appearing on the terminal is because they are being buffered by docker internally. You will get all of them once docker container has finished execution and stopped. You can read more about docker buffereing the output from these StackOverflow posts * disable-output-buffering * python-app-does-not-print-anything-when-running-detached-in-docker

To disable the output buffering we need to change the CMD in our docker file as

CMD [ "python3", "-u", "./hello.py"]
##
# app/Dockerfile

FROM python:3.8-slim-buster

# set the working directory in the container
WORKDIR /app

# copy the content of the local src directory to the working directory
COPY src/ .

# command to run on container start
CMD [ "python3", "-u", "./hello.py"]

We need to rebuild our docker image and then run it. Let’s do that will the following commands

docker build --tag python-docker .
docker run --name helloworld python-docker

this time you can see the output directly on the terminal docker-run-output

We don’t have to run the docker container in an attached mode, and can still get the logs from a running container using docker logs command. Let’s do that then

first run the docker image in a detached mode

docker run -d --name helloworld python-docker

this command will give the running container ID which will look something like this 06918cf824210c015b08ef072feed01768be30ee569e5dbc0eef8d3cd855ab47. We can pass this ID to our next command to view the logs stream from a running container. Use this command

docker logs -f 06918cf824210c015b08ef072feed01768be30ee569e5dbc0eef8d3cd855ab47

Also note that we don’t need to provide the full container id, and can can also provide the first 2 or 3 digits of the ID that can uniquely identify the running container. So following command will also work

docker logs -f 069

Output on the terminal will look like this docker-run-detached

All the project files till this point can be found at * project code files

Log Files Physical Location

Docker STDOUT / STDERR logs are also stored on host system as JSON files.

On linux You can find them at

/var/lib/docker/containers/<container-id>/<container-id>-json.log

On Windows You can find them at

\\wsl$\docker-desktop-data\version-pack-data\community\docker\containers\<container-id>/<container-id>-json.log

In the above paths replace <container-id> with your full cotaniner ID. Also note that on Windows 10 you can open the folder location by directly pasting the path in folder explorer search bar.

You can read more about them in the following StackOverflow post * windows-10-and-docker-container-logs-docker-logging-driver

Application Log Files

So far we have talked about the docker logs that were generated by application feed to STDOUT or STDERR. But now we are interested in app logs generated by logging module like the one we saw in our first example “logfile.log”. For this let’s first connect to running docker container and see where is this file located inside the docker running instance.

Run a new docker container again using command

docker run -d --name helloworld python-docker

If container already exists you can just start it using command

docker start <container-id>

Or you can also use the docker desktop to start existing container by click the play button over it.

docker-start-container

Once the docker container is running, you can connect to it by either using the CLI from docker desktop

docker-desktop-cli

or from the command below

docker exec -it <cotainer-id> /bin/bash

Remember that container-id of a running container can be obtained by command docker ps. Output from the above command will look like this

docker-command-cli

Note that the location of “logfile.log” is under app/ folder

Docker Volume

We have seen the application logs by connecting to the docker container, but we would like to have these logs readily available outside the docker container so they can be consumed in real time for debugging. Fot this docker recommends using volume. Let’s create a volume and then mount our application logs to it.

To create a volume use command

docker volume create applogs

To remove a volume use command

docker rm create applogs

To inspect a created volume use command

docker volume inspect applogs

Output of this command will be like this where Mountpoint is the location of logs on host system.

[
    {
        "CreatedAt": "2022-03-11T13:12:57Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/applogs/_data",
        "Name": "applogs",
        "Options": {},
        "Scope": "local"
    }
]

On Windows 10 you can find the location of these volumes at

\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\applogs\_data

Now let’s run our docker container with this volume mounted to it. Command for this is

docker run -d --name helloworld -v applogs:/app python-docker

Docker container will run as before, but if we check the volume on our host we will find that this time all the files and folders available in app/ folder from inside the docker container are available. And they will persist on the host OS even if the container is stopped.

docker-volume-mounted

Note that the log file ‘logfile.log’ is also available outside the container, and we can poll it for debugging. But having all the contents of app/ folder exposed can be a security issue, as they can contain secrets and passwords. We should only mount the logfile.log file on the volume as a best practice. So let’s do that next.

Application log files from the docker container

To do this we need to slightly update our application, and create the logfile.log file in a designated log/ folder inside the app/. This way we can only mount app/log/ folder on the volume. In our application we will update the logging module as

##
# app/src/commons/logger.py

import logging
import os


if not os.path.exists("logs"):
    os.makedirs("logs")

logformat = "%(levelname)s %(asctime)s - %(message)s"
filename = "logs/logfile.log"


# Setting the config of the log object
logging.basicConfig(
    format=logformat,
    filename=filename,
    level=logging.INFO,
)

This is all we need to change in our application. To cleanup the volume we can either remove the old one and recreate a new one. Or we can directly delete all the files & folders from the host OS from the directory

\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\applogs\_data

Since we have updated the application code we need to rebuild our docker image. Then create a new docker container, and this time only mount the logs/ folder on the volume. Command for this is

# delete the old volume
docker volume rm applogs

# create a new volume
docker volume create applogs

# build the docker image
docker build --tag python-docker .

# run the docker container and mount a specific folder on volume
docker run -d --name helloworld -v applogs:/app/logs python-docker

If we check the mounted volume, this time only logfile.log is exposed.

docker-logs-mounted

All the code for this post can be found at GitHub repository https://github.com/hassaanbinaslam/snapshots-docker-post-11032022 * Project code files * Project zip file