Init-Container fails, read-only filesystem - configMap volumes no longer writable

[albamoro]

Just to give you the heads up in case you didn't already know, configMap volumes have been read-only since kubernetes 1.9.4.

kubernetes/kubernetes#58720

[solsson]

Thanks, I read that in the release notes but failed to draw the conclusion. It's a major bummer for this kind of experimentation, and maybe we need to prioritize something like #155.

[solsson]

I think I saw some kind of flag to keep the legacy behavior, but right now I can't find it. In case someone wants to keep upgrading.

[solsson]

The first fix that comes to mind, apart from #155, is to mount the configmap only on the init container and share an emptyDir within the pod. init.sh would have to begin with a cp. Other ideas?

[clintfred]

I also ran into this yesterday. I ended up mounting the ConfigMap to /etc/opt and then wrote the modified files to an empty config volume. I'm not claiming this is the best solution, but it does seem to work.

If anyone wants to save themselves some time, you can grab the code at https://github.com/clintfred/kubernetes-kafka/tree/gke-startup-patch.
Warning: That branch also contains the changes for #160

@solsson I'm happy to PR my fix back here if you want it.

[clintfred]

I also noticed that the pzoo init container is using a full kafka docker image, but I don't think that's needed. Maybe it could use something slimmer? zoo does use a different container for init.

[albamoro]

Hehe, I resorted to the same hack.

…

On 22 Mar 2018 20:11, "clintfred" ***@***.***> wrote: I also ran into this yesterday. I ended up mounting the ConfigMap to /etc/opt and then wrote the modified files to an empty config volume. I'm not claiming this is the best solution, but it does seem to work. If anyone wants to save themselves some time, you can grab the code at https://github.com/clintfred/kubernetes-kafka/tree/gke-startup-patch. *Warning:* That branch also contains the changes for #160 <#160> @solsson <https://github.com/solsson> I'm happy to PR my fix back here if you want it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#162 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADzLahA-9nbVvR4iqkkD4bgEViFXUDLpks5thAVcgaJpZM4S2wcp> .

[solsson]

@clintfred @albamoro Does #163 match your solutions?

I also noticed that the pzoo init container is using a full kafka docker image, but I don't think that's needed

I want to keep using the zk release shipped with Kafka, due to things like https://issues.apache.org/jira/browse/ZOOKEEPER-2960 making it into zk releases. The idea with the default image here is that it should be as plain a Kafka distribution as possible.

[albamoro]

yup, #163 describes the same workaround I used.

[clintfred]

It's close to my solution. The one functional difference I see is that mine leaves only the .properties file in /etc/kafka, instead of all the files in the configmap, which I would consider slightly cleaner.

@solsson regarding the container for the init-config I'm just noting the discrepancy between:

50kafka.yml

initContainers:
      - name: init-config
        image: solsson/kafka-initutils@sha256:c98d7fb5e9365eab391a5dcd4230fc6e72caf929c60f29ff091e3b0215124713

vs

50pzoo.yml and 51zoo.yml

initContainers:
      - name: init-config
        image: solsson/kafka:1.0.1@sha256:1a4689d49d6274ac59b9b740f51b0408e1c90a9b66d16ad114ee9f7193bab111

Maybe the whole kafka:1.0.1 is needed in the init container for pzoo and zoo but not for kafka?

[solsson]

Thanks @clintfred for spotting this. I've pushed 2621880 to address this.

With a growing interest for tools like skaffold I think it wil be more commonplace for kubernetes users to iterate on config using image builds. It can be as productive as the ConfigMap use in this repo, but with better support for rolling updates and a more seamless transition to production builds (compared to #155).

We do need to iterate on config in production-to-be clusters, i.e. minikube etc won't do, for things like #78. Maybe the way to go for this repo is to include Dockerfiles and recommend a couple of tools that make config iterations as easy as with docker-compose.

[gladiatr72]

@clintfred yeah. That does work, but the problem is that this is now just part of the scaffolding of nearly every deployment. I guess that's why Jesus gave us init-containers, hm? So now we can have a shared emptydir with an init-container that says "cp /opt/config/thing /etc/thing/config" for everything. I believe that most file operations will try to use the directory of the target file for any work-related functions unless the program's author explicitly directs such operations to not that directory. (e.g.: sed's in-place editing (maybe perl's too) actually writes its changes to a new file (in the same directory) and then renames it to overwrite the original.

I don't have a magic solution for the mentioned CVE, but this erm... solution was naively imagined and contrary to the other bugs that have to wait a year before a alpha/beta/GAmeh/GA release-cycle, this one seems to have been back-ported all the way to 1.7.

[solsson]

Closed with #163