[Nix-dev] Fwd: nixos-container networking
Danylo Hlynskyi
abcz2.uprola at gmail.com
Wed Mar 15 10:17:04 CET 2017
Host side:
networking.bridges.${bridgeName}.interfaces = [];
networking.interfaces.${bridgeName}.ip4 = [ { address = "${subnet}.1";
prefixLength = 24; } ];
# Each container takes at least 4 inotify file handles, so you quickly
reach limit 128 when spawning many containers
boot.kernel.sysctl."fs.inotify.max_user_instances" = 2048;
# internet in containers!
networking.nat.enable = true;
networking.nat.internalInterfaces = [ "ve-+" "vb-+" bridgeName ]; #
TODO check if bridgeName is required
networking.nat.externalInterface = "eth0";
Container side:
privateNetwork = true;
hostBridge = bridgeName;
localAddress = container_ip + "/24";
config = {
networking.defaultGateway = "${subnet}.1";
networking.extraHosts = ''
${subnet}.1 ${hostConfig.networking.hostName}
'';
};
There is also some code to set-up hostname resolving via /etc/hosts,
because I know little about DHCP
, mDNS, nss-* and so on (https://github.com/NixOS/nixpkgs/issues/16754). It
is a wrapper around `containers`
and monitors IP clashes (which are set manually in my setup)
In general, I have 1 bridge with multiple IPs/subnets, so containers in one
subnet don't see containers in other.
2017-03-14 20:12 GMT+02:00 Tomasz Czyż <tomasz.czyz at gmail.com>:
>
> ---------- Forwarded message ----------
> From: Tomasz Czyż <tomasz.czyz at gmail.com>
> Date: 2017-03-14 18:12 GMT+00:00
> Subject: Re: [Nix-dev] nixos-container networking
> To: Danylo Hlynskyi <abcz2.uprola at gmail.com>
>
>
> Hey Danylo,
>
> yup, I hit 13 char limit and because I was removing - I thought it's that.
>
> I also had the issue with recreating containers, but this happened only
> sometimes and didn't happen when I restart the machine so I was not sure
> why is that. Thanks for your mail, it's very useful.
>
> Would you share your bridget networking?
> I was trying that but I'm wondering if you have one shared bridge or you
> have bridge per container and how you access containers from host (or route
> traffic to them).
>
> Cheers,
> Tom
>
>
> 2017-03-14 6:01 GMT+00:00 Danylo Hlynskyi <abcz2.uprola at gmail.com>:
>
>> Strange, I have lot's of containers with "-" and experience no problems.
>> But maybe you've exceeded by accident limit 13 symbols per container name?
>>
>> Also, last time I tried "veth" networking, I was struggling from
>> https://github.com/NixOS/nixpkgs/issues/16330. My container experience
>> was awful when I tried container renames. That's why I've already switched
>> to bridged networking
>>
>> ---
>>
>> BTW, I highly recommend patch to switch-to-configuration.pl
>> <https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf#diff-0a057d6ff3f6f83f68b859178484f4fe>
>> from https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b2
>> 77f78ece1bb81b79b5651897e46a2bf
>>
>> It isn't clear from commit message, but it does the following: makes
>> declarative containers truly reloadable (when you change
>> container config, it activates new configuration for container). The
>> culprit is *it should be* default behavior, because of
>>
>> 1. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v
>> irtualisation/containers.nix#L225-L230
>> 2. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v
>> irtualisation/containers.nix#L676
>>
>> I'd like to PR this, but got no time to test properly other parts of
>> Nixos.
>>
>> 2017-03-14 4:42 GMT+02:00 Tomasz Czyż <tomasz.czyz at gmail.com>:
>>
>>> Michael, Ian, thank you for your answers.
>>>
>>> Looks like my problem was with the container name. I tried bunch of
>>> different setups which didn't work and I discovered that when I'm using "-"
>>> in container name it doesn't work (I had impression that worked one or two
>>> times when I started machine from scratch, but most of the time didn't).
>>>
>>> After I removed "-" from the name, looks like private network is working
>>> (I can access private IP of container) so I don't need NAT actually.
>>>
>>> Tom
>>>
>>> 2017-03-13 23:54 GMT+00:00 Ian-Woo Kim <ianwookim at gmail.com>:
>>>
>>>> I've recently made nixos-container port forwarding easier (both
>>>> imperative and declarative) and it's now merged into master.
>>>>
>>>> https://github.com/NixOS/nixpkgs/pull/20869
>>>>
>>>> Hope that this helps.
>>>>
>>>> Ian
>>>>
>>>> On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker <mike at barrucadu.co.uk>
>>>> wrote:
>>>> > Tomasz,
>>>> >
>>>> > I have declarative container networking set up and working on a VPS,
>>>> > but I wrote most of the configuration as I was learning things, so it
>>>> > may not be the best way.
>>>> >
>>>> > Here's the configuration.nix for the VPS:
>>>> > https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix
>>>> > Each container has a config file here:
>>>> > https://github.com/barrucadu/nixfiles/tree/master/containers
>>>> >
>>>> > Containers have ports forwarded to them via NAT; each container is
>>>> > running a web server on port 80 with the host reverse-proxying via
>>>> > nginx; the host also does https and letsencrypt for all the proxied
>>>> > containers.
>>>> >
>>>> > At the top of the innsmouth.nix file, I have a "containerSpecs" record
>>>> > which has all the details for each container. The relevant bits of the
>>>> > config are:
>>>> >
>>>> > 1. Set up the networking and NAT:
>>>> >
>>>> > networking.nat.enable = true;
>>>> > networking.nat.internalInterfaces = ["ve-+"];
>>>> > networking.nat.externalInterface = "enp0s4";
>>>> >
>>>> > 2. Forward ports to containers:
>>>> >
>>>> > networking.nat.forwardPorts = concatMap
>>>> > ( {num, ports, ...}:
>>>> > map (p: { sourcePort = p; destination =
>>>> > "192.168.255.${toString num}:${toString p}"; }) ports
>>>> > ) containerSpecs';
>>>> >
>>>> > 3. Define all the containers:
>>>> >
>>>> > containers = mapAttrs
>>>> > (_: {num, config, ...}:
>>>> > { autoStart = true
>>>> > ; privateNetwork = true
>>>> > ; hostAddress = "192.168.254.${toString num}"
>>>> > ; localAddress = "192.168.255.${toString num}"
>>>> > ; config = config
>>>> > ; }
>>>> > ) containerSpecs;
>>>> >
>>>> > 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt
>>>> > certificates, and forward HTTP to HTTPS.
>>>> >
>>>> > This is a little complex as I have a fairly custom nginx config (see
>>>> > the services/nginx.nix file in the repository), but the
>>>> > reverse-proxying is fairly straightfoward. Here is the generated
>>>> > nginx.conf: https://misc.barrucadu.co.uk/nginx.txt
>>>> >
>>>> > On 13 March 2017 at 02:12, Tomasz Czyż <tomasz.czyz at gmail.com> wrote:
>>>> >> Hey,
>>>> >>
>>>> >> could anyone using nixos-container (declarative style) share how you
>>>> setup
>>>> >> networking?
>>>> >>
>>>> >> I'm trying to setup few containers with private network and http
>>>> proxy at
>>>> >> the front. Each container potentially could run application on port
>>>> 80 and I
>>>> >> would like to expose them through proxy.
>>>> >>
>>>> >> I tried to set this up with
>>>> >>
>>>> >> privateNetwork=true;
>>>> >> hostAddress
>>>> >> localAddress
>>>> >>
>>>> >> and I tried to also run nat on the host with (just to enable outbound
>>>> >> traffic)
>>>> >> internalInterfaces = ["ve-+"];
>>>> >> externalInterfaces = "eth0";
>>>> >>
>>>> >> but no luck.
>>>> >> My next try will be creating bridge on the host and add containers
>>>> to that
>>>> >> bridge. Is that how you do stuff or are better ways of doing
>>>> container
>>>> >> networking?
>>>> >>
>>>> >> Tom
>>>> >>
>>>> >> _______________________________________________
>>>> >> nix-dev mailing list
>>>> >> nix-dev at lists.science.uu.nl
>>>> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Michael Walker (http://www.barrucadu.co.uk)
>>>> > _______________________________________________
>>>> > nix-dev mailing list
>>>> > nix-dev at lists.science.uu.nl
>>>> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>>
>>>
>>>
>>>
>>> --
>>> Tomasz Czyż
>>>
>>> _______________________________________________
>>> nix-dev mailing list
>>> nix-dev at lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>
>
>
> --
> Tomasz Czyż
>
>
>
> --
> Tomasz Czyż
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20170315/ceb69b0a/attachment-0001.html>
More information about the nix-dev
mailing list