[Nix-dev] Fwd: nixos-container networking
Tomasz Czyż
tomasz.czyz at gmail.com
Mon Mar 20 19:32:40 CET 2017
Arnold, Danylo, thank you for info, links and examples!
2017-03-15 9:17 GMT+00:00 Danylo Hlynskyi <abcz2.uprola at gmail.com>:
> Host side:
>
> networking.bridges.${bridgeName}.interfaces = [];
> networking.interfaces.${bridgeName}.ip4 = [ { address =
> "${subnet}.1"; prefixLength = 24; } ];
>
> # Each container takes at least 4 inotify file handles, so you quickly
> reach limit 128 when spawning many containers
> boot.kernel.sysctl."fs.inotify.max_user_instances" = 2048;
>
> # internet in containers!
> networking.nat.enable = true;
> networking.nat.internalInterfaces = [ "ve-+" "vb-+" bridgeName ]; #
> TODO check if bridgeName is required
> networking.nat.externalInterface = "eth0";
>
> Container side:
>
> privateNetwork = true;
> hostBridge = bridgeName;
> localAddress = container_ip + "/24";
> config = {
> networking.defaultGateway = "${subnet}.1";
> networking.extraHosts = ''
> ${subnet}.1 ${hostConfig.networking.hostName}
> '';
> };
>
> There is also some code to set-up hostname resolving via /etc/hosts,
> because I know little about DHCP
> , mDNS, nss-* and so on (https://github.com/NixOS/nixpkgs/issues/16754).
> It is a wrapper around `containers`
> and monitors IP clashes (which are set manually in my setup)
>
>
> In general, I have 1 bridge with multiple IPs/subnets, so containers in
> one subnet don't see containers in other.
>
> 2017-03-14 20:12 GMT+02:00 Tomasz Czyż <tomasz.czyz at gmail.com>:
>
>>
>> ---------- Forwarded message ----------
>> From: Tomasz Czyż <tomasz.czyz at gmail.com>
>> Date: 2017-03-14 18:12 GMT+00:00
>> Subject: Re: [Nix-dev] nixos-container networking
>> To: Danylo Hlynskyi <abcz2.uprola at gmail.com>
>>
>>
>> Hey Danylo,
>>
>> yup, I hit 13 char limit and because I was removing - I thought it's that.
>>
>> I also had the issue with recreating containers, but this happened only
>> sometimes and didn't happen when I restart the machine so I was not sure
>> why is that. Thanks for your mail, it's very useful.
>>
>> Would you share your bridget networking?
>> I was trying that but I'm wondering if you have one shared bridge or you
>> have bridge per container and how you access containers from host (or route
>> traffic to them).
>>
>> Cheers,
>> Tom
>>
>>
>> 2017-03-14 6:01 GMT+00:00 Danylo Hlynskyi <abcz2.uprola at gmail.com>:
>>
>>> Strange, I have lot's of containers with "-" and experience no problems.
>>> But maybe you've exceeded by accident limit 13 symbols per container name?
>>>
>>> Also, last time I tried "veth" networking, I was struggling from
>>> https://github.com/NixOS/nixpkgs/issues/16330. My container experience
>>> was awful when I tried container renames. That's why I've already switched
>>> to bridged networking
>>>
>>> ---
>>>
>>> BTW, I highly recommend patch to switch-to-configuration.pl
>>> <https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf#diff-0a057d6ff3f6f83f68b859178484f4fe>
>>> from https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b2
>>> 77f78ece1bb81b79b5651897e46a2bf
>>>
>>> It isn't clear from commit message, but it does the following: makes
>>> declarative containers truly reloadable (when you change
>>> container config, it activates new configuration for container). The
>>> culprit is *it should be* default behavior, because of
>>>
>>> 1. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v
>>> irtualisation/containers.nix#L225-L230
>>> 2. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v
>>> irtualisation/containers.nix#L676
>>>
>>> I'd like to PR this, but got no time to test properly other parts of
>>> Nixos.
>>>
>>> 2017-03-14 4:42 GMT+02:00 Tomasz Czyż <tomasz.czyz at gmail.com>:
>>>
>>>> Michael, Ian, thank you for your answers.
>>>>
>>>> Looks like my problem was with the container name. I tried bunch of
>>>> different setups which didn't work and I discovered that when I'm using "-"
>>>> in container name it doesn't work (I had impression that worked one or two
>>>> times when I started machine from scratch, but most of the time didn't).
>>>>
>>>> After I removed "-" from the name, looks like private network is
>>>> working (I can access private IP of container) so I don't need NAT actually.
>>>>
>>>> Tom
>>>>
>>>> 2017-03-13 23:54 GMT+00:00 Ian-Woo Kim <ianwookim at gmail.com>:
>>>>
>>>>> I've recently made nixos-container port forwarding easier (both
>>>>> imperative and declarative) and it's now merged into master.
>>>>>
>>>>> https://github.com/NixOS/nixpkgs/pull/20869
>>>>>
>>>>> Hope that this helps.
>>>>>
>>>>> Ian
>>>>>
>>>>> On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker <mike at barrucadu.co.uk>
>>>>> wrote:
>>>>> > Tomasz,
>>>>> >
>>>>> > I have declarative container networking set up and working on a VPS,
>>>>> > but I wrote most of the configuration as I was learning things, so it
>>>>> > may not be the best way.
>>>>> >
>>>>> > Here's the configuration.nix for the VPS:
>>>>> > https://github.com/barrucadu/nixfiles/blob/master/hosts/inns
>>>>> mouth.nix
>>>>> > Each container has a config file here:
>>>>> > https://github.com/barrucadu/nixfiles/tree/master/containers
>>>>> >
>>>>> > Containers have ports forwarded to them via NAT; each container is
>>>>> > running a web server on port 80 with the host reverse-proxying via
>>>>> > nginx; the host also does https and letsencrypt for all the proxied
>>>>> > containers.
>>>>> >
>>>>> > At the top of the innsmouth.nix file, I have a "containerSpecs"
>>>>> record
>>>>> > which has all the details for each container. The relevant bits of
>>>>> the
>>>>> > config are:
>>>>> >
>>>>> > 1. Set up the networking and NAT:
>>>>> >
>>>>> > networking.nat.enable = true;
>>>>> > networking.nat.internalInterfaces = ["ve-+"];
>>>>> > networking.nat.externalInterface = "enp0s4";
>>>>> >
>>>>> > 2. Forward ports to containers:
>>>>> >
>>>>> > networking.nat.forwardPorts = concatMap
>>>>> > ( {num, ports, ...}:
>>>>> > map (p: { sourcePort = p; destination =
>>>>> > "192.168.255.${toString num}:${toString p}"; }) ports
>>>>> > ) containerSpecs';
>>>>> >
>>>>> > 3. Define all the containers:
>>>>> >
>>>>> > containers = mapAttrs
>>>>> > (_: {num, config, ...}:
>>>>> > { autoStart = true
>>>>> > ; privateNetwork = true
>>>>> > ; hostAddress = "192.168.254.${toString num}"
>>>>> > ; localAddress = "192.168.255.${toString num}"
>>>>> > ; config = config
>>>>> > ; }
>>>>> > ) containerSpecs;
>>>>> >
>>>>> > 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt
>>>>> > certificates, and forward HTTP to HTTPS.
>>>>> >
>>>>> > This is a little complex as I have a fairly custom nginx config (see
>>>>> > the services/nginx.nix file in the repository), but the
>>>>> > reverse-proxying is fairly straightfoward. Here is the generated
>>>>> > nginx.conf: https://misc.barrucadu.co.uk/nginx.txt
>>>>> >
>>>>> > On 13 March 2017 at 02:12, Tomasz Czyż <tomasz.czyz at gmail.com>
>>>>> wrote:
>>>>> >> Hey,
>>>>> >>
>>>>> >> could anyone using nixos-container (declarative style) share how
>>>>> you setup
>>>>> >> networking?
>>>>> >>
>>>>> >> I'm trying to setup few containers with private network and http
>>>>> proxy at
>>>>> >> the front. Each container potentially could run application on port
>>>>> 80 and I
>>>>> >> would like to expose them through proxy.
>>>>> >>
>>>>> >> I tried to set this up with
>>>>> >>
>>>>> >> privateNetwork=true;
>>>>> >> hostAddress
>>>>> >> localAddress
>>>>> >>
>>>>> >> and I tried to also run nat on the host with (just to enable
>>>>> outbound
>>>>> >> traffic)
>>>>> >> internalInterfaces = ["ve-+"];
>>>>> >> externalInterfaces = "eth0";
>>>>> >>
>>>>> >> but no luck.
>>>>> >> My next try will be creating bridge on the host and add containers
>>>>> to that
>>>>> >> bridge. Is that how you do stuff or are better ways of doing
>>>>> container
>>>>> >> networking?
>>>>> >>
>>>>> >> Tom
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> nix-dev mailing list
>>>>> >> nix-dev at lists.science.uu.nl
>>>>> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Michael Walker (http://www.barrucadu.co.uk)
>>>>> > _______________________________________________
>>>>> > nix-dev mailing list
>>>>> > nix-dev at lists.science.uu.nl
>>>>> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Tomasz Czyż
>>>>
>>>> _______________________________________________
>>>> nix-dev mailing list
>>>> nix-dev at lists.science.uu.nl
>>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>>
>>>>
>>>
>>
>>
>> --
>> Tomasz Czyż
>>
>>
>>
>> --
>> Tomasz Czyż
>>
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
>
--
Tomasz Czyż
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20170320/d2c9523e/attachment-0001.html>
More information about the nix-dev
mailing list