[Chugalug] Brainstorming help: CI / CD / Automated Testing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[Chugalug] Brainstorming help: CI / CD / Automated Testing

Lisa Harrison Ridley
Hey Chugalugers,

I work on several projects where we have adopted a test driven development process incorporated with continuous testing.  We typically utilize a service like TravisCI to spin up ephemeral testing environments that get destroyed once the test runs are complete.

I’m running into more and more cases where we have multiple systems that interact over APIs (not all systems are within our control), those APIs have limited access that requires whitelisting servers and/or IP addresses with the API in question.  This complicates continuous testing with ephemeral environments, as IP addresses can change from instance to instance.  Currently, we’ve taken to spinning up dedicated testing servers and whitelisting those IP addresses, and using VPN connections where the IP address of the VPN is whitelisted for local development environments.  However, having a dedicated server requires care / maintenance / feeding of those servers, which has an infrastructure cost associated with it to keep servers updated / patched / hardened,  etc.

Ideally I would love to be able to spin up a VM at TravisCI / CircleCI that connects to our company VPN and routes all internet traffic through that VPN (which is whitelisted).  Less ideal, but workable, would be spinning up a Digital Ocean droplet (we default to Digital Ocean for our company infrastructure) that gets assigned a whitelisted IP address from a bank of “floating IPs” and have that droplet exist for the life of the test run, then destroyed.

Those are just some of my initial thoughts.  If you’ve solved this issue somehow in your organization I would love to hear how you solved it, or if you have an idea that’s worth exploring I’d love to hear that too.
_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug
Reply | Threaded
Open this post in threaded view
|

Re: [Chugalug] Brainstorming help: CI / CD / Automated Testing

Eric Wolf
For lower-level testing, we mock APIs for CI/CD. But the products I work on don't rely on many external APIs.

One of the things I've been pushing for is more switch automation so that a new VLAN can be created with all test resources isolated to that VLAN. And tear it all down when the test completes. But that's a long ways off and it requires the networking guys to let us run code against the switch.

-Eric

-=--=---=----=----=---=--=-=--=---=----=---=--=-=-
Eric B. Wolf                           720-334-7734




On Tue, Oct 16, 2018 at 11:04 AM Lisa Harrison Ridley <[hidden email]> wrote:
Hey Chugalugers,

I work on several projects where we have adopted a test driven development process incorporated with continuous testing.  We typically utilize a service like TravisCI to spin up ephemeral testing environments that get destroyed once the test runs are complete.

I’m running into more and more cases where we have multiple systems that interact over APIs (not all systems are within our control), those APIs have limited access that requires whitelisting servers and/or IP addresses with the API in question.  This complicates continuous testing with ephemeral environments, as IP addresses can change from instance to instance.  Currently, we’ve taken to spinning up dedicated testing servers and whitelisting those IP addresses, and using VPN connections where the IP address of the VPN is whitelisted for local development environments.  However, having a dedicated server requires care / maintenance / feeding of those servers, which has an infrastructure cost associated with it to keep servers updated / patched / hardened,  etc.

Ideally I would love to be able to spin up a VM at TravisCI / CircleCI that connects to our company VPN and routes all internet traffic through that VPN (which is whitelisted).  Less ideal, but workable, would be spinning up a Digital Ocean droplet (we default to Digital Ocean for our company infrastructure) that gets assigned a whitelisted IP address from a bank of “floating IPs” and have that droplet exist for the life of the test run, then destroyed.

Those are just some of my initial thoughts.  If you’ve solved this issue somehow in your organization I would love to hear how you solved it, or if you have an idea that’s worth exploring I’d love to hear that too.
_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug

_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug
Reply | Threaded
Open this post in threaded view
|

Re: [Chugalug] Brainstorming help: CI / CD / Automated Testing

Dave Brockman
On 10/16/2018 2:13 PM, Eric Wolf wrote:
> For lower-level testing, we mock APIs for CI/CD. But the products I work
> on don't rely on many external APIs.
>
> One of the things I've been pushing for is more switch automation so
> that a new VLAN can be created with all test resources isolated to that
> VLAN. And tear it all down when the test completes. But that's a long
> ways off and it requires the networking guys to let us run code against
> the switch.

If your network guys aren't into ACI/SDN (not entirely certain I blame
them), have you given any thought to virtualizing your networking?  Or
have a permanent "TEST-VLAN" that doesn't change, but you can (re) use
as you see fit?  If you are in the habit of spinning up test servers on
a regular basis, do you really need to create/destroy/recreate the VLAN?

Regards,

dtb


_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Chugalug] Brainstorming help: CI / CD / Automated Testing

Dave Brockman
In reply to this post by Lisa Harrison Ridley
On 10/16/2018 1:03 PM, Lisa Harrison Ridley wrote:
> Ideally I would love to be able to spin up a VM at TravisCI / CircleCI that connects to our company VPN and routes all internet traffic through that VPN (which is whitelisted).

I use a server in front of whatever is connecting to act as the VPN
endpoint termination.  Requires the configuration of a "back end"
network with your cloud provider.  Works great as long as your cloud
provider is giving your AES-NI CPUs to work with.  Much less of a hassle
than keeping a "VPN all traffic endpoint" configured on the device itself.

Regards,

dtb






_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Chugalug] Brainstorming help: CI / CD / Automated Testing

Eric Wolf
In reply to this post by Dave Brockman
Dave,

Our problem is that the product we are testing is "spinning up servers" with somewhat complicated and diverse networking requirements. And the product is about to start configuring the network switch and OOB interfaces. We are working on virtualizing the entire environment but that only goes so far.

-Eric
-=--=---=----=----=---=--=-=--=---=----=---=--=-=-
Eric B. Wolf                           720-334-7734




On Tue, Oct 16, 2018 at 1:13 PM Dave Brockman <[hidden email]> wrote:
On 10/16/2018 2:13 PM, Eric Wolf wrote:
> For lower-level testing, we mock APIs for CI/CD. But the products I work
> on don't rely on many external APIs.
>
> One of the things I've been pushing for is more switch automation so
> that a new VLAN can be created with all test resources isolated to that
> VLAN. And tear it all down when the test completes. But that's a long
> ways off and it requires the networking guys to let us run code against
> the switch.

If your network guys aren't into ACI/SDN (not entirely certain I blame
them), have you given any thought to virtualizing your networking?  Or
have a permanent "TEST-VLAN" that doesn't change, but you can (re) use
as you see fit?  If you are in the habit of spinning up test servers on
a regular basis, do you really need to create/destroy/recreate the VLAN?

Regards,

dtb

_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug

_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug
Reply | Threaded
Open this post in threaded view
|

Re: [Chugalug] Brainstorming help: CI / CD / Automated Testing

Dave Brockman
On 10/16/2018 3:37 PM, Eric Wolf wrote:
> Dave,
>
> Our problem is that the product we are testing is "spinning up servers"
> with somewhat complicated and diverse networking requirements. And the
> product is about to start configuring the network switch and OOB
> interfaces. We are working on virtualizing the entire environment but
> that only goes so far.

Eric,
  I'm not entirely sure what you are up to now, but NFV is pretty solid
(with the correct hardware, Emulex has some nice cards) up to 40Gb.
There are players beyond that, but they don't play with my checkbook :)
From what you describe, I would think the Dev teams needs their own
switch to play with, isolated from things the Network team manages.

Regards,

dtb




_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Chugalug] Brainstorming help: CI / CD / Automated Testing

Eric Wolf
Dave,

The hardware our product deploys has 2 Emulex 40GB ports (and 2 Intel 40GB ports plus 2 Intel 10GB ports and out-of-band) per Compute server. The product is a private cloud infrastructure (HCI). What a shop might use to virtualize their environment.

-Eric

-=--=---=----=----=---=--=-=--=---=----=---=--=-=-
Eric B. Wolf                           720-334-7734




On Tue, Oct 16, 2018 at 1:50 PM Dave Brockman <[hidden email]> wrote:
On 10/16/2018 3:37 PM, Eric Wolf wrote:
> Dave,
>
> Our problem is that the product we are testing is "spinning up servers"
> with somewhat complicated and diverse networking requirements. And the
> product is about to start configuring the network switch and OOB
> interfaces. We are working on virtualizing the entire environment but
> that only goes so far.

Eric,
  I'm not entirely sure what you are up to now, but NFV is pretty solid
(with the correct hardware, Emulex has some nice cards) up to 40Gb.
There are players beyond that, but they don't play with my checkbook :)
From what you describe, I would think the Dev teams needs their own
switch to play with, isolated from things the Network team manages.

Regards,

dtb



_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug

_______________________________________________
Chugalug mailing list
[hidden email]
http://chugalug.org/cgi-bin/mailman/listinfo/chugalug